Terraform Professional (002): Secrets, Rotation and Pipeline Identities

The core practice is simple: secret values never live in Terraform code or in anything that ships with it. They are supplied at run time from a protected store, read without being persisted where possible, and written with write-only arguments when a resource needs them. Everything else in this lesson builds on that rule.

Places a secret must not sit, even in a private repository:

• A default on a variable block, or a value in locals — both are committed with the code.
• A terraform.tfvars or *.tfvars file in any repository, including a separate "private" platform repository.
• A default inside a private registry module.

sensitive = true on a variable is not a storage control. It redacts the value in plan and apply output, but the value still sits wherever you typed it and, if it reaches an ordinary resource argument, in state.

Where a secret should come from instead:

The order of preference is roughly bottom-up: a secret nobody handles beats a secret Terraform reads ephemerally, which beats a secret stored as a variable.

Where secrets leak: state, plans, logs and uploads

A secret leaks whenever it reaches something Terraform persists or something a run uploads. Knowing the paths is what lets you judge any design.

• State. Every ordinary resource argument and every data source attribute is recorded in state in plain text. Passing a sensitive variable to password on a database puts that password in state, regardless of the variable's sensitive flag. Nothing redacts values inside the state file — protecting state is about who can read it.
• Saved plan files. A saved plan contains the same values as state for ordinary arguments.
• Logs. Sensitive values are redacted in normal plan output, but HashiCorp documents that TF_LOG=TRACE can print environment variable values into run logs. Never leave trace logging on in a workspace that holds credentials as environment variables.
• CLI-driven uploads. A CLI-driven run (a cloud block, then terraform plan/apply locally) uploads an archive of the working directory to HCP Terraform. The upload is filtered by .terraformignore, not .gitignore, and remote runs load any *.auto.tfvars files in the uploaded configuration. Git ignore rules therefore protect the repository, not the upload: any local variable file that is not excluded by .terraformignore travels with the configuration version, and an auto-loaded one feeds the run. Keep secret files out of the working directory (or exclude them in .terraformignore) and supply the value as a sensitive workspace variable.

The VCS-driven case is the mirror image: a VCS-driven run is built only from what is committed in the connected repository, so a git-ignored file keeps the secret out of Git but also out of the run, which then fails for a missing value. That is why the run-time source has to be the platform, not a file.

Environment variables are different from Terraform variables here: environment variables (such as AWS_SECRET_ACCESS_KEY) never reach state, but they can reach trace logs.

A workspace (or variable set) variable marked sensitive is write-only for everyone — users, organization owners and API clients alike. Once saved, its value can never be read back in the UI or through the Variables API.

• Rotating the value is an overwrite. Anyone with permission to write variables (the fixed Write set includes it) can enter a new value and save it. Neither the old nor the new value is visible afterwards.
• Other attributes are frozen. The key and the sensitive flag cannot be edited on a sensitive variable. To "un-sensitive" a variable marked by mistake, delete it and create a new one with the same key and the settings you want. There is no reveal option, even for owners.
• Write-only protects the variable, not the value's journey. The run receives the real value. If the configuration passes it to an ordinary resource argument, it lands in state; if it is an environment variable and trace logging is on, it can land in logs.
• Sensitive is not read-only. Marking a variable sensitive hides it; it does not stop someone with variable write access from overwriting or deleting it. Preventing changes is a permissions decision (see team access below).

Ephemeral resources (Terraform 1.10 and later) read a value for the current run only and never write it to state or plan files. They are the right way to pull a secret from a secrets manager; a data source for the same secret persists every attribute in state, and wrapping its use in sensitive() only changes display.

Examples from current providers:

• ephemeral "aws_secretsmanager_secret_version" — reads a Secrets Manager value (the data source of the same name stores it in state; the aws_secretsmanager_secret data source returns metadata, not the value).
• ephemeral "aws_secretsmanager_random_password" — generates a password without storing it.
• ephemeral "vault_kv_secret_v2" — reads a Vault KV v2 secret.
• ephemeral "vault_database_secret" and ephemeral "vault_aws_access_credentials" — request dynamic secrets from Vault's database and AWS secrets engines (see the Vault section).

Where an ephemeral value may be referenced

Terraform fails at plan time if an ephemeral value is used anywhere that would persist it. Permitted contexts:

The rule is enforced, not advisory: an ephemeral value routed into any ordinary argument stops the plan with an error; Terraform never silently stores or redacts it. Configuring a provider (for example a database or Kubernetes provider) with an ephemeral credential, on the other hand, is exactly the intended use.

Passing ephemeral values into modules

A local derived from an ephemeral value is itself ephemeral. When it is passed to a child module, the receiving variable must declare ephemeral = true; sensitive = true is not a substitute. Terraform enforces this at the module call, even if the child would only hand the value to a write-only argument.

variable "password" {
  type      = string
  sensitive = true
  ephemeral = true
}

Write-only arguments (Terraform 1.11 and later), named with a _wo suffix, are sent to the provider during the operation and then discarded — never stored in state or plan files. Each is paired with a _wo_version argument that is stored. Examples: password_wo / password_wo_version on aws_db_instance, and secret_string_wo / secret_string_wo_version on aws_secretsmanager_secret_version.

Because Terraform keeps no copy of the write-only value, it has nothing to compare. Two consequences follow:

• A new value alone causes no change. If the source secret rotates (in Vault KV, say) or an ephemeral generator produces a different random password each run, terraform plan reports no changes.
• Rotation is triggered by bumping the version. Changing the _wo_version value makes the provider send the current write-only value in an in-place update. That turns rotation into a deliberate, reviewable change — a one-line pull request — without replacing the resource.

A secret stays out of state only if every hop is non-persistent: an ephemeral source or generator feeding a write-only argument. A managed random_password resource stores its result in state even if the next hop is write-only, and an ephemeral value cannot go into an ordinary argument like secret_string at all.

ephemeral "aws_secretsmanager_random_password" "api" {
  password_length = 32
}

resource "aws_secretsmanager_secret_version" "api" {
  secret_id                = aws_secretsmanager_secret.api.id
  secret_string_wo         = ephemeral.aws_secretsmanager_random_password.api.random_password
  secret_string_wo_version = 1   # bump to rotate
}

The general rule: the version argument is the only in-place rotation trigger for a write-only value. Recreating the resource would also send the current value, but that is a destroy-and-create, which is rarely acceptable for stateful infrastructure.

When rotation must happen on a schedule, even with no Terraform run, the strongest design is one where Terraform never generates or handles the secret at all. For Amazon RDS, setting manage_master_user_password = true on aws_db_instance makes RDS create the master password in AWS Secrets Manager and rotate it automatically (every 7 days by default). Applications read it from Secrets Manager using the secret ARN exposed in the master_user_secret attribute.

• manage_master_user_password conflicts with password and password_wo. To switch an existing instance, add the flag and remove the password argument in the same change; because the two are mutually exclusive in the schema, Terraform reports a configuration error while both are set.
• master_user_secret_kms_key_id sets the customer managed KMS key that encrypts the secret. Don't confuse it with kms_key_id, which is the storage encryption key for the database instance.

Pick by the stated requirement: "never in state, rotated through pull requests" points to write-only; "rotated on a schedule, Terraform must never handle it" points to service-managed.

HCP Terraform integrates with HashiCorp Vault in three distinct ways, and the exam expects you to keep them apart.

1. Authenticating the Vault provider (Vault dynamic provider credentials)

Instead of a long-lived token in a variable, the run presents a workload identity token to Vault's JWT auth method and receives a short-lived Vault token per phase. Set workspace environment variables TFC_VAULT_PROVIDER_AUTH=true, TFC_VAULT_ADDR and TFC_VAULT_RUN_ROLE (or per-phase TFC_VAULT_PLAN_ROLE/TFC_VAULT_APPLY_ROLE). Then in the configuration:

• Remove address, token and namespace from the vault provider block — HCP Terraform supplies them as environment variables.
• Use Vault provider 3.18.0 or later with skip_child_token = true.
• If the JWT auth method lives in a namespace (common on HCP Vault Dedicated, under admin), set TFC_VAULT_NAMESPACE; otherwise the login targets a path that does not exist.
• The token's aud defaults to vault.workload.identity, or the value of TFC_VAULT_WORKLOAD_IDENTITY_AUDIENCE. The Vault role's bound_audiences must list it, so a custom audience per cluster means updating the role.
• Token lifetime: a non-renewable token simply expires at its TTL, which HCP Terraform caps at 2 hours, so a longer apply fails partway. HashiCorp recommends renewable tokens with a short TTL (such as 20 minutes): HCP Terraform keeps renewing them for as long as the phase runs and revokes them at the end.

2. Reading dynamic secrets in configuration

Vault secrets engines can mint credentials on demand (database users, AWS keys). Read them with the provider's ephemeral resources — vault_database_secret for a database secrets engine role, vault_aws_access_credentials for the AWS engine (which also exposes lease_id and lease_duration without persisting them). The data source forms (and generic reads such as vault_generic_secret) write the credentials into state, where they outlive their lease. Ephemeral resources need Terraform 1.10+.

HashiCorp's Vault dynamic credentials guide also says data sources that generate dynamic secrets must not be used with Vault dynamic provider credentials: each phase gets its own short-lived Vault token, so a secret generated under the plan's token can be revoked before apply uses it — the classic symptom is plans that succeed and applies that fail with invalid credentials. For cloud credentials, use Vault-backed dynamic credentials instead.

3. Vault-backed dynamic credentials for clouds

Here HCP Terraform authenticates to Vault with its workload identity token, and Vault's AWS, GCP or Azure secrets engine issues the cloud credentials for each plan and apply. HashiCorp gives two reasons to choose it over direct cloud dynamic credentials:

• Cloud credential management is consolidated centrally in Vault.
• Terraform Enterprise's OIDC metadata endpoints need not be exposed to the cloud provider — ideal when TFE sits on a private network the cloud cannot reach and Vault shares that network.

It still relies on a Vault JWT auth role, and credentials are never stored. For AWS, TFC_VAULT_BACKED_AWS_AUTH_TYPE is iam_user, assumed_role or federation_token. With iam_user, Vault creates a brand-new IAM user and key per phase, and IAM's eventual consistency can make a fresh key fail with InvalidClientTokenId for a moment; HashiCorp suggests TFC_VAULT_BACKED_AWS_SLEEP_SECONDS to make the run wait after receiving the keys.

Direct dynamic provider credentials only stay secure if the cloud's trust policy admits exactly the runs you intend. HCP Terraform's workload identity token carries a sub claim in the form:

organization:ORG:project:PROJECT:workspace:WORKSPACE:run_phase:plan|apply

HashiCorp says the trust must check, at a minimum, the aud claim (so tokens meant for another platform are refused) and a sub that includes your organization name (so other organizations cannot match). Project, workspace and run phase narrow it further. Wildcarding the organization segment is the classic mistake: the remaining segments are just names, and anyone in any HCP Terraform organization can create a project or workspace with the same name.

On AWS, operator choice matters:

• StringEquals compares literally — a * in the value is just a character, so run_phase:* under StringEquals matches nothing and every run fails.
• StringLike treats * as a wildcard. Keep the organization, project and workspace literal and wildcard only what you mean to (for example the run phase).

The mechanics of configuring multiple credential sets per workspace and project are covered in the CI/CD lesson on dynamic credentials; this lesson is about getting the security decision right.

Separate identities for plan and apply

A plan is not harmless: terraform plan executes code with the run's credentials. During plan, providers refresh existing objects and read data sources by calling cloud APIs, and the external data source runs an arbitrary program. Creates, deletes and provisioners (including local-exec) happen only during apply. So configuration from an unreviewed pull request can use whatever the plan identity is allowed to do.

The response is a read-only plan identity and a separate write identity for apply:

• In HCP Terraform, set TFC_AWS_PLAN_ROLE_ARN to a read-only role and TFC_AWS_APPLY_ROLE_ARN to the write role, and pin each role's trust policy to its own run_phase. If the write role trusts run_phase:*, a plan token can assume it and the split is cosmetic. Separate roles need separate trust policies.
• Team permissions (who may queue or apply) and policy checks (what the plan proposes) act on people and plan results, not on code executing during the plan — they do not replace an identity split.

Least privilege for a plan role with an S3 backend

Plans take the state lock by default. With use_lockfile = true, the S3 backend creates and deletes a <key>.tflock object beside the state, so the plan role needs s3:GetObject, s3:PutObject and s3:DeleteObject on that lock object only, plus s3:ListBucket and s3:GetObject on the state key. Least privilege here means scoping each write action to the single object that needs it rather than to the bucket or prefix.

GitHub Actions can exchange a short-lived OIDC token for cloud credentials, removing stored keys from the pipeline. Two things decide whether it works and whether it is safe.

The job must be allowed to request a token: declare permissions: id-token: write on the workflow or job. Without it, the credentials step fails; no other permission scope grants token issuance.

The trust policy must match the right sub. GitHub builds sub from the job context (the classic format; repositories created after GitHub introduced its immutable subject format use a different form, so check which one a repository issues):

The environment form takes precedence: once a job references an environment, its token no longer carries the branch or pull-request form, so any trust policy written against those forms stops matching that job until it is updated to the environment form. That is also the stronger design: trusting a protected environment ties the write role to the environment's protection rules, such as required reviewers. Trust the read-only plan role from pull_request; avoid wildcard environments, which also admit unprotected ones.

API-driven pipelines need a token, and the right one is the one that survives staffing changes and reaches only what the pipeline needs.

The selection rule: match the token to the work and to its lifetime. Anything that queues or applies runs needs a token whose permissions are workspace permissions — scoped through a team, so it outlives any one person. Organization-level management that never runs anything fits the organization token.

Rotation without downtime: under the current team token API a team can hold multiple tokens, each with a unique description, and creating one does not invalidate the others. Create a second token, switch the pipeline, then delete the old one. The legacy single-token API invalidates the old token on regeneration, which causes an outage. Give tokens the shortest practical lifetime and rotate before expiry (HCP Terraform sends reminders); expiry is fixed at creation, not extended.

Because state can contain secrets, state access is a secrets decision. Every fixed workspace permission set — Read, Plan, Write and Admin — includes reading complete state versions. A team given Plan to "preview but never see secrets" can still download a password that sits in state. Sensitive variables stay unreadable to all of them, and plan logs redact sensitive values; state is the leak.

Custom workspace access sets each dimension independently. Use it when no fixed set fits, translating each stated restriction into the one dimension that controls it — "must not see full state" is a state setting, "must not change credentials" is a variables setting:

• Runs — Read, Plan or Apply: what the team may do with runs.
• Variables — No access, Read, or Read and write: whether the team can change (or delete) variables, sensitive or not.
• State versions — No access, Read outputs only, Read, or Read and write. Only Read outputs only withholds the full file; custom Read still exposes it.
• Extras — such as Lock/unlock workspace, granted separately from runs.

Project roles apply to every workspace in a project. Read is view-only (runs and state, no plans or changes). Write cannot create or delete workspaces. Maintain can create and delete workspaces (and Stacks) and manage them day to day, but cannot move workspaces, manage team access or manage the project's variable sets — those stay with project Admin. Maintain is the least privilege for a product team that runs its own workspaces while a platform team keeps access decisions.

Sharing outputs without sharing state

terraform_remote_state returns only outputs to the configuration, but the caller must be able to read the entire state snapshot; remote state sharing only decides which workspaces may do that. HashiCorp recommends the tfe_outputs data source, which reads outputs without full state access. Combined with a team token whose team has Read outputs only on the source workspace, the consuming runs can never fetch the full state.