Terraform Professional: Remote Backends and Remote State

A backend is the part of Terraform that decides where state is stored and how it is locked; a remote backend keeps that state in a shared service (an S3 bucket, an Azure storage container, a GCS bucket, HCP Terraform) instead of a terraform.tfstate file on one engineer's disk. You declare it once, inside the terraform block:

terraform {
  backend "s3" {
    bucket       = "acme-state"
    key          = "network/terraform.tfstate"
    region       = "eu-west-1"
    use_lockfile = true
  }
}

Three rules shape every backend question:

• Literals only. Terraform initializes the backend before it evaluates the rest of the configuration, so the block cannot refer to any named value: no var., no local., no data source attributes and no terraform.workspace. A reference is an error at terraform init — a variable's default value or a terraform.tfvars file does not rescue it.
• One backend per configuration. A root module has at most one backend block, and a cloud block (HCP Terraform) cannot appear alongside it.
• Init owns it. Adding, removing or changing any backend argument means running terraform init again; other commands refuse to run until you do.

Values that genuinely differ between environments therefore come from partial configuration at init time (next section), and per-workspace separation comes from the backend's own workspace layout, never from an expression in the block.

Partial configuration means leaving some backend arguments out of the committed block and supplying them when you run terraform init. This is HashiCorp's answer to "one configuration, several environments, different buckets or accounts" without duplicating Terraform files. The missing values can come from:

• a file: terraform init -backend-config=prod.s3.tfbackend — HashiCorp recommends the *.backendname.tfbackend naming; the file holds top-level key = value lines;
• a key/value pair: terraform init -backend-config="bucket=acme-prod-state";
• an interactive prompt for required arguments still missing (only when a terminal is attached).

Merge order: Terraform starts from the block, then applies each -backend-config option in the order given. Command-line options override the block, and when two options set the same argument the later one wins. So -backend-config=qa.s3.tfbackend -backend-config="region=eu-central-1" ends with eu-central-1, whatever the file or the block said. The merged result is saved in the .terraform directory.

-backend-config is an init-only option; plan and apply do not accept it. A pipeline that cannot edit its command line can inject it through TF_CLI_ARGS_init. Terraform inserts that value directly after the init subcommand and before any flags typed on the command line, so under the later-wins rule a -backend-config typed on the command line still overrides one injected through the variable.

Why the alternatives fall short follows from the same mechanism. Backend values are fixed at init, before any expression is evaluated, so anything that relies on evaluation (variables, locals, terraform.workspace) is rejected, and commands after init have no backend options to take. Keeping a separate backend_override.tf per environment does change the backend mechanically, but it trades one committed block for a set of files that have to be swapped in before each init.

Every value you put into backend configuration — in the block, in a .tfbackend file or as a -backend-config pair — is written in plain text to .terraform/terraform.tfstate in the working directory and captured in every saved plan file (terraform plan -out=...). That includes access keys and secrets.

It is not written to the remote state object, to the dependency lock file .terraform.lock.hcl (which records provider versions and checksums) or to terraform.tfstate.backup. Those hold different data.

HashiCorp's guidance is therefore to supply backend credentials through environment variables (for S3, the standard AWS_* variables or, better, a role or instance profile the runner already has), never through backend arguments, and never to commit the .terraform directory or saved plans. An uncommitted .tfbackend file still lands in .terraform/terraform.tfstate, so "keep the file out of git" does not solve the problem.

When an initialized working directory's backend configuration changes — a new bucket, a new key, a different backend type — terraform init must be told what to do with the existing state, and you pick one of two flags:

Non-interactive jobs: migration asks for confirmation, and with -input=false (or no terminal) Terraform aborts rather than guess. The CI form is terraform init -migrate-state -force-copy -input=false (or just -force-copy -input=false). -reconfigure -force-copy still copies nothing, because -reconfigure means "ignore the old state".

Recovering from a mistaken -reconfigure: it does not delete anything. The state object at the old location is untouched, so you point the configuration back at it, re-initialize, and then perform the move properly with -migrate-state. A plan that suddenly wants to create everything after a backend edit almost always means state was not migrated, not that resources drifted.

Two other init flags are often confused with these: -upgrade refreshes provider and module versions within their constraints, and -backend=false skips backend initialization entirely. Neither moves state.

State locking stops two runs from writing the same state at once: a run that changes state takes the lock first, and a second run that cannot get it fails with "Error acquiring the state lock". On the S3 backend, locking is configured with use_lockfile = true (generally available since Terraform 1.11). It is off by default, so you must opt in.

• The lock is an S3 object at the state key with .tflock appended — for prod/app.tfstate it is prod/app.tfstate.tflock. No extra AWS resource is needed.
• The role needs s3:GetObject, s3:PutObject and s3:DeleteObject on that lock object. A policy scoped tightly to the state object fails as soon as locking is enabled, and one without DeleteObject could take a lock it can never release.
• If kms_key_id is set, the lock file is encrypted with the same key.

The older dynamodb_table locking still works but is deprecated. The S3 backend documents that both may be configured at the same time to support migration: runs with both settings acquire both locks, so they still conflict with older runs that only check DynamoDB. The safe cutover is therefore: add use_lockfile while keeping dynamodb_table; once no job (including rollbacks) can run the old configuration, remove dynamodb_table and delete the table. Switching in one step leaves a window where old and new runs check different locks and can write concurrently.

Waiting instead of failing: by default a run fails immediately when the lock is held. -lock-timeout=10m makes it retry for up to that duration, so overlapping pipeline runs queue rather than fail. -lock=false removes the protection and is never the answer for shared state; breaking a stale lock (terraform force-unlock) is covered in the lesson on inspecting state and remediating drift.

State often contains secrets in plain text (database passwords, private keys, outputs marked sensitive), and it is the only record of what Terraform manages, so it needs both recoverability and confidentiality. Each is a different control:

With a customer managed key, reading state needs kms:Decrypt and writing it needs kms:GenerateDataKey (with kms:Encrypt), which is why a role that can read the state at init can still fail at the first plan or apply that writes state or a lock object (with use_lockfile, even a plan writes the encrypted lock). An explicit deny in a bucket policy overrides any Allow elsewhere, which is what makes "only the pipeline role may read" achievable in an account where other roles have wide S3 permissions; a KMS key policy limiting kms:Decrypt is the alternative route. Audit features such as logging are detective controls: they tell you who read the state afterwards, but neither restore it nor prevent the read.

On the S3 backend, the default workspace stores state at the key exactly as written, and every other CLI workspace stores it at <workspace_key_prefix>/<workspace>/<key>, where workspace_key_prefix defaults to env:. With key = "network/terraform.tfstate":

Consequences you should be able to reason about:

• Separate state per workspace needs no expression. A literal key already gives every workspace its own object, so engineers move between environments with terraform workspace select alone. Partial configuration or override files would also separate keys, but each switch would then need a new init.
• IAM scoped to a prefix breaks for non-default workspaces. A role allowed only under network/ works for the default workspace and fails with AccessDenied in staging, because that state sits under env:/. Setting workspace_key_prefix to a path inside the allowed prefix fixes it without moving the default workspace's state.
• Lock permissions follow the workspace path too: grants written for the default workspace's .tflock do not cover another workspace's lock.

The cloud block, nested in the terraform block, is HashiCorp's recommended way to connect a configuration to HCP Terraform (or Terraform Enterprise) for state storage and CLI-driven runs. It replaces a backend: HCP Terraform manages state itself, so no backend block may sit beside it. The older backend "remote" is still supported but is not the recommended path.

terraform {
  cloud {
    organization = "acme"
    workspaces {
      tags = { app = "billing" }
    }
  }
}

• name vs tags: workspaces maps the directory either to one named workspace (name) or to every workspace carrying the given tags (tags, as a key/value map or a list of key-only tags). They are mutually exclusive — setting both is an init error. To target one workspace from a tags-based block, use terraform workspace select (after init) or TF_WORKSPACE.
• Environment variables: TF_CLOUD_ORGANIZATION, TF_CLOUD_HOSTNAME, TF_CLOUD_PROJECT and TF_WORKSPACE are read only when the matching attribute is omitted; if both are set, the configuration wins. When every value comes from the environment you still commit an empty cloud {} block — without it there is no HCP Terraform integration at all.
• TF_WORKSPACE with tags: allowed, as long as it names an existing workspace that matches the tags. It is how a job with no terminal picks one workspace at init while the committed block keeps serving all of them. TF_CLOUD_PROJECT scopes the project, not the workspace.
• Remote execution: new workspaces default to remote execution, so a CLI plan uploads the configuration and runs on HCP Terraform workers. Your shell's environment variables are not sent; provider credentials must be workspace environment variables, a variable set or dynamic provider credentials — or the workspace is switched to local execution.

To move state from a backend into HCP Terraform with the CLI, replace the backend block with a cloud block and run a normal terraform init. Init detects the change and offers to copy the existing state into the HCP Terraform workspace.

The migration works because init compares the newly configured target with the backend recorded in .terraform and copies state only when both are known and different. That explains the common failures: a configuration cannot declare the old backend and the cloud block at once, an init option that tells Terraform to forget the recorded backend leaves nothing to copy, and the TF_CLOUD_* variables only fill in attributes of a cloud block that exists, so they cannot switch a configuration away from its backend on their own.

Several CLI workspaces at once: a cloud block with name maps the directory to a single workspace, so it can receive only one state. With tags, init offers to migrate each local or backend workspace into its own HCP Terraform workspace (optionally renaming them with a pattern), and the same committed block can target any of them afterwards. Migrating dev, staging and prod in one init therefore calls for a tags-based block.

The terraform_remote_state data source reads the root-module outputs of another configuration's latest state snapshot, using that configuration's backend settings:

data "terraform_remote_state" "network" {
  backend   = "s3"
  workspace = terraform.workspace
  config = {
    bucket = "acme-state"
    key    = "network/terraform.tfstate"
    region = "eu-west-1"
  }
  defaults = {
    log_bucket = "acme-logs-placeholder"
  }
}
# data.terraform_remote_state.network.outputs.vpc_id

• Root outputs only. Values appear under the outputs attribute. A child module's output is invisible until the producer re-exports it with a root-level output (for example value = module.vpc.vpc_id); there is no outputs.module... path.
• workspace defaults to default, regardless of the workspace the consumer runs in. If producer and consumer both use workspaces per environment, set workspace = terraform.workspace or a non-default consumer silently reads the default workspace's values. (Hard-coding the key to env:/prod/... would read prod's state, but it is a brittle fix that pins one environment and ignores the consumer's workspace; the data source already applies the backend's own workspace layout.)
• defaults supplies values for outputs that are missing or when the remote state is empty, so a plan in a fresh environment succeeds without editing every reference — the fallback lives in one place, beside the data source, rather than at each reference.
• Full state access. The reader needs access to the entire snapshot, which often contains secrets — marking an output sensitive does not change that.

In HCP Terraform, access to a workspace's state by other workspaces' runs is controlled by remote state sharing, configured on the producer (the workspace being read). New workspaces share with nobody by default. The producer can share with all workspaces in the organization, all workspaces in its project (a workspace in another project is outside that share) or specific workspaces; choose the narrowest that works. Changing settings on the consumer grants nothing.

tfe_outputs (from the tfe provider) reads only a workspace's outputs, so HashiCorp recommends it over terraform_remote_state when the producer's state holds sensitive data. It calls the API with the provider's token, so it is governed by that token's permissions, not by remote state sharing — restricting sharing alone does not stop a token-based reader. The underlying principle is to give a consumer the narrowest view that works: outputs rather than whole state, and one producer rather than an organization-wide share.