Terraform Professional: Inspecting State and Remediating Drift

Five commands read state and never write a new snapshot or touch infrastructure: terraform state list, terraform state show, terraform show, terraform output and terraform state pull. Each answers a different question, and picking the wrong one usually fails because it takes the wrong kind of argument.

terraform state list is the inventory tool. A prefix such as module.network filters to every instance inside that module, and -id=vol-0f3c... filters to instances whose id attribute matches, which is how you turn a cloud ID from an alert into a Terraform address. state show cannot search: it needs the address already. terraform output never sees resource attributes unless a root output exposes them.

By contrast, terraform refresh, terraform state push, state rm, state mv, state replace-provider, taint and untaint all write a new state snapshot. During a change freeze that distinction is the whole rule.

What does terraform console do, and why can it block a pipeline?

terraform console opens an interactive prompt that evaluates any expression, including functions, against the current workspace's state: length(aws_lb_target_group.api), aws_route_table.private["az1"].id, cidrsubnet(var.cidr, 8, 2). It creates no plan and changes nothing, which makes it the right tool when you need to compute from state rather than just print it.

The catch is locking. With a remote backend, console reads state from the backend and holds a state lock for as long as the session is open. A console left running in a terminal therefore makes every state-writing operation on that workspace (a pipeline apply, for example) fail with "Error acquiring the state lock".

When the lock holder is known and still alive, release the lock from its owner: exit the console. terraform force-unlock LOCK_ID is for locks whose owner has died; using it against a live process overrides protection that process still relies on. -lock=false disables locking entirely, and -lock-timeout only waits for a lock that will never be released on its own.

A resource instance address is built segment by segment: the module path, then type.name, then an instance key on whichever block repeats. count keys are numbers ([0]); for_each keys are quoted strings (["green"]).

The key attaches to the segment that carries count or for_each. If the module call repeats and the resource inside does not, the key goes on module.site, never on the resource. Dropping the module path, using a numeric index for a for_each resource, or putting the key on the wrong segment produces an address that matches nothing.

Shell quoting matters. On Linux and macOS shells, wrap the whole address in single quotes so the inner double quotes reach Terraform intact:

terraform state show 'module.cache.aws_elasticache_cluster.node["blue"]'

Without the single quotes, bash strips the double quotes and Terraform receives an invalid key. Windows cmd.exe and PowerShell need different escaping; HashiCorp's state command pages show each form.

Sensitive values leak because Terraform stores them in state in plain text and only its human-readable output redacts them. Marking a variable, output or provider attribute sensitive hides it in plan, apply, plain terraform show and the plain terraform output listing, where it appears as (sensitive value) or <sensitive>. Machine-readable forms print the real value:

• terraform show -json prints every attribute in state, sensitive ones included, in plain text. The JSON carries a sensitive_values structure per resource that marks which attributes are sensitive, so a pipeline that must not log secrets can strip exactly those paths. Filtering for the string "(sensitive value)" removes nothing, because that placeholder never appears in JSON.
• terraform output -json and terraform output -raw NAME print sensitive root outputs in plain text. -raw exists so scripts can consume a single string value.

The consequence: anyone who can read state can read every secret in it. Protect the state store and its access policy, and treat show -json, output -json and output -raw in shared CI logs as exposure points. (Choosing a secrets strategy so that fewer secrets land in state belongs to the security domain.)

terraform state pull downloads the raw current snapshot from the configured backend and prints it to stdout, so terraform state pull > pre-refactor-backup.tfstate is the standard backup before risky state work. It changes nothing. The output is the real state format, with its lineage (a unique ID assigned when the state was first created) and serial (a counter that increases with every write), so it can be pushed back unchanged. terraform show -json is a different, read-only JSON representation and is not a restorable snapshot.

terraform state push FILE overwrites the remote snapshot and has two safety checks. It refuses when:

• the lineage differs: the file probably belongs to a different state altogether;
• the destination serial is higher than the file's: the remote state holds writes the file does not.

A snapshot pulled from the same workspace yesterday shares its lineage, so restoring it after newer writes trips the serial check. -force disables both checks and the remote snapshot is replaced entirely; nothing is merged. Before forcing, confirm that every change recorded after the file's serial is meant to be thrown away.

Drift is a difference between the real infrastructure and what Terraform recorded or declared, caused by changes made outside Terraform (a console edit, an autoscaler, another tool). You either revert it to the configuration or accept it as the new desired state.

Refresh-only mode is how you accept drift into state. terraform plan -refresh-only reads the real objects and shows how state would change to match them, proposing no infrastructure actions; it writes nothing. terraform apply -refresh-only shows the same and, once approved, writes the updated state. HashiCorp recommends it over the older terraform refresh, which applies the same update without a review step.

A refresh-only apply fixes state only. The configuration is still the desired state, so:

• if an operator grew an EBS volume from 50 to 80 GiB and the configuration still says 50, the next normal apply plans an in-place change back to 50 (which fails at the API, since EBS volumes cannot shrink). To keep 80, also change the configuration;
• if an object was deleted outside Terraform, the refresh-only apply drops it from state, and the next normal plan proposes creating it again while its block remains. To accept the deletion, remove the block too.

The refresh-only plan is also the complete drift report. A normal plan refreshes too, but its "Objects have changed outside of Terraform" note lists only external changes that may have affected the actions it proposes; a refresh-only plan lists every difference between state and reality.

How do you revert drift, and what hides it?

To revert drift, run a normal terraform plan and apply. The plan refreshes state from the real objects, compares them with the configuration and proposes whatever brings reality back: an in-place update for a changed attribute, or a create for an object that was deleted outside Terraform. No import or state edit is needed first; import is for adopting objects that exist but are untracked.

Drift only becomes visible when Terraform reads the real objects. Anything that skips that read cannot detect it:

The rule: drift is found only by a command that refreshes, meaning it asks the providers for the real objects. Flags that shape output, concurrency or locking leave the refresh in place.

-refresh=false trades correctness for speed on large workspaces; HashiCorp warns it can produce an incomplete or incorrect plan. Never use it in a job whose purpose is finding drift.

To forget an object while it keeps running, remove it from state with either a removed block (reviewed, in a plan) or terraform state rm (immediate CLI edit). Deleting the resource block on its own is not enough: Terraform then plans to destroy the object.

removed {
  from = aws_db_instance.reporting

  lifecycle {
    destroy = false
  }
}

HashiCorp recommends removed blocks because the change shows in the plan. Two traps:

• state rm with the block left in place. The next plan sees a declared resource with nothing tracked and proposes to create it; with a fixed unique name (an S3 bucket, say), the apply then fails because the surviving object holds the name.
• One instance of a for_each set. A removed block cannot target aws_iam_user.contractor["temp-ops"], and pointing it at aws_iam_user.contractor forgets every instance. Drop the key from the for_each collection and run terraform state rm 'aws_iam_user.contractor["temp-ops"]'.

When a resource is renamed, moved into a module or re-keyed, its address changes; unless state is told, Terraform plans to destroy the object at the old address and create a new one at the new address. A moved block or terraform state mv rebinds the existing object instead.

moved {
  from = aws_sqs_queue.jobs
  to   = aws_sqs_queue.tasks
}

• Direction. from is the old address that state currently tracks; to is the new address in configuration. Reversed, the block points at an address with nothing tracked.
• moved block vs state mv. A moved block is applied during a normal plan, so the move appears in review and works for every workspace that uses the code. terraform state mv SOURCE DESTINATION edits state immediately; add -dry-run first to list the instances that would match without changing anything. Moving a block into a child module observability is terraform state mv aws_cloudwatch_log_group.audit module.observability.aws_cloudwatch_log_group.audit.
• count to for_each. aws_sqs_queue.dlq[0] and aws_sqs_queue.dlq["orders"] are unrelated addresses, and Terraform does not map indexes to keys in any order. Write one moved block per instance. The only moves Terraform infers on its own are between an unkeyed address and index [0] when count is added or removed.

An import block is not a substitute: it adopts an untracked object at the new address but leaves the old address in state, planned for destruction of that same object. Import workflows themselves belong to the infrastructure-lifecycle domain.

How do you repoint state to a new provider source?

Every resource in state records the fully qualified source of the provider that manages it, such as registry.terraform.io/hashicorp/google. Changing required_providers to a fork and running terraform init installs the fork but leaves those records unchanged, so planning fails while state still names a provider the configuration no longer requires.

terraform state replace-provider \
  -auto-approve \
  hashicorp/google registry.example.net/platform/google

terraform state replace-provider FROM TO rewrites the provider address of every resource that uses FROM, in one step. The first argument is the provider in state now, the second is the replacement; reversed, nothing matches. It asks for confirmation unless you pass -auto-approve (needed in a pipeline) and writes a state backup. init -upgrade only selects newer versions for the lock file, refresh only updates attribute values, and state mv works on resource addresses, so none of them change the recorded provider. Removing and re-importing every resource also ends in correct state but takes far more steps and risk.

To rebuild one object whose configuration has not changed, use terraform plan -replace=ADDRESS (or apply -replace). Terraform plans a replace action for that address even when the configuration would cause no action, and the replacement appears in a plan you review first. Quote keyed addresses for the shell: -replace='aws_lambda_function.worker["ingest"]'.

terraform taint is deprecated in favour of -replace: it edits state immediately to mark the object tainted, outside any reviewed plan, and other runs can pick that up before you meant them to. -target does not force replacement; it only limits what is planned.

Terraform also taints objects itself. If a multi-step create fails partway, typically a creation-time provisioner erroring, the object is marked tainted and the next plan proposes to replace it. When you have confirmed the object is actually healthy, terraform untaint ADDRESS clears the flag in state without touching the real object. A refresh does not clear taint (it updates attribute values only), and state rm orphans the object and plans a new one.

HCP Terraform health assessments run on a schedule without anyone queuing a run: drift detection compares real infrastructure with the workspace's configuration, and continuous validation re-evaluates check blocks and conditions. They run a speculative plan, so they never change infrastructure, and results appear on the workspace's Health page.

Enablement. Workspace admins can turn assessments on per workspace; an organization owner can enforce them for all eligible workspaces, which overrides the workspace setting so no workspace can opt out.

Eligibility. A workspace is assessed only when:

• it uses remote or agent execution (local execution mode is not eligible);
• it runs a supported Terraform version (0.15.4 or later for drift detection, 1.3.0 or later for continuous validation);
• it has applied successfully at least once (with no real infrastructure there is nothing to assess), and its latest run succeeded; after an errored, canceled or discarded run, assessments pause until a run applies successfully.

The VCS-driven workflow and the auto-apply setting are not prerequisites. Health assessments are also a paid-edition feature, so check that your organization's HCP Terraform plan includes them.

Remediation. Detected drift is resolved in one of two ways: overwrite it by queuing a normal plan and apply from the current configuration, or accept it by changing the configuration to match reality and pushing a new configuration version. A refresh-state run updates state but leaves the configuration unchanged, so the next assessment still reports the difference, and an on-demand assessment simply re-reports it.

Some drift is expected because another system owns a value after creation: scaling policies change an Auto Scaling group's desired_capacity, and a compliance tool stamps its own tags on resources. Reverting it on every apply causes outages or endless diffs, so tell Terraform to tolerate it.

resource "aws_autoscaling_group" "workers" {
  # ...
  desired_capacity = 3

  lifecycle {
    ignore_changes = [desired_capacity]
  }
}

ignore_changes uses the configured value at creation, then stops planning updates for the listed arguments. It works in both directions: Terraform neither reverts external changes nor applies later edits you make to that argument in configuration. To push a new baseline once, remove the argument from ignore_changes, apply, then add it back. -target still honours ignore_changes. The other lifecycle arguments (prevent_destroy, create_before_destroy, replace_triggered_by) govern destruction and replacement and do not stop an in-place update.

For tags applied by other systems across many AWS resources, use the AWS provider's ignore_tags block instead of editing every resource:

provider "aws" {
  ignore_tags {
    key_prefixes = ["compliance-scan/"]
  }
}

keys matches exact tag names; key_prefixes matches every tag starting with the prefix. It applies to every resource using that provider configuration (each aliased configuration needs its own), and drift on your own tags is still reported. Ignoring the whole tags argument per resource would hide that drift too.