Terraform Professional: Troubleshooting Deployment Issues with Logs, Validation, Locks and State Recovery

Troubleshoot a failed Terraform run by first deciding which layer failed, because each layer has its own evidence and its own fix. Read the error's first line and the resource address it names, then place it in one of these groups:

Only when the message alone is not enough do you turn on detailed logging. A disciplined order is: read the diagnostic, reproduce with terraform plan, enable logs for the layer you suspect, then change one thing at a time. Once the state is unlocked, terraform plan is the safe next step after an interrupted or failed apply: it changes nothing and shows what Terraform still thinks it has to do (it does still read data sources and needs the lock).

Turn logs on with TF_LOG and a level, narrow them with TF_LOG_CORE or TF_LOG_PROVIDER, and, for HCP Terraform remote runs, set the variable on the workspace as an environment variable rather than in your shell.

Local runs: TF_LOG, TF_LOG_PATH, TF_LOG_CORE and TF_LOG_PROVIDER

TF_LOG turns Terraform's detailed logging on and sets its level; the other variables only refine where the logs go or which part of Terraform produces them.

• TF_LOG accepts TRACE, DEBUG, INFO, WARN, ERROR (most to least verbose) or OFF. Logs are written to stderr.
• TF_LOG=JSON produces TRACE-level entries encoded as JSON objects. HashiCorp documents that this format is not a stable interface: it can change between releases without notice, so do not build parsers or alerts that depend on its fields. It is not the documented machine-readable UI of terraform plan -json, and it does not replace the normal output on stdout.
• TF_LOG_PATH appends the log to a file. It does nothing on its own: if no level is enabled, no file is created. To capture DEBUG entries from every run into one file, set a level such as TF_LOG=DEBUG together with a path such as TF_LOG_PATH=logs/terraform.log.
• TF_LOG_CORE and TF_LOG_PROVIDER take the same levels but enable logging for only one side. Core logs describe Terraform itself: graph building and walking, evaluation, state handling. Provider logs come from the plugins: the API requests they build and the responses they receive.

Variables such as TF_INPUT, TF_IN_AUTOMATION, TF_DATA_DIR and TF_CLI_CONFIG_FILE are real but unrelated to logging: they control prompts, output wording for automation, the working-data directory and the CLI configuration file.

HCP Terraform remote runs

In an HCP Terraform workspace with remote execution, the run happens on a HashiCorp-managed (or agent) worker, so a TF_LOG exported in your local shell never reaches it. Add TF_LOG to the workspace as an environment variable (category env), and the detailed entries appear in the run's output in the UI and in the CLI-driven run stream.

• A Terraform variable named TF_LOG does nothing: Terraform variables feed variable blocks, not the process environment.
• Leave TF_LOG_PATH unset for remote runs. It sends the entries to a file on the remote worker's disk, which you cannot read, and removes them from the run output you can read.
• Sensitive does not hide the logs. Marking the TF_LOG variable Sensitive only hides its value (the word TRACE); the variable still reaches the run, and the trace entries it turns on are printed in the run output for everyone allowed to read runs. Trace entries can contain provider API requests and responses, so treat them as sensitive: enable them for the shortest time, on the narrowest workspace, and delete the variable when debugging ends.

terraform validate checks that a configuration is syntactically valid and internally consistent without contacting any remote service: no provider APIs, no backend, no remote state. It reports problems such as an argument name the provider schema does not define, a wrong type, a reference to an undeclared variable, resource or module output, and invalid blocks.

Because it needs the provider schemas and child modules, it must run after terraform init. In a pipeline without backend or cloud credentials use:

terraform init -backend=false
terraform validate

-backend=false skips backend configuration while still installing providers and modules, so the job needs only registry or mirror access. terraform fmt -check only checks formatting, and terraform plan needs credentials, so neither replaces this pair.

What validate cannot see is anything that only the platform knows: whether an AMI ID or bucket name exists or is free, whether the pipeline's IAM permissions allow an action, whether the state is locked. Those surface at plan or apply time. A clean validate followed by an API error such as InvalidAMIID.NotFound is therefore expected behaviour, not a validate bug.

For CI tooling, terraform validate -json prints a documented JSON object with a valid flag and a diagnostics array; each diagnostic carries its severity, summary, detail and a range with the file name and start and end positions, which is exactly what inline pull-request annotations need.

Fix them through the dependency lock file: run terraform init (with -upgrade when the locked version must move), pre-populate hashes for every platform you use, and commit the result rather than deleting or hand-editing it.

Version selection errors

Most initialization errors come from the dependency lock file, .terraform.lock.hcl, which records the exact provider version Terraform selected and the checksums it trusts. Terraform never changes a locked version on its own; you tell it to.

Plain terraform init reuses a locked version for as long as it still satisfies the constraints. Once a constraint excludes the locked release, plain init stops with locked provider … does not match configured version constraint and tells you to use terraform init -upgrade, which selects the newest release the constraints allow. Never hand-edit the version line in the lock file: its hashes would no longer match. Backend flags on init (-reconfigure, -migrate-state) do not affect provider selection, and terraform get only fetches modules. Modules are not recorded in the lock file at all. Version policy itself (constraint operators, when to pin) is covered in the lesson on sharing data and managing provider and module versions.

Checksum mismatches across platforms

A checksum mismatch at init means the provider package Terraform just obtained does not match any hash recorded in the lock file for that provider version. The usual cause in a mixed team is a lock file that only contains hashes for the platform that created it.

The lock file holds two kinds of hash:

When providers come from a network or filesystem mirror there is no signed checksum list, so Terraform records only the hash for the current platform. A lock file generated on a windows_amd64 workstation through a mirror then fails on a linux_arm64 runner. A lock file generated against the public registry would not, because its zh: lines cover every published platform.

The fix that keeps verification and needs no recurring work is to pre-populate hashes for every platform you use and commit the result:

terraform providers lock \
  -platform=windows_amd64 \
  -platform=linux_arm64

By default terraform providers lock fetches packages from each provider's origin registry and ignores the installation methods in your CLI configuration. A team that can only reach a mirror (for example an air-gapped network) points it there explicitly with -fs-mirror=DIR or -net-mirror=URL.

Deleting or ignoring the lock file, or running init -upgrade in CI, removes the protection the lock file exists for. Conversely, when a teammate on another OS runs init against the registry and the diff shows only new h1: lines at the same version, that is Terraform recording the verified hash for their platform: commit it.

Error: Cycle means two or more objects depend on each other, so Terraform cannot order them. You fix it by restructuring the references; nothing at apply time can work around it.

• Adding depends_on never breaks a cycle; it only adds edges. -target does not help either, because the graph is still built from the configuration.
• The classic case is two resources whose inline blocks reference each other, such as two security groups with inline ingress rules naming each other's ID. Move the mutual part into its own resource (for AWS, aws_vpc_security_group_ingress_rule) so both groups can be created first and the rules afterwards.
• Other common sources: a module output that feeds an input of the same module through another resource, a data source that reads something the same configuration creates, and create_before_destroy, which reverses destroy ordering and propagates to dependencies, so a destroy-time edge can close a loop.

For a large cycle spanning modules, render it. terraform graph prints DOT; -draw-cycles highlights the edges that form cycles, but it is supported only when you select a real evaluation graph with -type:

terraform graph -type=plan -draw-cycles > cycles.dot
dot -Tpng cycles.dot -o cycles.png

Without -type, Terraform prints a simplified resource-only graph that cannot show the cycle. -plan=FILE renders an apply graph for a saved plan, which a cycle prevents you from producing in the first place. terraform show -json emits state or plan JSON, not DOT.

Recover a stuck lock with terraform force-unlock after confirming no run still holds it, and recover a failed state save with terraform state push errored.tfstate. These are the two different problems an interrupted run can leave behind: a lock that was never released, and state that was never written.

Stale lock

When a runner is killed mid-apply, the backend lock may remain, and every later run fails with Error acquiring the state lock. The lock info shows who held it and when. First confirm that no Terraform process is still running against that state, then run terraform force-unlock LOCK_ID with the ID from the error. force-unlock only removes the lock record; it does not change state or infrastructure. Do not run later applies with -lock=false: that invites two writers. init -reconfigure does not clear locks.

After unlocking, run terraform plan before any apply. Resources the cancelled run already created or changed show as no-ops; anything it did not finish shows as pending changes. An object that was interrupted mid-create may have been recorded as tainted and then shows as a replacement (see the timeouts and tainted objects section below). Do not state push an older version to "rewind": the real infrastructure did not rewind, so state would lie about it.

Failed state save

If the backend becomes unreachable after resources are created, Terraform cannot save the new state and writes it to errored.tfstate in the working directory. The backend still has the old state, so anyone else's plan proposes to create those objects again. Recover with one operation once the backend is reachable:

terraform state push errored.tfstate

Do not apply again first (it would create duplicates or fail with "already exists"), and do not delete the file and re-import everything by hand.

Terraform needs a resource's provider configuration to destroy it. If you delete a provider block, or an aliased one such as aws.west, in the same change as the resources that used it, the state still holds those resources, plan cannot destroy them, and it fails with Provider configuration not present naming the missing configuration.

Remove things in two steps: restore the provider configuration, apply the removal (or destroy) of its resources, then delete the provider block in a later change. Resources first, their provider configuration last.

The same rule applies to a legacy child module that contains its own provider block. Deleting the module call deletes the provider configuration with it, so its objects become orphans Terraform cannot destroy. Restore the module call, destroy its objects (for example terraform destroy -target=module.old_network), then delete the call. Modules that receive providers from their caller avoid this trap, which is why provider blocks inside reusable modules are discouraged.

Related commands do different jobs: terraform state rm makes Terraform forget the objects, leaving them running and unmanaged; terraform state replace-provider changes the provider source address, not an alias; init -upgrade reinstalls plugins and cannot recreate a configuration block.

Messages that begin Provider produced inconsistent … indicate a defect in the provider plugin, not in your configuration. The two you will meet most are Provider produced inconsistent result after apply (the provider returned a value different from what it planned) and Provider produced inconsistent final plan (the plan changed between planning and applying). Terraform itself adds that this is a bug in the provider which should be reported in the provider's own issue tracker.

The right response is to check the provider's issue tracker for a known issue, reproduce with TF_LOG=TRACE (or TF_LOG_PROVIDER=TRACE) and report it with the log, redacting secrets. If it appeared right after a provider upgrade, step around the release as described in the lock file section. Running terraform validate, restoring old state or treating it as drift will not fix it.

By contrast, Unsupported argument, Invalid for_each argument and Provider configuration not present all describe problems in your configuration, even though some mention the provider.

Fix a provider wait timeout by raising the matching key in the resource's timeouts block; fix the tainted object it left behind with terraform untaint if the object turned out healthy. Long-running operations are bounded by the provider's own wait limit, and many resources expose that block to change it per operation:

resource "aws_db_instance" "main" {
  # ...
  timeouts {
    create = "75m"
    update = "60m"
    delete = "45m"
  }
}

Match the key to the operation that timed out: create for timeout while waiting for state to become 'available' during a create or restore, delete for timeout while waiting for resource to be gone during destroy. Which keys a resource supports is listed in its provider documentation. Similarly named settings control other things: -lock-timeout is how long to wait for a state lock, -parallelism limits concurrent operations, and a connection block's timeout applies to provisioners.

A create that times out leaves the object recorded in state as tainted, so the next plan proposes to destroy and re-create it, even if it finished provisioning later and is healthy. To keep it, run terraform untaint ADDRESS and then plan, after raising the timeout so it does not recur. Refreshing does not clear a taint, and removing and re-importing it is more work for the same result.

Both are platform API refusals with a state-shaped cause: a destroy fails because the argument that allows it was never applied to state, and a create fails because the object already exists outside this state.

Destroy-time arguments such as force_destroy

Destroy uses the arguments stored in state, not those in the configuration you just edited. force_destroy on aws_s3_bucket, which lets the provider delete every object before deleting the bucket, only takes effect once a successful apply has written it to state. So after BucketNotEmpty:

1. Set force_destroy = true and run terraform apply (an in-place update with no change to the real bucket).
2. Run terraform destroy, or remove the resource block and apply.

If you have already deleted the resource block, restore it with force_destroy = true, apply, then delete it and apply again. A removed block with destroy = true, a refresh, or -target all still destroy with the stored value false. prevent_destroy is a separate safeguard that blocks destroys; setting it to false changes nothing here. Other destroy-time arguments, such as RDS final-snapshot or deletion-protection settings, follow the same apply-then-destroy pattern.

Objects that already exist outside state

An EntityAlreadyExists, AlreadyExists or similar API error on create means a real object with that unique name exists outside this state. What you do depends on who owns it:

Changing the resource's name argument creates a second object (while renaming only the resource address, such as aws_iam_role.x to aws_iam_role.y, retries the same create and fails again); -replace and state rm act on objects Terraform already tracks; ignore_changes only suppresses updates. Import block syntax and lifecycle mechanics are covered in the lesson on importing infrastructure and managing its lifecycle.