Terraform Professional: CI/CD Pipelines and Deployment Strategies

A Terraform CI/CD pipeline runs the same small set of stages for every change: check formatting and validity, install providers, produce a plan, put that plan in front of a reviewer or a policy check, and apply it. The pipeline either runs the Terraform CLI itself on a runner (GitHub Actions, GitLab, Jenkins and similar) or hands the work to HCP Terraform through one of its three workflows.

Whatever the workflow, a professional pipeline keeps three properties: every run is non-interactive, the plan a person approves is the plan that is applied, and critical environments never apply without review. Saved plans, -detailed-exitcode, TF_IN_AUTOMATION and locking are covered in the lesson on planning and applying changes; dynamic provider credentials and variable files are covered in the sibling topic on that subject.

Run terraform fmt -check -recursive from the repository root. It never rewrites files and exits non-zero when any file is not in canonical format, which is what fails the job.

The general rule: only -check makes fmt read-only and turns unformatted files into a failing exit code; the other flags change what is printed, not whether files are rewritten. Scope is a separate decision, so a gate that omits -recursive passes while nested modules drift. terraform validate is a different gate: it checks syntax and internal consistency, not layout, so badly formatted but valid code passes it. A useful read-only gate with readable logs is terraform fmt -check -diff -recursive.

Use a provider plugin cache only where one init runs at a time and commit the dependency lock file; on isolated runners use a filesystem mirror or -plugin-dir instead.

The plugin cache

• Set with TF_PLUGIN_CACHE_DIR or plugin_cache_dir in the CLI configuration file; the two are equivalent.
• The cache is not guaranteed to be concurrency safe. Jobs that run terraform init at the same time must each have their own cache directory, or they can read half-written packages.
• Terraform uses a cached package only when it matches a checksum already recorded in .terraform.lock.hcl. With no lock entry, init installs from upstream so that it can record checksums, and the cache is bypassed. Commit the lock file to get cache hits.
• TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE lets init use the cache without a lock entry, but the new entry then holds only the cached package's checksum, which may be invalid on other platforms. It is an escape hatch, not a fix.
• Even with a cache, init still asks its installation methods (normally the registry) which versions exist. A cache saves downloads; it does not make a runner work offline.

Runners with no registry access

The mirror directory uses the hostname/namespace/type/ layout. -lockfile=readonly (verify the lock file, never update it) is a good CI setting but conflicts with -upgrade, and neither flag changes where providers come from.

CI runners run the Terraform CLI non-interactively: GitHub Actions usually installs it with hashicorp/setup-terraform, whose default wrapper changes what each step outputs, and CLI-driven runs against HCP Terraform authenticate with a TF_TOKEN_ environment variable.

The setup-terraform wrapper

By default hashicorp/setup-terraform installs a wrapper script around the terraform binary (input terraform_wrapper, default true). The wrapper captures each call's streams and exposes them as step outputs, and it adds its own lines to the step's output.

There is no output named after the command (no outputs.plan). Posting a plan as a pull-request comment reads outputs.stdout from the plan step, ideally run with -no-color.

The wrapper gets in the way when a step redirects or pipes Terraform's output, for example terraform output -json > outputs.json followed by jq: the extra lines break the JSON. Set terraform_wrapper: false for such jobs. Only that input removes the wrapper; other step settings change the output's content, not the extra wrapper lines.

With continue-on-error: true on a step, GitHub records two results: outcome (before continue-on-error, so failure when the command failed) and conclusion (after it, so success). The job has not failed, so failure() is false. To react to a failed but tolerated check, test steps.<id>.outcome == 'failure' or steps.<id>.outputs.exitcode != '0'.

Authenticating to HCP Terraform with TF_TOKEN_

Export a TF_TOKEN_<hostname> environment variable holding an API token; the CLI reads it for that host without any credentials file or interactive terraform login.

• Encode the host name: each period becomes a single underscore, and a hyphen may be written as a double underscore (hyphens are usually not allowed in variable names). app.terraform.io → TF_TOKEN_app_terraform_io; tf.my-company.net → TF_TOKEN_tf_my__company_net.
• Getting the encoding wrong names a different host: a single underscore where the hyphen was means a period, and an extra double underscore adds a hyphen that is not in the real name.
• Environment variable credentials take precedence over credentials stored in the CLI configuration by terraform login (credentials.tfrc.json), so a stale file token on a runner image is ignored when a valid variable is set.

Which kind of token to give the pipeline, and how to scope and protect it, belongs to securing pipeline identities; workload-identity credentials for providers belong to the sibling topic on dynamic provider credentials.

A VCS-driven workspace starts runs from repository events according to its trigger settings, while an API-driven workspace starts runs when a pipeline uploads a configuration version.

VCS trigger modes

A VCS-connected workspace queues runs according to one of three triggering modes, and separately runs speculative plans for pull requests.

Path filters must include everything the root module reads. If envs/prod calls local modules with relative paths such as ../../modules/network, a pattern for envs/prod alone never runs when those modules change, even though they change what prod deploys. The VCS branch field names one branch to track; it is not a tag filter, and trigger patterns match file paths, not tag names. Release-by-tag for production is the tag trigger mode.

Pull-request plans as a merge gate

With automatic speculative plans on (the default), HCP Terraform runs a plan-only speculative run for each pull request against the tracked branch and reports the result to the VCS provider as a commit status check. Making that check required in the branch protection rules blocks merging until the plan succeeds. Speculative runs can never be confirmed or applied, so auto-apply has no effect on them, and turning automatic speculative plans off affects only pull requests, not tag or merge runs.

The API-driven workflow

In the API-driven workflow the pipeline creates a configuration version, uploads a .tar.gz of the root module to its upload URL, and HCP Terraform then queues a run (by default) or the pipeline creates one through the Runs API.

The archive's root must be the configuration directory itself: archive the contents of the directory (change into it, or use tar's -C <dir>, and archive .), for example tar -C dist -czf release.tar.gz .. Archiving the directory by name stores every entry under that directory's prefix, so Terraform finds no configuration files at the top level. Filtering which files go into the archive does not change the prefix, and the archive must keep any subdirectories the module reads.

Workspace settings such as auto-apply decide what happens after a plan; they do not control whether a run is queued or whether it can be applied at all.

An apply is gated by run tasks, which let external services check a run at a chosen stage, and by approval rules, which decide whether a person must confirm the plan.

Run task stages and enforcement

Run tasks call an external service (security scanners, cost estimators, compliance tools) at a chosen stage of an HCP Terraform run, and the task's enforcement level decides whether its failure stops the run.

• Advisory: a failure is shown as a warning and the run continues. An advisory task can never block a run.
• Mandatory: a failure stops the run. Timeouts and unexpected remote errors count as failures, not skips.
• Each task keeps its own level: attaching a mandatory task does not make another, advisory task blocking.

To scan proposed changes and block on a critical finding, attach the task at post-plan (or pre-apply) with mandatory enforcement. Writing the policies themselves (Sentinel or OPA) is a separate governance topic.

Auto-apply, manual apply and one outstanding plan

Approval is controlled per workspace: auto-apply applies a successful plan with no human step, manual apply waits for an authorised user to confirm it. HashiCorp's guidance for running Terraform in automation is to auto-approve only non-critical infrastructure, such as development or disposable sandbox environments, and to keep human review for anything whose outage matters. Being earlier in the promotion order does not make an environment non-critical: a shared staging environment that blocks every release needs review too, and a policy check is not a substitute for a human reviewer on production.

• The workspace auto-apply setting covers runs created from the UI, the API and VCS. Runs that the CLI creates through a cloud block ignore it and need terraform apply -auto-approve; with -input=false and no flag, the apply errors instead of waiting.
• Runs created by run triggers from other workspaces have their own, separate auto-apply setting.

One outstanding plan per state

HashiCorp recommends allowing only one plan awaiting approval per state at a time, so the plan a person reviewed is the plan that is applied. A self-hosted pipeline that lets several saved plans wait for the same state invites approvers to apply the wrong one; re-planning inside the apply job breaks the link between review and apply, and -auto-approve removes the review entirely.

HCP Terraform enforces this with its run queue: a workspace processes one non-speculative run at a time, and a run that has planned and is waiting for confirmation still holds the queue. Later runs stay Pending, and new runs queue behind it too. Speculative plans do not use the queue, so they keep completing. To move on, confirm or discard the waiting run.

A blue/green deployment runs the current version (blue) and the new version (green) side by side and moves traffic between them at the load balancer, so release and rollback are both a routing change. HashiCorp's tutorial builds it in one configuration:

• Two instance sets, each in its own target group, with feature-toggle counts: count = var.enable_green_env ? var.green_instance_count : 0.
• One ALB listener whose forward action weights the two target groups from a lookup on var.traffic_distribution (blue, blue-90, split, green-90, green).

Weights move traffic; toggles move capacity. Changing the weight never creates or destroys instances, and setting a toggle to false makes count 0, so Terraform plans to destroy those instances (it does not stop them or merely forget them; forgetting needs a removed block or terraform state rm). count = 0 is valid.

The order matters because a target group that receives weight but has no healthy instances can return errors to the users routed to it: never move weight to an environment before it exists, never remove instances while any weight still points at them, and never remove blue while it carries the traffic. The trade-off of the design is cost: both environments run, and are billed, until blue is decommissioned. Both live in one state; the toggles and weights already limit what each apply touches.

A canary release sends a small share of requests to the new version and raises it in steps (for example 10%, 50%, 100%), each step being a small, reviewed plan and apply of a weight variable. Where you set the weights decides how quickly a change takes effect.

The DNS delay comes from caching for the record's TTL; it is a property of DNS itself. A load balancer's deregistration delay is a different setting that only drains targets being removed from a target group.

Listener weights have one catch: target-group stickiness in the forward action (stickiness { enabled = true duration = N }) keeps a returning client on the target group it was first sent to for N seconds. A long duration means weight changes reach returning users only after it expires; shorten the forward action's stickiness duration so shifts take effect quickly. A target group's own stickiness setting is different: it keeps a client on one target inside that group and does not decide which group the forward action picks.

A rolling replacement with create_before_destroy is not a canary: it swaps capacity in one step with no partial traffic stage.

Immutable images make every server of a release identical and pre-tested, and promotion makes production deploy exactly the artifact a lower environment tested.

Immutable deployments

An immutable deployment never changes a running server: each release is a new, pre-tested, versioned machine image, and deploying it means creating new instances from that image ID and destroying the old ones. Because an image ID change forces replacement, Terraform does this automatically.

• Provisioners are a last resort. HashiCorp recommends building configuration into images (for example with Packer) or passing it as cloud-init/user_data. remote-exec scripts that install packages leave half-configured, tainted instances when they fail and let servers launched at different times differ. Any fix that still configures servers after launch, wherever the provisioner lives or however its failures are handled, keeps the mutable model and its drift.
• On aws_instance, a change to user_data is applied in place by default (the instance is stopped and started). Set user_data_replace_on_change = true so a user_data-only release also produces new instances.
• create_before_destroy only orders a replacement that is already planned; it does not turn an in-place update into a replacement. ignore_changes on user_data hides the release instead of shipping it.

Promoting tested artifacts

Promotion means production deploys the exact module version, image or commit that a lower environment tested, never whatever is newest at plan time.

Dev and test environments can keep following the newest artifact. The general rule: a gate must record which artifact passed testing. Conditions that do not (for example on an artifact's age), or settings that freeze every change, either let untested artifacts through or block approved ones.

Terraform has no undo: you roll back infrastructure by applying the last known-good configuration, normally by reverting the change in version control and letting a normal reviewed run apply the reverted commit. In blue/green or canary designs, rollback is moving the weights back.

HCP Terraform's state rollback is not an infrastructure rollback. It replaces the state record only:

• It requires permission to create state versions and a locked workspace.
• It duplicates the chosen earlier state version and makes the copy the current state version; later versions are kept, so the rollback can itself be undone.
• The workspace stays locked afterwards until someone unlocks it.
• It changes the state record, not the resources. HashiCorp cautions that a rolled-back state may no longer match the real infrastructure, so after a bad apply it leaves the infrastructure broken and the record wrong; it is meant for recovering from a corrupted or wrongly written state.

The same principle rules out other shortcuts: anything that re-applies the current, broken configuration (including forced replacement) reproduces the problem, and tearing production down to rebuild it turns a bad release into an outage. Runs cannot be replayed from history; the known-good configuration has to be applied again.