Terraform Professional: Scaling Projects and Multiple Environments

There are three mainstream ways to run the same infrastructure design as several environments (dev, staging, prod) or regions, and the right one is decided by how much isolation each environment needs: CLI workspaces of one configuration, a separate root configuration per environment calling shared modules, or a Terraform Stack with one deployment per environment in HCP Terraform. Variable files alone (-var-file=prod.tfvars) only change input values; they are not an isolation mechanism.

The deciding signal in a scenario is usually a constraint. A requirement that one environment's state or credentials be out of reach of the others rules out workspaces, because they share a backend. A short-lived copy that may share backend and credentials is what workspaces are designed for. One design repeated across many environment and region instances is the case Stacks were built for.

A CLI workspace is a separate, named state inside the same backend for the same configuration — nothing more. Every workspace shares the configuration files, the backend block, the provider credentials in the environment and the modules that terraform init installed. A new workspace starts with empty state, so applying in it creates a second, independent set of objects.

HashiCorp's documented use case is creating a parallel, distinct copy of a set of infrastructure to test a change (for example, a feature branch) before touching the main deployment, then running terraform destroy and deleting the workspace. HashiCorp also states when not to use them:

• Deployments that need separate credentials and access controls. Because every workspace lives in one backend, anyone who can reach dev's state can reach prod's. Choosing a provider role per workspace with terraform.workspace changes only the provider side; the state is still shared storage.
• System decomposition. Workspaces are copies of the whole configuration, so a "workspace per team" duplicates the entire system instead of splitting it.

Value differences that do suit workspaces include instance sizes, counts and name suffixes such as -prod, all driven from terraform.workspace. Neither copying a state file (which binds the copy to the real objects) nor -target (which still acts on the real environment) gives you a disposable test copy.

Code is shared too: a workspace has no configuration of its own, so every workspace runs whatever commit is checked out. Holding one environment on older code while another moves ahead needs a different commit per run, or a layout in which environments pin versions independently.

Using terraform.workspace safely in configuration

terraform.workspace returns the name of the current workspace, and it returns default when no other workspace has been created or selected — so "orbit-cache-${terraform.workspace}" becomes orbit-cache-default on a fresh clone. The Git branch has no influence on it, and in a directory-per-environment layout every root runs in default, so it does not identify the environment there.

Indexing a map with it is strict. If a map has a key for each long-lived workspace and someone selects a workspace that is not one of them, indexing the map with terraform.workspace fails the plan with an invalid index error; Terraform does not fall back to any value on its own. Give ad-hoc workspaces a fallback:

locals {
  node_counts = { qa = 2, live = 6 }
  node_count  = lookup(local.node_counts, terraform.workspace, 1)
}

Backend blocks cannot reference terraform.workspace or variables, and a version constraint is not an expression, so the workspace name cannot pick a backend, a provider version or a module release.

In pipelines, workspace select fails on a missing workspace unless you add -or-create, workspace new fails on one that already exists, and workspace delete refuses the current workspace or one that still tracks resources:

The clean teardown for a short-lived workspace is therefore: terraform destroy in it, terraform workspace select default, then terraform workspace delete NAME.

Every configuration starts with one workspace called default, which can never be deleted. With a standard backend, terraform init does not create named workspaces; only the workspace subcommands do. (A cloud block that selects HCP Terraform workspaces by tags is the exception: init can prompt for a workspace name and create it.)

Selecting workspaces in CI with TF_WORKSPACE

In pipelines, set the TF_WORKSPACE environment variable to the job's workspace: every command in the job (init, plan, apply) then uses it without a separate select step, which also survives fresh containers where a select from an earlier stage is lost. HashiCorp recommends it only for non-interactive use, because in a local shell it is easy to forget the variable is set and apply changes to the wrong state.

While TF_WORKSPACE is set, it is the selection mechanism: terraform workspace select refuses with an error saying the selected workspace is overridden by the TF_WORKSPACE environment variable. To target another workspace in one step, set the variable to that workspace for the step (or unset it and select). No workspace subcommand flag gets around the override; the environment variable is the selection.

HashiCorp's alternative to workspaces for isolated environments is a separate root configuration per environment (for example environments/dev, environments/staging, environments/prod) that each call the same shared modules. Every isolation requirement is a property of a root configuration, so each environment gets its own:

• backend and state storage, so prod state can live where dev engineers cannot read it;
• credentials and the identity that runs applies;
• required_version and required_providers constraints, plus its own .terraform.lock.hcl recording the exact provider build chosen in that directory;
• module version pins, and code a reviewer can read to see exactly what prod runs.

Directories that symlink one shared set of .tf files are still one configuration with one backend block and one set of constraints. The trade-off of real separation is that a change reaches an environment only when someone applies that directory, so roots can drift apart; shared modules keep the duplicated code small.

Promotion works through exact pins. If each root calls a registry module with an exact pin such as version = "2.7.1", a new release goes to dev by bumping dev's pin and running terraform init there, then to staging, then to prod after the soak period. A pessimistic constraint such as "~> 2.7" lets every root jump on its next init, and init -upgrade cannot move past an exact pin. Workspaces cannot do this at all: they share one configuration and one module tree installed by init. The same independence lets dev raise its provider constraint and run init -upgrade to try a new major provider version while staging and prod keep their own lock files. Module constraints are combined with the root's, not used to override it, and provider aliases share one provider version per configuration. (Version-constraint syntax and lock-file mechanics in depth belong to objective 3i.)

Some teams keep a single root directory and switch environments by supplying a different partial backend file at init, such as dev.s3.tfbackend and prod.s3.tfbackend. Switching is not a migration — prod already has its own state — so the right command is:

terraform init -reconfigure -backend-config=prod.s3.tfbackend

-reconfigure discards the saved backend settings and connects to the new backend without copying anything. -migrate-state (and -force-copy, which answers its prompts with yes) copies the current state into the new location, which would overwrite prod's state with dev's. -upgrade concerns provider and module versions only.

Terraform saves the backend choice in the working directory's data directory (.terraform by default). Two jobs running from the same checkout share it, so whichever init ran last decides which state the other job's plan reads. Setting a different TF_DATA_DIR per job keeps their .terraform data apart while the code stays shared. The jobs lock different states, so locking cannot catch the mix-up; the data directory is what has to differ. Backend argument rules themselves are covered in the remote backends lesson (objective 2a).

When one root configuration grows to thousands of resources, HashiCorp's guidance is to split it into several root configurations along architectural boundaries. That shortens plans, shrinks the blast radius and removes lock contention in one move, because each part has its own state and lock. Good reasons to put resources in different roots:

• Rate of change: a VPC that changes a few times a year should not be in the same plan as weekly application releases.
• Ownership: different teams who approve and secure their own changes should own their own state.

On their own, these are weak reasons to split: region alone (provider aliases can manage several regions in one configuration, although blast radius sometimes justifies separating them), the number of providers (one configuration can use many), and heavy cross-references (tightly coupled resources belong together; splitting them just moves the coupling into data passed between projects).

Routine -target is not a scaling strategy. HashiCorp documents it for exceptional situations such as recovering from errors. Targeting module.app does plan that module and everything it depends on, so the run itself works — the damage is what it hides: changes and drift everywhere else never appear in a plan, and state gradually diverges from the code. Only separate root configurations give each part its own state and lock.

HashiCorp's module composition guidance is to keep the module tree flat: the root module calls the modules and wires them together, and each module receives the objects it depends on as input variables (dependency inversion) instead of creating or discovering them itself.

• If a database module internally calls a network module that creates its own VPC, it cannot be placed into a network that already exists. Refactor it to take the network IDs as inputs and call the network module beside it in the root; one root passes the new network's outputs, another passes an existing network's IDs. The module then has one code path and no nested dependency.
• A data source inside a module that looks up a VPC by tag has no reference to the module creating that VPC, so Terraform reads it during plan — before the VPC exists — and the first plan on an empty account fails with no matching VPC. Passing module.network.vpc_id into an input both orders the operations and keeps the module reusable. Ordering tricks that only delay the read leave the hidden coupling in place; the reference is the design fix.
• Data-only modules package a shared lookup (for example, the shared VPC's subnets) as data sources in one maintained module. Consumers need cloud read permissions but no access to the producer's state, unlike terraform_remote_state; importing the producer's objects creates competing ownership, and copied IDs drift.

-parallelism=N limits how many resource operations Terraform walks at once during plan and apply; the default is 10. Lower it when a provider's API returns rate-limit errors (HTTP 429, "Rate exceeded") — the run is slower but stays under the limit; raising it adds pressure. It does not change state locking or drift detection, and it does not cut lock contention between teams.

When you cannot edit the commands (a shared pipeline template), pass flags through the environment:

In HCP Terraform the TFE_PARALLELISM environment variable sets parallelism for plan and apply, but it is not supported on HCP Terraform agents. A workspace that must run on an agent pool (for example to reach a private network) uses the TF_CLI_ARGS_plan and TF_CLI_ARGS_apply environment variables instead.

Terraform Stacks (generally available in HCP Terraform) separate the definition of the infrastructure from its instances. Component blocks, each sourcing a module and wired to each other's outputs, live in .tfcomponent.hcl files. Each deployment block in a .tfdeploy.hcl file is one instance — an environment, a region, or an environment-region combination — with its own inputs and isolated state. This replaces the pattern of dozens of near-identical workspaces stitched together with run triggers and remote state sharing; for_each over environments inside one workspace would instead collapse everything into one state.

To retire a deployment, HashiCorp recommends setting destroy = true on its block (the argument defaults to false), applying the resulting destroy plan, and only then removing the block.

Approval rules come from two blocks in the deployment file:

deployment_auto_approve "small_plans" {
  check {
    condition = context.plan.changes.total <= 5
    reason    = "Plan changes more than five resources."
  }
}
deployment_group "sandboxes" {
  auto_approve_checks = [deployment_auto_approve.small_plans]
}
deployment "sandbox" {
  inputs           = { region = "eu-west-1" }
  deployment_group = deployment_group.sandboxes
}
deployment "live" {
  inputs = { region = "ap-southeast-2" }
}

• A deployment_auto_approve block holds checks over context.plan (such as changes.total, changes.add, changes.remove or applyable); a plan is auto-approved only if every check passes.
• The rule takes effect only through a deployment_group that lists it in auto_approve_checks, and a deployment joins a group with its deployment_group argument.
• A deployment with no group gets its own generated group, <name>_default (here live_default), which has no auto-approve checks of yours — so its plans with changes wait for manual approval. (HashiCorp's built-in rule can still auto-approve a plan that changes nothing.) A deployment inherits whatever checks its group lists, so which group it joins is the approval decision.

Auto-approve orchestration rules are a feature of the higher HCP Terraform editions, so confirm your organization's plan includes them before relying on them.

HCP Terraform projects are folders for workspaces and Stacks, and the scope at which you grant access and share configuration as an organization grows. Each workspace belongs to exactly one project (new ones go to the Default Project unless you choose otherwise).

• Team access at project level covers every workspace and Stack in the project, including those created later. Tags help filtering but do not carry permissions.
• Variable sets scoped to a project apply to all of its workspaces, current and future. A global variable set applies to every workspace in the organization, so scope a set to the project when only that project should receive it.
• Precedence: workspace-specific variables overwrite variables from standard (non-priority) variable sets, so a project set alone cannot enforce a value. Marking it a priority variable set makes it override same-key variables at more specific scopes, which is how a platform team forces, say, a compliance region.
• Tags: workspaces created in a project inherit the project's tags.
• Deleting: only an empty project — no workspaces or Stacks — can be deleted; nothing is moved or deleted automatically, so move or delete its workspaces first.