Terraform Professional (002): functions, expressions, dynamic blocks and custom conditions

To predict an expression's value, evaluate it the way Terraform does: resolve the innermost function call first, apply each function's exact rule (argument order, inclusive or exclusive bounds, type conversion, ordering), and only then the outer call. Most wrong predictions come from one of four places: which argument wins a conflict, what happens when a value is missing, what order a collection comes out in, and what type the result has.

Terraform has no user-defined functions, so every function you meet is either built in or provider-defined. Most built-in functions are pure: the same inputs always give the same result, and they read nothing you do not pass them. The exceptions are few and worth knowing: timestamp(), uuid() and bcrypt() return a different result on every run, and file(), fileexists(), fileset() and templatefile() read from disk. Everything else is predictable on paper. When you are unsure, terraform console evaluates an expression against the current configuration and state, and it is the quickest way to confirm a result before it reaches a plan.

• Conflicts: merge() lets later arguments win.
• Missing values: lookup(), try(), can() and coalesce() each handle a different kind of absence.
• Ordering: map keys and sets of strings come out in lexical order, not the order you wrote them.
• Types: sort() returns strings, slice() returns a list, jsonencode() keeps numbers and booleans as JSON numbers and booleans.

merge() combines maps or objects left to right, and when the same key appears in more than one argument, the value from the last argument that has it wins. It never errors on a duplicate key and never combines the two values into a list.

locals {
  base     = { team = "data", tier = "standard" }
  override = { tier = "premium" }
  a = merge(local.base, local.override)   # tier = "premium"
  b = merge(local.override, local.base)   # tier = "standard"
}

This gives you a simple design rule. To let callers override your defaults, pass the defaults as the earlier argument and the caller's map after them. To make a value impossible to override (a compliance or billing label that reports depend on), pass your fixed map as the final argument, after the caller's. Every non-conflicting caller key survives either way; only the conflicting keys change.

The merge is shallow. If two arguments both have a key whose value is itself a map, the later map replaces the earlier one completely; the nested keys are not combined.

locals {
  defaults = { logging = { level = "info", target = "s3" }, replicas = 2 }
  override = { logging = { level = "debug" } }
  config   = merge(local.defaults, local.override)
  # { logging = { level = "debug" }, replicas = 2 }  -- target is gone
}

To combine nested maps, merge them explicitly at that level, for example merge(local.defaults.logging, local.override.logging). Note that setunion() works on sets, not maps, and coalesce() picks one whole argument rather than combining keys, so neither is a substitute for merge().

Each fallback function handles a different kind of absence, and choosing the wrong one either errors or silently picks the wrong value.

Data decoded with yamldecode() or jsondecode() is the classic case for try(): if a team's file omits a whole section, try(local.cfg.backup.window_hours, 4) returns 4, and so does can(local.cfg.backup.window_hours) ? local.cfg.backup.window_hours : 4. Comparing the section to null does not help when the attribute is absent rather than null, because referencing it is already an error.

coalesce() treating an empty string as missing is what makes the common optional-name pattern work: with variable "display_name" { default = "" }, coalesce(var.display_name, "${var.service}.${var.region}") falls through to the generated name. If every argument is null or empty, coalesce errors. For lists, coalescelist() returns the first non-empty list.

Collection results depend on two rules: Terraform orders map keys and string sets lexically, and each list function has its own bounds behaviour (element() wraps, direct indexes do not, slice() excludes its end).

Collection ordering: sets, map keys and sort()

Terraform does not preserve the order you wrote map keys in, and it does not preserve list order once you convert to a set. Map keys are always iterated in lexical (dictionary) order, and a set of strings iterates in lexical order with duplicates removed.

toset(["queue", "cache", "queue", "auth"])     # iterates as "auth", "cache", "queue"
[for s in toset(["queue", "cache", "queue", "auth"]) : s]  # ["auth", "cache", "queue"]

locals { tiers = { z = "gold", m = "silver", b = "bronze" } }
[for k, v in local.tiers : v]                      # ["bronze", "silver", "gold"]
keys(local.tiers)                                   # ["b", "m", "z"]

A for expression over a map is valid without keys(): for k, v in map binds the key and value in that lexical key order. The same rule decides the result of keys() and values(), and the order of instance keys in a map built for for_each. Sets of non-string elements have no documented order, so never rely on one.

sort() takes a list of strings and sorts lexically. Give it numbers and Terraform converts them to strings first, so the result is a list of strings in dictionary order, not numeric order: sort([25, 3306, 110, 5432]) is ["110", "25", "3306", "5432"], because the character 1 sorts before 2. To order numbers numerically, pad them to a fixed width with format() before sorting, or avoid relying on the order at all.

Indexing lists: element(), direct indexes, slice() and length()

element(list, index) wraps around: an index at or past the end is taken modulo the length, which is why it is used to spread count instances across a shorter list of subnets or zones. A direct index list[i] does not wrap; an out-of-range index is an "Invalid index" error, and negative indexes such as list[-1] are not supported.

Because the index is computed from length(), both of those last-element forms work for a list of any length, which is what a module needs when the number of zones or subnets varies by Region.

slice(list, start, end) returns a list, includes start and excludes end. slice(["alpha", "beta", "gamma", "delta", "epsilon"], 2, 4) is ["gamma", "delta"]. Because it returns a list, even a one-element slice is a list, not the string inside it.

A for expression transforms one collection into another, and combined with flatten() or setproduct() it turns nested input into the flat, uniquely keyed map that for_each needs.

for expressions: lists, maps, filters and grouping

A for expression builds a list when wrapped in square brackets and a map (object) when wrapped in braces with a key => value pair. An optional if clause filters elements before they reach the result.

[for name, l in var.listeners : l if name != "admin"]  # values, admin removed
{ for s in local.services : s.name => s.owner }        # map keyed by name
{ for s in local.services : s.owner => s.name... }     # grouping mode

Two rules decide most results:

• Order follows the source. A list source keeps list order; a map source iterates in lexical key order, so a filtered list built from a map comes out in key order (grpc before metrics), not in the order the map was written.
• Map keys must be unique. If two elements produce the same key, Terraform raises a duplicate-key error; it does not keep the first or last value. Add the ellipsis ... after the value expression to switch to grouping mode, where each key maps to a list of every value produced for it, in source order: { ops = ["billing"], web = ["cart", "search"] }.

Wrapping each value in a list (s.owner => [s.name]) does not group; it still collides on duplicate keys. Converting with tomap() does not help either, because the error happens while the for expression is built.

Shaping nested data for for_each with flatten() and setproduct()

for_each needs a map or a set of strings with one element per instance, so nested input must be flattened into a single collection with a unique key per element. Two functions do most of this work.

flatten() removes nesting from a list of lists, producing one flat list of the inner elements. The usual pattern is a nested for expression that emits a list of objects per parent, then flatten, then a map keyed by a unique string:

locals {
  vpcs = { edge = ["172.16.0.0/20"], core = ["172.16.32.0/20", "172.16.48.0/20"] }
  subnets = flatten([
    for vpc, cidrs in local.vpcs : [
      for i, c in cidrs : { key = "${vpc}-${i}", cidr = c }
    ]
  ])
  # a list of three objects, each with key and cidr
}

resource "aws_subnet" "this" {
  for_each   = { for s in local.subnets : s.key => s }
  cidr_block = each.value.cidr
  # ...
}

setproduct() returns every combination of its arguments as a list of lists: setproduct(["blue", "green"], ["a", "b"]) gives four pairs such as ["green", "a"]. Each pair is a list, so a for expression reads its parts by index: { for pair in setproduct(local.colors, local.slots) : "${pair[1]}.${pair[0]}" => pair }. It binds one variable per element; for c, s in ... over a list would bind the index and the pair, not the two halves. The resulting map's keys, like any map's, iterate lexically, so keys() of the pairs map is sorted (a.blue, a.green, b.blue, b.green), while each value is still the original two-element list.

How the resulting instance addresses are read and traced through outputs is covered in the lesson on tracing variables, locals, outputs and for_each; this lesson is about building the collection.

String and network functions are deterministic formatters: format() and replace() follow printf and regex rules, and the CIDR functions follow binary address arithmetic, so each result can be worked out exactly.

String functions: format() and replace() with regular expressions

format(spec, values...) uses printf-style verbs: %s for a string, %d for an integer, and a width with a zero flag pads numbers, so format("%s_%04d", "node", 42) is "node_0042". formatlist() applies the same spec across lists.

replace(string, search, replacement) treats search as a literal substring unless it is wrapped in forward slashes, in which case it is a regular expression. That distinction decides many results:

Inside an HCL string, a regex backslash is itself escaped, so \\. in the source is the regex \. (a literal dot). Function order matters too: replace(lower(var.name), "/[^a-z]/", "_") lowercases first, so uppercase letters survive as letters; lower(replace(var.name, ...)) replaces the uppercase letters with underscores before they are ever lowercased. With "Ops-Metrics.Hub" only the first order produces "ops_metrics_hub"; the second gives "_ps__etrics__ub".

Network address functions: cidrsubnet() and cidrsubnets()

cidrsubnet(prefix, newbits, netnum) extends the prefix length by newbits and returns subnet number netnum of that new size. For a /16, 8 new bits gives /24 networks that step through the third octet, and 4 new bits gives /20 networks that step by 16 in the third octet.

A for expression such as [for i in range(2) : cidrsubnet("172.31.0.0/16", 4, i + 2)] therefore skips the first two /20s and returns the next two, "172.31.32.0/20" and "172.31.48.0/20".

cidrsubnets(prefix, newbits...) allocates several consecutive subnets in one call, each with its own size, and aligns each one to its own size boundary. cidrsubnets("192.168.0.0/16", 8, 4, 8) gives "192.168.0.0/24" first; the /20 cannot start at 192.168.1.0 because a /20 must start on a multiple of 16 in the third octet, so it is "192.168.16.0/20"; and the last /24 follows the end of that range at "192.168.32.0/24". Changing the list later shifts every following range, so append new sizes at the end rather than inserting them.

optional(type, default) in an object type constraint lets a caller omit an attribute. When the attribute is omitted or explicitly null, Terraform substitutes the default; without a default, it becomes null.

variable "bucket" {
  type = object({
    name          = string
    logging      = optional(bool, true)           # omitted or null -> true
    admin_roles  = optional(set(string), [])      # omitted -> empty set, safe for length()
  })
}

Three behaviours are worth knowing exactly:

• Null means "use the default". Passing logging = null gives true, not null and not false. To let callers turn it off, they pass false.
• Defaults apply top-down. If a nested object attribute has its own default (for example cors = optional(object({ max_age = optional(number, 3600) }), {})), Terraform first applies the outer default {} and then the inner attribute defaults to it, so cors.max_age is 3600 even when the caller omits cors entirely.
• Extra attributes are discarded. Converting a value to an object type keeps only the declared attributes; an undeclared owner_email attribute is silently dropped, not an error.

The default must convert to the attribute's type: an empty list [] is a valid default for list(string) or set(string), a string is not, and a default of null is the same as no default. Validation rules on input variables are covered in the lesson on turning requirements into configuration.

A dynamic block generates zero or more nested blocks of one type inside a resource, data source, provider or provisioner, from a collection in for_each. The content block is the body of each generated block.

resource "aws_security_group" "web" {
  dynamic "egress" {
    for_each = var.destinations      # list or map (or set)
    content {
      from_port   = egress.value.port
      to_port     = egress.value.port
      description = "to ${egress.key}"
    }
  }
}

• The iterator is named after the block label (egress) unless you set iterator = dest, in which case the content uses dest.value and dest.key. each is for resource-level for_each and is not available inside a dynamic block's content.
• .key is the list index (0, 1, 2...) for a list and the map key for a map; .value is the element. The second element of a list therefore has key 1.
• for_each accepts lists directly; toset() is not required.
• Nested dynamic blocks reference the outer iterator by its name. If the outer block sets iterator = tier, the inner for_each reads tier.value.hosts.
• Optional single block: to include a block only when a value is set, use a zero-or-one element list: for_each = var.kms_key_arn == null ? [] : [var.kms_key_arn]. A list holding null or "" still generates one block.

Dynamic blocks can only generate blocks that belong to the resource type's own arguments. They cannot generate meta-argument blocks such as lifecycle and provisioner, because Terraform processes those before it can evaluate expressions. HashiCorp also advises using them sparingly: over-used, they hide the configuration's shape from readers.

Custom conditions encode assumptions and guarantees in configuration, and the difference between them is when they run and whether they stop the run.

Use a precondition to stop before creating something on bad input, such as a subnet looked up with data.aws_subnet that belongs to the wrong VPC; it compares other objects (the data source's attributes), not self. Use a postcondition to verify the result of the object it sits on, through self: self.map_public_ip_on_launch on that subnet data source, or self.encrypted on a volume resource once it exists. Referring to the object by its own address inside its own block would be a self-reference, which is why self exists.

Use a check block for continuous validation that must not block delivery, such as confirming that a certificate is not close to expiry through a data source scoped to the check: a failed assertion is reported as a warning and later applies are not blocked. If a precondition or postcondition depends on a value unknown at plan time, Terraform does not skip it; it evaluates it during apply instead.

jsonencode() and templatefile() turn Terraform values into text for policies and user data, and provider-defined functions extend the built-in set with provider-specific helpers called through a provider:: prefix.

Encoding and templates: jsonencode() and templatefile()

jsonencode() converts a Terraform value to minified JSON. Object and map keys are emitted in lexical order, numbers stay JSON numbers and booleans stay JSON booleans, so jsonencode({ zone = "x", count = 2, on = false }) is {"count":2,"on":false,"zone":"x"}. That is why a policy rendered with jsonencode() may differ textually from a hand-written file while meaning the same thing.

templatefile(path, vars) renders a file with the ${...} and %{...} template syntax, and the template can see only the variables in the map you pass. It cannot read var., local. or resource attributes from the calling module, so a template that uses ${bucket} needs bucket = var.bucket_name in the map. Built-in functions are available inside the template, so the way to put a list into JSON is ${jsonencode(subnet_ids)}, written with the template's own variable name; join() would produce unquoted strings. file() reads a file verbatim and performs no interpolation at all. Template files conventionally use the .tftpl extension.

Provider-defined functions

Since Terraform 1.8, providers can ship their own functions, called with the syntax provider::<local name>::<function>(args). The middle part is the local name the module gives the provider in required_providers, not the source address and not necessarily aws.

terraform {
  required_providers {
    cloud = { source = "hashicorp/aws", version = ">= 5.40" }
  }
}

locals {
  role_arn_short = provider::cloud::trim_iam_role_path(var.role_arn)
}

• The function must exist in the provider version you install: the AWS provider added arn_parse, arn_build and trim_iam_role_path in 5.40, so an older pinned version fails even on a new Terraform.
• The provider must be declared in required_providers; a separate provider configuration block is not needed just to call a function.
• On Terraform 1.7 or earlier the syntax does not parse at all, so no provider upgrade or try() wrapper can hide it; only a Terraform upgrade to 1.8 or later fixes it.
• Configuring providers themselves (aliases, versions, authentication) belongs to the providers lesson in this domain.