CloudFormation, StackSets, and golden images: provisioning for SOA-C03

Golden images bake your OS hardening, agents, and runtime dependencies into a reusable image so every instance launches identical and boots fast. On EC2 the unit is the AMI. You create one from a configured instance; by default EC2 reboots the instance first so the filesystem is captured in a consistent state (the no-reboot option skips this at the risk of an inconsistent image). AMIs are Regional resources: to launch the same image elsewhere you must copy the AMI to each target Region, and to let other accounts use it you share it with specific account IDs or across an organization. Deregistration hygiene matters operationally: deregistering an AMI does not delete its backing EBS snapshots — you keep paying for them until you delete the snapshots separately, so stale-image cleanup is a two-step job.

EC2 Image Builder turns image maintenance into a managed pipeline: a source image, build components that install and configure software, test components that validate the result, and a distribution configuration that copies the finished AMI to your chosen Regions and shares it with your chosen accounts. Put the pipeline on a schedule and your golden image is rebuilt automatically as new patches land — the exam's answer to keeping AMIs patch-current without manual rebuilds.

Container images follow the same idea: build the image, push it to Amazon ECR, and let ECS or EKS pull from there. Operationally, use ECR lifecycle policies to expire old and untagged images so repositories do not grow without bound. Whether AMI or container image, the goal is identical: a versioned, tested artifact you can roll out — and roll back — deliberately.

A CloudFormation template is a declarative description of a stack, and for operations you mostly read five sections. Resources is the only required one: every resource has a logical ID, a type, and properties. Parameters make one template serve many environments — instance sizes, environment names, and CIDR ranges arrive at deploy time instead of being hard-coded. Mappings are static lookup tables, classically mapping each Region to its local AMI ID so one template deploys anywhere. Conditions switch resources on or off — create the expensive Multi-AZ database only when the environment parameter equals prod. Outputs surface values (a load balancer DNS name, a subnet ID) after deployment, and an output with an Export name can be consumed by other stacks through Fn::ImportValue.

That last mechanism has an operational sting the exam likes: you cannot delete a stack, or remove an exported output, while another stack imports it. Cross-stack references create real dependencies between stacks, so deletions have an order.

Read templates the way the engine does: CloudFormation works out dependencies (references between resources, plus explicit DependsOn) and creates resources in dependency order, in parallel where it can. When you can predict what a template will create, in what order, and which values are environment-specific, update behavior — and the failure modes covered in the rest of this lesson — become much easier to reason about under exam time pressure.

A stack's lifecycle is create, update, delete — and the operational discipline lives in how you update. You can update directly, but the safer path is a change set: CloudFormation compares the new template and parameters against the running stack and shows you every planned Add, Modify, and Remove before anything happens. The critical detail is Replacement. Some property changes update a resource in place; others force CloudFormation to delete the resource and create a new one, because the property is immutable after creation. For a stateless web server a replacement is a non-event; for a database it destroys data. Review change sets specifically for replacements on stateful resources, then execute.

Two mechanisms protect stateful resources. A stack policy is a JSON document evaluated during stack updates that denies update actions against protected logical IDs — with a policy protecting the production database resource, an update that would modify or replace it fails instead of proceeding, until you explicitly override the policy. DeletionPolicy is a per-resource attribute controlling what happens when the resource is removed from the stack or the stack is deleted: Delete (the default for most resources), Retain (leave the resource running outside the stack), or Snapshot (take a final snapshot before deletion — supported for snapshot-capable resources such as EBS volumes, RDS instances, and ElastiCache clusters).

The exam pattern: 'preview exactly what an update will change before running it' points to a change set; 'prevent the database from being replaced or deleted' points to a stack policy plus DeletionPolicy: Retain or Snapshot. Stack-level termination protection adds one more guard, blocking deletion of the whole stack until it is turned off.

Drift detection answers the ops classic: a resource was changed outside CloudFormation — someone edited a security group rule in the console — and now the template no longer matches reality. Run drift detection on the stack and CloudFormation compares the live configuration of each supported resource against the template's expected configuration, reporting each resource as IN_SYNC, MODIFIED, or DELETED, and the stack as DRIFTED when anything diverges. Drift detection reports; it does not revert. Fixing drift means updating the stack (or the resource) so template and reality agree again — and until you do, the next stack update can behave unpredictably, because CloudFormation assumes it owns the configuration.

Rollback behavior is the other state-machine topic. When a create fails, CloudFormation rolls back and deletes everything it created; the stack lands in ROLLBACK_COMPLETE. That state is terminal for the stack: you cannot update it — delete the stack and create it again once you have fixed the cause. When an update fails, CloudFormation automatically rolls back to the last known working configuration (UPDATE_ROLLBACK_COMPLETE), which is why updates are far less scary than creates. If the rollback itself fails — often because a resource was changed out of band or a dependency is gone — the stack enters UPDATE_ROLLBACK_FAILED. Recovery is continue update rollback after fixing the underlying problem, optionally skipping resources CloudFormation cannot reconcile.

Nested stacks package common patterns (a standard VPC, a standard load balancer tier) as child stacks created by a parent via AWS::CloudFormation::Stack. Remember that failures propagate: a child stack failure rolls back the parent, and you troubleshoot by finding the failed child's own events.

Work one failure end to end. An update to a production stack grows an Auto Scaling group, adding instances to an existing subnet. The update fails and rolls back. Where do you look? The stack's event history, oldest failure first. CloudFormation cancels dependent operations after the first failure, so the event list fills with cascade noise; the first CREATE_FAILED or UPDATE_FAILED event names the real culprit and carries the underlying service's error message.

Here the first failure says the subnet has insufficient free addresses. That is a subnet sizing issue: the CIDR is exhausted. Remember that AWS reserves 5 IP addresses in every subnet (the first four and the last one), so a /28 offers 16 addresses minus 5 — only 11 usable. A subnet sized for steady state runs dry the first time scaling, a parallel environment, or an ENI-hungry service pushes past its ceiling. The fix is structural — a larger CIDR or additional subnets — not a retry.

Two other first-failure families dominate the exam. CloudFormation errors: 'resource already exists' means a name collision — an explicitly named resource (a bucket name, a role name) clashes with one already in the account, which is why letting CloudFormation generate names is safer; insufficient-capacity errors come from the underlying service, not the template. Permissions issues: CloudFormation calls each service's API as some principal. By default that is the deploying user's credentials; if a service role is attached to the stack, CloudFormation uses the role instead. An access-denied error therefore means the deployer lacks permissions or the service role does — check whichever identity actually performed the call.

Provisioning at organization scale is about pushing one definition everywhere — or sharing one resource instead of duplicating it. CloudFormation StackSets deploy a single template as stack instances across many accounts and Regions from one administrator account. With self-managed permissions you create the trust yourself: an administration role in the admin account and an execution role in every target account that the administration role assumes. With service-managed permissions, StackSets integrates with AWS Organizations, provisions those roles for you, and unlocks automatic deployment: target an organizational unit and any new account joining it automatically receives the stack — the exam's answer to 'every new account must get this baseline'.

AWS RAM solves a different problem: some resources should exist once and be used by many accounts. Share subnets so multiple accounts launch workloads into one centrally managed VPC, or share a Transit Gateway or Route 53 Resolver rules. The owner keeps control; consumers use the resource in place. That distinction is the exam trigger: StackSets copy infrastructure into each account, while RAM shares one resource across accounts.

Service Catalog is the curated middle path, needed at mention level: administrators publish approved, CloudFormation-backed products into portfolios, and end users launch them self-service without holding the underlying permissions.

Deployment strategies at the infrastructure level trade speed against blast radius. An in-place deployment updates the instances you already have — fastest and cheapest, but the old version is gone once you start, so rollback means deploying backwards, and any failure lands on live capacity. A blue/green deployment builds a complete parallel environment (green) alongside production (blue), validates it, then shifts traffic — at the load balancer or DNS layer — and keeps blue running until you are confident. Rollback is nearly instant (shift traffic back), at the cost of temporarily doubled infrastructure.

For the fleet-rollout case the exam favors Auto Scaling group instance refresh. After you update the launch template with a new AMI version, instance refresh performs a rolling replacement of the group's instances, honoring a minimum healthy percentage so capacity never drops below your floor while old instances drain and new ones prove healthy. It is the operational bridge from the golden-image pipeline to the running fleet: Image Builder produces the new AMI, the launch template gets a new version, and instance refresh rolls it through the group without downtime. It also repairs configuration skew — if some instances are still running an older launch template version after manual interventions, a refresh brings the whole group back to the current definition. When a question asks how to roll a new AMI through an Auto Scaling group with minimal disruption and no parallel environment, instance refresh is the intended answer.

The AWS CDK is not a second engine — it is a developer-friendly front end to the same one. You define infrastructure in a general-purpose language (TypeScript, Python, and others), and cdk synth synthesizes a CloudFormation template; cdk deploy creates or updates an ordinary CloudFormation stack. Everything in this lesson — change sets, rollback states, drift detection, stack events — applies unchanged to CDK-deployed stacks, which is exactly how the exam tests it.

Terraform is the third-party alternative named in the exam guide. It is multi-cloud, declares infrastructure in its own configuration language, and — the key operational contrast — tracks what it has built in a state file that you must store, secure, and lock (commonly in an S3 backend), whereas CloudFormation's state is managed by AWS inside the stack itself. A Terraform plan previews changes the way a change set does, and it refreshes state against reality to surface drift at plan time, where CloudFormation offers on-demand drift detection against its managed state.

Underneath either tool, Git is the source of truth: infrastructure definitions live in a repository, changes arrive as reviewed commits, and the environment stays reproducible from a known revision. When a scenario wants template changes versioned, reviewed, and auditable, the answer is the repository workflow, not console edits.