CloudFormation, StackSets, and golden images: provisioning for SOA-C03

A golden image bakes your OS hardening, agents and runtime dependencies into a reusable image so every instance launches identical and boots fast; on EC2 that image is the AMI. You create one from a configured instance; by default EC2 reboots the instance first so the file system is captured consistently (the no-reboot option skips this at the risk of an inconsistent image). An AMI is a registration that points at one or more EBS snapshots — keep that two-layer model in mind, because most AMI operations questions turn on it.

AMIs are Regional. To launch the same image in another Region you copy the AMI to that Region. A copy is not instant: EC2 copies all of the backing snapshot data, so for disaster recovery the copy must already exist in the recovery Region before you need it (copy on every new image version, not at failover time). Copying the snapshots alone is not enough either — a snapshot is not launchable until an AMI is registered from it, which is what CopyImage does for you.

Encryption follows the Region. KMS keys are Regional, so an encrypted AMI copied to another Region is re-encrypted with a key in the destination Region — the account's default EBS encryption key in that Region (the AWS managed aws/ebs key unless you have set a customer managed default), or a customer managed key you specify in the copy request. You can also encrypt an unencrypted AMI during the copy.

A copy is a new AMI with its own ID and its own permissions. CopyImage does not carry over launch permissions (or S3 bucket permissions), so accounts that could launch the source cannot see the copy until you share the copy again — and if the copy is encrypted with a new key, they also need permission to use that destination-Region key.

You share a private AMI by adding launch permissions for specific AWS account IDs, or for an organization or organizational unit (OU) ARN. Sharing with an OU is the scalable choice: accounts created in, or moved into, that OU can launch the AMI without anyone editing permissions per account, and no account outside the scope sees it. Making an AMI public is a different, rarely appropriate action.

Encrypted AMIs add a key requirement. The target accounts must be allowed to use the KMS key that encrypts the AMI's snapshots, which means the AMI must be encrypted with a customer managed key whose key policy (or a grant) lets those accounts use it. The AWS managed aws/ebs key's policy cannot be changed, so an AMI encrypted with it cannot usefully be shared: copy or re-create it encrypted under a customer managed key, then share that AMI and the key.

For organization-wide distribution on a schedule, EC2 Image Builder's distribution settings (next section) share each new version automatically.

Retiring an image is a sequence of distinct actions, and each does less — or more — than its name suggests.

Deregistering an AMI does not delete its EBS snapshots by default — you keep paying for them until they are deleted, so cleanup is two steps in a fixed order. EC2 refuses to delete a snapshot that backs a registered AMI (a disabled AMI is still registered), so deregister first, then delete the snapshots; DeregisterImage can also optionally delete the associated snapshots in the same call. Disabling has one storage use: a snapshot behind an enabled AMI cannot be moved to the EBS Snapshots Archive tier, while one behind a disabled AMI can. Deregistration protection guards important images against accidental removal.

To make removal recoverable, use the Recycle Bin with two retention rules: one for AMIs and one for EBS snapshots, the snapshot rule retaining at least as long as the AMI rule. An AMI can only be restored once its snapshots have been restored, so an AMI rule alone does not bring a deleted image back after its snapshots are gone.

EC2 Image Builder turns image maintenance into a managed pipeline: a recipe (a source image plus build components that install and configure software), test components that validate the result, and a distribution configuration. An image recipe produces an AMI; a container recipe produces a container image that the pipeline pushes to an Amazon ECR repository — the same components can harden both.

• Schedule: a pipeline runs on a cron schedule, and can be set to start only when a dependency update is available (a newer base image or component version), so it doesn't rebuild identical images every night.
• Tests gate distribution: if a test component fails, the image is not distributed. Vulnerability scanning with Amazon Inspector reports findings on the image, but it doesn't run your application tests and doesn't stop distribution — only test components gate the image.
• Distribution: copies the finished AMI to chosen Regions and shares it with accounts, organizations or OUs (and can update launch templates); container images can be distributed to ECR repositories in other Regions.
• Lifecycle policies: deprecate, disable or delete old image versions and their resources automatically.

For container images outside Image Builder, the operational basics are ECR lifecycle policies to expire old and untagged images. ECR pull through cache only mirrors images from an upstream registry; it does not build or harden anything. Whether AMI or container image, the goal is a versioned, tested artifact you can roll out — and roll back — deliberately.

A CloudFormation template is a declarative description of a stack, and for operations you mostly read five sections. Resources is the only required one: every resource has a logical ID, a type and properties. Parameters make one template serve many environments — instance sizes, environment names and CIDR ranges arrive at deploy time. Mappings are static lookup tables. Conditions declare true/false tests (for example, the environment parameter equals prod); you attach one to a resource with the Condition attribute, and CloudFormation creates that resource only when the condition is true. Outputs surface values after deployment, and an output with an Export can be read by other stacks through Fn::ImportValue.

Latest AMI without editing Mappings. A Region-to-AMI mapping goes stale every time a new image is released. Instead, declare a parameter of type AWS::SSM::Parameter::Value<AWS::EC2::Image::Id> whose default is an AWS public Systems Manager parameter (for example the Amazon Linux 2023 latest-AMI path under /aws/service/ami-amazon-linux-latest/). CloudFormation resolves it to the current AMI ID for that Region on each create or update.

No hard-coded accounts or Regions. Build names and ARNs with Fn::Sub and the pseudo parameters AWS::AccountId, AWS::Region, AWS::Partition and AWS::StackName, so the same template works in every account and Region.

Exports have rules. An export name must be unique within an account and Region, so a template deployed twice in one Region must build its export names from AWS::StackName or a parameter. While any stack imports an export, CloudFormation refuses to delete the exporting stack and refuses to change or remove that export — update the importing stacks to stop using it first. That protection is the reason to prefer exports over copying IDs by hand between independent stacks; nested stacks pass values through their own parameters and outputs instead.

CloudFormation works out dependencies (references between resources, plus explicit DependsOn) and creates resources in dependency order, in parallel where it can.

A change set is how you preview an update: CloudFormation compares the new template and parameters with the running stack and lists every planned Add, Modify and Remove, which you review and then execute — nothing changes until you execute it. The critical column is Replacement (True, False or Conditional). Some property changes update a resource in place; changing an immutable property replaces it.

How replacement works: CloudFormation creates the new resource first, points dependent resources at it, and deletes the old one during the cleanup phase. The result is a new physical resource — a new instance ID, a new private IP address, and, for a database, a new empty database unless you restore from a snapshot. Anything outside the stack that referenced the old ID or IP breaks.

Custom names block replacement. If a resource has an explicit physical name (a TableName, a bucket name), CloudFormation cannot create the replacement alongside the original, so the update fails on that resource and the stack rolls back. Change the name in the same update, or remove the name and let CloudFormation generate one.

Replacement False is about the resource you changed. When you point an Auto Scaling group at a new launch template version, the change set shows the group itself modified in place. Running instances keep the old version — only new launches use the new one — unless the group's UpdatePolicy tells CloudFormation to replace them (see the deployment section).

Four mechanisms protect production, and each guards a different event — exam questions hinge on picking the one that matches the event.

DeletionPolicy does not fire on replacement and UpdateReplacePolicy does not fire on deletion, so a stateful resource normally gets both. Snapshot works for snapshot-capable resources such as EBS volumes, RDS DB instances and ElastiCache clusters. RetainExceptOnCreate retains the resource on every deletion except when the stack's own create fails and rolls back — then the new, empty resource is deleted instead of piling up. These attributes only protect events that happen after they are in the deployed template.

Drift detection compares the live configuration of a stack's supported resources with the configuration the template expects, marking each resource IN_SYNC, MODIFIED, DELETED or NOT_CHECKED, and the stack DRIFTED when anything diverges — the tool for a resource changed outside CloudFormation. Three limits matter:

• It only compares properties you set. Properties explicitly defined in the template or parameters are checked; a setting the template never declared (say, versioning on a bucket whose template sets only encryption) can change without showing as drift.
• It reports; it never reverts.
• A normal update doesn't fix it either. A standard stack update only acts on properties whose template values changed, so drifted properties you didn't touch stay drifted. Reconcile by changing the resource back, updating the template to match reality, or running a drift-aware change set, which compares the new template, the last-deployed template and the live state and reverts drifted properties when executed.

Rollback is the other state machine. When a create fails, CloudFormation deletes what it created and the stack lands in ROLLBACK_COMPLETE; it can't be updated from there — delete it and create it again once the cause is fixed. When an update fails, CloudFormation rolls back to the last working configuration (UPDATE_ROLLBACK_COMPLETE) and the stack stays usable. If that rollback itself fails — often because a dependency was changed or deleted out of band — the stack enters UPDATE_ROLLBACK_FAILED; fix the cause, then continue update rollback, optionally skipping resources CloudFormation cannot reconcile.

Nested stacks are child stacks a parent creates with AWS::CloudFormation::Stack. A child failure rolls back the parent, and you troubleshoot from the failed child stack's own events.

Troubleshooting starts in the stack's event history, at the first failure. After one resource fails, CloudFormation cancels dependent operations, so the list fills with cascade noise; the earliest CREATE_FAILED or UPDATE_FAILED event names the real culprit and carries the underlying service's error message. Three families of first failure dominate.

Subnet sizing

An 'insufficient free addresses' error means the subnet's CIDR is exhausted. AWS reserves 5 addresses in every subnet (the first four and the last), so usable addresses = 2^(32 − prefix) − 5: a /28 has 16 − 5 = 11, a /26 has 64 − 5 = 59. Every network interface uses one, and a rollout that launches replacements before terminating old instances needs headroom on top of steady state. A subnet's CIDR can't be changed after creation, so the fix is structural: add a secondary IPv4 CIDR block to the VPC if needed, create new subnets in it, and add them to the Auto Scaling group (or move the workload) — not a retry.

Name collisions

'Already exists' means an explicitly named resource clashes with an existing one. S3 bucket names are globally unique; IAM names are global to the account, so a template with a hard-coded RoleName deploys in one Region and then fails in a second Region of the same account. Let CloudFormation generate names, or build them with Fn::Sub from the account, Region or stack name.

Capacity errors

An insufficient-capacity error (for example InsufficientInstanceCapacity) comes from the underlying service, not from a mistake in the template: retry later, or allow another instance type or Availability Zone.

Permissions and capabilities

A template that creates IAM resources must be acknowledged: CAPABILITY_IAM for IAM resources with generated names, CAPABILITY_NAMED_IAM when they have custom names — otherwise the request is rejected with InsufficientCapabilities before anything is created. CloudFormation then calls each service's API as some principal: by default the caller's credentials; with a stack service role, the role's permissions alone. With a service role, the caller's own policies (and any permissions boundary on the caller) don't apply to the resource calls, so an administrator can still hit access denied if the role lacks a permission. The caller, in turn, needs iam:PassRole on that role (scoped to it) to use it — that is how a service role lets developers deploy without broad rights.

Provisioning at organization scale means pushing one definition everywhere — or sharing one resource instead of duplicating it. CloudFormation StackSets deploy a single template as stack instances across many accounts and Regions from one administrator account.

• Self-managed permissions work with any accounts, including ones outside your organization. You create an administration role in the administrator account (by convention AWSCloudFormationStackSetAdministrationRole) and, in every target account, an execution role (AWSCloudFormationStackSetExecutionRole) whose trust policy lets the administrator account assume it. If the execution role is missing or doesn't trust the administrator, the operation fails in that account.
• Service-managed permissions use AWS Organizations: StackSets creates the roles for you and supports automatic deployment — target an OU and every account that later joins it receives the stack (and, optionally, accounts that leave have it removed). Service-managed StackSets never deploy to the management account, even when you target the root; cover it with an ordinary stack.

AWS RAM solves a different problem: one resource used by many accounts. With VPC sharing, a network account shares subnets so other accounts launch workloads into one centrally managed VPC. After you enable RAM sharing with AWS Organizations, shares to the organization or an OU take effect without invitations to accept. Ownership boundaries in a shared VPC:

• The owner controls the VPC and subnets — route tables, network ACLs, gateways.
• Participants create, own, pay for and manage their own resources (instances, databases, their security groups) and can't modify the owner's network or each other's resources.
• Default subnets can't be shared.

Service Catalog, at mention level: administrators publish approved, CloudFormation-backed products into portfolios, and users launch them self-service without holding the underlying permissions.

Deployment strategies trade speed against blast radius. An in-place deployment updates the instances you already have — fastest and cheapest, but failures land on live capacity and rollback means deploying backwards. A blue/green deployment builds a complete parallel environment, validates it, then shifts traffic at the load balancer or DNS layer, keeping the old one until you're confident — near-instant rollback at the cost of temporarily doubled infrastructure. A rolling deployment replaces capacity in batches within the existing fleet.

Auto Scaling group instance refresh is the rolling tool for a new AMI: update the launch template, start a refresh, and the group replaces instances in batches. Key settings:

• Minimum healthy percentage — the capacity floor during the refresh (terminate-then-launch when below 100%).
• Maximum healthy percentage — above 100% (up to 200%) launches replacements before terminating old instances, so capacity never drops below desired; it needs spare subnet addresses.
• Auto rollback with CloudWatch alarms — if a listed alarm goes into ALARM, or the refresh fails, the group rolls back to the previous launch template version.

Instance refresh also repairs configuration skew when some instances still run an older launch template version.

When CloudFormation owns the group, its UpdatePolicy decides what a launch template change does to running instances:

The AWS CDK is a front end to CloudFormation, not a second engine. You define infrastructure in a general-purpose language; cdk synth synthesizes a CloudFormation template, cdk diff previews changes against the deployed stack, and cdk deploy creates or updates an ordinary stack — so change sets, rollback states, drift and stack events all apply. Before the first deploy to an account and Region, run cdk bootstrap once for that environment: it creates the CDKToolkit stack (asset bucket, ECR repository and deployment roles). A deploy to an un-bootstrapped account or Region fails before creating any stack.

Terraform is the third-party alternative named in the exam guide. It is multi-cloud and tracks what it manages in a state file that you must store, secure and lock, whereas CloudFormation keeps state inside the stack. For a team, use a remote backend — commonly an S3 bucket with versioning and encryption, plus state locking (S3-native lock files in current Terraform; a DynamoDB lock table in older setups) — so everyone shares one state and concurrent applies can't corrupt it. terraform plan previews changes and, by refreshing state, surfaces drift; resources change only on terraform apply, and nothing reverts drift on its own.

Underneath either tool, Git is the source of truth: definitions live in a repository, changes arrive as reviewed pull requests that show the plan or change set, and an approved merge triggers the apply from a pipeline rather than a laptop — reproducible from a known revision and auditable.