SAP-C03 Operational Automation and Resilience: Auto Scaling, Self-Healing, Safe Remediation and Runbooks

Health monitoring becomes an operational response when a signal (a CloudWatch alarm, an AWS Health event or a service event) reaches Amazon EventBridge and a rule starts automation such as an AWS Systems Manager Automation runbook. The architect's job is to pick the signal that actually carries the information, and the response that actually fixes the problem.

CloudWatch alarms and EventBridge

Every CloudWatch alarm state change is published to EventBridge as an event. A rule that matches a change to ALARM can target an SNS topic (people), a Lambda function, a Step Functions state machine, or a Systems Manager Automation runbook (machines). Alarm actions on the alarm itself are narrower: notify through SNS, invoke a Lambda function, run an EC2 action (stop, terminate, reboot, recover), drive Auto Scaling, or create an OpsItem or incident. None of them starts a Systems Manager Automation runbook directly; that path goes through EventBridge. Composite alarms combine several alarms so that one page represents one real impairment; designing those belongs to the observability task, but the alarms you build here (for example on quota usage) often feed them.

AWS Health events

AWS Health reports AWS-side events that affect your own resources: issues, account notifications and scheduled changes such as planned maintenance and instance retirement. Health sends these events to EventBridge (source aws.health), so one rule on the scheduledChange category covers EC2, RDS, ElastiCache, Direct Connect and many other services without polling. Other sources carry different data: Trusted Advisor reports best-practice check results, CloudTrail records API calls, and AWS Config evaluates resource configuration, none of which announce AWS maintenance.

A common automation is instance retirement. When AWS schedules an EBS-backed instance for retirement because its host is degraded, the Health event (for example AWS_EC2_PERSISTENT_INSTANCE_RETIREMENT_SCHEDULED) can start the AWS-owned AWS-RestartEC2Instance runbook with the instance ID from the event. A stop and start places an EBS-backed instance on new hardware; an operating-system reboot does not, because the instance stays on the same host. A CloudWatch recover action is different again: it reacts after a system status check has already failed.

One Health feed for an organization

By default an EventBridge rule in an account sees only that account's Health events. With AWS Health organizational view turned on from the management account, the management account, or an account registered as a delegated administrator for AWS Health, receives a single feed of Health events from every account in the organization. Rules written in that operations account then cover all accounts with nothing deployed in member accounts, which is the difference from forwarding rules pushed to each account with StackSets.

Service Quotas is the capacity-planning control for account limits: quotas apply per account and per Region, and the service lets you monitor usage against them, request increases, and pre-request increases for new accounts. A scale-out or failover that hits a quota fails just as surely as one that lacks hardware.

Alarming on quota utilization

Many quotas publish usage to the AWS/Usage CloudWatch namespace (for example the vCPU ResourceCount for Running On-Demand Standard instances). CloudWatch metric math has a SERVICE_QUOTA() function that returns the current applied quota for a usage metric, so an expression such as m1/SERVICE_QUOTA(m1)*100 gives a utilization percentage. An alarm on that expression keeps working after every quota increase, while a static threshold computed from today's quota becomes wrong the moment the quota changes. Because it is an ordinary CloudWatch alarm, it can feed a composite alarm or an EventBridge rule.

Quota request templates and automatic management

• Quota request templates: in an organization with all features turned on, you list up to 10 quota values in a template and associate it with the organization. Service Quotas then submits those increase requests automatically for every new account that is created, so accounts start with the quotas they need (supported in commercial Regions that are not opt-in).
• Automatic Management: Service Quotas watches usage and, in Notify only mode, alerts when usage approaches a quota (at 80% and 95% utilization); in Notify and auto-adjust mode it also requests increases. Notices arrive through AWS Health, EventBridge, email or chat. It reacts to usage that already exists, so it does not give a new account raised quotas on day one.

Quotas in disaster recovery

Because increases are per Region, a recovery Region where the same functions or instances sit idle usually still has default quotas. A Lambda function failing over to such a Region is throttled at the Regional concurrency quota; reserved and provisioned concurrency are both allocated from that quota and cannot raise it. Request the same increases in the recovery Region in advance, and make quota parity part of DR readiness checks.

Capacity reservations versus pricing commitments

Only a capacity reservation guarantees that instances can launch in a specific Availability Zone; quotas are permissions and Savings Plans are pricing commitments. Many capacity-planning scenarios are decided by keeping those three apart.

Two facts combine in cost questions. Capacity Reservations are billed at the On-Demand rate, but Savings Plans and regional Reserved Instances apply their discount to Capacity Reservations. So capacity that must be guaranteed for a long time, while the instance family may change, is best met with On-Demand Capacity Reservations covered by a Compute Savings Plan; a zonal Standard RI would also reserve capacity but ties the commitment to one instance type. For a short known peak (a sale, a launch), create reservations for just that period and release them afterwards. A scheduled action or a higher maximum size only changes when the launch is attempted, not whether capacity exists.

Open versus targeted reservations

A reservation with open eligibility is used automatically by any matching instance in the account. A reservation with targeted eligibility accepts only instances whose launch specifies it, either by reservation ID or through a capacity reservation resource group that contains it. An Auto Scaling group therefore uses targeted reservations only if its launch template's capacity reservation specification targets the reservation or the resource group; otherwise the reservations sit unused while the group launches ordinary On-Demand instances. Switching the reservations to open would fix utilization but lets other teams' matching instances consume the capacity, and an open preference in the launch template matches only open reservations.

A safe retry waits longer after each failure (exponential backoff), randomizes each wait (jitter), stops after a capped delay and number of attempts, and is only repeated for operations that are idempotent. Each part prevents a different failure.

• Backoff reduces the load each client adds while a dependency is struggling.
• Jitter stops clients that failed at the same moment from retrying at the same moment. Without it, thousands of clients on a fixed or purely exponential schedule retry in synchronized waves, and a recovering service is knocked over again. Fixed intervals, immediate retries and backoff without jitter all keep the waves.
• Caps on the maximum delay and the number of attempts keep the behaviour bounded.
• Idempotency makes a repeated request harmless. A retry can repeat a write whose first attempt succeeded but whose response was lost; a unique request ID enforced with a conditional write (or an idempotency token on the API) rejects the duplicate. Fewer or no retries trade duplicates for lost writes, which is not a fix.

Step Functions retriers

A Task state's Retry field lists retriers with ErrorEquals, IntervalSeconds (first wait), BackoffRate (multiplier per attempt), MaxAttempts, MaxDelaySeconds (a cap on any single wait) and JitterStrategy. Setting JitterStrategy to FULL randomizes each delay between zero and the computed backoff, which breaks up waves when many executions fail together; NONE is the default. Changing the backoff rate or the attempt count, or looping through a fixed Wait state, keeps every execution on the same schedule.

AWS SDK retry modes

Raising the attempt count in standard mode amplifies load during throttling; adaptive mode is the setting that makes each client back off its sending rate.

A circuit breaker stops calls to a failing dependency for a cool-down period and serves a fallback instead, while a bulkhead isolates resources so that one failing dependency or tenant cannot exhaust what others need. Retries handle brief faults; these patterns handle long ones.

Circuit breaker states

• Closed: calls flow normally and failures are counted.
• Open: after repeated failures, callers skip the dependency and return a fallback (a cached or default value, or a queued request) until the open period ends.
• Half-open: after the open period, one trial request is allowed. If it succeeds the circuit closes; if it fails the circuit reopens.

Without a half-open state, every waiting caller sends traffic the instant the open period ends, a fragile dependency fails again, and the circuit flaps. A longer open period or jittered retries still release full traffic at once.

Shared state in DynamoDB

In serverless designs (Step Functions and Lambda) the breaker state must be shared by every caller, and AWS Prescriptive Guidance uses a DynamoDB table for it: callers read the circuit item before calling, and a failure counter opens it. Two implementation details matter:

• Expiry must be checked on read. DynamoDB TTL deletes expired items in the background, typically within a few days, and reads return expired items until then. A breaker that treats "item exists" as "circuit open" can stay open for hours. Compare the expiry attribute with the current time on every read (or use a filter expression) and treat an expired item as closed.
• Exactly one probe in half-open. A conditional write, such as setting a probe owner only if none exists, lets exactly one caller win the right to send the trial request while the rest keep using the fallback.

Queues, lower concurrency and more retries only slow calls down; they do not stop them or provide a fallback.

Bulkheads

The symptom of a missing bulkhead is a shared resource exhausted by one slow or noisy party. Typical bulkheads on AWS:

Fair queues are not FIFO queues. On a FIFO queue the same attribute means strict ordering per group, FIFO throughput quotas apply to the queue, and an existing standard queue cannot be converted to FIFO.

Controlled remediation applies a change to many resources in limited batches and stops automatically when too many changes fail, so that a faulty fix damages a few resources instead of all of them. AWS offers this control in several services.

Systems Manager rate control

A rate-controlled automation runs a runbook against targets (tags, resource groups or parameter values) with two safety settings:

• MaxConcurrency: how many targets run at once, as a number or a percentage of the targets.
• MaxErrors: how many failures are tolerated before Automation stops starting new targets, also a number or a percentage. With an absolute value of N, it stops when error N+1 arrives; invocations already running may finish.

MaxErrors and MaxConcurrency each accept either an absolute count or a percentage of the targets, and on a large fleet the two give very different stop points. A simple execution that lists every instance ID has no batching or error threshold. Run Command and maintenance window tasks use the same two settings. Running the same control across many accounts and Regions is covered in the multi-account operations lesson.

Step Functions Distributed Map

For a large item list (for example a large item list stored in S3), a Map state in Distributed mode runs each item as a child workflow execution. MaxConcurrency limits how many run in parallel, and ToleratedFailurePercentage or ToleratedFailureCount sets how many items may fail before the whole Map Run fails, which is the stop switch. Inline mode runs inside the parent execution, is limited to 40 concurrent iterations and has no tolerated-failure setting; per-item retries never stop the run.

EventBridge API destinations

When remediation or ticketing calls a third-party HTTP API with a request ceiling, an API destination enforces it without code: you set an invocation rate limit per second, and events above it are delayed and delivered later within the target's retry policy (by default up to 24 hours and 185 attempts). EventBridge retries 429 and 5xx responses, and a dead-letter queue keeps events that still cannot be delivered. Retries without a rate limit break the ceiling, and Lambda concurrency limits cap parallel calls, not requests per second.

Target tracking scaling keeps a metric (average CPU, request count per target, or a custom metric) near a target value by creating and managing its own CloudWatch alarms and adjusting capacity proportionally. It is the default choice for metric-based scaling of Auto Scaling groups.

How target tracking behaves

• Several policies favour availability. When a group has more than one target tracking policy, it scales out if any policy calls for more capacity (the largest request wins) and scales in only when every policy is ready to scale in. Policies have no creation-order priority.
• Never edit the managed alarms. The alarms that target tracking creates belong to the policy; edits are overwritten or break it.
• Default instance warmup. New instances often run hot while initializing (compiling caches, loading data). Target tracking aggregates the metric over in-service instances, so that startup spike looks like load and triggers more scale-outs. A default instance warmup on the group keeps new instances' metrics out of the aggregate until the warmup ends; it is the setting AWS recommends for every group. The health check grace period is a different setting that only delays health checks.
• Scale-in protection. Termination policies choose which instance leaves (oldest, oldest launch template and so on). To keep a busy instance from being chosen at all, the workload sets instance scale-in protection on itself when a job starts and clears it when the job ends. Scale-in protection does not stop health-check replacement.

Custom scale-in with step scaling

When scale-in must be slower than scale-out (for example, only after the metric has stayed well below target for a set time, and one instance at a time), set DisableScaleIn on the target tracking policy so that it only adds capacity, and add a step scaling policy on your own CloudWatch alarm that removes capacity. A second target tracking policy at a lower target does not achieve this: it would also scale out whenever the metric passes its lower target. The group's default cooldown applies only to simple scaling policies, so it does not tune target tracking or step scaling.

Scaling ECS services and Lambda concurrency for performance, and choosing the metric for queue backlogs, are covered in the performance and scalability lessons.

Predictive scaling forecasts load from the metric history of an Auto Scaling group and launches capacity ahead of the forecast, scheduled actions set capacity at fixed times, and warm pools make a scale-out fast by keeping pre-initialized instances ready. Each answers a different question: recurring patterns, known times, or slow boots with unpredictable spikes.

Predictive scaling modes and limits

• Forecast only creates forecasts (visible in the console and as metrics) without changing capacity, so you can compare them with actual load before trusting them. Forecast and scale also launches capacity ahead of the forecast; a scheduling buffer controls how far ahead.
• By default predictive scaling honors the group's maximum size (MaxCapacityBreachBehavior HonorMaxCapacity). With IncreaseMaxCapacity it may raise the maximum when the forecast exceeds it, limited by MaxCapacityBuffer, a percentage above the forecast capacity. That ties growth beyond the maximum to the forecast, which a fixed larger maximum or daily scheduled maximums do not.
• Predictive scaling does not react to unexpected load, so AWS recommends keeping a dynamic policy such as target tracking alongside it; the group uses the higher of the two capacities.

Scheduled actions and time zones

A scheduled action sets minimum, maximum or desired capacity once or on a cron recurrence. The recurrence is evaluated in UTC unless the action has a time zone (an IANA name such as Europe/Berlin); with one, the cron expression is written in local time and follows daylight-saving changes automatically. A fixed UTC expression is right for only half the year in such zones. Use scheduled actions when a floor is required at a fixed time; a forecast does not guarantee one.

Warm pools

A warm pool holds instances that have already completed their slow first boot, so a scale-out only has to start them. Instances can wait in the Stopped state (you pay for volumes, not compute, usually the cost-effective choice), Hibernated (memory kept on the root volume) or Running (fastest, billed as running). An instance reuse policy can return instances to the pool on scale-in instead of terminating them.

Two details decide many designs:

• When an instance enters the warm pool, Auto Scaling does not wait for user data to finish before stopping or hibernating it, so long downloads or installs can be cut off. Add a launch lifecycle hook and have the script call CompleteLifecycleAction when initialization finishes (or let the heartbeat timeout pass).
• Warm pools need EBS root volumes and cannot be used in mixed instances groups that use Spot Instances or instance weighting.

Automated failure recovery means a health signal marks a unit unhealthy and the platform replaces or restarts it, in a way that keeps enough healthy capacity and lets in-flight work finish. Auto Scaling groups and ECS each have several knobs for this.

Auto Scaling health check sources

Types can be combined (for example EBS and ELB together). A TCP health check that only proves a listener accepts connections cannot see a hung disk, which is what the EBS type adds; an external monitoring tool can mark an instance Unhealthy through SetInstanceHealth without a load balancer. The health check grace period applies only to newly launched instances (it also covers VPC Lattice and container health checks); it adds no check and does nothing for a reboot later in the instance's life.

Replacing without losing capacity or work

• By default a health replacement terminates first, then launches. An instance maintenance policy with a minimum healthy percentage of 100 and a maximum above 100 makes the group launch the replacement and wait for it before terminating, so healthy capacity never drops below desired. It applies to health replacements, instance refresh, maximum instance lifetime and rebalancing, without raising steady-state capacity.
• Instance scale-in protection stops only scale-in, and Auto Scaling does not honour EC2 termination protection. A termination lifecycle hook applies to every termination by the group, including health replacement: the instance waits in Terminating:Wait while it uploads a checkpoint or drains, then calls CompleteLifecycleAction (or the timeout passes).
• Suspending ReplaceUnhealthy or HealthCheck stops replacements for every instance, which is rarely acceptable as a fix for one planned activity.

Amazon ECS restart and replacement

A process that hangs without exiting never triggers a restart policy; a health check (a liveness probe that checks the process is still doing work) is what detects it. Marking a sidecar non-essential stops it from taking down the task but leaves it dead.

Draining ECS container instances

With an Auto Scaling group capacity provider, managed termination protection only stops scale-in from removing instances that run tasks. Managed instance draining uses an ECS-managed lifecycle hook to put an instance into DRAINING before any group termination (scale-in, instance refresh, maximum instance lifetime, health replacement, Spot rebalancing and interruption), so tasks are rescheduled elsewhere first. The agent's Spot instance draining setting covers only Spot interruption notices, and a target group's deregistration delay does not move tasks.

How Lambda retries a failure depends on how the function is invoked: asynchronous invocations, stream event source mappings and SQS event source mappings each have their own retry settings and their own place for failed events. Most mistakes come from applying one invocation model's setting to another.

Keeping good records and setting bad ones aside

• Streams: bisect batch on function error splits a failing batch in two on each error, so good records are processed and the bad one is isolated; a finite retry count or record age lets Lambda discard it and move on. Both are configuration only. Partial batch responses (ReportBatchItemFailures) also work for streams but require function code that returns the failed sequence number. A higher parallelization factor adds concurrency per shard but keeps per-key ordering, so a poison record still blocks its key.
• SQS: with ReportBatchItemFailures turned on, the function returns the IDs of failed messages, so successful messages in the batch are deleted and not processed again (which prevents duplicate side effects). Combine it with the queue's redrive policy; AWS recommends a maxReceiveCount of at least 5 and a visibility timeout of at least six times the function timeout. Bisect and maximum retry attempts are stream-only settings, and a function's async DLQ never sees event source mapping traffic.

Automated runbooks turn an operational procedure (remediate, collect evidence, restart, patch, scale) into a Systems Manager Automation document or patch configuration that starts from an event or a schedule, with approvals and safety limits where a human must decide.

Starting runbooks from events

An EventBridge rule can target a Systems Manager Automation runbook directly. An input transformer maps fields from the event (such as the instance ID in an alarm or Health event) to the runbook's parameters, and EventBridge needs an IAM role it can assume to start the execution. This removes the human from routine, well-understood responses. An OpsItem with an associated runbook or an SNS notification still waits for a person, and an EC2 alarm action can only stop, terminate, reboot or recover.

Approvals and evidence

The aws:approve action pauses a runbook until a specified approver (IAM users, IAM roles or assumed-role sessions; a minimum number of approvals can be required) approves or rejects. Put it after steps that must happen immediately, such as collecting thread dumps and logs from every instance, and before disruptive steps such as a restart.

Tracking incidents

Systems Manager OpsCenter records operational issues as OpsItems, created from CloudWatch alarms (an alarm action) or EventBridge rules. It groups duplicates, shows the affected resources with their configuration and CloudTrail history, and lets an engineer run the associated runbook from the item. AWS Systems Manager Incident Manager is closed to new customers; existing customers keep it, but new accounts should use OpsCenter with alarms, runbooks and their own paging. Explorer only summarizes operational data, and the Health Dashboard reports AWS-side events.

AWS Config auto-remediation

An AWS Config rule can have an automatic remediation action, a Systems Manager Automation runbook that runs when a resource becomes noncompliant, with a configurable number of retry attempts and retry interval so that a transient failure does not leave the resource noncompliant. Rate-limited remediation of many resources is covered above.

Patching at scale

• Quick Setup patch policies define scan and install schedules, patch baselines (which classifications and severities, with approval delays) and reboot options once, and apply them to an organization, chosen OUs or accounts, and chosen Regions. New accounts in targeted OUs are covered automatically. Per-account maintenance windows deployed with StackSets work but cost more effort, and Amazon Inspector finds missing patches without installing them.
• Maintenance windows run AWS-RunPatchBaseline tasks on a schedule with the same concurrency and error thresholds as other rate-controlled tasks.
• Patching inside Auto Scaling groups: a reboot can fail ELB health checks, and the group replaces the instance from the old AMI. The grace period does not help (it covers only new launches). AWS-RunPatchBaselineWithHooks runs a pre-install hook before installing, a post-install hook before the reboot and an on-exit hook after the reboot, so each instance can enter Standby before patching and exit it afterwards, while other instances are still replaced if they fail.
• Immutable patching: patching in place leaves the AMI stale, so new launches start unpatched. An EC2 Image Builder pipeline can build a patched AMI on a schedule and, through launch template configurations in its distribution settings, set it in a new default launch template version. An instance refresh then replaces instances in batches while keeping its minimum healthy percentage in service, and auto rollback with a CloudWatch alarm specification returns to the previous configuration if the alarm fires.