Automated Remediation: EventBridge, Runbooks, and EC2 Recovery (SOA-C03)

Every automated-remediation design on this exam is the same four-step loop, and naming the steps makes unfamiliar scenarios easy to decompose. Detect: a signal says something is wrong — a CloudWatch alarm breaches, an EC2 status check fails, an AWS Health event announces degraded hardware, or an anomaly-detection band is exceeded. Route: the signal travels to whatever will act on it — an alarm's built-in action, an SNS topic fanning out to subscribers, or an EventBridge rule matching the event and forwarding it to targets. Act: something fixes the problem — the EC2 recover action, an Auto Scaling replacement, a Lambda function running custom logic, or a Systems Manager Automation runbook executing a documented procedure. Verify: confirm the fix worked — the alarm transitions back to OK, the runbook reports step success, or an OpsItem records the incident for review.

The verify step is the one candidates and junior engineers skip, and the exam notices. A Lambda that restarts a service but never checks the result is half a remediation; a design that closes the loop — act, then watch the same metric that detected the problem return to healthy — is the stronger answer.

Two scope boundaries keep this task tidy. Creating the metrics, filters, and alarms themselves belongs to Task 1.1; here they are inputs. And tuning the resource that keeps misbehaving — resizing an instance, changing an EBS volume type, adding RDS Proxy — is performance optimization, Task 1.3. Task 1.2 is the plumbing between a signal and a fix, and questions reward the design with the least ongoing human effort: managed loops over hand-run commands.

Before you can remediate, you have to identify what is failing, and Task 1.2 names three analysis tools. Metric math builds expressions over existing metrics so you can alarm on the ratio or aggregate that actually matters. Raw error counts mislead — 50 errors is noise at 100,000 requests and an outage at 60 — so an expression like 100*(errors/requests) produces an error-rate metric, and the alarm watches the expression instead of either raw series. Metric math also aggregates across resources: summing a metric over a fleet, or subtracting healthy hosts from desired capacity to alarm on the gap.

Alerting on two conditions at once has two answers. Inside one alarm, the metric math IF function gates the expression — IF(m2 >= 100, 100*m1/m2, 0) reports the error rate only when at least 100 requests arrived and otherwise returns 0, so a handful of failures at idle never breaches. Across alarms, a composite alarm with an AND rule fires only when every child alarm is in ALARM — for example an error-percentage alarm AND a request-volume alarm. The IF expression keeps it to one alarm; the composite keeps each condition visible and reusable.

Anomaly detection answers the question a static threshold cannot: what is normal right now? CloudWatch trains a model on the metric's history and draws an expected band around it, accounting for daily and weekly patterns; the alarm fires when the metric leaves the band. Choose it for metrics with strong cyclical behavior — request volume, queue depth — where any fixed threshold is either too tight at peak or too loose at night. Choose a plain threshold when the limit is genuinely absolute, like a disk filling up. Two tuning controls matter. The band's threshold is a number of standard deviations: a higher value widens the band (fewer alarms), a lower one narrows it. And you can exclude time periods from training — a deployment, a migration, a one-off event — so an abnormal stretch does not teach the model that abnormal is normal and leave the band too wide to catch the next real regression.

The third tool is correlation with CloudTrail. Incidents caused by change dominate real operations, so line the incident timeline up against recent API activity: latency spiked at 14:02, and CloudTrail shows a security group was modified at 14:01 — you have your suspect, and the remediation is to revert the change, not restart servers. CloudTrail management events record which principal called which API, against which resource, and when. CloudWatch tells you when it broke; CloudTrail tells you what changed and who changed it. Mind the retention window: the CloudTrail console's event history covers only the last 90 days. For anything older you need a trail delivering log files to Amazon S3 (an organization trail does this for every member account), and you query those files with Amazon Athena. (CloudTrail Lake also queries events with SQL, but it closed to new customers on 31 May 2026, so do not assume it exists in an account.)

EC2 runs two automated status checks per instance, and telling them apart is one of the highest-yield facts in Domain 1 — because each maps to a different fix.

The recover action can be attached only to an alarm on StatusCheckFailed_System: CloudWatch detects the host-level failure and migrates the instance to new underlying hardware. The recovered instance keeps its instance ID, private IP addresses, Elastic IP addresses, attached EBS volumes and instance metadata. It does not keep anything in RAM, any data on instance store volumes, or its OS uptime — recovery is a restart on a new host, not a live move. The trigger phrase is "underlying hardware failure" or "host-level issue". Recovery has limits worth memorising: it is not supported for instances in an Auto Scaling group (the group's own health checks replace them instead) or for instances on Dedicated Hosts. Supported instance types also get simplified automatic recovery by default; the alarm is how you make recovery explicit and visible.

A recover alarm can never fix an OS fault, and the reason is the metric, not the action: during a corrupted filesystem or a kernel out-of-memory hang, the system check keeps passing, so a StatusCheckFailed_System alarm never leaves OK. OS faults fail the instance check, so the matching automation is an alarm on StatusCheckFailed_Instance with the EC2 reboot action. One more prerequisite applies to every EC2 alarm action (stop, terminate, reboot, recover): the alarm must watch a per-instance metric — one with an InstanceId dimension. An alarm on a metric aggregated by AutoScalingGroupName cannot be offered an EC2 action, because there is no single instance to act on.

When a reboot is not enough

If the OS cannot boot at all — for example a broken kernel or bootloader after an update — rebooting only repeats the same failure, and no remote-access tool can reach an instance whose OS never comes up. Repair the root volume offline. The AWSSupport-ExecuteEC2Rescue Automation runbook automates this: it stops the instance, attaches its root volume to a temporary helper instance, runs the EC2Rescue tool to fix common boot problems, and puts the volume back (the runbook works only on unencrypted root volumes; for an encrypted one, use the manual method). The manual equivalent is the same idea: stop the instance, detach the root EBS volume, attach it to a healthy helper instance in the same Availability Zone, fix the files, then reattach it as the root device and start.

Fleets: Auto Scaling health checks

For fleets, Auto Scaling is the remediation layer: its health checks mark a bad instance unhealthy, terminate it and launch a replacement. EC2 status checks are always used; turning on ELB health checks adds the load balancer's view, so an instance that hangs at the application layer — status checks passing, target failing — is also replaced instead of lingering at reduced capacity. The health check grace period tells the group how long to wait after an instance enters service before acting on failed health checks. If instances need longer to warm up (load a cache, start an app server) than the grace period allows, the group terminates them before they become healthy and loops; set the grace period just above the real warm-up time. Auto Scaling replaces rather than repairs — the right model for stateless fleets, and the wrong one for a stateful single instance, which is exactly the distinction scenario questions probe.

The exam's remediation questions are usually pattern-selection questions: given a failure and a constraint (least operational overhead, no code, multi-step fix), pick the right wiring. Learn the menu.

Two habits resolve most choices. First, prefer managed capability over custom code: if a predefined runbook or a built-in alarm action does the job, it beats a Lambda you must write, test, and maintain — that is what "least operational overhead" means. Second, match the blast radius: instance-level failures get instance-level actions; fleet-level symptoms get Auto Scaling; anything needing sequenced steps with verification gets an Automation runbook. SNS stays in the picture even when automation acts, because remediation you were never told about is how small incidents hide until they become large ones — a topic can notify humans in parallel with the automated fix.

Amazon EventBridge is the routing layer of AWS operations. Every account has a default event bus in each Region that automatically receives events from AWS services — alarm state changes, EC2 instance state changes, AWS Health notifications — and you can add custom buses for your own applications or partner buses for SaaS sources. A rule attaches to one bus, matches events against an event pattern, and delivers each match to up to five targets. Patterns are JSON that must match the event's actual structure: every field you specify must be present in the event with one of the values you list, matching is exact and case-sensitive, and values sit in arrays ("state": {"value": ["ALARM"]}).

Delivery is immediate: when an event matches, each of the rule's targets is invoked independently, straight away. One rule can therefore fan out — an SNS topic to tell people and an Automation runbook to fix the problem, from the same match. A rule has no notion of a change window; if work must wait for an approved time, the rule should open a tracked item or notify, and the change itself should run from a schedule (a maintenance window or a scheduled rule).

Shaping what the target receives

Each target chooses how much of the event it gets. This is the exam's "enrich or shape events without adding a Lambda" skill.

Troubleshooting a rule that never fires

Troubleshooting is an explicit skill (1.2.2), and the rule's CloudWatch metrics (namespace AWS/Events) tell you which half is broken. MatchedEvents and TriggeredRules count events that matched the rule; Invocations counts deliveries to targets; FailedInvocations counts deliveries that failed. If nothing matches, the problem is upstream of the target. Work the checklist: the pattern doesn't match the real event (test it against a real sample event — a typo'd field name or wrong case silently matches nothing); the rule is disabled; the event is arriving on a different bus than the rule watches; or the event occurs in a different Region — EventBridge is regional, and a rule in us-east-1 never sees a eu-west-1 alarm unless that Region forwards it. If the rule matches but FailedInvocations climbs while the target sees nothing, the problem is the target's permissions: Lambda functions and SNS topics need a resource-based policy allowing events.amazonaws.com, while targets such as Systems Manager Automation, Step Functions and event buses in other accounts or Regions need an IAM role on the target that EventBridge assumes.

Scheduled automation — nightly cleanup, weekly patch checks — is started by a schedule expression, and EventBridge accepts two kinds. A rate expression runs at a fixed interval: rate(value unit), where the unit is minutes, hours or days. The unit is singular only when the value is 1 — rate(1 hour), but rate(6 hours); rate(6 hour) is invalid. Use rate when only the spacing matters, not the clock time.

A cron expression runs at specific clock times and has six fields: cron(minutes hours day-of-month month day-of-week year). You cannot specify both day-of-month and day-of-week; one of the two must be ?. So 02:00 every Sunday is cron(0 2 ? * SUN *), and 01:30 every day is cron(30 1 * * ? *).

So a job that must run at a fixed local time all year belongs in EventBridge Scheduler with a cron expression and the local time zone; a UTC-only rule would drift by an hour at every daylight saving change, and a 24-hour rate would too. AWS recommends Scheduler for new scheduled tasks.

Centralised remediation means getting events to one place, and EventBridge does it by letting a rule's target be another event bus — in another Region, another account, or both. The sending rule matches the event in its own account and Region and forwards it; rules on the receiving bus then act on it as if it had arrived locally.

• Cross-Region: a rule in each source Region targets the default bus in the central Region (for example, alarm events from eu-west-1 and ap-southeast-2 forwarded to us-east-1, where the runbooks live). No code and no duplicated runbooks; the target needs an IAM role that allows events:PutEvents on the destination bus.
• Receiving side (cross-account): the central bus needs a resource-based policy allowing events:PutEvents from the senders. For a whole organization, allow the organization with the aws:PrincipalOrgID condition rather than listing every account.
• Sending side (cross-account): a new cross-account event bus target also needs an IAM role in the sending account that allows events:PutEvents on the destination bus. The console creates this role for you; with the AWS CLI, CloudFormation or other IaC you must create it and set it on the target yourself — a common reason scripted deployments deliver nothing.
• One hop only: EventBridge does not forward events that a bus received from another account on to a third account. A chain such as workload account → security account → operations account stops at the second account; send events directly from each source to every account that needs them.

AWS Systems Manager is the act-and-verify half of the loop, and Task 1.2 names its pieces. Automation runbooks are step-by-step documents that execute operational procedures against AWS resources. AWS ships predefined runbooks with AWS- and AWSSupport- prefixes — AWS-RestartEC2Instance stops and then starts an instance, and others create images, snapshot volumes or repair instances — while custom runbooks you author in YAML or JSON sequence your own steps: invoke AWS API actions, run scripts, branch on results, pause for approval, and stop on a failed step. Runbooks can be started manually, from an EventBridge rule, on a schedule, from a State Manager association or maintenance window, or through the AWS SDKs and CLI (StartAutomationExecution). When a scenario needs a multi-step, repeatable, documented fix with recorded per-step results and no Lambda to maintain, the answer is an Automation runbook.

Whose permissions does a runbook use? A runbook's steps run with the permissions of its AutomationAssumeRole parameter when one is set. Without it, Automation uses the permissions of whoever started the execution — and when EventBridge starts it, that is the IAM role on the rule's target. A target role that allows only ssm:StartAutomationExecution can start the runbook, but the first step that calls, say, ec2:CreateSnapshot fails as unauthorized. Either give the runbook an assume role with the step permissions, or grant them to the starting identity. Passing an assume role also requires iam:PassRole on that role for the identity starting the execution.

The State Manager versus maintenance window line is a favourite: an association with no schedule still runs once at creation, persists, and is applied automatically to new nodes that match its targets; a scheduled association reapplies on its cadence. A maintenance window task runs only when its window opens, so drift between runs persists until the next window. "Ensure this setting stays configured on every instance, including new ones" is State Manager.

OpsCenter aggregates operational work items as OpsItems — the tracked, auditable record of an issue, with related resources, operational data and suggested runbooks attached, open until someone resolves it. CloudWatch alarms can create an OpsItem directly as an alarm action, and EventBridge rules can create them as a target. For noisy sources, put a deduplication string in the OpsItem's operational data (the /aws/dedup key): while an OpsItem with the same string is open or in progress, OpsCenter adds to it instead of opening a new one. Systems Manager Explorer is the dashboard on top — a customizable view of OpsItems and other operations data (such as patch compliance and instance inventory) across accounts and Regions. Incident Manager (response plans, escalation and on-call engagement) closed to new customers on 7 November 2025; AWS directs new customers to OpsCenter and related capabilities, so do not assume it is available in a new account.

Not every remediation signal starts in your own metrics. The AWS Health Dashboard has two views. Service health is the public view of AWS service status across Regions. Your account health shows events that affect your account specifically: operational issues impacting your resources, and scheduled changes such as host maintenance, instance retirement or certificate expirations — with the affected resource IDs and dates. That account-specific view answers "which of my instances is scheduled for retirement next month"; the public view cannot.

Health events are also delivered to EventBridge, which turns the dashboard into a remediation source. A rule matching aws.health events can notify the operations channel through SNS, open an OpsItem in OpsCenter, and start a runbook that replaces an affected instance long before its retirement date, from the same match. Delivery is Regional, which catches people out:

• Each event goes to EventBridge in the Region it concerns, so a rule in us-east-1 sees only us-east-1 events. Events not tied to a Region (global events, such as IAM) arrive in us-east-1.
• Every event is also sent to a backup Region — us-west-2 for all other Regions (us-east-1 backs up us-west-2) — so rules in both Regions, de-duplicated, give a highly available setup.
• Simplified integration: a single aws.health rule in us-west-2 receives account-specific events from all standard Regions. It does not include public events and has no high-availability backup, which suits action-required notices like scheduled maintenance.

Kiro, AWS DevOps Agent and Amazon Q

The SOA-C03 guide names two agentic AI tools, Kiro and AWS DevOps Agent, as examples for analyzing performance metrics and automating remediation. Know what each is for, and keep them apart from the console assistant:

Stay at this level: the guide does not enumerate their features, so resist answer choices that depend on invented specifics.

Put the loop together on a realistic setup: an Auto Scaling group of stateless web servers in front of one stateful reporting instance that cannot simply be replaced.

The fleet largely heals itself. The Auto Scaling group uses ELB health checks, so an instance that stops answering the load balancer — not just one that fails EC2 status checks — is marked unhealthy, terminated, and replaced. Your job is detection coverage on top: an alarm on the group's unhealthy-host count routed to SNS, so a pattern of replacements gets investigated rather than silently absorbed.

The stateful instance gets two alarms. First, StatusCheckFailed_System with the recover action — if the underlying host fails, the instance migrates to healthy hardware keeping its instance ID, IPs, and EBS volumes. Second, StatusCheckFailed_Instance, 2-out-of-2 datapoints, with no EC2 action (the team wants a recorded stop-and-start rather than an in-place reboot); instead, an EventBridge rule matches this alarm's state change to ALARM and targets the AWS-RestartEC2Instance Automation runbook, with an input transformer extracting the instance ID from the event into the runbook's parameter. The rule's IAM role permits starting the automation; SNS notifies the on-call in parallel.

Verify, then troubleshoot. Success looks like: runbook execution succeeds, the status-check alarm returns to OK, and an OpsItem records the event. During the game-day test, the runbook never ran — so you walk the EventBridge checklist: the pattern had "state": ["ALARM"] instead of matching the event's actual nested state.value field, so it matched nothing. Fix the pattern, replay the test, watch the loop close. That debugging sequence — symptom, checklist, single root cause — is precisely how the exam frames these questions.