SAP-C03 Backup Governance, Ransomware Protection and Chaos Engineering

You enforce backups organization-wide by attaching an AWS Organizations backup policy to the root, an OU or an account. Organizations merges every policy that applies to an account into an effective policy, and AWS Backup renders it as a backup plan in each member account and listed Region. Member accounts can see that plan but cannot edit or delete it, and accounts that join the OU later receive it automatically. This is the difference from deploying ordinary backup plans with CloudFormation StackSets: a StackSet plan is a normal resource that an account administrator can change, while a policy-managed plan is controlled only from the management account (or a delegated administrator for backup policies).

What a backup policy contains

• Plans and rules: schedule, lifecycle (retention), target vault, and optional copy actions to another vault, Region or account.
• Resource selection: usually by tag (for example every resource tagged backup=gold), so new resources are protected as soon as they are tagged.
• Regions: the plan is created in each listed Region of each account.
• Variables: $account in an ARN is replaced with each member account's ID, so one policy can name "this account's vault" everywhere.

The policy creates the plan, not its dependencies

A backup policy references a vault and an IAM role by name, but it does not create them. Every vault the policy names, including copy-target vaults in a second Region, and the IAM role must already exist in every account and Region where the plan runs. If they are missing, the account shows the effective plan but its backup or copy jobs fail. The usual fix is a service-managed StackSet with automatic deployment to the target OU or organization, so accounts that are created, invited or moved in later get the vault and role without manual steps. Organization-level AWS Backup settings only permit behaviour; they never deploy vaults or roles into member accounts.

Stopping child OUs from weakening a parent policy

Backup policies inherit through the OU tree with inheritance operators: @@assign overwrites a value, @@append adds to a list and @@remove deletes one. By default a child policy can use any of them, so a business unit's OU policy can @@assign a shorter retention onto a plan defined at the root. The parent controls this with the child control operator @@operators_allowed_for_child_policies. Setting it to ["@@none"] on a node locks that node's value; set it on each level of the plan you want fixed (the plan, its rules, its lifecycle). Putting @@none on the top-level plans key instead would also stop children from adding plans, which is usually not what you want.

Cross-account copy sends each recovery point from a workload account into a vault in a separate, tightly controlled backup account, so a compromised workload account cannot reach the copies. It needs three things.

1. Opt in once from the management account. Cross-account backup is an AWS Backup setting turned on while signed in to the organization's management account; it applies to accounts in the organization. Registering a delegated administrator or sharing the vault does not replace this step.
2. A named destination vault with an access policy that allows backup:CopyIntoBackupVault, typically for any principal whose aws:PrincipalOrgID matches the organization. Copies are pushed by the source account's copy job; the destination does not pull them, and AWS RAM sharing is not how standard vaults accept copies. The account's Default vault cannot be a cross-account copy destination; create a separately named vault.
3. Keys the destination can use. For resource types that are fully managed by AWS Backup (for example Amazon EFS, Amazon S3 and DynamoDB with AWS Backup advanced features), the recovery point is encrypted with the vault's key, so copies just work. For other types, such as Amazon EBS, Amazon EC2 and Amazon RDS, the backup keeps the source resource's encryption. AWS managed keys such as aws/ebs and aws/rds have key policies you cannot edit, so they cannot be shared and those copies fail. Encrypt the source with a customer managed KMS key whose key policy allows the destination account.

Restoring into a different account

A recovery point in a standard vault can be restored only by the account that holds it. To recover a retired or compromised account's data into a new account, first create a vault in the new account whose access policy allows backup:CopyIntoBackupVault from the backup account, copy the recovery points from the central vault into it, and then restore there. A backup policy only governs future backup jobs; it never moves existing recovery points anywhere. The exception is a logically air-gapped vault shared through AWS RAM, which a recovery account can restore from directly (see below).

One view of backup jobs across accounts

AWS Backup cross-account monitoring, turned on in the management account, shows backup, copy and restore jobs from every member account in one place, viewed from the management account or a delegated administrator. It is the built-in answer to "show me failed jobs everywhere" without signing in to each account or building a log pipeline.

You prove compliance with AWS Backup Audit Manager and restorability with AWS Backup restore testing. Having a recovery point is not evidence that it can be restored, or restored fast enough.

AWS Backup Audit Manager

Backup Audit Manager evaluates frameworks made of controls, for example: resources are protected by a backup plan, the plan meets a minimum frequency and retention, recovery points are encrypted, recovery points are copied to another account or Region, vaults are locked, and restore time meets target (restore testing jobs finish within a stated time). Controls can be scoped by resource type or tag. Report plans deliver daily reports (job reports and compliance reports) to an Amazon S3 bucket. It relies on AWS Config, so AWS Config resource tracking must be turned on for the backup resource types. It detects and reports; it does not change backup plans.

Do not confuse it with AWS Audit Manager, a separate service that is in maintenance mode and closed to new accounts and new Regions. For a new organization, backup evidence comes from Backup Audit Manager, and general framework checks from AWS Config conformance packs.

Restore testing

A restore testing plan runs scheduled restores of selected recovery points (for example the latest, or a random one within a time window), selected by resource type, ARN or tag. AWS Backup records each restore job's duration, keeps the restored resource for a configurable validation window, and then deletes it, so test databases are not left running. Combined with the restore-time control in a Backup Audit Manager framework and a scheduled report, this produces automatic evidence that restores work and finish within the target.

Validating the data, not only the restore job

A completed restore job proves the resource came back, not that it holds the right data. For data-level checks, create an Amazon EventBridge rule that matches restore job state changes to COMPLETED for the testing plan, and invoke an AWS Lambda function that queries the restored resource (expected tables, row counts, file checksums) and calls PutRestoreValidationResult with a successful or failed status. The result is recorded on the restore job in AWS Backup, inside the validation window.

AWS Backup Vault Lock makes recovery points in a vault write-once-read-many (WORM): nobody can delete them before their retention ends or change their lifecycle. The mode decides whether the lock itself can be removed.

The grace time

In compliance mode, the lock can still be deleted or replaced with corrected values (by running the command again) until the grace time ends. After that it is immutable: it cannot be removed or edited by the root user, by AWS or by AWS Support, and running the command again does not replace it. Mistakes must therefore be caught inside the grace time; test settings in governance mode first.

Minimum and maximum retention

MinRetentionDays and MaxRetentionDays act on new backup and copy jobs. A job whose lifecycle falls outside the range fails; AWS Backup does not raise or lower the retention to fit. Recovery points already in the vault keep their own lifecycle: a lock with a 35-day minimum neither extends existing 7-day recovery points nor shortens existing ten-year ones under a lower maximum. After locking, fix the backup rules so their retention falls inside the range.

Why policies alone are not WORM

A vault access policy that denies backup:DeleteRecoveryPoint can be edited by an administrator, and an SCP can be detached and never applies to the management account or the root user of a standalone account. Neither protects against someone with full control of the account; compliance-mode Vault Lock does.

A logically air-gapped vault is an AWS Backup vault type that is always locked in compliance mode, is encrypted with an AWS owned key or a customer managed key, and can be shared with other accounts through AWS Resource Access Manager (AWS RAM) so that those accounts restore from it directly. It is designed for ransomware recovery into a clean account.

Why it lowers recovery time

With a compliance-locked standard vault in a backup account, recovery into a clean account takes two steps: copy recovery points into the recovery account, then restore. Copying large backups adds hours. A logically air-gapped vault shared with the recovery account through AWS RAM skips the copy: the recovery account starts restore jobs straight from the shared vault. Standard vaults cannot be shared through AWS RAM for restores, and a governance-locked vault is not immutable against an administrator.

Recovering when the owning account is lost

If the account that owns the air-gapped vault is compromised, locked out or closed, you can still reach the backups through multi-party approval: associate the vault with an approval team of named approvers, and when enough of them approve, a recovery account is granted access to the vault. This avoids keeping a second full copy of the backups and ensures no single person can grant access.

Backing up straight to an air-gapped vault

A backup plan or Organizations backup policy can target an air-gapped vault directly (in a backup policy, target_logically_air_gapped_backup_vault_arn). Resource types fully managed by AWS Backup, such as Amazon S3 and Amazon EFS, back up straight into it. Other types, such as Amazon EBS, are first created as a temporary recovery point in the standard vault and then copied across; that temporary recovery point is billed until it is removed. So a standard vault must still exist, and briefly holds those backups. As with any cross-account copy, resource types that keep their source encryption need a customer managed key rather than an AWS managed key.

S3 Object Lock stores object versions in WORM form: a locked version cannot be deleted or overwritten until its retention expires or while a legal hold is on it. It requires S3 Versioning, and it can be enabled on new or existing buckets.

A legal hold is independent of retention: a version can have both, and it stays protected while either applies. Holds are placed on individual object versions; there is no bucket-level legal hold. Removing the hold returns the version to its normal retention.

Object Lock protects versions, not keys

A DELETE request that does not name a version ID succeeds even on a locked object: S3 just adds a delete marker, and the locked version stays retrievable by its version ID. A request that names the locked version ID fails until the retain-until date passes (or, in governance mode, unless the caller bypasses it). Lifecycle expiration rules cannot remove a locked version either; they act only after the lock expires. MFA delete is a different control: it requires an MFA code from the bucket owner's root user to permanently delete a version or change the bucket's versioning state, and only the root user can turn it on. It is not WORM retention.

Default retention is forward-only

A bucket's default retention applies to object versions written after it is set. Versions already in the bucket stay unprotected. To protect them without creating new versions, run an S3 Batch Operations job with the Object Lock retention operation (or the legal hold operation) over an inventory or manifest of the existing versions. Copying each object onto itself would also apply default retention, but it creates new versions and leaves the originals unprotected.

Amazon EBS snapshot lock prevents a locked snapshot from being deleted until the lock expires; Recycle Bin lets you recover a snapshot after it was deleted. One prevents deletion, the other undoes it, and ransomware-resistant designs often use both.

Snapshot lock modes

• Governance mode: users with the right IAM permissions (for example ec2:UnlockSnapshot) can unlock the snapshot or change the lock's mode and duration at any time.
• Compliance mode: you can add an optional cooling-off period of 1 to 72 hours. During it, authorized users can still unlock the snapshot or change the lock, which gives a window to correct a mistaken lock. After the cooling-off period (or immediately, if none was set) nobody can unlock it, shorten it or change its mode, regardless of IAM permissions; the only change allowed is extending the lock duration.

The lock has a duration (1 to 36,500 days) or an expiry date. A compliance-mode lock past its cooling-off period means you keep, and pay for, the snapshot until the lock expires; the only way to remove it earlier is to close the AWS account. Try lock settings in governance mode before using compliance mode. When protection must apply with no window to undo it, use compliance mode without a cooling-off period.

Recycle Bin

A Recycle Bin retention rule for EBS snapshots (and EBS-backed AMIs) keeps deleted snapshots for a retention period from 1 day to 1 year, scoped to the Region or to tagged resources. Within that period a deleted snapshot can be restored. Retention rules can themselves be locked so an attacker cannot simply delete the rule first.

An Amazon Data Lifecycle Manager retention setting deletes snapshots on schedule rather than protecting them, and the snapshot archive tier only lowers storage cost; neither prevents deletion.

A resilient architecture has redundancy (no single copy of anything critical), fault tolerance (it keeps serving through a component loss) and self-healing (it replaces what failed without a person). On AWS that means health checks that drive the right replacement, and capacity and dependencies provisioned in every Availability Zone ahead of time.

Self-healing compute

A self-healing fleet detects a failed component and replaces or recovers it without a person. On EC2 that means the right health checks driving Auto Scaling replacement, or EC2 automatic recovery for an instance that cannot be replaced.

Give the Auto Scaling group ELB health checks

By default an Auto Scaling group uses only EC2 status checks, which pass as long as the instance is running. If the application hangs, the load balancer marks the target unhealthy and stops routing to it, but the group never replaces it. Turning on Elastic Load Balancing health checks for the group makes it terminate and replace instances that fail the target group's health check.

Shallow versus deep health checks

The health check that drives replacement should be shallow: it tests only what a new instance would fix, such as the local process responding. A deep check that also calls the database or a third-party API makes every instance fail at once when that dependency slows, and the group then replaces the entire healthy fleet, turning a partial outage into a full one. Keep the target group health check local, and monitor dependencies with separate CloudWatch alarms that alert or degrade gracefully. When a group uses ELB health checks with several target groups, an instance must pass every one of them, so a deep check anywhere still drives replacement.

Grace period, warm-up and lifecycle hooks

The health check grace period is how long after an instance enters InService the group ignores failed health checks. A fixed grace period fits poorly when start-up time varies: too short and slow instances are killed in a loop, too long and a genuinely hung instance stays in service for that long. For variable warm-up, add a launch lifecycle hook: the instance waits in Pending:Wait, receives no traffic, and calls CompleteLifecycleAction when it is ready; the grace period can then be close to zero. Default instance warmup controls when an instance's metrics count towards scaling decisions, not health checks.

EC2 automatic recovery for a single, irreplaceable instance

When an instance cannot be replaced, for example a licence bound to its instance ID or clients that use its private IP address, use EC2 automatic instance recovery. When a system status check fails (underlying host problem), the instance is recovered onto new hardware with the same instance ID, private IP addresses, Elastic IP addresses, EBS volumes and metadata. Supported instance types have simplified automatic recovery by default, and you can also use a CloudWatch alarm with the recover action. It does not help with an instance status check (software) failure. An Auto Scaling group with a size of one replaces the instance with a new one, so the instance ID and private IP change.

Static stability: survive a zone loss without changes

A system is statically stable when it keeps working through the loss of a component, such as an Availability Zone, without making any changes: no instance launches, no scaling, no re-creation of resources. Control-plane actions (launching instances, starting stopped ones, creating NAT gateways) are the ones most likely to be slow or throttled during a large event, so a statically stable design does not depend on them.

Sizing capacity for the loss of one zone

Spread the fleet evenly over N zones and provision enough that the remaining N−1 zones carry peak load alone. Each zone needs peak ÷ (N−1) capacity, so the total is peak × N ÷ (N−1). More zones means less idle headroom:

Scaling out after a zone fails, or starting instances from a warm pool, still depends on launching or starting instances during the event, so it does not meet a static-stability requirement.

Every zone needs its own copy of each dependency

Capacity in each zone is useless if all zones route through something in one zone. A single NAT gateway in one zone is the classic example: lose that zone and every private subnet loses internet egress. Use a zonal NAT gateway in each zone with each zone's private route table pointing at its own gateway (or, when instances already run in every zone, a regional NAT gateway, which AWS manages across zones; it can take time to expand into a zone where new resources appear). An alarm that recreates the gateway after a failure is a recovery action, not static stability.

Databases

For Amazon RDS, a Multi-AZ deployment keeps a synchronous standby running in another zone and fails over automatically, typically within a minute or two, with no new database created during the event. Restoring a backup into another zone or promoting a read replica by hand both create or change resources during the outage and take longer. Cross-Region DR design against RTO and RPO belongs to Task 4.2.

Chaos engineering tests a stated expectation about how a system behaves under failure by injecting a controlled fault and observing a business metric. AWS Fault Injection Service (AWS FIS) runs these experiments from experiment templates.

The method

1. Define steady state with a business metric customers feel, such as successful orders per minute or checkout success rate, and measure its baseline before injecting faults. CPU or instance counts are not steady state.
2. State a hypothesis: "if one zone loses power, successful orders per minute stay within the normal band."
3. Start with a small blast radius, then widen it.
4. Run, observe, and fix what the experiment reveals; repeat regularly (game days).

FIS does not evaluate the hypothesis for you; you compare the metric with its baseline. FIS also does not roll back damage: there is no "undo" action for terminated instances.

What an experiment template contains

• Actions, such as stopping or terminating EC2 instances, CPU stress, network latency on ECS tasks, or scenarios such as AZ Availability: Power Interruption.
• Targets: a resource type selected by ARNs, tags and filters, plus a selection mode that limits the blast radius: ALL, COUNT(n) (a fixed number) or PERCENT(n) (a share, whatever the group's size).
• Stop conditions: CloudWatch alarms.
• Experiment role (roleArn): FIS performs every action with this IAM role. If an action fails with an authorization error, add the permission to this role, not to the person who started the experiment, the targets' instance profile, or the service-linked role.
• Optional logging and an experiment report configuration.

Stop conditions and the safety lever

A stop condition is a CloudWatch alarm, ideally on the steady-state metric. If it enters the ALARM state during the experiment, FIS stops running actions and ends the experiment in the stopped state. A stopped experiment never resumes, even when the alarm returns to OK, and completed actions are not reversed (terminated instances stay terminated; the Auto Scaling group launches replacements). An EventBridge rule or Route 53 health check cannot be a stop condition, and wiring an alarm to Lambda to call StopExperiment just rebuilds what stop conditions do.

For an incident, the FIS safety lever is a single control per account and Region: engaging it stops every running experiment and prevents new ones, including scheduled ones, from starting until it is disengaged.

Multi-account experiments

To disrupt resources in several accounts with one experiment, run it from an orchestrator account with account targeting set to multi-account, and add a target account configuration per workload account, each with that account's ID and an IAM role FIS assumes there. Identify Availability Zones by AZ ID (for example use1-az4), because AZ names such as us-east-1a map to different physical zones in different accounts.

You design a failover test by triggering the same path a real outage uses, then measuring how long service takes to recover against the requirement. Two tools do most of the work: AWS FIS for injecting the fault and recording evidence, and Route 53 health check settings that determine DNS failover time.

FIS experiment reports

An experiment report configuration in the template makes FIS produce a report (PDF, stored in S3) for each run: what the experiment did, its timeline, and graphs from a named CloudWatch dashboard covering a pre-experiment window, the experiment and a post-experiment window. With latency, error rate and healthy host count on the dashboard, the report shows when recovery happened, which gives auditors evidence without manual work. Logs show actions, not metrics, and a Resilience Hub assessment estimates recovery instead of measuring it.

Route 53 failover timing

Approximate client failover time = detection time + DNS TTL, where detection time is roughly the request interval × failure threshold.

• Request interval: 30 seconds (standard) or 10 seconds (fast). It is set when the health check is created and cannot be changed; create a new health check to change it.
• Failure threshold: 1 to 10 consecutive failures before the endpoint is unhealthy; this can be edited.
• TTL on non-alias records: resolvers keep serving the old answer until it expires. Alias records use the target's TTL.

Worst-case failover time is roughly interval × threshold + TTL. Example: a 30 s interval with a threshold of 5 plus a 120 s TTL is 150 + 120 = 270 s, about four and a half minutes; a 10 s interval with a threshold of 3 plus a 30 s TTL is about one minute. Only those three values move the number: settings that change where checks run from or what Route 53 graphs leave detection time as it was.

Forcing a failover for a test

To test the real health-check-driven path without touching the primary application, invert the primary health check so Route 53 treats it as unhealthy, time the failover, then clear the inversion. Disabling a health check does the opposite of what testers expect: Route 53 then considers it always healthy, so no failover happens.

Health checks that reflect what matters

HTTP and HTTPS health checks can use string matching: the endpoint is healthy only if a given string appears in the response body. Route 53 searches only the first 5,120 bytes of the body, so a marker in the footer of a large page is never found. Return a small status response with the marker near the start. For multi-Region failover design with Amazon Application Recovery Controller and Global Accelerator, see Task 4.2.