Responding to AWS Security Events: Compromised EC2, Credentials, and S3

Live response starts with validation, not action. A GuardDuty finding, a Security Hub alert or an AWS Health notification is a claim about your environment, and Task 2.2 tests whether you can validate findings from AWS security services to assess the scope and impact of an event before you contain anything. Confirm the resource exists and matters (tags give the owner and data classification), read the finding's details — the API calls, the remote IP, the credential involved — and pivot into Amazon Detective or CloudTrail to see whether the activity is isolated or one thread of something larger. Containing one instance while the attacker still holds the instance role's credentials achieves nothing.

Benign findings are validated, then narrowly suppressed. When the investigation proves a recurring finding is expected behaviour — an approved vulnerability scanner, a sanctioned penetration test — record the evidence and create a GuardDuty suppression rule whose filter names the finding type and the validated resource (instance ID, IP address or principal). Suppressed findings are still generated and still visible in the archive; they simply stop alerting. A filter on the finding type alone would silence the same behaviour from a real attacker on every other resource, and disabling the detector or the data source removes coverage entirely.

Impact for data exposure is an intersection. Scope tells you what was reachable; impact is what was sensitive and actually read. Amazon Macie sensitive data discovery tells you which objects hold personal or regulated data; S3 server access logs or CloudTrail data events tell you which objects were retrieved, by whom, during the exposure window. The objects that appear in both sets drive notification decisions. Neither source alone answers the question.

Then hold the three response stages apart, because answer options blur them. Containment stops the spread while preserving evidence: isolate the instance, deactivate the key, block the public access. Eradication removes the threat: delete malware and backdoor principals, terminate the compromised host. Recovery restores service from known-good sources: clean AMIs, recovery points from before the intrusion, rotated secrets. Eradication never precedes evidence capture, and recovery never reuses artifacts you have not verified as clean.

Isolating a compromised instance means cutting its network paths and its ties to automation while leaving it running and intact. Terminating it first destroys memory, instance store data and, with delete-on-termination set, the EBS volumes too.

Security groups and connection tracking. The standard per-instance move is to replace all of the instance's security groups with a dedicated isolation security group that has no inbound and no outbound rules. Security groups are stateful, so the result depends on whether each existing flow is tracked:

• A TCP or UDP flow is untracked only when a rule allows it from all addresses (0.0.0.0/0 or ::/0) and the other direction has a rule allowing all addresses on all ports (0–65535). ICMP is always tracked, and flows through NAT gateways, Network Load Balancers, Network Firewall endpoints, interface endpoints and similar paths are automatically tracked. An untracked flow is evaluated packet by packet against the rules, so it is dropped immediately when the rule that allowed it is removed or changed.
• Every other flow is tracked, and a tracked connection is not interrupted when you change or remove the rules that allowed it. It keeps working until it ends or idles out. An attacker's interactive session that arrived through a narrow inbound rule therefore survives the switch to an empty group.

The documented way to cut tracked connections with security groups alone is two steps: first move the instance to a temporary group that allows all traffic from and to 0.0.0.0/0 on all ports, which turns its flows untracked, then move it to the empty isolation group, which drops those now-untracked flows at once.

Idle timeouts do not cut active sessions. The idle connection tracking timeout for established TCP connections can be set per network interface, from 60 seconds up to 432,000 seconds (5 days); the default is 5 days on most instance types and much shorter on the newest Nitro generation. It only expires connections that are idle — a session that keeps exchanging packets is never timed out.

Network ACLs cut established sessions deterministically. Network ACLs are stateless and evaluate every packet, so a deny breaks an existing connection immediately. Rules are evaluated in ascending number order and the first match wins, so a deny must be numbered lower than any allow that would match the same traffic. Weigh each option by its blast radius:

Auto Scaling and load balancers. An Auto Scaling group replaces instances that fail health checks by terminating them, and it does so regardless of EC2 API termination protection — so the instance must leave the group's control before anything else touches it. Two actions do that correctly:

• Detach removes the instance from the group (optionally launching a replacement) and deregisters it from the group's load balancer target groups.
• Standby keeps it in the group but stops health checks on it and deregisters it from the target groups, so no user traffic reaches it and the group does not replace it as unhealthy.

Two look-alikes do not do the job. Instance scale-in protection only stops termination during scale-in; an unhealthy instance is still replaced. Suspending the ReplaceUnhealthy process does keep the instance, but for the whole group, so every genuinely failed instance stops being replaced too — and neither action removes the instance from the load balancer. After detaching or moving to standby, enable termination protection (DisableApiTermination) against accidental or malicious terminate calls.

With the instance contained, capture evidence in order of volatility — memory and local disks before anything durable — and only then destroy anything.

Snapshot before you touch. Create EBS snapshots of every attached volume while the instance keeps running, before any scan, patch, reboot, stop or terminate. A snapshot is a point-in-time, block-level copy; taken now, it preserves the disk as the attacker left it. Every action you run on the live system afterwards alters state, so the snapshot is your defensible baseline. Multi-volume snapshots (CreateSnapshots) capture all attached volumes at the same instant.

Capture volatile memory on the running instance. RAM survives neither stop nor reboot. Memory acquisition means running a capture tool on the instance — ideally pre-staged in the AMI, or pushed with Systems Manager — and writing the image to evidence storage; the Automated Forensics Orchestrator for Amazon EC2 automates this capture-and-store flow. Two shortcuts fail:

• Hibernation writes RAM to the encrypted EBS root volume, but it must be enabled when the instance is launched and it is off by default, so it is rarely available during an incident and cannot be turned on afterwards.
• Creating an AMI (CreateImage) reboots the instance by default, which clears memory, unless you choose the no-reboot option. An AMI captures EBS volumes only.

Instance store data needs its own copy. EBS snapshots and AMIs capture EBS volumes only. Instance store volumes are erased when the instance stops, hibernates or terminates, so anything an attacker staged there must be copied while the instance runs — for example with a disk-imaging tool run through Systems Manager that writes an image of the instance store device to an attached EBS volume, which you then snapshot.

Cut its identity. The instance profile's temporary credentials may already be exfiltrated — GuardDuty's UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS finding exists for exactly this. Revoke the role's active sessions with the aws:TokenIssueTime deny pattern described in the credentials section.

Tag, record and hand off. Tag the instance and snapshots as evidence under investigation, record who did what and when, and move analysis to the forensics account. Only after capture is complete do you eradicate — terminate the instance — and recover with a replacement built from a known-clean, patched AMI (see the recovery section), never by reusing the compromised host or an image taken from it.

Non-EC2 resources are contained with the control that stops them executing or communicating while keeping their state for investigation — not by deleting them.

Lambda. Setting a function's reserved concurrency to 0 throttles every invocation, whatever the source — API Gateway, event source mappings, direct Invoke calls — without deleting anything. Provisioned concurrency is a performance setting, not a limit, so changing it does not stop invocations. Before you replace the malicious code, preserve it: publish a version or download the deployment package (GetFunction returns a presigned link), and record the function's configuration, environment variables and execution role. Then revoke the role's sessions if the code could have used them, redeploy known-good code, and restore concurrency.

EKS pods. The aim is to stop the one compromised pod communicating while the rest of the Deployment keeps serving:

• Give the pod a unique label and apply a deny-all Kubernetes network policy (no ingress, no egress) whose selector matches only that label; this needs network policy enforcement enabled in the cluster (for example the VPC CNI's network policy support). A namespace-wide deny would also cut the healthy replicas.
• Cordon the worker node so no new pods are scheduled there. Do not drain it, and do not apply a NoExecute taint: both evict running pods — a NoExecute taint evicts every pod that lacks a matching toleration — destroying the evidence on the compromised one.
• Do not delete the pod or scale the Deployment to zero before capture; the container's processes, memory and file system go with it. Capture what you need (container file system, node volume snapshots, memory where tooling allows), then eradicate.

Databases are covered in the recovery section: a compromised data store is usually recovered by restoring to a new instance from a point before the malicious change.

Credential incidents split on one axis: long-term credentials can be deactivated; temporary credentials cannot, and must be denied.

Long-term IAM user access keys — leaked in a repository, on a stolen laptop, on a paste site — are contained by deactivating the key immediately. Deactivation is instant and reversible, and it keeps the key ID you will pivot on in CloudTrail; delete the key only after scoping. When AWS detects a publicly exposed key it opens an AWS Health event and can attach the AWSCompromisedKeyQuarantineV3 managed policy to the user. That policy is a deny on a list of high-risk actions — IAM changes such as creating keys or users and attaching policies, minting new sessions with GetSessionToken or GetFederationToken, launching compute, and some S3 actions — so the key can still call anything the user is allowed that is not on the list. It is partial containment and a signal to run the playbook, not remediation.

Temporary credentials — a compromised role, stolen STS session tokens, exfiltrated instance-profile credentials — cannot be recalled. Attach an inline deny to the role: "Effect": "Deny", "Action": "*", "Condition": {"DateLessThan": {"aws:TokenIssueTime": "<now>"}}. This is what the console's Revoke active sessions does. What it does and does not reach:

• Every session issued before the cutoff loses all permissions at once — including the role's legitimate users and workloads, which see AccessDenied until they obtain credentials issued after the cutoff. SDKs pick up new credentials on their own, but for instance-profile credentials that may not happen until the next automatic refresh, so plan for the disruption.
• It does nothing to long-term access keys: requests signed with them carry no token issue time, so the condition never matches.
• Removing the role from the instance profile or lowering the role's maximum session duration affects only future credentials; sessions already issued keep working until they expire.
• To stop instance-role credentials working off the instance in future, policies can require the credential to be used from its own VPC and address with the aws:EC2InstanceSourceVPC and aws:EC2InstanceSourcePrivateIPv4 condition keys.

Sessions minted from a long-term key outlive it. An attacker holding a user's key can call GetSessionToken or GetFederationToken to obtain temporary credentials (ASIA… key IDs, lasting up to 36 hours). Deactivating or deleting the original key does not end those sessions, and the quarantine policy only stops new ones being minted. Their permissions are bounded by the IAM user's policies, though, so an aws:TokenIssueTime deny attached to the user ends them — and, because long-term keys carry no issue time, a newly issued key for the legitimate owner keeps working.

Tracing an attacker through a shared role. After AssumeRole, CloudTrail records each request under the role session, not the original user. The link is in the AssumeRole event itself: its responseElements include the temporary access key ID that was issued, and that same ID appears in userIdentity.accessKeyId on every request the session made. Filtering on it separates the attacker's activity from other callers of the same role.

Hunt and classify. Query CloudTrail for everything the credential did, then sort calls by their effect:

A console password added to a user that never had one (CreateLoginProfile) explains console sign-ins after the keys were deactivated. Finally, rotate every secret the credential could read, because scoping tells you what was reachable, not what was copied.

An S3 exposure — a bucket made public by a policy change, public ACLs, or access granted to an outside account — has a short containment step and a long analysis tail.

Block Public Access. S3 Block Public Access can be set on the account, the bucket and access points, and S3 applies the most restrictive combination of them. Of its four settings, two act on existing exposure: IgnorePublicAcls makes S3 ignore public ACLs on the bucket and its objects, and RestrictPublicBuckets limits a bucket with a public policy to AWS service principals and authorized users in the owner's account. (BlockPublicAcls and BlockPublicPolicy reject new public grants.) Turning it on removes public access without rewriting a single object or policy, so the evidence of how the exposure happened stays intact.

Non-public, cross-account grants are a different problem. Block Public Access acts only on grants that are public. A policy statement or ACL that grants one named outside account is not public, so it keeps working. Remove those grants at the source:

• Delete the bucket policy statement (record it first — it is evidence).
• Set Object Ownership to bucket owner enforced, which disables ACLs: bucket and object ACLs stop affecting access at once, without rewriting any object. If the bucket ACL itself grants access to another account, S3 rejects the setting until that bucket ACL is reset to private — record it first, as evidence. Bucket owner preferred leaves ACLs enabled, so existing object grants still work.
• A deny on s3:PutObjectAcl stops only new ACL grants; existing ones still apply.

Analysis: what was actually read? Preparation decides what is knowable:

When data events cover only some prefixes, server access logs are the complete record for the rest of the bucket. Query either source with Athena to list what was retrieved, by whom and from where during the exposure window, then intersect with Macie's sensitive-data results. If neither was enabled, you have to treat the whole bucket as exposed.

Recovery. Rotate credentials or secrets stored in exposed objects, notify per your obligations, and feed the root cause — how the bucket became accessible — into preventive controls.

Analysis happens in a dedicated forensics account, not in the compromised one — the attacker's credentials have no reach there, tooling cannot contaminate production, and access is limited to the forensics team.

Moving the evidence. Share the EBS snapshot with the forensics account through its create-volume permissions, copy it inside the forensics account (an independent copy the source account cannot delete), create a volume from the copy, and attach it to an isolated analysis instance as a secondary, non-boot device, so the evidence's own operating system never runs and the original stays untouched.

Encryption decides whether sharing works. Snapshots encrypted with the AWS managed key aws/ebs cannot be shared, because its key policy cannot be changed to trust another account. Copy the snapshot, re-encrypting it to a customer managed key, and share that copy. Cross-account use of a customer managed key needs both sides:

• the key policy in the owning account must allow the forensics account (for example kms:Decrypt, kms:DescribeKey, kms:CreateGrant, kms:ReEncrypt*), and
• an IAM policy in the forensics account must allow the analyst role those same KMS actions on that key ARN.

Either side alone fails. A grant cannot be created by an account the key policy does not trust, and KMS keys are not shared through AWS RAM. Prepared teams standardize on customer managed keys for exactly this reason.

Chain of custody. Record who captured each artifact, when, from which resource, and every later access; hash images when you capture them and again before analysis; work on copies, never originals. Your CloudTrail record of the snapshot, share and copy calls is itself part of the custody record.

Forensic artifacts — disk images, memory captures, exported logs — are stored as copies in the forensics or log-archive account, in storage that cannot be altered, with proof that they were not.

S3 Object Lock stores objects write-once-read-many. It is enabled on the bucket (with versioning) and applied per object version:

A bucket's default retention applies only to objects written after it is set; existing objects keep their own settings. A legal hold is independent of retention, so it suits preservation with no known end date — a lawsuit or a regulator's inquiry — on top of whatever retention the object already has.

CloudTrail log file integrity validation proves log files were not changed or deleted after delivery. With it on, CloudTrail delivers a digest file every hour that holds the SHA-256 hash of each log file delivered in that hour, signed with CloudTrail's private key and chained to the previous digest. Running aws cloudtrail validate-logs checks the hashes and signatures and reports files that were modified, deleted or missing. Trails created in the console have validation on by default; it can be turned on for other trails at any time, but it only covers files delivered after it was enabled. S3 ETags, server access logs and CloudTrail event history are not proof of integrity: ETags are not signed and change with the object, and the other two show activity, not content. Store copies of logs, digests and hashes of disk images together under Object Lock.

Building an incident timeline means querying several log sources over the same window and joining them on shared values: time, source IP address, access key ID or role session name, instance ID and request ID. Pick the tool by where the logs live and whether the data already exists.

A typical correlation: an ALB log shows a suspicious request from an address; VPC Flow Logs show the instance then connecting outbound to another address; CloudTrail shows the instance role's session (same time window, the instance ID in the session name) calling APIs from that external address. Ordering every event by time builds the minute-by-minute account of the attack that root cause analysis needs.

Amazon Detective supports root cause analysis by building a behavior graph: a linked model of your accounts' IAM principals, IP addresses, EC2 instances, S3 buckets, EKS objects and findings, built from data it ingests itself.

• Entity profiles show one entity's history — API calls, IP addresses used, geolocations, volumes of traffic — against its own baseline, so you can see when behaviour changed.
• Finding groups correlate related findings and entities into one view of a probable attack chain, so you investigate the campaign rather than findings one by one.
• Source data: the core sources are CloudTrail management events, VPC Flow Logs (for EC2) and GuardDuty findings. Optional packages add EKS audit logs and AWS security findings from Security Hub CSPM. The EKS audit log package was on by default only for behavior graphs created after 26 July 2022; an older graph must have it turned on.

Three limits shape what Detective can answer. It consumes independent copies of these streams — it does not read your trails or flow log destinations, and you do not need to configure them for it. It keeps up to a year of history, but only from the day it was enabled: it does not backfill, so for earlier periods you query your retained CloudTrail logs in S3 with Athena. And it does not ingest DNS (Route 53 Resolver) logs or S3 object-level data events, so those questions go to the log sources in the previous section.

Root cause analysis walks backwards from the impact to the first cause — not just "an instance was compromised" but the chain: an unpatched package, a security group open to the internet, an instance role with too much access. Each link becomes a corrective action.

Recovery restores service from sources known to be clean: a patched image and data from a recovery point taken before the intrusion began.

• Compute: launch replacements from a known-good, patched AMI or a rebuilt container image, never from the compromised host or an image of it.
• Data: the investigation timeline tells you when the intrusion started. Restore from the most recent recovery point (AWS Backup, EBS snapshots) taken before that time, onto the clean image. A later point may carry the attacker's changes; a much older one is clean but loses more legitimate data. That data-loss trade-off is a business decision informed by the timeline.
• Databases: Amazon RDS point-in-time recovery always creates a new DB instance; it never rewinds the existing one. Choose the time just before the malicious change shown in the timeline — not the latest restorable time, which includes the change, and not an older snapshot that loses more data — then point the application at the new instance and keep the original for evidence.

Before returning to service, rotate the secrets and credentials the attacker could reach and close the root cause, or the restored system is simply compromised again.

Automated containment and the working relationship with AWS are designed in advance, as part of the plan (Task 2.1, covered in the lesson Design and Test an AWS Incident Response Plan). During a live event they matter in three ways.

• Automation is an actor in your timeline. GuardDuty findings routed through EventBridge to Lambda functions or Systems Manager Automation runbooks may already have isolated an instance, taken snapshots or revoked sessions. Check what ran before acting, and include those actions in the timeline and custody record.
• AWS notifications are findings too. AWS Health events (for example an exposed access key) and abuse notices from AWS enter the same triage as GuardDuty findings, and are validated the same way.
• Close the loop. After recovery, record the root cause, what was detected late, and which runbook steps failed; then fix the control, add the missing detection and update the playbook.