Designing Security Logging Solutions: CloudTrail, Flow Logs, Central Archives

Every logging design starts from the same inventory: know what each source records, where it can deliver, and — just as important on the exam — what it does not capture.

Map requirement language to a row: “who deleted the bucket” is CloudTrail management events; “which object was read” is S3 data events; “was the connection rejected” is flow logs; “which domain did the instance resolve” is Resolver query logs.

CloudTrail is the control-plane record and the backbone of every AWS investigation. Management events capture control-plane operations — RunInstances, PutBucketPolicy, AssumeRole, console sign-ins — and a trail records them by default. Data events capture data-plane operations — GetObject and PutObject on S3, Lambda Invoke, DynamoDB item-level actions — and are off by default because volume and cost are dramatically higher. Enable them selectively, scoped with selectors to the sensitive buckets or functions that justify the volume; advanced event selectors give fine-grained matching on fields like resource ARN. A design that enables all data events everywhere is usually the distractor.

Make trails multi-Region — the default when creating a trail in the console — so activity in a Region you never use (a favorite attacker hiding spot) is still recorded, including global service events like IAM calls.

At organization scale, use an organization trail: created from the management account (or a CloudTrail delegated administrator account), it automatically logs every existing and future member account to one central S3 bucket. Member accounts can see the trail but cannot modify, stop, or delete it — the property that makes it the answer whenever a scenario requires logging that individual account administrators cannot tamper with or opt out of. Delivery to S3 typically lands within about 15 minutes of the API call, which is why CloudTrail is an audit and investigation source, not a real-time alerting mechanism on its own — pair it with CloudWatch Logs delivery and metric filters when latency matters.

Network telemetry answers the questions CloudTrail cannot: what actually moved, between which addresses, and was it allowed. VPC Flow Logs record IP traffic metadata at the VPC, subnet, or ENI level. Each record carries the source and destination addresses and ports, protocol, byte and packet counts, and the action — ACCEPT or REJECT — telling you whether security groups or network ACLs permitted the flow. The default format is only the beginning: custom formats add fields that matter in security work, such as pkt-srcaddr and pkt-dstaddr (the original addresses behind a NAT gateway, without which you attribute traffic to the NAT rather than the real source), tcp-flags (SYN scans stand out), flow-direction, and traffic-path.

Know the limitations cold: flow logs capture metadata only, never packet payloads; they are not real time; and certain traffic is excluded, including traffic to the Amazon DNS resolver, instance metadata requests to 169.254.169.254, and DHCP. Transit gateway flow logs fill a related gap, recording flows crossing a transit gateway — the vantage point for inter-VPC and hybrid traffic that a single VPC's logs cannot show end to end.

DNS is its own detection channel. Route 53 Resolver query logging records the domains VPC resources look up and the responses returned — often the earliest indicator of malware calling home. Note the division of labor: GuardDuty already analyzes DNS query activity directly from AWS infrastructure whether or not you enable Resolver query logging; you enable query logging when you need the records — for your SIEM, for Athena hunts, or for retention.

Above the network layer, each tier contributes its own record. ELB access logs capture every request a load balancer handles — client IP, request line, latencies, backend status, and TLS cipher and protocol version, which matters when auditing for legacy TLS clients. They deliver to S3 only, and a detail worth remembering: the ALB access-log bucket must use SSE-S3 encryption, as SSE-KMS is not supported for that delivery path. CloudFront standard access logs record viewer requests at the edge into S3, while real-time logs stream selected fields to Kinesis Data Streams when seconds matter. AWS WAF logs record each inspected request, the rules it matched, and the action taken (allow, block, count) — the evidence base for tuning rules and investigating application-layer attacks; they deliver to CloudWatch Logs, S3, or Firehose, with the log group or bucket name required to begin with aws-waf-logs-.

Amazon EKS control-plane logging sends five log types to CloudWatch Logs — API server, audit, authenticator, controller manager, and scheduler. The audit log is the security-critical one: it records who did what against the Kubernetes API, the cluster-level equivalent of CloudTrail. Each type is enabled individually on the cluster.

RDS engine logs — error, audit, and slow-query logs, varying by engine — export to CloudWatch Logs for retention and search, and operating-system-level logs from EC2 instances reach CloudWatch Logs through the CloudWatch agent, which you configure to ship security-relevant files such as /var/log/secure. For the exam, know these exist and where they land; deep tuning belongs to the database and compute domains.

Centralization is the structural defense: logs that leave the account they were generated in cannot be destroyed by compromising that account. The standard design is a dedicated log archive account in the organization holding the S3 buckets that receive the organization trail, flow logs, and service logs, plus — where streaming is needed — a dedicated CloudWatch logging account receiving cross-account log flows. Producers write in; almost nobody reads; security tooling consumes through tightly scoped roles.

Choose the destination per log by how it will be used. S3 is the durable archive: cheapest at rest, lifecycle-managed, queryable in place with Athena — the default for anything kept for compliance. CloudWatch Logs is the operational store: near-real-time ingestion, metric filters that turn patterns into alarms, Logs Insights for interactive queries, and per-group retention settings. Many sources justifiably go to both — CloudTrail to S3 for the archive and to CloudWatch Logs for alerting.

To move logs onward, subscription filters stream matching events from a log group to Kinesis Data Streams, Amazon Data Firehose, or Lambda. The classic SIEM feed is a subscription filter into Firehose, which batches, optionally transforms via Lambda, and delivers to S3, OpenSearch Service, or an HTTP endpoint such as Splunk. Cross-account streaming works through a CloudWatch Logs destination resource in the receiving account, fronting its stream with a resource policy that permits sender accounts.

Amazon Security Lake is the managed alternative to building this pipeline yourself: it collects CloudTrail, flow logs, Resolver logs, EKS audit logs and more org-wide, normalizes them to OCSF in Parquet, and hands SIEMs the data through subscribers. Weigh it whenever the requirement is multi-source, multi-account normalization rather than a single feed.

A log archive is a target — an attacker covering tracks attacks the logs first — so the exam tests its protection in detail. Start with the service-delivery bucket policy: CloudTrail delivery requires statements allowing the cloudtrail.amazonaws.com service principal s3:GetBucketAcl on the bucket and s3:PutObject on the delivery prefix, conditioned on s3:x-amz-acl being bucket-owner-full-control. Harden delivery policies with aws:SourceArn or aws:SourceAccount conditions so only your trail can write — the mitigation for confused-deputy delivery from someone else's account.

Encrypt with SSE-KMS and understand the key-policy consequences, a repeat exam scenario: the key policy must allow CloudTrail to call kms:GenerateDataKey (scoped with an encryption-context condition on the trail), or delivery fails silently from the bucket's perspective; and analysts need kms:Decrypt or Athena queries return access errors even with full S3 read. The customer managed key should live in the log archive account, administered separately from log readers.

Enforce least privilege and deny-delete: an explicit bucket-policy Deny on s3:DeleteObject (and denying s3:PutBucketPolicy except for a break-glass role) means even a compromised administrator in the log account cannot purge evidence. Versioning preserves prior versions against overwrite. For true write-once-read-many, S3 Object Lock in compliance mode makes objects undeletable and unmodifiable for the retention period by anyone, including the root user — governance mode, bypassable with s3:BypassGovernanceRetention, is the weaker cousin. Object Lock requires versioning.

Finally, CloudTrail log file integrity validation proves the record itself: hourly digest files containing SHA-256 hashes of each delivered log file, signed by CloudTrail, chained so each digest references its predecessor. Running validate-logs detects any modified, deleted, or forged log file — the answer to “prove the logs weren't altered.”

Retention is a compliance decision implemented as lifecycle automation. In S3, lifecycle rules transition aging log objects to cheaper storage classes — Glacier Flexible Retrieval or Glacier Deep Archive for logs kept for multi-year mandates — and expire them when the retention period ends. Design the transition points around access patterns: recent logs get queried during investigations and belong in Standard or Intelligent-Tiering; year-old logs are almost never read and can tolerate retrieval delays. When Object Lock enforces retention, align the lock period with the lifecycle expiration so expiry doesn't silently fail against locked objects. In CloudWatch Logs, retention is per log group and defaults to never expire — an unbounded cost and data-minimization problem, so set explicit retention on every group, keeping the durable copy in S3.

Querying splits by where the logs live. Amazon Athena runs SQL directly over CloudTrail, flow logs, and ELB logs in S3 — define the table once and query years of history in place; partitioning (or partition projection) by date and Region keeps queries from scanning the whole archive. It is the tool for broad, historical questions: every action by an access key over six months, every source address that reached a port across the fleet. CloudWatch Logs Insights runs a purpose-built query language interactively over log groups — the tool for fast, recent, operational investigation where the data already streams to CloudWatch Logs. The selection rule the exam rewards: Athena for deep history in S3, Logs Insights for recent data in CloudWatch Logs; and when the requirement outgrows both into multi-source normalized analytics, that is Security Lake with Athena over OCSF tables.

A regulated company with 30 accounts must retain security logs for seven years, prove logs were never altered, guarantee no single account administrator can destroy them, and feed its SIEM in near real time. The design:

1. Centralize with an organization trail. From the management account, create a multi-Region organization trail with log file integrity validation enabled, delivering to a bucket in the dedicated log archive account. Every member account, present and future, is logged; members cannot stop or alter the trail.
2. Scope data events deliberately. Add advanced event selectors for S3 data events on the buckets holding regulated data — not on every bucket.
3. Add the network and service sources. Organization-wide VPC Flow Logs with a custom format including pkt-srcaddr and tcp-flags, Route 53 Resolver query logging, and WAF and ELB access logs, all delivering into the archive account.
4. Harden the buckets. Service-delivery policies conditioned on aws:SourceArn; SSE-KMS with a key in the archive account whose policy grants CloudTrail kms:GenerateDataKey and analysts kms:Decrypt; versioning plus Object Lock in compliance mode with seven-year retention; an explicit deny on s3:DeleteObject.
5. Lifecycle for cost. Transition objects to Glacier Deep Archive after 90 days; expire after seven years, aligned with the lock period.
6. Stream to the SIEM. Send CloudTrail to CloudWatch Logs in parallel and attach a subscription filter into Firehose delivering to the SIEM endpoint.

An auditor asking “prove these logs are complete and unmodified” gets a validate-logs run over the digest chain; an attacker who owns a member account finds nothing there to delete.