Designing Security Logging Solutions: CloudTrail, Flow Logs, Central Archives

Every logging design starts from the same inventory: know what each source records, where it can deliver, and — just as important on the exam — what it does not capture.

Map requirement language to a row: “who deleted the bucket” is CloudTrail management events; “which object was read” is S3 data events; “was the connection rejected” is flow logs; “which domain did the instance resolve” is Resolver query logs.

CloudTrail is the control-plane record and the backbone of every AWS investigation. Management events capture control-plane operations — RunInstances, PutBucketPolicy, AssumeRole, console sign-ins — and a trail records them by default. Data events capture data-plane operations — GetObject and PutObject on S3, Lambda Invoke, DynamoDB item-level actions — and are off by default because volume and cost are dramatically higher. Enable them selectively, scoped with selectors to the sensitive buckets or functions that justify the volume; advanced event selectors give fine-grained matching on fields like resource ARN. A design that enables all data events everywhere is usually the distractor.

Make trails multi-Region — the default when creating a trail in the console — so activity in a Region you never use (a favorite attacker hiding spot) is still recorded, including global service events like IAM calls.

At organization scale, use an organization trail: created from the management account (or a CloudTrail delegated administrator account), it automatically logs every existing and future member account to one central S3 bucket. Member accounts can see the trail but cannot modify, stop, or delete it — the property that makes it the answer whenever a scenario requires logging that individual account administrators cannot tamper with or opt out of. Delivery to S3 typically lands within about 15 minutes of the API call, which is why CloudTrail is an audit and investigation source, not a real-time alerting mechanism on its own — pair it with CloudWatch Logs delivery and metric filters when latency matters.

Know what CloudTrail event history is not. Every account gets a free, searchable view of the last 90 days of management events, per Region, with no trail at all — but it never contains data events, is not delivered to S3, and cannot be centralized or retained longer. A question about who read or deleted objects weeks ago is answered from the data events a trail delivered to S3 (queried with Athena), not from event history. CloudTrail Lake is the managed, SQL-queryable event store alternative, but AWS closed it to new customers on 31 May 2026: existing event data stores keep working (with critical fixes only), while new accounts cannot create one and AWS points them to CloudWatch for similar capabilities. Where it exists, an event data store records events from the time it is created unless you explicitly import existing trail logs into it.

Network telemetry answers the questions CloudTrail cannot: what actually moved, between which addresses, and was it allowed. VPC Flow Logs record IP traffic metadata at the VPC, subnet, or ENI level. Each record carries the source and destination addresses and ports, protocol, byte and packet counts, and the action — ACCEPT or REJECT — telling you whether security groups or network ACLs permitted the flow. The default format is only the beginning: custom formats add fields that matter in security work, such as pkt-srcaddr and pkt-dstaddr (the original packet addresses where an intermediate interface rewrites or relays traffic), tcp-flags (SYN scans stand out), flow-direction, and traffic-path. A flow log's format cannot be edited after creation — to add fields, you create a new flow log.

Reading flow logs behind a NAT gateway. Traffic from a private instance to the internet crosses a (zonal) NAT gateway's network interface as two legs. On the instance-to-NAT leg, srcaddr is the instance and dstaddr is the NAT gateway's private IP — but pkt-dstaddr carries the real internet destination. On the NAT-to-internet leg, the NAT gateway's private IP is the source and the internet host the destination. So in a default-format log, every record touching a malicious address shows only the NAT gateway; with pkt-srcaddr/pkt-dstaddr added, you match pkt-dstaddr to the bad IP and read the instance's address from srcaddr on the same leg. The NAT gateway's interface is requester-managed, so its records show - for instance-id; attribute traffic by address, or with flow logs on the instance's own ENI. Regional NAT gateways log differently — their records identify the gateway through resource-id, and the first record already shows the instance and the internet host in srcaddr/dstaddr.

Know the limitations cold: flow logs capture metadata only, never packet payloads; they are not real time; and certain traffic is excluded, including traffic to the Amazon DNS resolver, instance metadata requests to 169.254.169.254, and DHCP. When a requirement demands the actual packet contents — forensic capture, deep packet inspection, an IDS appliance — the feature is VPC Traffic Mirroring: a mirror session copies the traffic of a source ENI (on supported instance types — in practice Nitro-based, plus a few older families), optionally narrowed by a mirror filter, to a target — another ENI, a Network Load Balancer, or a Gateway Load Balancer endpoint fronting the inspection appliances. Mirrored packets travel VXLAN-encapsulated, so the appliance must decapsulate them. Traffic Mirroring is a capture pipe, not a log: nothing is stored unless the appliance stores it.

Transit gateway flow logs record flows crossing a transit gateway — the vantage point for inter-VPC and hybrid traffic that a single VPC's logs cannot show end to end. They are created on the transit gateway itself, by the account that owns it, and publish to S3, CloudWatch Logs or Firehose; nothing needs to be configured in the accounts that own the attached VPCs. Use them when the network account must see traffic between attachments; use VPC flow logs when you need ENI-level detail inside a VPC.

DNS is its own detection channel. Route 53 Resolver query logging records the domains VPC resources look up through the Route 53 Resolver and the responses returned — often the earliest indicator of malware calling home. It logs only queries the Resolver actually handles: queries from the VPCs in the logging configuration, queries arriving through inbound endpoints, and queries forwarded through outbound endpoints. If a DHCP options set points instances at self-managed DNS servers that resolve internet names themselves, those lookups never reach the Resolver and never appear; the fix is to return the VPC to AmazonProvidedDNS and use Resolver forwarding rules with an outbound endpoint for on-premises names. Repeat queries answered from the Resolver's cache within the TTL are also not logged. Note the division of labor with GuardDuty: GuardDuty analyzes DNS activity for queries that go through the Route 53 Resolver whether or not you enable query logging; you enable query logging when you need the records — for your SIEM, for Athena hunts, or for retention.

Above the network layer, each tier contributes its own record. ELB access logs capture every request a load balancer handles — client IP, request line, latencies, backend status, and TLS cipher and protocol version, which matters when auditing for legacy TLS clients. They deliver to S3 only, and a detail worth remembering: the ALB access-log bucket must use SSE-S3 encryption, as SSE-KMS is not supported for that delivery path. CloudFront standard access logs record viewer requests at the edge, while real-time logs stream selected fields to Kinesis Data Streams when seconds matter. AWS WAF logs record each inspected request, the rules it matched, and the action taken (allow, block, count) — the evidence base for tuning rules and investigating application-layer attacks. Logging is enabled on the web ACL; logs deliver to CloudWatch Logs, S3, or Firehose, with the log group, bucket or stream name required to begin with aws-waf-logs-. Sensitive request fields such as an Authorization header can be listed as redacted fields in the logging configuration, and logging filters can keep only the requests you care about, such as blocked ones.

Amazon EKS control-plane logging sends five log types to CloudWatch Logs — API server, audit, authenticator, controller manager, and scheduler — each enabled individually on the cluster and all off by default. The distinction the exam probes: CloudTrail records calls to the EKS API (creating clusters and node groups, updating configuration), not requests to the Kubernetes API. Who ran kubectl exec into a pod, created a role binding, or read a secret is in the audit log; the authenticator log records how IAM identities authenticated to the cluster, not what they then did.

RDS engine logs — error, audit, and slow-query logs, varying by engine — export to CloudWatch Logs for retention and search. Operating-system and application log files from EC2 instances (and on-premises servers) reach CloudWatch Logs through the CloudWatch agent: you list each file, such as /var/log/secure or an application audit log, in the collect_list of the agent configuration with its target log group. At fleet scale, install the agent with Systems Manager (the AmazonCloudWatchAgent package via Distributor or State Manager) and store one configuration in Parameter Store for every instance to fetch. Two prerequisites decide whether anything arrives: the instance role needs CloudWatch Logs write permissions — typically the CloudWatchAgentServerPolicy managed policy, which the Systems Manager core policy does not include — and the instance needs a network path to the CloudWatch Logs endpoint, which in a private subnet without a NAT gateway means an interface VPC endpoint for logs. Diagnosing an agent that is installed but silent is Task 1.3 territory, covered in the troubleshooting lesson.

Centralization is the structural defense: logs that leave the account they were generated in cannot be destroyed by compromising that account. The standard design is a dedicated log archive account in the organization holding the S3 buckets that receive the organization trail, flow logs, and service logs, plus — where streaming is needed — a dedicated CloudWatch logging account receiving cross-account log flows. Producers write in; almost nobody reads; security tooling consumes through tightly scoped roles.

Choose the destination per log by how it will be used. S3 is the durable archive: cheapest at rest, lifecycle-managed, queryable in place with Athena — the default for anything kept for compliance. CloudWatch Logs is the operational store: near-real-time ingestion, metric filters that turn patterns into alarms, Logs Insights for interactive queries, and per-group retention settings. Many sources justifiably go to both — CloudTrail to S3 for the archive and to CloudWatch Logs for alerting. Log groups come in two general-purpose classes chosen at creation and never changed (a third, Delivery class, exists only for vending logs such as Lambda logs on to S3 or Firehose): Standard, with every feature, and Infrequent Access, which costs less per GB ingested and still supports Logs Insights queries and export to S3, but not metric filters, subscription filters or Live Tail — so it suits after-the-fact forensic logs, never a group that must drive alarms or feed a SIEM.

Thresholds over time versus single events. An EventBridge rule matches individual events as they arrive — right for “alert on any StopLogging call” or “any root sign-in”. It cannot count. “More than ten AccessDenied errors within fifteen minutes” needs CloudTrail delivered to a CloudWatch Logs log group, a metric filter that emits a value for each matching event, and a CloudWatch alarm on the metric’s sum over the period. On an organization trail’s log group, a plain filter sums every account together; adding recipientAccountId as a metric dimension publishes a separate time series per account. A standard alarm watches one metric — one dimension value, so one account — so per-account alerting means either one alarm per account or a single Metrics Insights multi-time-series alarm (GROUP BY recipientAccountId), in which each account is a separate contributor with its own alarm state.

To move logs onward, subscription filters stream matching events from a log group to Kinesis Data Streams, Amazon Data Firehose, or Lambda (the console’s OpenSearch Service option is a Lambda function under the hood). The classic SIEM feed is a subscription filter into Firehose, which batches, optionally transforms via Lambda, and delivers to S3, OpenSearch Service, or an HTTP endpoint such as Splunk — fully managed, with no consumer to run. Cross-account streaming works through a CloudWatch Logs destination in the receiving account: the destination fronts the Kinesis stream or Firehose with an IAM role that CloudWatch Logs assumes to put records, and a destination access policy lists the sender accounts allowed to call logs:PutSubscriptionFilter against it. Each sender then creates a subscription filter whose target is the destination ARN — the senders never get write access to the stream itself.

Amazon Security Lake is a managed security data lake: it collects security logs and findings across an organization’s accounts and Regions into S3 buckets in your account, normalizes them to the Open Cybersecurity Schema Framework (OCSF) in Apache Parquet, manages the AWS Glue and Lake Formation catalog, and applies your retention rules. It is the managed alternative to the hand-built pipeline above whenever the requirement is multi-source, multi-account normalization rather than a single feed. You enable it from a delegated administrator account, and can roll several Regions up into one rollup Region.

Sources. Native AWS sources are switched on, not built: CloudTrail management and data events, VPC Flow Logs, Route 53 Resolver query logs, EKS audit logs, AWS WAF logs, and Security Hub CSPM findings (the original Security Hub, renamed when AWS launched the new Security Hub). One caveat: CloudTrail management events require a multi-Region CloudTrail trail that records read and write management events to already exist; the other native sources need nothing configured beforehand. Anything else — an on-premises firewall, a SaaS identity provider — arrives as a custom source: you register it, Security Lake provisions its location and catalog entry, and the producer writes OCSF-conformant Parquet objects there. Security Lake does not convert arbitrary formats for custom sources; normalization to OCSF is the producer’s job.

Subscribers. Consumers get access per source through subscribers, and the two types differ in whether the consumer copies the data:

Security Lake complements, rather than replaces, the organization trail: auditors who require native CloudTrail log files with integrity validation still get them from the trail’s own bucket.

A log archive is a target — an attacker covering tracks attacks the logs first — so the exam tests its protection in detail. Start with the service-delivery bucket policy: CloudTrail delivery requires statements allowing the cloudtrail.amazonaws.com service principal s3:GetBucketAcl on the bucket and s3:PutObject on the delivery prefix, conditioned on s3:x-amz-acl being bucket-owner-full-control. Harden delivery policies with aws:SourceArn or aws:SourceAccount conditions so only your trail can write — the mitigation for confused-deputy delivery from someone else's account.

Encrypt with SSE-KMS and understand the key-policy consequences, a repeat exam scenario: when the trail itself is configured with a KMS key, the key policy must allow CloudTrail to call kms:GenerateDataKey*, scoped with the aws:cloudtrail:arn encryption-context condition (or aws:SourceArn) to your trail, or delivery fails; and analysts need kms:Decrypt or Athena queries return access errors even with full S3 read. The customer managed key should live in the log archive account, administered separately from log readers.

Enforce least privilege and deny-delete: an explicit bucket-policy Deny on s3:DeleteObject, s3:DeleteObjectVersion and s3:PutLifecycleConfiguration (and on s3:PutBucketPolicy), except for a break-glass role, means a compromised administrator in the log account cannot purge evidence. Denying s3:DeleteObject alone is not enough on a versioned bucket: it only blocks delete markers, while permanent deletion uses s3:DeleteObjectVersion. The weakness is the account's own root user, which can always replace a bucket policy. A service control policy attached to the log archive account closes that gap: SCPs restrict every principal in a member account, including its root user, and are managed from the organization's management account, outside the reach of anyone in the log archive account. (SCPs never restrict the management account itself — one more reason the archive lives in a member account.) MFA Delete adds friction for version deletion but can only be configured by the root user, through the API or CLI, and blocks nothing once that user is compromised.

For true write-once-read-many, S3 Object Lock in compliance mode makes object versions undeletable and unmodifiable for the retention period by anyone, including the root user — governance mode, bypassable by principals with s3:BypassGovernanceRetention, is the weaker cousin. Object Lock requires versioning and can be enabled on an existing bucket, but a bucket's default retention applies only to objects written after it is set; objects already stored need retention applied to them explicitly, at scale with an S3 Batch Operations Object Lock retention job.

Finally, CloudTrail log file integrity validation proves the record itself: hourly digest files containing SHA-256 hashes of each delivered log file, signed by CloudTrail with a private key, and chained so each digest references its predecessor. Running validate-logs recomputes the hashes, verifies each digest's signature with CloudTrail's public key, and walks the chain — so it detects any modified, deleted, or forged log file. Editing a log file and also rewriting its hash in the digest does not help an insider: the altered digest no longer matches its signature, which only CloudTrail can produce. Validation is what answers “prove the logs weren't altered”; Object Lock and deny statements are what stop the alteration in the first place.

Retention is a compliance decision implemented as lifecycle automation. In S3, lifecycle rules transition aging log objects to cheaper storage classes — Glacier Flexible Retrieval or Glacier Deep Archive for logs kept for multi-year mandates — and expire them when the retention period ends. Design the transition points around access patterns: recent logs get queried during investigations and belong in Standard or Intelligent-Tiering; year-old logs are almost never read and can tolerate retrieval delays. When Object Lock enforces retention, align the lock period with the lifecycle expiration so expiry doesn't silently fail against locked objects. In CloudWatch Logs, retention is per log group and defaults to never expire — an unbounded cost and data-minimization problem, so set explicit retention on every group, keeping the durable copy in S3.

Querying splits by where the data lives and what shape it is in:

Amazon Athena runs SQL directly over logs in S3 — define the table once and query years of history in place; partitioning (or partition projection) by date, account and Region keeps queries from scanning the whole archive. It is the tool for broad, historical questions: every action by an access key over eighteen months, every source address that reached a port across the fleet. CloudWatch Logs Insights runs a purpose-built query language (and OpenSearch PPL or SQL) interactively over log groups — the tool for fast, recent investigation where the data already streams to CloudWatch Logs. Security Hub findings are a different layer: not raw logs but conclusions other services drew from them, aggregated across accounts and Regions, filterable by severity, resource and account, and grouped with insights — the place to decide what to investigate, after which Athena or Logs Insights supplies the underlying events. When the requirement outgrows all of these into multi-source normalized analytics, that is Security Lake with Athena over OCSF tables.

Correlation fails when sources describe the same thing differently — one log calls the client srcaddr, another client_ip; timestamps arrive in different formats. Normalizing means mapping every source onto one schema before analysis, and AWS gives you three building blocks for it, each with a distinct job:

A common pattern: VPC Flow Logs and application logs flow from CloudWatch Logs through subscription filters into Firehose; a Lambda transformation function maps each record onto a shared field set; Firehose delivers to OpenSearch Service, with failed records backed up to S3; analysts correlate in OpenSearch Dashboards, while Managed Grafana puts those panels next to CloudWatch metrics and Athena queries over the S3 archive. Firehose buffers incoming data and invokes the function synchronously once per buffered batch, not once per record; the function must return every recordId in that batch with a result of Ok, Dropped or ProcessingFailed. Records it omits or marks ProcessingFailed go to the S3 error/backup prefix instead of the destination, so a function that silently drops records breaks the “every record must arrive” requirement. Firehose features such as record-format conversion to Parquet and dynamic partitioning apply to S3 destinations only. When the requirement is normalization at organization scale into an open schema, Security Lake’s OCSF output is the managed answer instead of a custom Lambda mapping.

A regulated company with 30 accounts must retain security logs for seven years, prove logs were never altered, guarantee no single account administrator can destroy them, and feed its SIEM in near real time. The design:

1. Centralize with an organization trail. From the management account, create a multi-Region organization trail with log file integrity validation enabled, delivering to a bucket in the dedicated log archive account. Every member account, present and future, is logged; members cannot stop or alter the trail.
2. Scope data events deliberately. Add advanced event selectors for S3 data events on the buckets holding regulated data — not on every bucket.
3. Add the network and service sources. Organization-wide VPC Flow Logs with a custom format including pkt-srcaddr and tcp-flags, Route 53 Resolver query logging, and WAF and ELB access logs, all delivering into the archive account.
4. Harden the buckets. Service-delivery policies conditioned on aws:SourceArn; SSE-KMS with a key in the archive account whose policy grants CloudTrail kms:GenerateDataKey and analysts kms:Decrypt; versioning plus Object Lock in compliance mode with seven-year retention; an explicit deny on s3:DeleteObject, s3:DeleteObjectVersion, lifecycle changes and bucket-policy edits — repeated in an SCP on the log archive account so even its root user is bound.
5. Lifecycle for cost. Transition objects to Glacier Deep Archive after 90 days; expire after seven years, aligned with the lock period.
6. Alert and stream. Send CloudTrail to CloudWatch Logs in parallel, add metric filters and alarms for threshold conditions, and attach a subscription filter into Firehose delivering to the SIEM endpoint.

An auditor asking “prove these logs are complete and unmodified” gets a validate-logs run over the digest chain; an attacker who owns a member account finds nothing there to delete.