Preparing Logs for Audit: CloudTrail, CloudTrail Lake, and CloudWatch Logs

AWS CloudTrail answers the auditor's first question — who did what, when, from where — by recording API calls across your account as events containing the caller identity, source IP, timestamp, request parameters, and response elements. Every account gets Event history for free: the last 90 days of management events in the current Region, searchable by a single attribute at a time (user name, event name, resource name, event source and similar) and downloadable as CSV or JSON. Event history has no SQL, no joins and no data events, and it cannot be extended — anything older than 90 days, and anything beyond management events, requires a trail or an event data store that was configured before the activity happened.

Know the event taxonomy cold. Management events (control plane) record operations on resources: creating a Glue job, modifying a Redshift cluster, attaching an IAM policy. Trails log them by default. Data events (data plane) record operations on the data inside resources: S3 object-level GetObject and PutObject, Lambda function invocations, DynamoDB item-level actions. Data events are high-volume, off by default, selected per resource with event selectors (advanced event selectors can narrow by bucket, prefix or read/write type), and billed separately — so when a question asks who read or deleted a specific S3 object, the answer is that data events must have been enabled beforehand; Event history alone will never show it. Insights events are the third type: CloudTrail Insights learns a baseline of your normal API call rate and API error rate and records an Insights event when activity departs from it, such as a sudden burst of delete calls or access-denied errors — anomaly detection with no custom logic. Insights analyses management events, and on trails it can also baseline data events.

S3 offers a second record of object-level access: S3 server access logs. They are free apart from storage, but delivered on a best-effort basis (some records can arrive late or not at all) in a plain-text line format. CloudTrail data events are the structured, identity-rich record — full IAM principal, request parameters, delivered reliably — and they are what an audit should rely on.

A trail delivers events as compressed JSON files to an S3 bucket you choose (typically within about 5 minutes of the API call — the files are not a real-time feed), optionally also to a CloudWatch Logs log group. Best practice — and the assumed exam default — is a multi-Region trail, which covers every current Region and any Region enabled later, so activity in Regions you never use still lands in one place. From the S3 archive you can retain events for as many years as compliance demands, at S3 prices, with lifecycle policies tiering old logs to Glacier storage classes.

An audit-ready log is complete, centralized, and provably unaltered. Completeness and centralization come from an organization trail: created in the AWS Organizations management account (or a delegated administrator account), it automatically logs every member account — including accounts created later — into one S3 bucket. Member accounts can see the trail but cannot modify or delete it, so the people generating the activity cannot switch off its recording. Per-account trails that each deliver to a central bucket do work, but they are neither future-proof (every new account needs its own trail) nor tamper-proof (each account's administrators can stop or edit their own trail).

Where the logs live: the AWS Security Reference Architecture

The AWS SRA gives log custody its own account. The Log Archive account holds the central log bucket and nothing else; it is kept separate from the Security Tooling account (the delegated administrator for GuardDuty, Security Hub CSPM, Config aggregation and similar) and from the management account, which should run as few workloads and store as little data as possible because its principals cannot be restricted by service control policies. The bucket policy allows writes only from the CloudTrail service principal (scoped with aws:SourceArn/aws:SourceAccount conditions), reads only from designated audit roles, and log files are encrypted with SSE-KMS under a tightly scoped key.

Proving integrity: log file validation

CloudTrail log file validation proves whether log files were changed after delivery. When enabled, CloudTrail delivers a digest file every hour; each digest holds SHA-256 hashes of the log files delivered in the previous hour, is signed with CloudTrail's private key, and references the previous digest, forming a chain. Running aws cloudtrail validate-logs checks the chain and reports each file as valid, modified, or — when a referenced file has been deleted — INVALID: not found. Know its limits:

• Coverage lags: the newest log files are not covered until the next hourly digest is delivered.
• No retroactive coverage: only log files delivered after validation was enabled are covered; turning it on today cannot vouch for last month's files.
• It detects tampering; it does not prevent it. Encryption (SSE-KMS-encrypted log files validate normally) and bucket versioning do not prove integrity on their own.

Preventing deletion: Object Lock versus MFA Delete

To make stored logs undeletable, use S3 Object Lock (WORM — write once, read many). It requires versioning, can be enabled on a new bucket or on an existing versioned bucket, and once enabled cannot be disabled. Its two retention modes differ in who can override them:

The standard multi-account pattern therefore is: organization trail (multi-Region, log file validation on) delivering to an SSE-KMS-encrypted, Object Lock compliance-mode bucket in the Log Archive account, with AWS Config and application log archives landing in the same locked-down account.

Amazon CloudWatch Logs is the default home for application logs — the output of your Lambda functions, Glue jobs, Amazon MWAA DAGs, Step Functions executions, EMR steps, ECS containers, and anything on EC2 running the CloudWatch agent. Logs are organized into log groups (one per application or job, typically) containing log streams (one per instance, container, or execution). Lambda and Glue write here automatically; for audit purposes your job is mostly configuration, not plumbing.

Retention is the first setting to fix: log groups default to never expiring, which silently accumulates cost. Set a retention period per log group to match your requirement — but understand that expiry is permanent deletion. Once events age out, nothing in CloudWatch can bring them back: Logs Insights only queries stored events, an export task copies only events still in the log group, and raising the retention setting afterwards only keeps future events longer. History beyond retention exists only if a copy was streamed out beforehand, typically a subscription filter into Amazon Data Firehose delivering to S3, where Athena can query it in place. For multi-year retention, that continuous stream is better than periodic export tasks (which are one-off, asynchronous, and limited to one running task per account at a time).

Two filter mechanisms turn stored logs into signals. Metric filters match patterns in incoming log events and increment a CloudWatch metric — the classic audit example counts unauthorized API calls or root-account usage in CloudTrail logs delivered to CloudWatch Logs, with an alarm notifying security. Subscription filters stream matching log events in near real time to a destination: Kinesis Data Streams, Amazon Data Firehose, or a Lambda function directly. Amazon OpenSearch Service is not a direct subscription target — the console's OpenSearch option creates a Lambda function that loads the domain, or you route through Firehose to OpenSearch.

Subscription filters are also how one central account aggregates logs from many workload accounts. The receiving account creates a CloudWatch Logs destination (a resource pointing at a Kinesis stream or Firehose stream, with a policy allowing the sender accounts), and each sender adds a subscription filter targeting it. A destination must be in the same Region as the log groups that subscribe to it, although the stream it forwards to may live in another Region — so log groups in two Regions need a destination in each Region.

Near-real-time alerting on a specific API call means acting on the event stream, not on the S3 log files: trail files land in S3 minutes after the call and in batches, so polling the bucket is the slow, fragile answer. There are two standard event-driven paths.

• Amazon EventBridge rule on CloudTrail events. API calls recorded by CloudTrail are emitted to EventBridge with the detail type AWS API Call via CloudTrail. A rule matching the event source and event name (for example a DeleteTrail, PutBucketPolicy or DeleteTable call) can invoke SNS, Lambda, Step Functions or other targets within seconds. This requires a trail with logging enabled in the account.
• Trail to CloudWatch Logs, metric filter, alarm. Configure the trail to also send events to a log group, define a metric filter that matches the pattern (root sign-in, unauthorized calls, security group changes) and attach a CloudWatch alarm to the metric. This suits counting and threshold patterns, such as more than five access-denied errors in five minutes.

CloudTrail Insights complements both: it flags unusual rates of activity you did not think to write a rule for, while EventBridge and metric filters catch the specific calls you named in advance.

AWS CloudTrail Lake is a managed audit data lake: you create an event data store, choose which events it ingests (management events, data events, AWS Config configuration items, and audit events from outside AWS via its integration APIs), set a multi-year retention period, and query it directly with SQL from the CloudTrail console or API — no S3 bucket to secure, no Glue table to define, no partitioning to maintain.

Availability caveat: CloudTrail Lake closed to new customers on 31 May 2026. Accounts that already use it keep their event data stores and can keep creating and querying them, but the service receives only critical fixes (and an account-level event data store does not start covering member accounts added to the organization later — only an organization event data store does), and AWS points new adopters to CloudWatch for centralized log analytics. The exam guide still lists CloudTrail Lake, so know what it does — but for a brand-new account, the realistic design is an organization trail to S3 queried with Athena, or CloudTrail delivered to CloudWatch.

Its audit-specific properties are the selling point for existing users. Ingested events are stored immutably — you cannot edit or delete individual events — and an event data store can be organization-scoped, capturing all member accounts like an organization trail does. Because the store is queryable in place, answering an auditor's question about activity from two years ago is a single SQL statement rather than a restore-and-catalog exercise. It charges for ingestion and for data scanned by queries, so it is priced for audit and investigation workloads, not general analytics.

Compare it with its near twins: Event history gives 90 days of management events with single-attribute lookups and no SQL; CloudTrail Lake gives multi-year, immutable, SQL-queryable storage with no pipeline; a trail to S3 plus Athena gives raw log files you own (some regulators want the artifacts), the cheapest long-term archive, joins with other data-lake tables — and remains open to every account.

Amazon Athena is the query engine for logs archived in S3. For CloudTrail, the console can generate the Athena table definition for your trail bucket; from there you write standard SQL to answer questions like which principals called DeleteObject last quarter, or which source IPs used a given access key. Athena bills per data scanned, so the exam-relevant optimizations are the usual S3 analytics ones: partition the table (by account, Region, and date — partition projection computes partitions from the path pattern instead of crawling or registering thousands of prefixes), compress, and convert heavily queried logs to Parquet with a Glue or EMR job. Athena is equally the tool for VPC Flow Logs, ALB access logs, and S3 server access logs sitting in S3.

Redshift Spectrum can also query log files in S3 in place through external tables, and is the natural choice when the logs must be joined with tables already loaded in a Redshift warehouse. For one-off or occasional log investigations it costs more than Athena, because it needs a provisioned Redshift cluster running and billing in addition to the per-TB scan charge (on Redshift Serverless the scans are billed as RPU usage instead), whereas Athena is serverless with nothing to keep running.

CloudWatch Logs Insights is the interactive query tool for data still in CloudWatch Logs — no table definitions, no export. Queries can be written in the Logs Insights query language (pipe commands: filter, parse, stats, sort, limit), in OpenSearch PPL, or in OpenSearch SQL. It discovers fields in JSON log events automatically, queries many log groups at once, and is priced per data scanned. It is the right answer for operational and troubleshooting queries: find the error spike in last night's Glue job logs, count Lambda timeouts per function this week, trace a request ID across services — as long as the events are still within retention.

The dividing line: where do the logs live, how old are they, and how often are they queried? Logs in CloudWatch Logs within retention — Logs Insights. Logs archived to S3, joins with other datasets, scheduled reporting — Athena (or Spectrum when the warehouse is already part of the query). Both Athena and Logs Insights charge per byte scanned, so cost grows with query volume; when many users search logs all day, a capacity-billed OpenSearch domain becomes the cheaper and faster option, covered next.

Amazon OpenSearch Service fits when the requirement is continuous, interactive log exploration: full-text search across log lines, filterable dashboards in OpenSearch Dashboards, and alerting on patterns — the live operations view, as opposed to Athena's retrospective SQL. Logs typically arrive through Amazon Data Firehose (which batches, optionally transforms with Lambda, and delivers into the domain with failed records backed up to S3) or a CloudWatch Logs subscription filter using a Lambda function. Its cost model is different from the query-in-place tools: a provisioned domain is billed for its instances and storage whether queried once or ten thousand times, so it wins when many users query constantly and loses for occasional questions. Keep a recent window hot, use UltraWarm and cold storage to extend retention more cheaply, and rely on the S3 archive with Athena for deep history.

Amazon EMR is the exam's stated answer for processing very large volumes of log data. When years of multi-terabyte log archives need sessionization, enrichment, joining against reference data, or reformatting into Parquet, a Spark job on EMR (or EMR Serverless) reads the raw logs from S3, transforms them at cluster scale, and writes curated output back to S3 — after which Athena queries become fast and cheap. EMR is a batch transformation engine here, not an interactive query tool.

AWS Glue versus EMR for big log jobs. Both run distributed Spark, and Glue is the lower-operations choice when the job needs only Spark plus libraries: Glue's managed workers accept extra JAR files and Python wheels or modules, but you cannot run node-level setup on them, such as installing operating-system packages with a package manager or a native parsing binary. EMR gives you the cluster: bootstrap actions run your scripts on every node as it launches, before applications start, so custom OS packages, agents and native libraries are installed everywhere. When a log-processing job depends on that kind of setup, EMR is the fit.

A complete audit-logging architecture layers the tools rather than choosing one: CloudTrail and application logs stream to durable, validated, Object Lock-protected S3 archives; a recent slice flows to OpenSearch for live search; EventBridge rules and metric filters raise alarms on named security events; Athena serves deep historical queries; and EMR batch-processes the archive when scale demands it.

Most questions in this task reduce to matching the scenario's requirement to one service. Use this table as the tie-breaker.

Worked scenario: an auditor asks who deleted objects from a regulated S3 bucket fourteen months ago, and compliance rules require proving the logs were never altered. The working chain is: S3 data events were enabled on an organization trail with log file validation, delivered to an Object Lock compliance-mode bucket in the Log Archive account; you query the archive with Athena (or, in an account that already has an organization event data store, run one SQL query in CloudTrail Lake); and you run validate-logs to demonstrate integrity for the period. Any answer relying on Event history fails twice — it only spans 90 days and never included data events.