Monitoring Data Pipelines: CloudWatch, CloudTrail and Log Analysis

Every monitoring question on this exam resolves to one of three telemetry sources: metrics, logs or API activity. Amazon CloudWatch metrics are numeric time series — a Glue job's DPU utilization, an EMR cluster's IsIdle flag, a Lambda function's error count. They answer how much and how fast, and they drive alarms. CloudWatch Logs stores the text your applications and AWS services emit — Spark driver output, Airflow task logs, Lambda print statements. Logs answer what happened inside the code. AWS CloudTrail records API calls made against your account — who started a Glue job, who deleted an S3 object, which role modified a security group. CloudTrail answers who did what, when, and from where.

The exam deliberately blurs these. A question about "auditing which user terminated the EMR cluster" is a CloudTrail question, not a CloudWatch one — no metric or application log records the caller's IAM identity. A question about "why the job ran slowly last night" is metrics-plus-logs. A question about "alerting the on-call engineer when the pipeline fails" is an alarm or EventBridge rule publishing to Amazon SNS. Anchor every scenario to the source that actually holds the answer before you look at the options.

Traceability, a term the exam guide uses explicitly, means being able to reconstruct a pipeline run end to end: the trigger (EventBridge rule or CloudTrail event), the execution (job run ID and its metrics), the application behavior (CloudWatch Logs) and the data it touched (S3 data events in CloudTrail, if enabled). Deploying logging and monitoring "to facilitate auditing and traceability" means turning all three sources on before you need them — you cannot retroactively collect logs that were never written.

Choose the tool by matching what it stores and how you query it to the question being asked. This table is worth memorizing — most Domain 3 monitoring questions are a row-lookup against it.

Two distinctions trip candidates up. First, Logs Insights versus OpenSearch: both analyze logs, but Logs Insights queries data in place in CloudWatch Logs with no infrastructure, while OpenSearch requires an ingestion pipeline and a domain to manage — in exchange you get sub-second full-text search and rich dashboards. For occasional troubleshooting, Logs Insights is the cheaper, simpler answer; for a continuously used operational search experience, OpenSearch wins. Second, CloudTrail versus CloudWatch Logs: CloudTrail can deliver its events to CloudWatch Logs (and to S3), but the source of API-audit truth is always CloudTrail. Athena querying a CloudTrail trail in S3 is the standard pattern for historical audits beyond the 90-day event history window.

Configuring CloudWatch Logs means deciding what gets written, where it lands, how long it lives and what watches it. Log events land in log streams inside log groups; the group is where you set retention (from one day to indefinite — the default is never-expire, which quietly accumulates cost, so set retention explicitly), encryption with AWS KMS and the log class. Managed services write logs when you enable them: AWS Glue jobs write driver and executor logs to Glue-specific log groups, and continuous logging streams Spark output while the job is still running instead of only at the end — enable it so you can watch a long job live rather than waiting for failure. Lambda logs automatically via its execution role. On EC2 or on-premises hosts, the CloudWatch agent ships application log files.

Log classes and long-term retention

A log group is created in the Standard or the Infrequent Access log class, and the class cannot be changed afterwards. Infrequent Access lowers the ingestion price, not the storage price, and supports a smaller feature set than Standard (you can still query it with Logs Insights, but features such as metric filters are Standard-only). It suits high-volume logs you keep for occasional forensics, not logs that drive alarms. When the requirement is to keep logs for years at the lowest cost, the answer is a short retention period in CloudWatch Logs plus a long-term copy in Amazon S3 (with S3 lifecycle rules to colder storage classes) — not a multi-year retention setting on the log group.

Metric filters

A metric filter turns a log pattern into a CloudWatch metric — for example, count occurrences of ERROR in a pipeline's log group, then alarm when the count exceeds zero in five minutes. That is the canonical "alert on an application error that has no built-in metric" answer. Two details decide the harder questions. First, a metric filter publishes a value only when matching events are ingested; set a default value (usually 0) so periods with log traffic but no matches report zero instead of nothing. If the log group receives no events at all, there is still no data point, so the alarm's missing-data treatment (for example notBreaching) decides whether it sits in INSUFFICIENT_DATA. Second, metric filters are not retroactive: they act only on events ingested after the filter is created. To count matches in logs that already exist, run a Logs Insights query instead.

Subscription filters and exports

A subscription filter streams matching log events continuously, in near real time, to one of three destinations: Amazon Kinesis Data Streams (your own consumers, fan-out), Amazon Data Firehose (formerly Kinesis Data Firehose — the managed, no-code path to Amazon S3 or Amazon OpenSearch Service), or an AWS Lambda function (custom processing per batch). Like metric filters, subscription filters only see events ingested after they exist; they do not backfill. An export task is the opposite shape: a one-off batch copy of a log group's time range to S3, suited to a single audit extraction, not to continuous delivery. Once logs sit in S3 as objects, Athena can query them with SQL — which is exactly how you satisfy an auditor asking for "all pipeline errors in Q3" without paging through the console.

Centralizing logs across accounts

To copy logs from many accounts into one, the receiving account creates a CloudWatch Logs destination (backed by a Kinesis data stream or a Firehose stream) with a destination access policy that allows the sending accounts or the organization; each sending account then creates a subscription filter that targets the destination's ARN. Do not confuse this with CloudWatch cross-account observability, which links source accounts to a monitoring account so you can view their metrics, logs and traces in dashboards and queries — the data stays in the source accounts and nothing is copied.

AWS CloudTrail tracks API calls by recording every management operation in your account — console clicks, CLI commands and SDK calls all surface as CloudTrail events carrying the caller's identity, timestamp, source IP and request parameters. Management events (control-plane operations like StartJobRun, CreateCluster, DeleteTable) are recorded by default and viewable for 90 days in the event history. For anything longer — and every compliance scenario implies longer — create a trail that delivers events continuously to an S3 bucket, optionally also to CloudWatch Logs for real-time alarming on specific calls.

Data events are the high-volume data-plane operations: S3 object-level reads and writes (GetObject, PutObject, DeleteObject), Lambda Invoke, DynamoDB item activity. They are not logged by default and cost extra, so you enable them explicitly on a trail. Note that logging read-only management events does not capture S3 object reads — GetObject is a data event. When a question asks how to know who accessed specific S3 objects, the answer is CloudTrail data events, not server metrics and not application logs.

To keep data-event cost down, use advanced event selectors, which filter on fields such as eventCategory (Data), resources.type (for example AWS::S3::Object), resources.ARN with StartsWith (one bucket or prefix) and readOnly (false logs only writes and deletes). Logging only the write calls on one sensitive prefix costs a fraction of logging every read on every bucket.

CloudTrail Insights learns a baseline of normal API call rates and error rates and records an Insights event when activity deviates from it — no thresholds to write. It covers management events and, since November 2025, data events. Contrast it with a metric filter alarm on CloudTrail logs delivered to CloudWatch Logs: that you build yourself, per API pattern, with a static threshold or a CloudWatch anomaly detection band. Use Insights for "tell me when API activity is unusual"; use a metric filter alarm for "page me when this specific call happens".

One gotcha: CloudTrail records the API call to start a Glue job, but the job's internal behavior (rows processed, transformations applied) lives in CloudWatch Logs and job metrics. Audit questions about API actions go to CloudTrail; audit questions about data processing detail go to logs.

Audit-grade CloudTrail means three things: every account is covered, the logs are queryable for years, and the logs can be proven untouched. An organization trail, created in the AWS Organizations management account (or a delegated administrator account), logs every member account — including accounts that join later — into one central bucket, and member accounts can see it but cannot modify or delete it. That beats creating a trail per account, which misses new accounts and can be switched off locally.

The standard query pattern is a trail delivering to S3 and an Athena table over its JSON. CloudTrail writes objects under an account, Region and date path (AWSLogs/<account>/CloudTrail/<region>/YYYY/MM/DD/), so define the table with partition projection on Region and date: Athena computes the partitions from the path pattern, with no crawler or ALTER TABLE ADD PARTITION to maintain. Queries must filter on those partition columns, or Athena scans the whole archive. CloudTrail Lake also offers SQL over events, but it closed to new customers on 31 May 2026 (existing event data stores keep working); for a new account, use a trail to S3 with Athena, or CloudWatch, which AWS points new users to.

For evidence, CloudTrail is preferred over S3 server access logs because it records the authenticated IAM identity and supports log file integrity validation: signed hourly digest files let you prove a log file was not modified or deleted after delivery. Validation detects tampering; to prevent deletion, store the logs in a bucket with S3 Object Lock in compliance mode, where no user, including the root user, can delete or shorten retention on a locked object. Governance mode can be bypassed by anyone with s3:BypassGovernanceRetention, and MFA delete only requires the bucket owner's (root user's) MFA code to permanently delete a version or change versioning — it does not stop that root user from deleting — so neither meets a "nobody can delete it" requirement.

Alerts on AWS follow two main paths, and the exam expects you to know when each applies. Path one: a CloudWatch alarm watches a metric against a threshold — Glue job duration, a metric-filter error count, EMR cluster memory — and on breach publishes to an Amazon SNS topic. SNS fans one message out to every subscriber: email, SMS, HTTPS endpoints, Lambda functions and Amazon SQS queues (the way to keep a durable copy for a ticketing system to pull). Alarms suit anything expressible as a number crossing a line, including composite alarms that combine several conditions to cut alert noise. Alarm actions publish to SNS, invoke a Lambda function, or run Auto Scaling, EC2 and Systems Manager actions; they cannot target an SQS queue directly — subscribe the queue to the topic.

Path two: an Amazon EventBridge rule matches structured events emitted by services as state changes happen. AWS Glue emits Glue Job State Change events (SUCCEEDED, FAILED, TIMEOUT), Amazon EMR emits cluster and step state changes, and AWS Step Functions emits Execution Status Change events whose detail carries the execution ARN, status, and the error and cause of a failure. A rule pattern like "Glue Job State Change where state is FAILED" routed to an SNS topic is the cleanest possible "notify the team when the nightly job fails" answer — no polling, no custom code, near-real-time. An input transformer on the target reshapes the event into a readable message (job name, run ID, error) before it reaches SNS. EventBridge has no direct email target, so human notification goes through SNS; other targets include Lambda (automated remediation, such as restarting a job) and Step Functions (failure-handling workflows).

Why an alert never arrives

Three configuration traps explain most silent alerting. Region: AWS service events are delivered to the default event bus in the Region where they occur, so a rule in us-east-1 never sees a Glue job that runs in eu-west-1 — a rule whose MatchedEvents metric stays at 0 is usually in the wrong Region (create it in the job's Region, or forward events between buses). Encryption: if the SNS topic uses SSE-KMS, CloudWatch alarms can publish only when the topic uses a customer managed key whose key policy allows cloudwatch.amazonaws.com to call kms:Decrypt and kms:GenerateDataKey*; the AWS managed aws/sns key cannot be used because its key policy cannot be edited. Missing data: alarms have three states (OK, ALARM, INSUFFICIENT_DATA); a metric that stops reporting lands in INSUFFICIENT_DATA, so choose the missing-data treatment deliberately — notBreaching for sparse error counts, breaching when silence itself means the pipeline is dead.

Also distinguish SNS from SQS in options: SNS pushes notifications to subscribers immediately, while SQS queues messages for a consumer to pull — a "send an alert to the operations team" requirement means SNS.

Troubleshooting a Glue job means matching the question you are asking to the Glue tool that answers it. Each one is enabled on the job and answers something different:

The Spark UI needs Spark event logs written to S3 (a job setting). Glue renders the Spark UI for a job run directly in the console from those event logs, so a self-hosted Spark history server is optional, not required. Note that a Glue failed-task count measures Spark task failures, which Spark retries; it is not a count of failed job runs.

Reading the failure signatures

An out-of-memory driver often traces to collecting too much data to the driver or to listing millions of small files. Undersized capacity versus data skew is read from the per-executor heap metrics: when every executor runs near its memory limit, the job needs more or larger workers; when one executor's heap and shuffle climb while the others sit idle, a hot key has put most of the data in one partition — more workers will not help, and the fix is in the data (covered in the data quality lesson). In the Spark UI the same skew shows as a stage whose maximum task time dwarfs its median.

Maintaining Glue pipelines

Small files: thousands of tiny S3 objects create one task per file and swamp the driver. Glue's file grouping reads many files into one in-memory partition — set groupFiles to inPartition and optionally groupSize (bytes per group); Glue turns grouping on automatically when an input has more than 50,000 files, so below that you set it yourself. Compacting into larger Parquet files fixes it at the source. Capacity: Glue Auto Scaling (Glue 3.0 and later) adds and removes workers during the run up to a maximum you set — the fit for variable data volumes with a deadline. The Flex execution class runs on spare capacity at a lower price, but jobs may start late and run times vary, so it suits non-urgent work such as nightly backfills, not SLA-bound pipelines. Job bookmarks track already-processed data between runs: a reset or disabled bookmark reprocesses everything (duplicates), while a stale or mis-set one silently skips new data.

Concrete scenario: a nightly Glue job that ran in 30 minutes now takes four hours and occasionally fails with executor lost errors. Metrics show one executor's heap and shuffle write dwarfing the rest — skew on the join key. The trail runs: CloudWatch alarm on job duration fired, EventBridge caught the eventual FAILED state and notified SNS, job run insights named the failing line, and the Spark UI confirmed a single monster task. That chain — alarm, event, log, Spark UI — is exactly the diagnostic order the exam rewards.

Amazon EMR troubleshooting works with different artifacts because you manage more of the stack. EMR writes cluster, step and application logs to the S3 location you set as the cluster's log URI (LogUri) at launch — set it every time. Without it there is no copy of the logs in your own S3 bucket once the cluster terminates: logs on instance disks go with the instances, and although EMR keeps certain logs in EMR-managed storage, that does not replace a LogUri: you cannot rely on it for a step's stderr or for audits. Terminated-cluster debugging is a recurring exam setup.

For Spark and YARN history, EMR's persistent application user interfaces host the Spark history server and YARN timeline server off-cluster in the EMR console, and remain available for 30 days after the cluster terminates — no SSH tunnel and no history server of your own. For live investigation, the on-cluster YARN ResourceManager and Spark UIs show container allocation and stage timings.

Reading EMR metrics

EMR publishes CloudWatch metrics that point at the right fix. IsIdle detects clusters burning money doing nothing (alarm on it, or use an auto-termination policy). Low YARNMemoryAvailablePercentage with a rising ContainerPending count means the cluster is short of compute — jobs are waiting for memory. High HDFSUtilization means it is short of storage. Compute shortage is fixed with task nodes, which run YARN containers but hold no HDFS data, so they can run on Spot and be added and removed freely. Storage shortage needs core nodes (they host HDFS DataNodes) or, better, moving data to S3.

With EMR managed scaling, set a maximum for core nodes below the overall maximum: EMR then adds peak capacity as task nodes, which can use Spot, while the HDFS-bearing core fleet stays small and On-Demand. Other buckets: wrong instance profile (memory-hungry Spark jobs on compute-optimized instances thrash — match instance family to workload) and Spot interruptions (task nodes on Spot are a cost win, but core nodes on Spot risk losing HDFS blocks mid-job).

For step failures, read the step's stderr log under the S3 log URI first — it holds the Spark or Hive exception. And remember EMR emits step and cluster state-change events to EventBridge just like Glue, so "alert when the EMR step fails" is again an EventBridge rule to SNS, not a custom polling script.

Once logs exist, four AWS services analyze them, and the right one depends on where the logs live, how often you query, and how fast answers must arrive. CloudWatch Logs Insights queries log groups in place and is the default for interactive troubleshooting: no setup, pay per query by data scanned, results in seconds, across one or many log groups. Because it reads what is already stored, it also answers questions about the past that a newly created metric filter cannot.

Writing Logs Insights queries

A query is a pipeline of commands: fields chooses what to show, filter keeps matching events, parse extracts fields, stats aggregates, sort and limit shape the output. JSON logs are discovered as fields automatically; plain-text logs need parse first, either with a glob pattern (parse @message "rows=* duration=*ms" as rows, ms) or a regular expression with named groups. In stats, count() counts matching events while sum(field) adds a numeric field's values — "how many batches" versus "how many rows". Group by time with bin(), for example stats count() by bin(1h) for failures per hour.

Athena, EMR and OpenSearch Service

Amazon Athena is the answer when logs are in S3 — exported CloudWatch Logs, CloudTrail trails, S3 server access logs, VPC Flow Logs, or application logs a pipeline wrote directly. Define a table (partition it by date, ideally with partition projection, and convert high-volume logs to Parquet if you query them repeatedly) and analyze with standard SQL. Athena is serverless and priced per data scanned, making it the audit workhorse for long-retention archives queried occasionally.

Amazon EMR earns its place when log analysis becomes a heavy transformation job in its own right — parsing terabytes of raw application logs into structured datasets, sessionizing clickstreams, or running Spark ML over log-derived features. Big data application logs are a common case: Athena's text SerDes treat each line as a record, so a multi-line Java stack trace splits into many rows, and custom classification logic does not fit SQL well; Spark on a transient EMR cluster can group multi-line records in code and costs nothing between runs. You would not spin up EMR to grep for an error. Amazon OpenSearch Service fits continuous operational search: stream logs in with a subscription filter to Amazon Data Firehose, index them, and give operators full-text search and OpenSearch Dashboards with near-real-time latency. The trade-off is a managed domain (or serverless collection) to size and pay for continuously. Exam shorthand: in place and ad hoc — Logs Insights; in S3 with SQL — Athena; massive or multi-line parsing jobs — EMR; interactive real-time dashboards — OpenSearch.