Monitoring Data Pipelines: CloudWatch, CloudTrail and Log Analysis

Every monitoring question on this exam resolves to one of three telemetry sources: metrics, logs or API activity. Amazon CloudWatch metrics are numeric time series — a Glue job's DPU utilization, an EMR cluster's IsIdle flag, a Lambda function's error count. They answer how much and how fast, and they drive alarms. CloudWatch Logs stores the text your applications and AWS services emit — Spark driver output, Airflow task logs, Lambda print statements. Logs answer what happened inside the code. AWS CloudTrail records API calls made against your account — who started a Glue job, who deleted an S3 object, which role modified a security group. CloudTrail answers who did what, when, and from where.

The exam deliberately blurs these. A question about "auditing which user terminated the EMR cluster" is a CloudTrail question, not a CloudWatch one — no metric or application log records the caller's IAM identity. A question about "why the job ran slowly last night" is metrics-plus-logs. A question about "alerting the on-call engineer when the pipeline fails" is an alarm or EventBridge rule publishing to Amazon SNS. Anchor every scenario to the source that actually holds the answer before you look at the options.

Traceability, a term the exam guide uses explicitly, means being able to reconstruct a pipeline run end to end: the trigger (EventBridge rule or CloudTrail event), the execution (job run ID and its metrics), the application behavior (CloudWatch Logs) and the data it touched (S3 data events in CloudTrail, if enabled). Deploying logging and monitoring "to facilitate auditing and traceability" means turning all three sources on before you need them — you cannot retroactively collect logs that were never written.

Choose the tool by matching what it stores and how you query it to the question being asked. This table is worth memorizing — most Domain 3 monitoring questions are a row-lookup against it.

Two distinctions trip candidates up. First, Logs Insights versus OpenSearch: both analyze logs, but Logs Insights queries data in place in CloudWatch Logs with no infrastructure, while OpenSearch requires an ingestion pipeline and a domain to manage — in exchange you get sub-second full-text search and rich dashboards. For occasional troubleshooting, Logs Insights is the cheaper, simpler answer; for a continuously used operational search experience, OpenSearch wins. Second, CloudTrail versus CloudWatch Logs: CloudTrail can deliver its events to CloudWatch Logs (and to S3), but the source of API-audit truth is always CloudTrail. Athena querying a CloudTrail trail in S3 is the standard pattern for historical audits beyond the 90-day event history window.

Configuring CloudWatch Logs means deciding what gets written, where it lands, how long it lives and what watches it. Log events land in log streams inside log groups; the group is where you set retention (from one day to indefinite — the default is never-expire, which quietly accumulates cost, so set retention explicitly) and encryption with AWS KMS. Managed services write logs when you enable them: AWS Glue jobs write driver and executor logs to Glue-specific log groups, and continuous logging streams Spark output while the job is still running instead of only at the end — enable it so you can watch a long job live rather than waiting for failure. Lambda logs automatically via its execution role. On EC2 or on-premises hosts, the CloudWatch agent ships application log files.

Automation is the exam's emphasis. A metric filter turns a log pattern into a CloudWatch metric — for example, count occurrences of ERROR in a pipeline's log group, then alarm when the count exceeds zero in five minutes. That is the canonical "alert on an application error that has no built-in metric" answer. A subscription filter streams matching log events in near real time to a destination — Lambda for custom reactions, Amazon Data Firehose for delivery to S3 or OpenSearch, or Kinesis Data Streams for fan-out. Subscription filters are how you centralize logs from many accounts or feed an OpenSearch domain.

For audit extraction, remember the export paths: one-off exports of a log group go to S3 with an export task, while continuous delivery uses a subscription filter through Firehose. Once logs sit in S3 as objects, Athena can query them with SQL — which is exactly how you satisfy an auditor asking for "all pipeline errors in Q3" without paging through the console.

AWS CloudTrail tracks API calls by recording every management operation in your account — console clicks, CLI commands and SDK calls all surface as CloudTrail events carrying the caller's identity, timestamp, source IP and request parameters. Management events (control-plane operations like StartJobRun, CreateCluster, DeleteTable) are recorded by default and viewable for 90 days in the event history. For anything longer — and every compliance scenario implies longer — create a trail that delivers events continuously to an S3 bucket, optionally also to CloudWatch Logs for real-time alarming on specific calls.

Data events are the high-volume data-plane operations: S3 object-level reads and writes, Lambda invocations, DynamoDB item activity. They are not logged by default and cost extra, so you enable them selectively — for example, object-level logging on the bucket holding regulated data, so you can prove exactly which principal read which object. When a question asks how to know who accessed specific S3 objects, the answer is CloudTrail data events, not server metrics and not application logs.

For data engineering audits the standard architecture is: trail delivering to S3, a table over the trail's JSON in Athena, and SQL queries answering questions like "which IAM roles ran Glue jobs against the production catalog last month." Integrity matters too — enable log file validation on the trail so digest files can prove the logs were not tampered with after delivery. One gotcha: CloudTrail records the API call to start a Glue job, but the job's internal behavior (rows processed, transformations applied) lives in CloudWatch Logs and job metrics. Audit questions about API actions go to CloudTrail; audit questions about data processing detail go to logs.

Alerts on AWS follow two main paths, and the exam expects you to know when each applies. Path one: a CloudWatch alarm watches a metric against a threshold — Glue DPU usage, a metric-filter error count, EMR cluster memory — and on breach publishes to an Amazon SNS topic, which fans out to email, SMS, an HTTPS endpoint or a Lambda function. Alarms suit anything expressible as a number crossing a line, including composite alarms that combine several conditions to cut alert noise.

Path two: an Amazon EventBridge rule matches structured events emitted by services as state changes happen. AWS Glue emits job state-change events (SUCCEEDED, FAILED, TIMEOUT), Amazon EMR emits cluster and step state changes, and AWS Step Functions emits execution status changes. A rule pattern like "Glue Job State Change where state is FAILED" routed to an SNS topic is the cleanest possible "notify the team when the nightly job fails" answer — no polling, no custom code, near-real-time. EventBridge targets also include Lambda (for automated remediation, such as restarting a job) and Step Functions (for failure-handling workflows).

Choosing between them: state changes and discrete events favor EventBridge; thresholds and trends favor alarms. Both converge on SNS for human notification. Also distinguish SNS from Amazon SQS in options: SNS pushes notifications to subscribers immediately, while SQS queues messages for a consumer to pull — a "send an alert to the operations team" requirement means SNS. Finally, remember alarms have three states (OK, ALARM, INSUFFICIENT_DATA) — a metric that stops reporting entirely lands in INSUFFICIENT_DATA, and you can configure how missing data is treated so a silent, dead pipeline still pages someone.

Troubleshooting a Glue job starts with three artifacts: the job run metrics in CloudWatch, the driver and executor logs in CloudWatch Logs, and the Spark UI (enable persisting Spark event logs to S3 so you can open the history server after the run). Metrics reveal the shape of the problem — executor memory pressure, uneven work distribution, low parallelism; logs reveal the exact exception; the Spark UI reveals which stage and which tasks consumed the time.

Know the classic failure signatures. An out-of-memory driver often traces to collecting too much data to the driver or to listing millions of small files — the small-files problem also destroys read performance, and the fixes are compacting input, using larger files in columnar formats like Parquet, or Glue's grouping options for reads. Executor OOM or straggler tasks point at data skew — one partition far larger than the rest (covered in depth in the data quality lesson). Underprovisioning shows as sustained full utilization of all workers: scale the worker count or move to a larger worker type. Conversely, paying for idle executors means you overprovisioned — Glue auto scaling can right-size executors during the run. If a job that previously succeeded suddenly reprocesses everything or nothing, check job bookmarks, Glue's mechanism for tracking already-processed data between runs; a reset bookmark reprocesses history, a stale one silently skips new partitions.

Concrete scenario: a nightly Glue job that ran in 30 minutes now takes four hours and occasionally fails with executor lost errors. Metrics show one executor's shuffle write dwarfing the rest — skew on the join key. The trail runs: CloudWatch alarm on job duration fired, EventBridge caught the eventual FAILED state and notified SNS, the driver log named the failing stage, and the Spark UI confirmed a single monster task. That chain — alarm, event, log, Spark UI — is exactly the diagnostic order the exam rewards.

Amazon EMR troubleshooting works with different artifacts because you manage more of the stack. EMR writes cluster, step and application logs to an S3 location you configure at launch — enable this every time, because logs on the cluster's local disks vanish when the cluster terminates, and terminated-cluster debugging is a recurring exam setup. For live investigation, the YARN ResourceManager and Spark history server UIs (reachable through SSH tunneling or the console's persistent application UIs) show container allocation and stage timings. EMR also publishes CloudWatch metrics: IsIdle for detecting clusters burning money doing nothing (alarm on it to auto-terminate or notify), YARN memory availability, HDFS utilization and apps pending.

Performance issues on EMR usually fall into four buckets. Resource starvation: pending YARN applications and full memory metrics mean the cluster is too small — resize the core or task fleets, or rely on EMR managed scaling. Wrong instance profile: memory-hungry Spark jobs on compute-optimized instances thrash; match instance family to workload. HDFS pressure: shrinking core nodes (which host HDFS DataNodes) risks data loss and rebalancing storms — scale task nodes for compute elasticity instead, since they carry no HDFS. Spot interruptions: task nodes on Spot are a cost win, but core nodes on Spot risk losing HDFS blocks mid-job; keep core nodes on On-Demand for reliability-sensitive pipelines.

For step failures, read the step's stderr log in S3 first — it holds the Spark or Hive exception. And for traceability, remember EMR emits step and cluster state-change events to EventBridge just like Glue, so "alert when the EMR step fails" is again an EventBridge rule to SNS, not a custom polling script.

Once logs exist, four AWS services analyze them, and the right one depends on where the logs live, how often you query, and how fast answers must arrive. CloudWatch Logs Insights queries log groups in place with a purpose-built language — fields, filter, parse, stats — and is the default for interactive troubleshooting: no setup, pay per query by data scanned, results in seconds. Use it to answer "how many timeouts per hour did the pipeline log yesterday" without moving a byte.

Amazon Athena is the answer when logs are in S3 — exported CloudWatch Logs, CloudTrail trails, S3 server access logs, VPC Flow Logs, or application logs a pipeline wrote directly. Define a table (partition it by date to control scan costs, and convert high-volume logs to Parquet if you query them repeatedly) and analyze with standard SQL. Athena is serverless and priced per data scanned, making it the audit workhorse: long-retention log archives queried occasionally.

Amazon EMR earns its place when log analysis becomes a heavy transformation job in its own right — parsing terabytes of raw application logs into structured datasets, sessionizing clickstreams, or running Spark ML over log-derived features. You would not spin up EMR to grep for an error; you would use it when the "analysis" is really batch processing. Amazon OpenSearch Service fits continuous operational search: stream logs in via subscription filters and Firehose, index them, and give operators full-text search and OpenSearch Dashboards with near-real-time latency. The trade-off is a managed cluster (or serverless collections) to size and pay for continuously. Exam shorthand: in place and ad hoc — Logs Insights; in S3 with SQL — Athena; massive parsing jobs — EMR; interactive real-time dashboards — OpenSearch.