Root Cause Analysis on AWS: Reading Logs, Metrics, and Error Codes

Every root cause analysis starts by choosing the right signal, and each of the three answers a different question. Metrics answer "is something wrong?" — they are numeric time series (invocation counts, error rates, latency percentiles) that are cheap to store, easy to graph, and the only signal you can alarm on. Logs answer "why is it wrong?" — discrete, timestamped events carrying full detail: stack traces, request fragments, the branch the code actually took. Traces answer "where is it wrong?" — they follow a single request across service boundaries and show which hop contributed the latency or threw the error.

On AWS these map to concrete services. Amazon CloudWatch collects metrics in namespaces with dimensions; AWS Lambda publishes Invocations, Errors, Throttles, and Duration automatically. CloudWatch Logs stores log events in log groups (one per function or application) divided into log streams. AWS X-Ray stores traces assembled from the segments each instrumented service records.

The exam expects you to run the diagnostic loop in that order: a metric alarm tells you something broke and when it started; a Logs Insights query over the same time window tells you what the code reported; a trace pins the failure to one service in the chain. Producing this telemetry well is its own skill — in a root cause analysis your job is to consume the signals that exist and name the defect they point at.

CloudWatch Logs Insights is a purpose-built query language for log groups, and the exam expects you to read and assemble its five core commands. fields selects what to display, filter keeps only matching events, parse extracts ad hoc fields from unstructured text, stats aggregates (count(), avg(), max(), grouped with by), and sort orders results. Commands chain with the pipe character, and system fields such as @timestamp, @message, and @logStream are always available.

A worked example against a Lambda log group, confirming a timeout hypothesis:

fields @timestamp, @requestId, @message
| filter @message like /Task timed out/
| sort @timestamp desc
| limit 20

Lambda also writes a REPORT line for every invocation with auto-discovered fields, so you can profile duration and memory without touching code:

filter @type = "REPORT"
| stats avg(@duration), max(@duration),
        max(@maxMemoryUsed / 1000 / 1000) by bin(5m)

When logs are unstructured, parse rescues you: parse @message "order=* status=*" as orderId, status creates two ephemeral fields you can filter and aggregate on. Note the direction of that trade — parsing at query time works, but it is fragile compared with logs structured at write time.

When the exam asks how to find "all errors for a given order ID in the last hour," a Logs Insights query with filter (and parse if the logs are free-form) is the intended answer — not exporting log files or scanning streams by hand.

API-facing failures usually reach you as an HTTP status code, and the exam loves asking which layer produced it. Learn this table as a first-look diagnostic map for Amazon API Gateway fronting Lambda or another integration.

Two pairs are deliberately confusable. 403 versus 429: candidates guess throttling for both, but throttling and quota exhaustion return 429, while 403 is always an authorization-shaped problem — credentials, policies, API keys, or WAF. And 502 versus 504: a 504 means the backend never answered in time (for example, a Lambda function that runs past the 29-second integration limit even though its own timeout allows more), while a 502 means the backend answered with something API Gateway could not use.

Here is the classic scenario, worth internalizing end to end. Your REST API returns intermittent 502 Bad Gateway responses. You start with metrics: the API stage's 5XXError metric confirms the spikes, but the Lambda function's Errors metric is flat at zero and Duration is normal. The function is not failing — so why is the gateway reporting a bad response from it?

Query the function's log group in Logs Insights for the failing request IDs and look at what the code returned. The defect: one code path returns its result object directly — { "status": "ok", "items": [...] } — instead of the envelope the Lambda proxy integration contract requires. With proxy integration, API Gateway does not transform anything: your function must return an object with a numeric statusCode, an optional headers map, and a body that is a string (call JSON.stringify on your payload), plus isBase64Encoded for binary content. Return a raw object, omit statusCode, or pass a JSON object as body, and API Gateway cannot map it to an HTTP response — it emits 502 with "Internal server error" while the function itself completed successfully.

The fix is one line: serialize the body and return the full envelope on every code path. The exam trigger phrasing is direct: "API Gateway returns 502 but the Lambda function shows no errors" points at a malformed proxy response. An unhandled exception in the function surfaces the same way, because the error payload Lambda returns is not a valid proxy envelope either.

Application logs full of SDK exceptions are an exam staple, and each exception name encodes its own root cause. ThrottlingException — and its DynamoDB-specific cousin ProvisionedThroughputExceededException — means you are calling faster than the service or table allows. The immediate mitigation is retries with exponential backoff and jitter, behavior every AWS SDK builds in by default, which is why the exam answer is usually "let the SDK retry, then fix the cause" rather than hand-writing retry loops. The durable fix is capacity: raise provisioned throughput, switch to on-demand, or redesign a hot partition key.

AccessDenied and AccessDeniedException are never capacity problems: some policy in the chain — the caller's IAM identity policy, a resource policy, a KMS key policy, a permissions boundary — refuses the action. Read the error message, which names the principal and the denied action, and fix the policy rather than retrying; a denied call stays denied.

ConditionalCheckFailedException comes from Amazon DynamoDB when a write's condition expression evaluates false — an optimistic-locking update requiring version = :expected after another writer got there first, or an attribute_not_exists guard rejecting a duplicate insert. It is not an outage and not blindly retryable: it is your own guard doing its job, and the correct response is application logic — re-read the item, reconcile, then decide whether to reapply the write with fresh state.

The general split to carry into the exam: throttling and 5xx errors are transient and retryable; 4xx-shaped errors — access denials, validation failures, failed condition checks — are deterministic and need a code, policy, or data fix.

When built-in metrics cannot answer your question — checkout failures per payment provider, cache hit ratio, orders per tenant — you publish custom metrics, and CloudWatch embedded metric format (EMF) is the way to do it from high-volume code. EMF is a JSON log convention: instead of calling a metrics API, your code writes a structured log line whose _aws metadata block declares which fields are metrics, in which namespace, with which dimensions. CloudWatch Logs recognizes the format and extracts real metrics from it asynchronously.

The advantages are exactly what the exam probes. Publication rides your existing logging path, so there is no PutMetricData call in the hot path — no API quota to throttle you, no added request latency, no batching code to maintain. And because every datapoint originates from a log event, the full request context stays queryable in Logs Insights alongside the metric, turning "this metric spiked" into "these specific requests made it spike" in one step. On Lambda, printing an EMF line to standard output is all it takes.

A minimal example emitting one metric with one dimension:

{
  "_aws": { "Timestamp": 1700000000000, "CloudWatchMetrics": [{
    "Namespace": "OrderService",
    "Dimensions": [["PaymentProvider"]],
    "Metrics": [{ "Name": "CheckoutFailure", "Unit": "Count" }] }] },
  "PaymentProvider": "cardco",
  "CheckoutFailure": 1,
  "orderId": "o-8817"
}

Once extracted, the metric behaves like any other: graph it on dashboards, alarm on it, break it down by dimension. The trigger phrase to remember: "publish custom metrics from Lambda without throttling or added latency" means embedded metric format.

Reviewing application health starts wide before it goes deep. A CloudWatch dashboard is the wide view: one page of the metrics that define "healthy" for your workload — API error rates and latency percentiles, Lambda throttles and duration, DynamoDB throttled requests, queue depth — so an on-call developer sees at a glance which layer is misbehaving before opening a single log. Alarm status widgets on the same dashboard show what has already crossed a threshold.

CloudWatch ServiceLens tightens the loop by correlating all three signals in one place: it overlays metrics, logs, and X-Ray traces on a service map, so from a spiking error rate you pivot straight to the traces and log events behind it without hopping between consoles.

The X-Ray service map itself is the exam's favorite "where" tool. Each node is a service — an API stage, a Lambda function, a DynamoDB table, an external HTTP endpoint — and each edge is the calls between two of them. Nodes and edges carry request rates, error and fault percentages, and latency distributions, and unhealthy nodes are visually highlighted. When a distributed application is slow or erroring and the question is which hop is responsible, you open the service map, find the node where errors or latency concentrate, and drill into its traces, where individual subsegments show exactly which downstream call consumed the time.

Trigger phrasing: "identify which service in a microservices chain is causing the latency" is the X-Ray service map — not a dashboard, and not a Logs Insights query.

Troubleshooting a failed deployment means reading the deploying service's own output, because each tool records exactly where it stopped. AWS CloudFormation records stack events: find the first CREATE_FAILED or UPDATE_FAILED event — later failures are usually rollback noise — and read its status reason: a permission denial, an invalid property value, a resource limit. AWS CodeBuild streams every build to CloudWatch Logs and reports which buildspec phase failed (install, pre_build, build, post_build), so a dependency-resolution failure and a failing unit test land in different, clearly labeled phases. AWS CodeDeploy runs your lifecycle event hooks (such as BeforeInstall, ApplicationStart, ValidateService); a deployment that dies in a hook records the failed script, its exit code, and its output in the deployment's event details and the agent's logs on the instance.

Service integration failures are the cases where every component is individually healthy but the wiring between them is not — and they classify three ways. Permission failures: an EventBridge rule matches but its target Lambda function was never granted resource-based permission to be invoked, or a function's execution role lacks the SQS actions its event source mapping needs to poll the queue. Configuration failures: the event source mapping or the rule is disabled, the batch size is inappropriate, or an event pattern silently mismatches the real event shape — a classic is matching a literal value the producer capitalizes differently. Payload-shape failures: the event arrives but the consumer expects different fields. When messages pile up in SQS and Lambda never fires, check the event source mapping's state and the execution role's queue permissions before you touch the function code.