DVA-C03 Root Cause Analysis: Lambda and API Errors, Logs Insights, X-Ray, Deployment Logs and AI Troubleshooting

A root cause analysis is the work of tracing a symptom (an error rate, a slow request, a failed deployment) back to the one defect or change that produced it, using the evidence AWS already records: metrics say when and how much, logs say what happened in one request, traces say where the time or error sits across services, and CloudTrail says who changed what. Task 4.1 tests whether you can read that evidence correctly, not whether you can set it up (setting up logging, custom metrics, alarms and tracing is Task 4.2; tuning concurrency, memory and caching is Task 4.3).

A repeatable order works for almost every scenario:

1. Locate the symptom in metrics. Which service, which metric, which statistic, starting when?
2. Narrow with traces. Which hop is slow or failing, and is it an Error, a Fault or a Throttle?
3. Read the evidence in logs. Query the exact failing requests and read the message: timeout, out of memory, AccessDenied, malformed response.
4. Look for a change. A deployment, a configuration update or a policy edit just before the symptom started (CloudTrail, deployment logs, stack events).
5. Fix the cause the evidence names, not a neighbouring setting. A timeout is fixed by the timeout or the slow dependency, an AccessDenied by the policy of the principal the message names.

Many settings sit close to any given symptom, and most of them are real, useful features. A change only fixes the problem when it acts on the mechanism the evidence shows, so always ask which signal a proposed fix would actually move before choosing it.

Each Lambda failure mode writes a distinct signature to the function's CloudWatch Logs log group, and the signature tells you which setting or defect to fix. Every invocation ends with a REPORT line (Duration, Billed Duration, Memory Size, Max Memory Used, and Init Duration on a cold start), so compare it with the error line.

Three distinctions are tested repeatedly. A timeout always names the configured limit; an out-of-memory kill happens before it. A throttled invocation never starts, so it writes nothing to the function's logs. And retries and concurrency do not fix a timeout: every retry or extra copy hits the same limit against the same slow dependency.

An AWS AccessDenied message names four things, and together they tell you exactly which policy to change: the principal that made the call, the action, the resource, and the reason (for example because no identity-based policy allows the action, or with an explicit deny in a named policy type).

Read the principal first. arn:aws:sts::444455556666:assumed-role/invoice-render-role/invoice-render is a session of the role invoice-render-role, which for a Lambda function is its execution role. That role's policy is what must change; the developer's own IAM user, which may work fine from the AWS CLI, is irrelevant because the function signs requests with the role's credentials. When the resource named in the message is one the code only recently started using, check whether the policy's Resource element still lists the resource it used before; the action can be allowed and the call still denied because the ARN does not match.

Know which role each service uses, because the message's principal is easy to misattribute:

• Lambda: the execution role for everything the code calls.
• ECS: the task execution role for pulling images, fetching injected secrets and writing awslogs before the container starts; the task role for calls the running application makes.
• CodeBuild: the project's service role (sessions appear as assumed-role/<role>/AWSCodeBuild-…). Some actions are not tied to a resource, such as ecr:GetAuthorizationToken, which must be granted on Resource "*" in the identity policy; a repository policy cannot grant it.

Encrypted and cross-account access needs two sides. Reading an SSE-KMS object makes S3 call kms:Decrypt on behalf of the requester, so the caller needs KMS permission as well as S3 permission; CloudTrail then shows the denied kms:Decrypt. When the key is in another account, the key policy in that account must allow the external principal and the caller's IAM policy must allow the action on the key ARN. A bucket policy cannot grant KMS permissions, and a key policy cannot grant S3 actions. Similarly, a resource policy written for a service principal (such as lambda.amazonaws.com) does not match a role session.

Separate permission errors from network errors: an explicit AccessDeniedException is a policy problem; a timeout or connection error reaching an endpoint is a network path problem (route, security group or VPC endpoint).

API Gateway's status codes and its two latency metrics locate an integration fault without guesswork.

Latency vs IntegrationLatency. IntegrationLatency is the time API Gateway spends waiting on the backend. Latency is the whole request inside API Gateway. When Latency is far larger than IntegrationLatency, the time is spent in the gateway itself, and the usual suspect is a Lambda authorizer: it runs before the integration, and with an authorization caching TTL of 0 it runs on every request, so its duration and cold starts land directly on clients. The authorizer is a Lambda function with its own metrics and log group, separate from the backend's, and that is where the extra time shows.

Long-running work. When work legitimately takes longer than clients can wait, the fix is architectural: accept the request, start the work asynchronously (for example through a queue), return a job ID and let the client poll. AWS can raise the REST integration timeout for Regional and private APIs through a quota request, but that only makes clients wait longer. Caching helps only for repeated identical requests.

Metrics answer when and how much; the pattern across several metrics, read together, usually points at one cause. The statistic you choose matters as much as the metric.

Percentiles, not averages, for latency. If 1 request in 100 is slow, the Average of Duration barely moves. The p99 statistic shows that tail. Maximum shows only the single worst data point in each period, which is too noisy to describe a tail. Once p99 confirms the slow requests, logs can tell you why (see the Logs Insights section on REPORT lines and @initDuration for cold starts).

Reading Lambda and SQS metrics together:

Throttled requests are rejected, not slow, so a Throttles alarm cannot explain slow requests that eventually succeed. And ConcurrentExecutions measures how many copies run at once, not how long a request takes. Fixing a concurrency limit itself belongs to Task 4.3; here the skill is recognising it from the metrics.

An X-Ray trace shows one request's path across services as segments (one per service) containing subsegments (one per downstream call), each with its timing and outcome, so the slow or failing hop is visible directly.

Error, Fault and Throttle are different. X-Ray classifies responses as:

Reading a timeline. For a Lambda function with active tracing, the function segment holds an Initialization subsegment (present only on a cold start) and an Invocation subsegment for the handler. Inside the invocation, instrumented AWS SDK and HTTP clients add a subsegment per call. Subtract: if an HTTP call to a shipping-rates API takes 1.6 seconds of a 1.8-second invocation, that dependency is where the time goes, and a 30 ms S3 GetObject beside it is not worth tuning. No Initialization subsegment means no cold start. Calls to services that are not themselves instrumented (such as DynamoDB, or an external HTTP API) appear as inferred nodes on the trace map, built from the caller's subsegment.

Finding the right traces: annotations vs metadata. Filter expressions (and X-Ray groups, which are named filter expressions) search built-in keywords such as responsetime, duration, http.status, http.url, user, error, fault and throttle. Of the custom data you attach yourself, they can search only annotations, which are indexed key-value pairs, for example annotation[customer_tier] = "gold" (the current documented syntax uses brackets, which are required when a key contains dots; the older dotted form annotation.key also appears in examples). Metadata is stored with the trace but not indexed, so it cannot be searched. With OpenTelemetry instrumentation (for example the AWS Distro for OpenTelemetry), span attributes become X-Ray metadata by default; a key becomes a searchable annotation only when it is listed in the span's aws.xray.annotations attribute. If a filter on a field returns nothing while the trace details show the field under metadata, fix the instrumentation for future traces; changing the expression, the sampling rate or a group does not index a field. The X-Ray SDKs and daemon entered maintenance mode in February 2026, and AWS recommends OpenTelemetry for new instrumentation (how to instrument is covered in the Task 4.2 lesson).

CloudWatch Logs Insights runs a pipeline of commands over one or more log groups, left to right, so command order decides the result: each command sees only what the previous one produced. The core commands are fields (choose fields), filter (keep matching events), parse (extract fields from text), stats (aggregate), sort and limit.

filter @message like /Exception/
| stats count(*) as exceptions by bin(1h)

This counts exception events per hour, and a time-binned count like this shows when a spike began. Grouping by a field such as @logStream counts per execution environment instead of per time bucket, and sort @timestamp desc | limit N lists individual events without counting them. After stats, only the aggregated fields remain, so a filter on a raw field placed after stats no longer works.

Plain-text logs need parse. Logs Insights discovers fields automatically for JSON logs and for some AWS log types, but a plain-text line such as user=u-19 action=checkout result=DECLINED ms=870 has no result field until you create one. In glob mode, each * is captured into a name listed after as, and the number of names must match the number of wildcards:

parse @message "user=* action=* result=* ms=*" as user, action, result, ms
| filter result = "DECLINED"
| stats count(*) by action

A filter on result placed before the parse matches nothing, because the field does not exist yet. parse also supports regular expressions with named capture groups.

Lambda REPORT lines. For Lambda log groups in the default plain-text format, Logs Insights discovers fields from the REPORT line: @type, @requestId, @duration, @billedDuration, @memorySize, @maxMemoryUsed and, only on cold starts, @initDuration. START and END lines carry no memory data, and @memorySize is the configured setting, the same for every invocation of a version, so it cannot rank usage.

filter @type = "REPORT"
| fields @requestId, @duration, @maxMemoryUsed
| sort @duration desc
| limit 10

filter @type = "REPORT" and ispresent(@initDuration)
| stats count(*) as coldStarts, max(@initDuration) by bin(1h)

The first lists the 10 slowest invocations with their request IDs and memory use (sorting on @maxMemoryUsed instead ranks them by memory); the second counts cold starts per hour so you can relate slow periods to them. Binned stats lose the request IDs, so use fields and sort when you need individual invocations. Logs Insights can also be written in OpenSearch PPL or SQL, and the pattern command groups similar log lines into recurring patterns.

The right tool depends on where the data lives and whether you need history or a live view. Logs Insights cannot read files in S3, and CloudTrail does not record application traffic.

Other features in the same family are not query tools: a metric filter turns matching events into a metric count, a subscription filter forwards events to another destination such as Amazon Data Firehose, and Contributor Insights ranks top contributors with a rule you write. Amazon OpenSearch Service can search logs only after they are ingested into a domain.

CloudTrail for change investigation. CloudTrail event history shows the last 90 days of management events in each Region with no setup, including userIdentity, eventTime, source IP and request parameters. So a changed Lambda environment variable (an UpdateFunctionConfiguration call) can be traced to its caller even in an account with no trails. Function logs and traces never record configuration changes. Two limits matter:

• Data events (object-level operations such as S3 GetObject/DeleteObject, Lambda Invoke, DynamoDB item operations) are never in event history; they are recorded only by a trail or event data store configured to capture them, from that point on.
• S3 Lifecycle actions (expirations and transitions) are carried out by S3 itself rather than requested through an API call by a principal, so CloudTrail does not record them as DeleteObject events, even with data events on. S3 server access logs do record lifecycle actions.

CloudTrail Lake closed to new customers on 31 May 2026 (existing event data stores keep working), so it is not the choice for a new account; AWS points new users to CloudWatch.

Deployment tools log each phase, lifecycle event or resource operation, and the first real failure plus the output of its failing command or its reason text is the root cause. Later entries are usually consequences of it.

AWS CodeBuild

The build log (in CloudWatch Logs and/or S3) shows each command as [Container] Running command …, its output, and phase transitions such as Phase complete: PRE_BUILD State: FAILED. Read the command output just above the failed phase:

• An AccessDeniedException naming assumed-role/<service-role>/AWSCodeBuild-… is a missing permission on the project's service role, not the pipeline's role. Pushing to Amazon ECR needs ecr:GetAuthorizationToken on Resource "*" plus repository actions such as ecr:PutImage.
• An error that the build cannot connect to the Docker daemon means privileged mode is off for a build that runs Docker commands; that is a different message from an IAM denial.

Phase behaviour after a failure. When a command in the build phase fails, the build is marked as failing but the post_build phase still runs. A docker push in post_build can therefore publish an image whose tests failed. Two buildspec fixes stop that:

• Check the environment variable CODEBUILD_BUILD_SUCCEEDING (1 while succeeding, 0 once failing) before pushing.
• Set on-failure: ABORT on the build phase so the build ends there (on-failure applies to EC2 compute; CONTINUE keeps going to the next phase, which is the problem behaviour).

Commands in a phase's finally block run even when a command in that phase fails, so moving a push there does not protect anything.

AWS CodeDeploy

Where to look depends on the compute platform:

SNS triggers, EventBridge deployment state-change events and CloudTrail tell you that a deployment failed or who started it, not what a script printed.

AWS CloudFormation stack events

The root cause of a failed stack operation is the earliest failure event whose reason is not a cancellation. When one resource fails during creation, CloudFormation cancels the resources still in progress and logs each as CREATE_FAILED with Resource creation cancelled. Those newest entries are noise; scroll back to the first real failure.

cancel-update-stack works only while an update is in progress. Termination protection and stack policies protect a stack from deletion by users and from updates; neither stops a failed create from rolling back.

Amazon ECS stopped tasks

When an ECS deployment keeps replacing tasks, the stopped reason of a stopped task (console or describe-tasks) names the stage that failed:

• CannotPullContainerError: the image could not be pulled (wrong name or tag, missing permission, or no network path to the registry).
• ResourceInitializationError: setup failed before the container started. Read the rest of the message. An AccessDeniedException on secretsmanager:GetSecretValue means the task execution role lacks access to a secret referenced in the container definition (plus kms:Decrypt if the secret uses a customer managed key); the task role is only for the running application. An i/o timeout reaching the ECR API means no network path. In private subnets with no internet route, Fargate platform version 1.4.0 or later needs interface endpoints for ecr.api and ecr.dkr and a gateway endpoint for s3 (image layers come from S3), plus logs for the awslogs driver. The ECS interface endpoints serve the agent on EC2 container instances, not Fargate image pulls.
• Essential container in task exited: the container started and its main process ended. Read the container's exitCode and reason in the stopped task details (an out-of-memory kill shows in the reason) and the last lines of its log stream in CloudWatch Logs (stack trace, missing setting).

CloudTrail does not explain these: ECS stops such tasks itself, and CloudTrail records API calls, not why a process exited. VPC Flow Logs and utilisation charts describe packets and load, not the cause.

DVA-C03 adds AI-assisted troubleshooting: tools that gather telemetry, write queries or read your code and errors, then propose explanations and fixes that you review before acting on. Three are worth knowing.

CloudWatch investigations is a generative AI feature of Amazon CloudWatch. You start an investigation from an alarm, a metric or a Logs Insights query result. It scans related telemetry (metrics, logs, deployment events, AWS Health events, CloudTrail change events and X-Ray traces), shows observations and proposes root-cause hypotheses. It differs from single-signal features: anomaly detection models one metric's expected band, Logs Insights pattern analysis groups similar log lines, and a Synthetics canary confirms an endpoint is failing; none of them reasons across sources. How it is set up matters:

Natural-language query generation in Logs Insights. The query editor's Generate query option turns a plain-English prompt ("find the 10 slowest requests") into a Logs Insights QL, OpenSearch PPL or SQL query, explains it line by line and refines it from a follow-up prompt. It is authorized by cloudwatch:GenerateQuery, a separate action from logs:StartQuery and logs:GetQueryResults; it is included in the CloudWatchLogsReadOnlyAccess and CloudWatchLogsFullAccess managed policies. For least privilege, add only that action to a custom policy. The caller does not invoke Amazon Bedrock directly. Read a generated query before trusting its results.

Kiro is an AI IDE whose chat can analyse failures when you attach context with # providers instead of pasting text:

Providers can be combined in one message: pairing the evidence of a failure with the change that may have caused it gives the chat both halves of the analysis. Review every suggested fix before applying it. Using AI tools to find optimization opportunities belongs to Task 4.3.