Troubleshooting and Optimization practice questions

An AccessDenied message names the principal, the action, the resource and the type of policy that is missing.

The principal is assumed-role/orders-fn-role, which is the function's execution role, and the message says no identity-based policy allows dynamodb:PutItem on Orders-v2. After the refactor the role's policy most likely still lists the old table ARN. Adding the new ARN to the role fixes it. The developer's own permissions do not matter, because Lambda uses the execution role's credentials. A resource policy for the Lambda service principal would not match a role session.

1. Read the principal in the error: it is the execution role session.
2. Read the action and resource: PutItem on the new table ARN.
3. Note the reason: no identity-based policy allows it.
4. Add the new table ARN to the execution role's policy.

Exam tip: Fix the policy of the principal named in the AccessDenied message, which for Lambda is the execution role.

Using a KMS key across accounts requires permission on both sides: the key policy and the caller's IAM policy.

S3 decrypts SSE-KMS objects by calling kms:Decrypt for the requester, so the role needs KMS permission as well as S3 permission. Across accounts, a key is usable only when its key policy (in account B) allows the external principal and the principal's IAM policy (in account A) allows the action on the key ARN. Either one alone fails. A bucket policy cannot grant KMS permissions, and switching default encryption leaves existing objects encrypted with the KMS key.

1. Note that the denied call is kms:Decrypt, not s3:GetObject.
2. Recall that cross-account KMS access needs the key policy and the caller's IAM policy.
3. Add the role to the key policy in the key's account.
4. Add kms:Decrypt on the key ARN to the role's policy.

Exam tip: Cross-account KMS access: the key policy must allow the principal and the principal's IAM policy must allow the action.

With Lambda proxy integration, API Gateway expects a specific response shape from the function.

In a Lambda proxy integration, the function must return an object with statusCode, optional headers and multiValueHeaders, and body as a string. A bare object such as {"orderId": "123"} is malformed, so API Gateway returns 502 Bad Gateway even though the function succeeded. A timeout would return 504, and a missing invoke permission would stop the function from running at all.

1. The function runs and succeeds, so look at what it returns.
2. Recall the proxy response shape: statusCode, headers, string body.
3. Rule out a timeout, which would be 504.
4. Wrap the result in the proxy response format and serialize the body.

Exam tip: A 502 with a successful Lambda run usually means a malformed proxy response.

API Gateway stops waiting at its integration timeout no matter how long the backend is allowed to run.

The 504s line up with IntegrationLatency at 29,000 ms, the default integration timeout for REST APIs. The backend sometimes needs 45 seconds, so the gateway gives up first. Raising the Lambda timeout does not change the gateway's limit, and throttling would return 429 instead. With clients needing an answer in seconds, the fix is to start the work asynchronously (for example by queueing it) and return a job ID to poll. AWS can raise the REST integration timeout for Regional and private APIs by quota request, but responses would still be slow.

1. Match the 504s to IntegrationLatency at the 29-second ceiling.
2. Note that the backend needs longer than that.
3. Apply the constraint: responses within a few seconds.
4. Move the long work to an asynchronous job pattern.

Exam tip: 504 with IntegrationLatency near 29 seconds means the backend is slower than the API Gateway integration timeout.

Several metrics read together show whether a backlog comes from errors, retries or a capacity limit.

ApproximateAgeOfOldestMessage shows the backlog. Errors at zero rules out failing batches, and Throttles rising while ConcurrentExecutions stays exactly at 20 shows the function hitting a hard ceiling: reserved concurrency of 20. Raising or removing the reservation lets the poller scale. An event source mapping's maximum concurrency setting would also hold concurrency flat, but it limits the pollers so the function is not throttled, and the Throttles metric would not climb. Visibility timeout problems cause duplicates, and retention problems would remove old messages, which lowers the age metric.

1. Confirm the backlog with ApproximateAgeOfOldestMessage.
2. Use Errors = 0 to rule out processing failures.
3. Read the flat ConcurrentExecutions line together with rising Throttles as a reserved concurrency cap.
4. Check the function's reserved concurrency and raise or remove it.

Exam tip: Flat concurrency with rising throttles and no errors means a concurrency cap, not a code defect.

Averages hide tail latency. Percentiles reveal it, and Lambda REPORT lines show which invocations had a cold start.

When 1% of requests are slow, the Average barely moves. The p99 statistic of Duration shows the slow tail. To link that tail to cold starts, a Logs Insights query such as filter @type = "REPORT" | stats count(*), avg(@initDuration) by bin(5m) works, because Init Duration is written only for invocations that started a new execution environment. Maximum shows one worst case, Throttles are rejected rather than slow, and these requests finish, so they never log a timeout.

1. Recognise that a 1% tail does not show in an average.
2. Switch the Duration statistic to p99 to confirm the slow tail.
3. Recall that REPORT lines carry Init Duration only on cold starts.
4. Query @initDuration to see whether slow requests line up with cold starts.

Exam tip: Use percentile statistics for tail latency and REPORT @initDuration to confirm cold starts.