DVA-C03 Logging, Monitoring and Observability: CloudWatch, EMF, Alarms, X-Ray and Health Checks

Effective application logging means writing structured entries (JSON objects with consistent field names) that record what the application did and the state it was in, so the entries can be filtered, counted and joined later without parsing free text. In AWS Lambda the quickest route is the function's advanced logging controls, which need no code change for supported runtimes and their built-in logging methods (for example Python's logging library or Node.js console methods).

• Log format: Text (the default) or JSON. With JSON, Lambda writes each application entry as a JSON object with fields such as timestamp, level, message and requestId.
• Application log level: filters the entries your code writes (TRACE, DEBUG, INFO, WARN, ERROR, FATAL). Setting it to WARN drops TRACE, DEBUG and INFO entries. Lambda applies this filter only when the function uses the JSON log format; with Text format, application entries are not filtered by level.
• System log level: filters the platform records Lambda itself writes (START, END, REPORT and similar). It does not affect application entries.

Correlation IDs: following one request across functions

To follow one business request through several components, every entry must carry a field that is the same in all of them. Lambda's requestId does not do that job: it is generated for each invocation, so a producer function and the consumer that later processes the same order have different values, and it is also different from the API Gateway request ID that a client sees in an error response.

• Log business keys (order ID, tenant ID, user action) and a correlation ID as top-level JSON keys, and pass the correlation ID along in messages (for example as an SQS message attribute or a field in the body).
• Powertools for AWS Lambda Logger produces structured JSON and can set a correlation_id on every entry from a path in the event, such as correlation_paths.API_GATEWAY_REST (the API Gateway request ID).
• CloudWatch Logs Insights discovers fields in JSON events automatically, so a query can filter on orderId or correlation_id across several log groups without parse expressions. Writing those queries during an investigation is covered in the root cause analysis lesson (Task 4.1).

Log stream names identify execution environments, not requests, and should not be used to group a request's entries. Never log secrets or credentials; protect personal data with the controls in the next section.

Every log group should have a retention period, the right destination for how the logs are used, and a data protection policy if they can contain personal data. Log groups that Lambda creates default to never expire, which is why storage costs grow every month; setting retention (for example 90 days) makes CloudWatch Logs delete older events automatically with no code or scheduled jobs.

Where Lambda can send its logs

Writing logs to S3 from the function code with PutObject adds latency and code, and a subscription filter to your own Firehose stream adds a stream to operate.

Subscription filters stream copies

A subscription filter sends matching log events, as they arrive, to Kinesis Data Streams, Amazon Data Firehose or a Lambda function. It delivers copies: it never removes or changes what is stored in the log group, and it does not process events that were ingested before it existed.

Masking sensitive data

A CloudWatch Logs data protection policy, attached to a log group or to the account, uses managed data identifiers (email addresses, credit card numbers, credentials and many others) to find sensitive data. Matching values are masked when people view the events, and only principals allowed logs:Unmask can see the original values. This needs no application change. Encrypting the log group with an AWS KMS key is a different control: it decides whether a principal can read the log group at all, not which fields it sees. Metric filters and subscription filters cannot redact stored events.

Code can publish custom metrics to CloudWatch in two ways: call the PutMetricData API, or write log entries in CloudWatch embedded metric format (EMF) and let CloudWatch Logs extract the metrics. For high-volume code such as a busy Lambda function, EMF is usually the better choice.

Libraries such as Powertools for AWS Lambda Metrics and the EMF client libraries write the format for you.

Dimensions define the metric

CloudWatch identifies a metric by its namespace, name and the full set of dimensions. Two consequences matter for developers:

• No cross-dimension aggregation for custom metrics. If a value is published only with the dimensions Environment and Endpoint, CloudWatch stores it only under that combination; asking for the metric by Environment alone returns no data, and an alarm defined that way stays in INSUFFICIENT_DATA. Every aggregate you want to graph or alarm on must be published as its own dimension set (one EMF event can declare several dimension sets), or computed with metric math over the published metrics.
• Every unique combination of dimension values is a separate, billed custom metric. A high-cardinality dimension such as an order ID or user ID creates a new metric for every value. Keep such identifiers as ordinary top-level fields in the EMF event: fields not declared as dimensions or metrics stay in the log and can still be filtered in Logs Insights.

Resolution and fast alarms

Custom metrics are standard resolution (one-minute granularity) unless published with StorageResolution set to 1, which makes them high resolution with one-second data. Resolution is decided at publish time, not by how often values are sent. Alarms on high-resolution metrics can use periods of 10, 20 or 30 seconds (or any multiple of 60); a sub-minute period makes it a high-resolution alarm, which costs more. Fast detection needs both the high-resolution metric and the short alarm period.

For built-in aggregates without custom code, CloudWatch Lambda Insights and Container Insights collect runtime and container metrics, but they do not replace business metrics such as basket size per sales channel.

A metric filter turns matching log events into a CloudWatch metric, which is how you count or alarm on something the application already logs without changing its code. For JSON logs the filter pattern selects a field, for example { $.level = "ERROR" }; each match publishes a value (often 1, or a number taken from the event), and an alarm on the Sum over a period gives a count-based alert. The selector must name the JSON property that actually holds the value being tested, using the same spelling and nesting as the log event (for example $.http.status for a nested status code).

• Metric filters only see new events. They evaluate events as they are ingested and never backfill data that was already in the log group. For a one-time historical count, run a CloudWatch Logs Insights query.
• Sparse events need a default value. When nothing matches in a period, a filter publishes nothing, so an alarm on a rare error sees missing data. Setting the filter's default value to 0 publishes a zero for periods with log activity but no match; alternatively treat missing data as not breaching on the alarm.

A useful alarm evaluates the right metric, treats missing data deliberately, combines signals where one alone is noisy, and can actually deliver to its Amazon SNS topic. Alarm states are OK, ALARM and INSUFFICIENT_DATA, and the alarm action (usually an SNS topic in the same Region as the alarm) runs on state changes.

Missing data treatment

Composite alarms

A composite alarm has a rule expression over the states of other alarms, joined with AND, OR and NOT. It reduces alarm noise by notifying only when a combination of conditions holds, for example a queue backlog alarm AND a consumer-error alarm. The notification action goes on the composite alarm; the underlying alarms usually have no actions of their own but keep evaluating, so they still show on dashboards and in alarm history. A single metric alarm evaluates one metric or one metric math expression, so a rule over the states of several alarms needs a composite alarm.

Encrypted SNS topics

When an SNS topic uses server-side encryption, the publishing service must be allowed to use the KMS key. The key policy of the AWS managed key aws/sns cannot be edited, so CloudWatch (cloudwatch.amazonaws.com) and EventBridge (events.amazonaws.com) cannot publish to a topic encrypted with it. Encrypt the topic with a customer managed key whose key policy allows those service principals kms:GenerateDataKey* and kms:Decrypt, in addition to the topic policy allowing sns:Publish. Alarms do not run under an execution role of their own.

Alerts for specific actions are built from two AWS mechanisms: CloudWatch alarms on usage metrics for quota limits, and Amazon EventBridge rules on state-change events for things that happen, such as a deployment finishing. Both usually target an SNS topic for email or chat delivery.

Quota limits

• For quotas that support it, the Service Quotas console creates a CloudWatch alarm on the quota's utilization at a threshold you choose (for example 75% of the running On-Demand EC2 instance vCPU quota); add an SNS action to get notified before throttling starts.
• In templates, use metric math on a usage metric from the AWS/Usage namespace: (m1 / SERVICE_QUOTA(m1)) * 100 gives utilization as a percentage. Because SERVICE_QUOTA() returns the currently applied quota, a percentage threshold keeps its meaning after the quota is raised.
• AWS Budgets alerts track spend, not quotas, and throttling errors only show up after the limit has been reached.

Deployment completions and failures

For ECS the two pieces depend on each other: without the circuit breaker a failing rolling deployment keeps trying to launch tasks and no deployment-failed event is sent. How the deployments themselves are configured is covered in the CI/CD lesson (Task 3.3).

Tracing records the path and timing of an individual request through each service it touches, which aggregate metrics and per-service logs cannot show. AWS X-Ray stores traces made of segments (one per service) and subsegments or child spans (for example a downstream AWS SDK call), and shows them in the X-Ray and CloudWatch trace views.

Lambda sends its segments with the execution role; X-Ray never assumes that role, so no trust-policy change is needed. PassThrough mode only traces requests that an upstream service has already sampled; Active makes Lambda sample and trace invocations itself. To see time spent inside your code's downstream calls, instrument the code so AWS SDK and HTTP calls produce subsegments or child spans (see the OpenTelemetry section).

Annotations are indexed key-value pairs that make traces searchable; metadata is unindexed context attached to a trace; sampling rules decide which requests are traced.

With OpenTelemetry instrumentation, span attributes are stored as X-Ray metadata unless their keys are configured to be converted to annotations. X-Ray filter expressions search what X-Ray indexes in the traces themselves (annotations, service names, response times, HTTP status and similar); data held elsewhere, such as in logs, is searched with the tool that stores it.

Centralized sampling rules

X-Ray does not trace every request by default. Centralized sampling rules are defined in X-Ray and fetched by the instrumentation (SDK, ADOT, CloudWatch agent or collector), so they change without redeploying code. Each rule matches requests (service name, HTTP method, URL path and so on) and has:

• a priority: rules are evaluated from the lowest number up, and the default rule always comes last;
• a reservoir: the number of matching requests traced each second;
• a fixed rate: the share of matching requests beyond the reservoir that is traced.

The default rule traces the first request each second and 5% of the rest. To trace every request on one path while leaving others at the default, add a higher-priority rule for that path with a fixed rate of 1.0. Raising the default rule's rate affects every path, and a reservoir smaller than the traffic still leaves gaps. Local sampling rules files ship with the application and need a redeploy to change.

For new and existing applications, AWS now recommends instrumenting with OpenTelemetry: an OpenTelemetry SDK or the AWS Distro for OpenTelemetry (ADOT) in the code, sending OTLP spans to the CloudWatch agent or an OpenTelemetry collector, which exports them to X-Ray. The X-Ray SDKs and the X-Ray daemon entered maintenance mode on 25 February 2026 (security fixes only). X-Ray as a service, its trace views, sampling rules, annotations, and Lambda and API Gateway tracing remain current; only the X-Ray-specific instrumentation libraries and daemon are being superseded.

• On Amazon ECS (including AWS Fargate), run the CloudWatch agent or an OpenTelemetry collector as a sidecar container in the task, and point the application's exporter at it.
• Auto-instrumentation in OpenTelemetry SDKs records AWS SDK and HTTP calls as child spans, which become subsegments in X-Ray.
• The X-Ray daemon receives X-Ray segment documents over UDP and does not accept OTLP; OpenTelemetry exporters send to the CloudWatch agent or a collector instead.
• Container Insights collects container metrics and logs; it does not trace requests.

Which role needs the X-Ray permissions

Every AWS API call made by code running in the task, including a sidecar, is authorized against the task role, so permissions an application or agent needs at run time belong on that role.

Workload health is monitored at three levels: load balancer health checks test each target from the network, container and pod self-checks (ECS container health checks, Kubernetes probes) test the process from inside, and outside-in checks (CloudWatch Synthetics canaries, Route 53 health checks) test what users actually reach.

Load balancer health checks

Container and pod self-checks

An ECS container healthCheck in the task definition is a command run inside the container, with interval, timeout, retries and startPeriod (time during which failures do not count). Repeated failures mark the container UNHEALTHY, and a service replaces tasks whose essential container is unhealthy. Because it runs a local command, it can detect a process that is still running but no longer doing work, and it needs no load balancer.

Kubernetes, including Amazon EKS, has three probe types:

Putting a shared dependency, such as a database, in the liveness check turns a short database failover into a restart of every pod and a long recovery. For a slow start, a startup probe whose failureThreshold × periodSeconds covers the longest start lets the liveness probe keep tight settings once the application is running.

Outside-in checks

• CloudWatch Synthetics canaries are scripts that run on a schedule and can drive a headless browser through multi-step flows (for example search a catalogue, then submit a contact form), publishing success and duration metrics you can alarm on. Because canaries generate their own traffic, they test availability whether or not users are active.
• Route 53 health checks test a single endpoint from several locations. Route 53 publishes their metrics (such as HealthCheckStatus) to CloudWatch in US East (N. Virginia) only, so the alarm must be created there, and its SNS action must use a topic in that Region too.
• CloudWatch RUM collects performance and error data from real users' browser sessions.