Instrument Code for Observability: CloudWatch, X-Ray, and Alarms

These three terms form a ladder, and the exam tests the rungs directly. Logging is the act of recording discrete events as they happen — a request arrived, a payment failed, an exception was thrown. A log is raw material: it states what occurred at a point in time, with whatever context the code chose to include.

Monitoring is watching known signals against expectations — collecting metrics, graphing them, and alarming when a threshold is crossed. Monitoring answers questions you knew to ask in advance: is the error rate acceptable, is p99 latency in bounds, is the queue draining. It is reactive by design and blind to anything you did not think to measure.

Observability is a property of the system rather than an activity: a system is observable when its outputs — logs, metrics, and traces together — are rich enough that you can work out its internal state and diagnose problems you never anticipated, without shipping new instrumentation first. Monitoring tells you a known symptom appeared; observability lets you ask brand-new questions of telemetry you already have. Those three signal types are commonly called the three pillars of observability, and instrumenting all of them from code is what this task is about.

The exam phrasing to recognize: "known issues and predefined thresholds" describes monitoring; "understand internal state from external outputs" and "investigate previously unknown issues" describe observability; "a record of discrete events" describes logging. Questions often present all three as options and reward the candidate who keeps the definitions straight.

An effective logging strategy decides, before any incident, what gets recorded, at what level, and for how long. What to log: state transitions (order created, payment authorized, message rejected), the identifiers needed to follow one transaction (request ID, order ID, user ID), decision outcomes with the inputs that drove them, and every error with enough context to reproduce it. Log at boundaries — incoming requests, outgoing calls, queue interactions — because boundaries are where failures get diagnosed.

What never to log: secrets, API keys, session and bearer tokens, passwords, full payment card numbers, and personally identifiable information beyond what your data policies allow. Logs fan out to many systems and many readers; a credential in a log line is a credential leaked, and scrubbing after the fact is unreliable. Sanitize at the logging call site, not downstream.

Levels: use the conventional ladder — DEBUG for high-volume diagnostic detail, INFO for normal state transitions, WARN for handled anomalies, ERROR for failures needing attention — and make the active level configurable through an environment variable, so production runs at INFO and can be dropped to DEBUG during an investigation without a code change or redeploy of new logic.

Retention: CloudWatch Logs groups keep events indefinitely by default, which quietly becomes a storage bill. Set a retention policy on each log group that matches the data's diagnostic and compliance value. Retention is configuration on the log group, not something application code manages.

Structured logging means emitting each log event as a machine-parseable record — in practice, one JSON object per line with consistent field names — instead of free-form text. The payoff is queryability: CloudWatch Logs Insights automatically discovers JSON fields, so filter level = "ERROR" and orderId = "o-8817" just works, with no brittle parse patterns over hand-formatted strings. Consistent fields across services (level, service, requestId, durationMs) mean one query style and one mental model for the whole system. Version 2.1 of the exam guide calls this skill out explicitly, so expect it to appear.

The keystone field is the correlation ID: one identifier minted at the edge of the system — commonly the API Gateway request ID, or an ID your first service generates — and passed along with the work through every hop: HTTP headers to downstream services, message attributes on SQS messages, the detail of EventBridge events. Every service writes it into every structured log line it emits for that transaction. The result is that a single Logs Insights query across log groups reassembles the complete story of one transaction, no matter how many services touched it. Without propagation, each service's logs are an island and cross-service debugging becomes guesswork.

You can implement this by hand with a logging wrapper that merges standard fields into every record, but on Lambda the idiomatic implementation is AWS Lambda Powertools: its logger emits structured JSON, enriches records with function context automatically, and appends correlation IDs extracted from the incoming event. Know it at concept level — a structured logger, EMF-based metrics, and tracing utilities in one library.

Business-level signals — checkouts per minute, cache hit ratio, failures per payment provider — do not exist until your code emits them, and you have two publication paths to choose between.

PutMetricData is the CloudWatch API: your code calls it, ideally batching datapoints, and the metric exists once the call succeeds. It works from anywhere with credentials, but the call sits in your request path: it adds latency, it is subject to API quotas and throttling, and under load you end up maintaining buffering and flush logic that has nothing to do with your business domain.

Embedded metric format (EMF) inverts the model: your code writes a structured JSON log line whose _aws block declares the namespace, dimensions, and metric fields, and CloudWatch extracts the metrics from the log stream asynchronously. Nothing synchronous happens in your hot path, and every datapoint keeps its full log context queryable afterward.

Two more knobs the exam checks. Resolution: standard metrics store at one-minute granularity; high-resolution metrics accept one-second granularity for latency-sensitive alarming. Dimensions: name-value pairs that qualify a metric — but every unique combination of dimension values is a separate metric, so an unbounded dimension such as a per-user ID explodes cardinality and cost.

AWS X-Ray records the path of individual requests across your services so latency and errors can be attributed to a specific hop. The moving parts: the X-Ray SDK in your code records segments — one per service handling the request — and subsegments for the work inside them, especially downstream calls. The SDK does not call the X-Ray API directly; it emits segment documents over UDP to the X-Ray daemon, a local agent that buffers and uploads them in batches, keeping tracing overhead out of your request path. A trace header propagated on outbound calls stitches every service's segments into one trace.

Instrumenting is mostly wrapping. Patch your AWS SDK clients and subsegments for every DynamoDB, S3, or SQS call appear automatically; wrap your HTTP client and outbound calls to third-party APIs are captured too. On Lambda you enable active tracing on the function and the platform runs the daemon for you; API Gateway can enable tracing per stage, so the trace begins before your code runs.

Tracing every request would be costly, so X-Ray samples: the default rule records the first request each second plus 5% of any additional requests, and you can define sampling rules to trace more of the traffic that matters most.

Concretely, an instrumented order flow — API Gateway with tracing enabled, an order Lambda function with active tracing and a patched SDK client, a DynamoDB write, and an HTTPS call to a payment provider — produces one trace with a segment per hop and subsegments for the table write and the payment call. When checkout is slow, that trace shows whether the time sits in your function's own code, the table write, or the third-party API.

Both annotations and metadata attach your own data to X-Ray segments, and the difference between them is one of the most reliably tested distinctions in this domain: annotations are indexed, metadata is not.

Annotations are simple key-value pairs — strings, numbers, or booleans — that X-Ray indexes for search. Record customerId, orderId, or tier as annotations and filter expressions can pull up exactly the traces that match: every trace for one customer, every failed trace where paymentProvider = "cardco". X-Ray supports up to 50 annotations per segment, so reserve them for the handful of business identifiers you will actually filter or group by.

Metadata accepts arbitrary JSON — request payloads, configuration snapshots, structured debugging context. It is visible when you open an individual trace, but it is not indexed and can never appear in a filter expression. Use it for rich context you want available once you have already found the trace by other means.

The exam phrasing is nearly a password: any question about "filtering traces by customer," "searching traces for a specific order," or "grouping traces by a business attribute" is answered by annotations. A value stored as metadata can never be found by a filter expression — that is the trap the wrong options rely on.

Telemetry only helps if someone finds out in time, and the exam expects you to route two different kinds of signal through two different mechanisms.

Threshold breaches go through CloudWatch alarms. An alarm watches one metric — built-in or custom; a metric emitted through embedded metric format alarms like any other — and changes state when the value crosses a threshold for a configured number of evaluation periods; requiring several consecutive datapoints keeps a single noisy spike from paging anyone. Alarm actions publish to an Amazon SNS topic, which fans out to email, SMS, HTTPS endpoints, or a Lambda function that opens a ticket or posts to chat. Composite alarms combine other alarms with AND/OR logic — alert only when the error-rate alarm and the latency alarm fire together — which cuts noise during incidents that trip many related alarms at once.

Discrete operational events go through Amazon EventBridge. A deployment completing or failing, a pipeline changing state, a service quota approaching its limit — these are not metrics crossing thresholds; they are events that occur. AWS services publish them to EventBridge, where a rule matches on the event pattern — a CodeDeploy deployment entering a failed state, a CodePipeline execution succeeding, a Service Quotas or Trusted Advisor notification about an approaching limit — and routes to targets such as an SNS topic or a Lambda function.

The split to carry into the exam: "notify when a metric is too high" is an alarm plus SNS; "notify when a deployment completes" or "when nearing a quota" is an EventBridge rule. Choosing an alarm for a discrete event, or polling an API for state changes, is the distractor.

A health check is an endpoint or command that reports whether one instance of your application can do its job, and the current exam guide added the skill to this task. Two purposes must not be conflated. A liveness check asks "is this process alive at all?" — failing it means the process is hung or dead and should be restarted. A readiness probe asks "can this instance take traffic right now?" — failing it means the instance should be temporarily removed from rotation (still warming up, still loading configuration, momentarily overloaded) but not killed. Restarting an instance that merely was not ready yet is how warmup problems become outages.

AWS provides health checking at several layers. An Application Load Balancer probes each target on a configured path and routes only to targets that pass; healthy and unhealthy thresholds — consecutive successes or failures required to change state — prevent flapping. Amazon Route 53 health checks operate at the DNS layer, steering resolution away from unhealthy endpoints for failover between regions or sites. In Amazon ECS, the task definition can declare a container health check command that runs inside the container, letting the scheduler replace containers that fail it.

Shallow versus deep is the design decision the exam probes. A shallow check returns healthy if the process is up and serving requests. A deep check also verifies dependencies — database reachable, downstream API answering. Deep checks look responsible but carry a failure mode: when one shared dependency blips, every instance's deep check fails simultaneously, the load balancer marks the whole fleet unhealthy, and a partial degradation becomes a total outage. The usual guidance: keep liveness shallow, scope readiness to dependencies the individual instance owns, and surface shared-dependency health through metrics and alarms instead.