SAP-C03 Observability and Monitoring: OpenTelemetry, Container Insights, Cross-Account Logs, Alarms, AI Metrics, RUM and SLOs

Standardized instrumentation on AWS now means OpenTelemetry: services emit traces, metrics and correlated logs through the OpenTelemetry APIs and SDKs (or the AWS Distro for OpenTelemetry, ADOT, which packages them with AWS-specific components), and a collector or the CloudWatch agent receives the data over OTLP and forwards it to Amazon CloudWatch and AWS X-Ray. Because OTLP is vendor-neutral, the same instrumentation can later feed a third-party backend by changing the exporter, not the code.

The older path, the AWS X-Ray SDKs plus the X-Ray daemon, entered maintenance mode in February 2026. AWS recommends OpenTelemetry SDKs or ADOT for new and existing applications and an OpenTelemetry collector or the CloudWatch agent in place of the daemon. X-Ray itself is not going away: it remains the trace store and analysis service that OpenTelemetry spans are sent to. The X-Ray daemon does not accept OTLP, so it is not a receiver for OpenTelemetry SDKs.

Where the telemetry goes first

• ADOT Collector (or an upstream OpenTelemetry Collector): run as a sidecar in an ECS task, a DaemonSet on EKS, or a process on EC2. Flexible pipelines, processors and multiple exporters, at the cost of one more component to run and patch.
• CloudWatch agent with an otlp receiver: add an otlp section under traces_collected and under metrics_collected in the agent configuration (default endpoints 127.0.0.1:4317 for gRPC and 4318 for HTTP). An agent that already collects host metrics and log files then also accepts OTLP and sends traces to X-Ray and metrics to CloudWatch, so a fleet gets OpenTelemetry without a second agent.
• CloudWatch embedded metric format (EMF): structured log lines from which CloudWatch extracts metrics. Useful for high-cardinality custom metrics from Lambda and containers, but it is a metrics technique, not a tracing standard.

Trace context propagation in mixed estates

A distributed trace survives only if every hop understands the same propagation header. OpenTelemetry defaults to the W3C Trace Context header (traceparent); the X-Ray SDKs read and write X-Amzn-Trace-Id. During a gradual migration, a migrated service that sends only traceparent makes every downstream X-Ray-SDK service start a new trace, so one request shows up as two traces. The fix is to configure the OpenTelemetry SDK with the X-Ray propagator alongside W3C, so both headers are sent and accepted. Sampling rules, collector choice and annotations do not change which headers services exchange.

Also remember what does not create spans: an Application Load Balancer adds a trace header to requests but does not record segments of its own, so tracing has to be instrumented in the services (API Gateway stages and Lambda functions do have their own tracing switches).

CloudWatch Transaction Search lets you keep and search every span cheaply, separately from how many traces X-Ray indexes. When it is on, X-Ray writes all ingested spans as structured logs to the aws/spans log group, where they can be searched by any span attribute (an order ID, a customer ID) and analysed with Logs Insights. Separately, an indexing rule decides what percentage of spans also become X-Ray trace summaries for the traces page and trace analytics; the default is 1%, which is included at no extra charge.

Two settings are therefore independent, and many scenarios turn on keeping them apart:

So when every request must stay findable by a business identifier, without paying to index every trace, the design is 100% sampling with the business ID as a span attribute, Transaction Search on, and indexing left low. Raising X-Ray sampling to 100% without Transaction Search pays to index everything and keeps X-Ray traces for only 30 days; keeping a low sampling rate misses most requests whatever the indexing level. And a traces page that lists only a small share of requests while attribute search finds them all reflects the indexing percentage at work, not dropped spans.

Turning it on as code

The console does it in one step. Through the API or CLI (for example from an account-vending pipeline) it is two required actions: a CloudWatch Logs resource policy that allows xray.amazonaws.com to write to aws/spans and /aws/application-signals/data, then update-trace-segment-destination with the destination CloudWatchLogs. update-indexing-rule is optional and only changes the indexing percentage. X-Ray uses a resource policy here, not an IAM service role.

CloudWatch Container Insights with enhanced observability collects metrics down to the individual container (cluster, node or instance, service, task or pod, and container) and provides curated dashboards that drill from a cluster to a single container, with no monitoring stack to run yourself.

Amazon EKS

Install the Amazon CloudWatch Observability EKS add-on. It deploys the CloudWatch agent and Fluent Bit as managed DaemonSets, turns on Container Insights with enhanced observability, and also ships logs (see the next section). Because it is an EKS add-on, AWS manages the agent versions, which is the least-overhead answer compared with running Prometheus, kube-state-metrics and Grafana yourself. Two near-misses: EKS control plane logging sends API server, audit, authenticator, scheduler and controller manager logs, not workload metrics; the Kubernetes Metrics Server gives point-in-time CPU and memory for autoscaling and kubectl top, not history or dashboards.

Amazon ECS

On ECS, Container Insights is a cluster setting (containerInsights = enhanced, enabled or disabled). The account setting (put-account-setting or put-account-setting-default) only sets the default for clusters created afterwards. Existing clusters keep their own value until you run update-cluster-settings on each one; redeploying services does not change it. On Fargate no agent is needed; on EC2 container instances the ECS agent supplies the data.

One view across many accounts

For a central operations team, combine Container Insights in each workload account with CloudWatch cross-account observability: an Observability Access Manager sink in the monitoring account and links from source accounts (which can be created for a whole organization or OU, so new accounts are linked as they join). The monitoring account then views and drills into metrics, logs and traces that stay in the source accounts. That beats copying metrics through metric streams or Prometheus remote-write pipelines, and it beats giving operators a role to switch into each account, which gives no single view.

Container log design is about where logs are routed and what happens to the application when delivery is slow.

Routing

• awslogs log driver (ECS): sends stdout and stderr to one CloudWatch Logs log group. Simple, but every line is ingested into CloudWatch Logs.
• FireLens (ECS): a Fluent Bit (or Fluentd) sidecar that receives the container's output and can filter and route lines to several destinations, for example audit events to CloudWatch Logs and high-volume access lines to Amazon S3 through Amazon Data Firehose. Because the split happens before ingestion, the lines routed elsewhere are never charged as CloudWatch Logs ingestion.
• EKS: the CloudWatch Observability add-on's Fluent Bit DaemonSet sends container stdout/stderr to /aws/containerinsights/<cluster>/application, node logs to …/host and data plane logs to …/dataplane, with no agent configuration to write. Hand-built node agents or per-pod log sidecars work but add maintenance.

Anything that acts after ingestion, such as a subscription filter, a log class or an export task, cannot stop a line from being ingested in the first place.

Blocking versus non-blocking delivery

ECS has defaulted to non-blocking since June 2025 (the defaultLogDriverMode account setting), but an explicit mode option in a task definition, often copied from an old template, overrides the default. When availability matters more than every log line, set mode to non-blocking and size the buffer for bursts. Raising PutLogEvents quotas, adding stream prefixes or changing the log class leaves blocking behaviour in place.

Streaming logs from many accounts to one place in near real time uses CloudWatch Logs subscriptions through a logical destination. A subscription filter on a log group can deliver directly to a Kinesis data stream, Firehose stream, Lambda function or OpenSearch Service in the same account; to reach another account it must target a CloudWatch Logs destination owned by the receiving account.

Building the destination (receiving account)

1. An IAM role that trusts logs.amazonaws.com and allows writing to the target stream (for example kinesis:PutRecord, or Firehose put actions).
2. PutDestination, which takes the target stream ARN and that role ARN.
3. PutDestinationPolicy, a destination access policy that allows logs:PutSubscriptionFilter for the sending account IDs, or for the whole organization with the aws:PrincipalOrgID condition so new accounts are admitted without edits.

The workload accounts then create subscription filters that reference the destination ARN. A resource policy on the Kinesis stream is not part of this flow, and pull-based designs (a role that polls FilterLogEvents) are neither real time nor low effort. CloudWatch cross-account observability is a different tool: it lets a monitoring account view data that stays in the source accounts, it does not deliver a feed to a SIEM.

The Region rule

A subscription filter can reference only a destination in the same Region as the log group. The destination itself can point at a stream in another Region. So adding a new workload Region means creating another destination in that Region in the receiving account; it can target the existing stream elsewhere if the SIEM must keep one feed. Policy conditions do not relax the Region rule.

Covering every log group: account-level subscription filter policies

Per-log-group filters miss groups that developers create later. An account-level subscription filter policy (PutAccountPolicy with type SUBSCRIPTION_FILTER_POLICY, or AWS::Logs::AccountPolicy) applies to every existing and future log group in that account and Region; there is one such policy per account per Region, and it can target a cross-account destination. Its only selection criterion is LogGroupName NOT IN [...], which exists to prevent recursion: if a component of the delivery pipeline writes its own logs (for example an S3-triggered Lambda function that enriches Firehose output), those logs would be forwarded, land in S3 and trigger it again forever. Exclude such log groups by name. When the destination is itself a Lambda function, CloudWatch Logs does not apply the policy to that function's own log group, but other functions in the pipeline are not protected.

A policy covers only its own account, so at organization scale deploy it with a service-managed CloudFormation StackSet with automatic deployment to the OU, in every Region, each pointing at that Region's destination. Event-driven automation (an EventBridge rule on CreateLogGroup) or AWS Config remediation can also reach coverage, but they are more moving parts than one declarative policy.

A cost-effective log archive keeps recent logs in CloudWatch Logs for querying and keeps the long-term copy in Amazon S3, moved to the cheapest storage class whose retrieval time is acceptable. CloudWatch Logs storage is far more expensive per GB than S3 Glacier classes, so multi-year retention inside CloudWatch Logs is rarely the cheapest design.

Typical pattern

• Set a CloudWatch Logs retention period that matches the interactive query window (for example 90 days for Logs Insights investigations).
• Deliver the same events continuously to S3 through a subscription filter and Amazon Data Firehose. A CloudWatch Logs export task is a batch job (and only one can run at a time per account), so it suits occasional exports, not a continuous archive or a near-real-time feed.
• Add an S3 Lifecycle rule that transitions objects after the query window and expires them at the end of the mandated period.

Immutability

When a regulator requires that nobody, including the root user, can delete or overwrite logs before the period ends, use S3 Object Lock in compliance mode with a default retention period. Governance mode can be bypassed by principals with s3:BypassGovernanceRetention, and an SCP is a permissions guardrail, not a write-once guarantee. Object Lock retention stays in force when Lifecycle transitions objects to Glacier classes.

The Infrequent Access log class

CloudWatch Logs offers a Standard and an Infrequent Access log class. Infrequent Access has a lower ingestion price and still supports Logs Insights queries, but it lacks several Standard features, including subscription filters, metric filters, Live Tail, embedded metric format and Container Insights or Lambda Insights log ingestion, and a log group's class is chosen at creation and cannot be changed. If logs must also stream through a subscription-based pipeline (a SIEM feed or the Firehose archive), the log group has to be Standard.

Good alerting pages a human once, for a condition that really needs action, while keeping detailed signals visible for diagnosis. CloudWatch provides the levers to do that without custom code.

Composite alarms

A composite alarm evaluates a rule over the states of other alarms, for example ALARM(p99Latency) AND (ALARM(Error5xx) OR ALARM(OrdersDrop)). Put the paging action on the composite alarm only and remove paging from the children; the children still change state, so engineers see every signal on dashboards. Tuning each child (more evaluation periods, anomaly detection bands) improves individual alarms but still sends one page per alarm, and SNS filtering cannot combine conditions across alarms.

Actions suppression

A composite alarm can name an actions suppressor alarm (for example a ReleaseWindow alarm). While the suppressor is in ALARM, the composite alarm still changes state but takes no actions, and actions resume on their own afterwards, so no pipeline has to call DisableAlarmActions. Two timers handle the edges:

Adding AND NOT ALARM(ReleaseWindow) to the rule also stops pages, but it changes the state the dashboard shows, which suppression does not.

Datapoints and missing data

An alarm evaluates M out of N datapoints over N evaluation periods; requiring several breaching datapoints tolerates one-off spikes. Missing data treatment decides what silence means: missing (default; an alarm whose whole range is missing goes to INSUFFICIENT_DATA), breaching, notBreaching or ignore (keep the current state). For a heartbeat metric that stops when a worker crashes, treat missing data as breaching and require, for example, 2 of 2 ten-minute datapoints, so the alarm fires after 20 minutes of silence but not for one late heartbeat. notBreaching and ignore hide the failure, and on a single-period alarm an INSUFFICIENT_DATA action fires on any one gap.

Anomaly detection alarms compare a metric with a band learned from its history; they suit metrics with daily or weekly patterns, but they do not decide which alarms should page together.

An alarm becomes useful through what it triggers: a notification, a tracked work item, or an automated fix. Match the requirement to the right wiring.

Every alarm state change is also published to Amazon EventBridge, so one alarm can notify through SNS and remediate through an EventBridge-started runbook at the same time, with no code. Pass the instance ID or other dimension from the event to the runbook as an input parameter.

AWS Systems Manager Incident Manager is closed to new customers (existing customers keep it). It can still appear in scenarios, but for a new account the tracked-response choice is OpsCenter OpsItems (with SNS for paging).

EC2 instance recovery

StatusCheckFailed_System reports problems with the underlying host; StatusCheckFailed_Instance reports problems inside the instance (a reboot may fix those). The recover action moves an EBS-backed instance to healthy hardware and keeps its instance ID, private IP addresses, Elastic IP addresses, metadata and EBS volumes; terminate replaces it and reboot stays on the same host. Most supported instances now get simplified automatic recovery by default, but it excludes some cases such as bare metal (.metal) sizes; those need the CloudWatch alarm with the recover action, which can also notify SNS.

AI workloads need three kinds of monitoring beyond normal service health: inference latency broken into its parts, traces of agent steps, and model quality that can drift without any error.

Amazon Bedrock inference latency

• InvocationLatency: time from request to the last token. It grows when the model generates more slowly and when prompts or answers get longer.
• TimeToFirstToken: the wait before the first token appears, published for the streaming operations (ConverseStream, InvokeModelWithResponseStream). This is what users mean by "nothing appears for several seconds".
• Output tokens per second via metric math: OutputTokenCount / (InvocationLatency − TimeToFirstToken) × 1000. Removing the first-token (prefill) time and dividing only output tokens gives generation speed, which stays stable when answers simply get longer and drops when generation slows. Including input tokens or dividing by total latency mixes prompt length back in.
• InvocationThrottles, InputTokenCount, OutputTokenCount: capacity and volume, not latency.

Amazon SageMaker AI endpoints

ModelLatency is the time the model container takes to respond; OverheadLatency is the time SageMaker AI adds around it (request handling, routing). Comparing the two before and after a release shows which side of the boundary added the delay. CPUUtilization and InvocationsPerInstance describe load, not where time goes.

AI agent observability

Amazon Bedrock AgentCore Observability records OpenTelemetry spans for agent sessions, model calls and tool calls and shows them on the CloudWatch generative AI observability page (sessions, traces, token usage, latency and errors, with drill-down into one session). Prerequisites and edges:

• Transaction Search must be turned on once in the account, because agent spans are ingested through it.
• Agents hosted on AgentCore Runtime include the ADOT package in the agent code, and the runtime provides the log group and telemetry pipeline. Agents hosted elsewhere (EKS, ECS, EC2) add ADOT themselves and start with OpenTelemetry auto-instrumentation, so frameworks such as Strands Agents emit spans for model and tool steps.
• AgentCore Memory and Gateway resources need tracing turned on per resource, and log delivery (for example APPLICATION_LOGS) configured to a destination; AgentCore does not set those up for you.
• Bedrock model invocation logging records prompts and responses to CloudWatch Logs or S3; it is an audit log, not a trace of agent steps.

Model drift and model quality

A model can fail silently: latency and error rates stay normal while predictions get worse. Two signals differ in timing:

The building blocks are endpoint data capture to S3, a baseline computed from training data, scheduled analysis jobs that compare captured data with the baseline (and join delayed labels for quality), and metrics published to CloudWatch with alarms. Amazon SageMaker Model Monitor packaged these steps but is no longer open to new customers, so a design for a new account assembles them from data capture and scheduled jobs. Bias and explainability reports on training data, or retraining on a schedule, do not watch production.

User-experience monitoring combines synthetic transactions, which test journeys on a schedule, with real user monitoring, which measures what actual visitors experience; SLOs then turn these signals into measurable targets with error budgets.

Synthetics canaries

Canaries are scripts (Node.js, Python or Java runtimes) and created from blueprints such as heartbeat monitoring, API canary, broken link checker, canary recorder and GUI workflow builder. Visual monitoring compares each run's screenshot with a baseline and fails when the difference exceeds a threshold, catching a page that returns HTTP 200 but hides a button. To test a private endpoint (an internal ALB), run the canary in the VPC with private subnets and a security group that can reach the target. The canary must still publish metrics to CloudWatch and write artifacts to S3: without internet access that needs an interface VPC endpoint for CloudWatch monitoring and an endpoint for Amazon S3 (IAM permissions alone give no network path).

CloudWatch RUM

Create an app monitor and add its web client snippet to the application. The session sample rate (100% by default) sets what share of sessions send data, and charges scale with events, so a high-traffic site can sample a fraction and still get reliable breakdowns. Turning on X-Ray active tracing in the app monitor makes the web client add trace headers to HTTP requests, so a slow page request can be followed through API Gateway and Lambda (the backend must have tracing on and its CORS configuration must allow the header). RUM keeps data for 30 days; for longer retention, turn on sending a copy to CloudWatch Logs and set that log group's retention.

SLOs with CloudWatch Application Signals

An SLO has a service level indicator (latency or availability of a service operation discovered by Application Signals, or any CloudWatch metric), an attainment goal (for example 99.9%) and an interval (rolling or calendar). The error budget is the share the goal allows to be bad.

• Period-based SLOs count good versus bad time periods: one failed request in a sparse minute makes the whole minute bad, which overweights low-traffic periods.
• Request-based SLOs count good versus total requests, so the budget burns in proportion to failed requests.
• Time window exclusions (one-time, or recurring with a cron expression) leave planned maintenance out; for request-based SLOs the requests inside the window are excluded.

Burn rate is the observed error rate divided by the rate that would use exactly the budget over the interval (1 − goal). A burn rate of 1 spends the whole budget in exactly one interval. To page when a fraction f of the budget would be spent within a look-back window w, the threshold is f × interval ÷ w: spending 5% of a 720-hour budget in 6 hours is 0.05 × 720 ÷ 6 = 6. A short look-back with a threshold well above 1 pages on fast burns that would exhaust the budget within days; alarming on attainment below the goal fires only after the budget is already gone. Application Signals also offers attainment and SLI health alarms, and its service dashboards show SLO status beside the traces that explain a breach.