CloudWatch Metrics, Alarms, and Log Filters for SOA-C03

A CloudWatch metric is a time-ordered series of datapoints identified by three things: a namespace (a container such as AWS/EC2 or AWS/Lambda, or a custom one like MyApp/Orders), a metric name (such as CPUUtilization), and zero or more dimensions — key-value pairs like InstanceId=i-0abc123 that scope the metric to a specific resource. This matters operationally because CloudWatch treats every unique combination of namespace, name, and dimensions as a separate metric. If you publish a custom metric with dimension Env=prod and later query it with no dimensions, or a new build adds or renames a dimension, the alarm or dashboard watching the old combination gets nothing — a classic reason it shows no data.

Resolution determines how often datapoints arrive. EC2 standard monitoring sends instance metrics every 5 minutes at no charge. Detailed monitoring raises that to 1-minute metrics for an extra cost, and you enable it per instance (or in a launch template). Detailed monitoring does not add any new metrics — it only makes the existing ones more frequent, which lets alarms and Auto Scaling react faster.

Custom metrics you publish yourself with PutMetricData come in two resolutions, chosen per call with the StorageResolution parameter. Omitting it (or setting 60) gives standard resolution, stored at 1-minute granularity. Setting StorageResolution to 1 makes it a high-resolution metric stored down to 1-second granularity. Only a high-resolution metric can back a high-resolution alarm — one whose period is 10 or 30 seconds; a standard-resolution metric has nothing finer than a minute to evaluate, so its alarms use periods of 60 seconds or multiples of it. You need both halves when a workload can fail faster than a 1-minute alarm could notice. Faster resolution and high-resolution alarms cost more, so reserve them for the metrics that gate paging or scaling.

Metrics can also leave CloudWatch continuously. A metric stream sends selected namespaces in near real time to an Amazon Data Firehose delivery stream, which forwards them to Amazon S3 or a third-party observability tool. It is an export path, not a sharing mechanism: to let another AWS account see your metrics, use cross-account observability (covered under dashboards).

The metrics EC2 publishes by default come from the hypervisor, which can only see the instance from the outside: CPU, network bytes and packets, disk I/O operations, and status checks. The hypervisor cannot see inside the guest operating system, so the default metric set never includes memory utilization, disk-space utilization, or swap usage — and EBS volume metrics describe I/O, not how full a file system is. When memory or free-disk metrics are "missing" from CloudWatch, the fix is to install and configure the CloudWatch agent on the instance.

The agent runs inside the OS and reads a JSON configuration that lists which metrics and log files to collect. Two plugins come up often:

• Disk space — the disk section publishes disk_used_percent per file system, with the mount point as the path dimension, so you can alarm on the root volume filling up separately from a data volume.
• Processes — the procstat plugin watches named processes (by executable name, pattern, or PID file) and can publish a process count such as procstat_lookup_pid_count. An alarm when that count drops below 1 detects a daemon that exited while the operating system — and therefore every status check — stayed healthy.

Managing the agent across a fleet. For instances that are already Systems Manager managed nodes, the standard pattern has three parts:

1. Store the agent configuration as a Parameter Store parameter whose name starts with AmazonCloudWatch- (for example AmazonCloudWatch-linux). One parameter is the single place to change what the fleet collects.
2. Install the agent package with the Systems Manager AWS-ConfigureAWSPackage document (package name AmazonCloudWatchAgent), typically through Run Command or a State Manager association.
3. Apply the configuration with the AmazonCloudWatch-ManageAgent document, pointing it at the Parameter Store parameter. After a configuration change, run it again (or let a State Manager association do so) to push the change to every running instance.

Baking the agent into an AMI or launch-template user data is fine for new instances, but neither reaches instances that are already running, and neither gives you one place to change the configuration later.

Permissions. The instance profile role needs the AWS managed policy CloudWatchAgentServerPolicy: it allows cloudwatch:PutMetricData, the CloudWatch Logs writes the agent needs (create log groups and streams, put log events), a few EC2 describe calls for dimensions, and ssm:GetParameter on parameters named AmazonCloudWatch-*. CloudWatchAgentAdminPolicy adds ssm:PutParameter and belongs on the identity that authors and saves the configuration, not on every instance. A common diagnosis: AmazonSSMManagedInstanceCore alone lets Systems Manager manage the node and even read parameters, so the agent can be installed and can load its configuration — but its PutMetricData and log writes are denied until CloudWatchAgentServerPolicy is attached.

Container Insights is CloudWatch's managed collection of container metrics — CPU, memory, network, and storage at cluster, service, task, pod, and container level — together with the performance log events behind them. With enhanced observability it adds finer-grained (container-level) metrics and curated views. How you switch it on depends on the platform.

Two ECS details are easy to miss. First, the containerInsights account setting is only a default for clusters created after it changes; an existing cluster keeps its own value until you change that cluster with update-cluster-settings (or the console). Second, Container Insights collects performance data, not your application's log output: on ECS, application logs still go through the task definition's log driver — awslogs to CloudWatch Logs, or FireLens for routing elsewhere. Running the CloudWatch agent as a sidecar in Fargate tasks also works for custom metrics, but every task definition has to be edited and redeployed, which is why the cluster setting is the low-effort answer for standard container metrics.

If a team has standardized on Prometheus — remote-write, PromQL queries and alerting rules, Grafana dashboards — Amazon Managed Service for Prometheus ingests and stores the Prometheus-format metrics and Amazon Managed Grafana hosts the dashboards, so the team keeps its existing tooling while AWS operates, patches, and scales the servers. Choose it when Prometheus compatibility is the requirement; choose Container Insights when the goal is native CloudWatch metrics, alarms, and dashboards.

Serverless and AI services publish their core metrics to CloudWatch automatically — no agent is involved — so monitoring them is mostly about reading the right metric and switching on the deeper telemetry when you need it.

AWS Lambda publishes these built-in metrics in the AWS/Lambda namespace, each answering a different question:

So "requests fail but the function logs show nothing" points to Throttles, while high ConcurrentExecutions only says the function is busy. The built-in metrics do not include memory or CPU used per invocation. Lambda Insights (the enhanced-monitoring option) adds those — memory utilization, CPU time, network, and cold-start information — by attaching the Lambda Insights extension layer to the function; you turn it on in the function configuration with no code change, and the execution role needs the permissions to publish (the managed policy CloudWatchLambdaInsightsExecutionRolePolicy).

Amazon Bedrock publishes runtime metrics in the AWS/Bedrock namespace, dimensioned by model ID — among them Invocations, InvocationLatency, InvocationClientErrors, InvocationServerErrors, InvocationThrottles, InputTokenCount, and OutputTokenCount. Alarms on the token counts are how you catch a sudden rise in consumption (and cost). Metrics carry no content, though. To keep the actual prompts and responses, enable model invocation logging in the Bedrock settings: it is off by default, is configured per account and Region, and delivers request and response data to CloudWatch Logs, Amazon S3, or both. CloudTrail still records that each Bedrock API call happened and who made it, but not the prompt and response bodies — so audit of content needs invocation logging, not CloudTrail.

CloudWatch Logs organizes everything into log groups and log streams. A log group is the unit of management — retention, log class, access control, metric filters, and subscription filters all attach at the group level. A log stream is one ordered sequence of events from a single source, such as one EC2 instance or one Lambda execution environment. Operationally you almost always work at the group level: one group per application or component, with each instance writing its own stream inside it.

Retention is set per log group and defaults to never expire. That default is a cost trap: log storage grows forever until someone sets a retention policy, which you can configure from 1 day up to 10 years. CloudWatch Logs has no cheaper archive tier of its own, so when older events must stay retrievable but are rarely read, the pattern is to export them to Amazon S3 (where lifecycle rules can move them to colder storage classes) and let a shorter log-group retention expire them from CloudWatch — exporting first, so nothing is lost when retention removes it.

Each log group also has a log class, chosen when the group is created and not changeable afterwards — to switch, you create a new group. Standard supports the full feature set. Infrequent Access costs less per GB ingested and still supports Logs Insights queries, but drops features such as metric filters, subscription filters, anomaly detection and Live Tail (it can still be exported to S3), so it suits logs you keep for occasional investigation rather than logs you alarm on.

Two tools read logs interactively, and they answer different questions:

Logs Insights queries are built from commands like fields, filter, stats, and sort (for example stats count(*) by bin(5m)) and can span several log groups. Neither tool changes your monitoring configuration — which is what separates them from the metric filters in the next section.

Logs arrive from many producers: the CloudWatch agent ships OS and application log files, Lambda writes function logs automatically, ECS tasks write through the awslogs driver, VPC Flow Logs can target a log group, and CloudTrail can deliver its events into CloudWatch Logs for alarming.

A metric filter watches a log group for a pattern and publishes a CloudWatch metric value every time the pattern matches. It converts unstructured log lines into something you can alarm on. Two rules to remember: metric filters apply only to log events ingested after the filter is created (they never backfill history — use Logs Insights to look back), and the metric they publish behaves like any other custom metric, with a namespace, name, and value you define.

Walk through the classic setup. The CloudWatch agent ships an application log file to the log group /myapp/prod. You create a metric filter on the group with the pattern ERROR, publishing metric ErrorCount in namespace MyApp with a metric value of 1 per match — and a default value of 0, so the filter still emits a datapoint during quiet periods instead of the metric going missing. Then you alarm on it.

The alarm's statistic matters. Every match contributes the value 1, so within a period:

• Sum is the number of matches — the right statistic for "five or more errors in 5 minutes".
• Average of a stream of 1s is 1 no matter how many errors occurred (and with a default value of 0 mixed in, it becomes a ratio, not a count), so a count threshold on Average never behaves as intended.
• SampleCount counts datapoints, including the default-value zeros, so it is not an error count either.

An alarm of ErrorCount Sum ≥ 5 over a 5-minute period, with the default value of 0 keeping the metric continuous, reads OK during healthy hours and pages within minutes of a real burst.

A subscription filter is the other kind of filter. Instead of counting matches into a metric, it delivers the matching log events themselves, in near real time, to Kinesis Data Streams, Amazon Data Firehose, or a Lambda function — no code needed for Kinesis or Firehose targets. Use metric filters when you need a number to alarm on; use subscription filters when the log data must flow somewhere for central analysis, transformation, or storage. (CloudWatch has also added log alarms, which run a scheduled Logs Insights query, but a metric filter feeding a metric alarm remains the standard, cheapest way to alarm on a log pattern.)

Cross-account log delivery uses a CloudWatch Logs destination. In the receiving account you create the destination, which wraps the target (for example a Kinesis data stream), names an IAM role that CloudWatch Logs assumes to write to that target, and carries a destination access policy that lists the sender account IDs (or an organization ID) allowed to subscribe. In each sending account you then create a subscription filter on the log group whose target is the destination's ARN. Both halves are required: the senders cannot write to the stream directly, and the destination accepts only the accounts its policy names.

AWS CloudTrail records API calls: who made the call, from where, when, against which resource, and whether it succeeded. That makes it an audit service, distinct from CloudWatch. "Which user modified the security group?" is CloudTrail. "Why did CPU spike at 14:00?" is CloudWatch. Sort a scenario by asking whether you need actions taken (CloudTrail) or how the system is performing (CloudWatch).

Event history is on by default with no setup: it keeps the last 90 days of management events, per Region, viewable and searchable in the console. Its retention cannot be extended. Management events are control-plane operations — launching or terminating instances, changing IAM policies, modifying security groups. Data events are high-volume data-plane operations such as S3 object-level reads and writes or Lambda invocations; they never appear in event history and are recorded only when you explicitly add them to a trail (at extra cost).

To keep events longer, create a trail that records management events (a new trail includes them by default) and delivers them continuously to an S3 bucket, where S3 retention and lifecycle rules meet multi-year requirements. A multi-Region trail captures every Region; an organization trail created in the management account logs every member account to one bucket. (CloudTrail Lake, the managed event data store, closed to new customers on 31 May 2026 — existing users keep it, but for a new account the trail-to-S3 or trail-to-CloudWatch Logs paths are the options.)

CloudTrail Insights is an opt-in feature on a trail that learns a baseline of API call volume and error rates and records an Insights event when activity departs from it. It is good for spotting the unusual; it does not implement a fixed rule such as "three failures in five minutes".

When the question is "who did it?" under default settings, know what each source can answer:

From audit record to page. To alert on a rule over API activity — say, three or more failed console sign-ins in 5 minutes — send the trail to CloudWatch Logs, add a metric filter that matches failed ConsoleLogin events, and alarm on the Sum over a 5-minute period. An EventBridge rule on sign-in failures would react to each event individually (one notification per failure, no counting threshold), and CloudTrail Insights fires on deviation from a learned baseline rather than on your threshold. The metric-filter pattern is CloudTrail for the record and CloudWatch for the reaction.

A CloudWatch alarm watches one metric (or metric math expression) and sits in one of three states: OK (threshold not breached), ALARM (breached), or INSUFFICIENT_DATA (not enough datapoints to decide). New alarms start in INSUFFICIENT_DATA, and an alarm returns there when the metric stops arriving — a stopped instance, a broken agent, or a custom metric published with a different namespace or dimension set than the alarm specifies.

Three settings control when an alarm fires. The period is how long each evaluated datapoint covers. Evaluation periods (N) is how many recent datapoints the alarm examines. Datapoints to alarm (M) is how many of those must breach — the "M out of N" setting, and the breaching datapoints do not have to be consecutive. With a 60-second period, EvaluationPeriods 5 and DatapointsToAlarm 3 alarm when any 3 of the last 5 minutes breach. Requiring 3 of 3 means three consecutive breaching minutes; 2 of 3 tolerates a single recovered blip while still catching a sustained problem within about three minutes. M out of N tames flapping alarms without dulling detection — lengthening the period instead slows every detection.

Treat missing data decides what a gap in the metric means:

One more distinction: disabling alarm actions does not stop the alarm. It keeps evaluating and changing state (visible in alarm history); only the actions are skipped. That is useful during maintenance, and a common reason an alarm "fired" without anyone being notified.

Alarm actions run on state transitions (one exception: an Auto Scaling action keeps being invoked about once a minute for as long as the alarm stays in the new state), and you attach them per target state — usually on entering ALARM, sometimes on OK or INSUFFICIENT_DATA. A metric alarm can directly:

• publish to an Amazon SNS topic;
• invoke an AWS Lambda function;
• run an EC2 Auto Scaling scaling policy;
• take an EC2 action — stop, terminate, reboot, or recover — on the instance behind a per-instance metric;
• create a Systems Manager OpsItem or Incident Manager incident (Incident Manager closed to new customers in November 2025 — existing users keep it, but a new account uses OpsCenter OpsItems);
• start a CloudWatch investigation.

Anything else — a Systems Manager Automation runbook, a Step Functions workflow, a third-party target — is reached through Amazon EventBridge: every alarm state change is emitted as an event that a rule can match and route, using an IAM role that lets EventBridge start the target. A direct Lambda action that calls the target's API (for example StartAutomationExecution) is the other route. Building those remediation flows is Task 1.2; here, the point is knowing which actions are native.

Permissions differ by route. An alarm has no IAM role of its own. For a direct Lambda action, the function's resource-based policy must allow the service principal lambda.alarms.cloudwatch.amazonaws.com to invoke it (not cloudwatch.amazonaws.com); without that statement the alarm changes state, its history shows a failed action, and the function logs show no invocation. For EventBridge targets, the rule's target role (or the target's resource policy) grants access.

A composite alarm combines other alarms with AND, OR, and NOT logic and exists to cut noise. If a load balancer's 5xx alarm and its latency alarm both fire during real incidents but each alone is usually a harmless wobble, a composite of 5xx AND latency pages once, on the real event: keep the child alarms (other automation may depend on them) and move the notification action to the composite. A composite can also act as an actions suppressor for its children's actions, for example during known maintenance.

So when an instance must be rebooted only when a composite condition is met, the composite cannot do it itself: match the composite's state-change event with an EventBridge rule and target a Systems Manager Automation runbook such as AWS-RestartEC2Instance (or have a Lambda action do the work).

A notification reaches people only if three links hold: the alarm (or other service) can publish to the topic, the topic can be read under its encryption settings, and the subscription is live. Check them in that order of likelihood when "the alarm fired but nobody got an email".

Topic and subscription. An alarm notifies an SNS topic in the same Region, and the topic needs at least one confirmed subscription — an email endpoint stays "pending confirmation" until someone clicks the link in the confirmation message, and receives nothing until then. Also confirm that the alarm's ALARM action references the intended topic ARN and that actions are enabled.

Topic access policy. When another AWS service publishes on your behalf — an EventBridge rule, S3 event notifications, and similar — the topic's resource-based access policy must allow that service principal (for EventBridge, events.amazonaws.com) to call sns:Publish. The console adds this statement when you pick the topic as a target; creating the rule with the CLI or an API does not. The symptom is a rule whose metrics show matched events and FailedInvocations while subscribers receive nothing.

Encrypted topics. With server-side encryption on the topic, the publishing service must also be allowed to use the KMS key (kms:GenerateDataKey* and kms:Decrypt). The AWS managed key aws/sns has a key policy you cannot edit, and it does not let services such as CloudWatch or EventBridge use it — so encrypting an alarm topic with aws/sns silently stops the alarm notifications while test messages from IAM users still work. Use a customer managed key whose key policy allows the publishing service principal (cloudwatch.amazonaws.com for alarms, events.amazonaws.com for EventBridge).

CloudWatch dashboards are the customizable, shareable home page for a workload's health, and they can display metrics and alarms across multiple accounts and Regions. Dashboards are global resources, and each widget has its own Region setting, so one dashboard can show your us-east-1 API fleet next to its eu-west-1 replica — you simply choose the Region when adding each widget.

Crossing accounts uses CloudWatch cross-account observability: you designate a central monitoring account (which holds a sink) and link source accounts to it, after which the monitoring account can search, graph, dashboard, and alarm on metrics, logs, and traces from every linked account without switching roles. Sinks and links are Regional: a dashboard that graphs two Regions needs each source account linked to the monitoring account in each of those Regions. In an organization you can link member accounts in bulk with a template or policy.

Sharing a dashboard with people has several options that differ in scope and in who can view:

Build dashboards around decisions, not decoration: pair each headline metric with its alarm-status widget so the viewer sees both the value and whether it is breaching, and use text widgets for runbook links. Dashboards display metrics but evaluate nothing (alarms do that), and automatic dashboards give you an instant per-service view before you build anything custom.