Troubleshooting Security Monitoring, Logging, and Alerting on AWS

Triage every broken detection pipeline in the same order: source → delivery → destination permissions → consumer. First, confirm the event was generated at all — is the trail logging, is GuardDuty enabled in that Region, did the API call actually happen there? Second, check delivery: does the producing service still hold the permissions it needs to write, and is the delivery role or resource policy intact? Third, check the destination: does its bucket policy, log group resource policy, or KMS key policy still allow the writing principal, and does the destination still exist under the same name? Fourth, check the consumer: does the EventBridge pattern actually match the real event JSON, is the rule enabled and in the right Region, is the alarm on the right statistic, is the subscription confirmed and allowed to receive? Working in this order matters because later stages fail silently — a rule that matches nothing raises no error anywhere.

CloudTrail stops delivering for one of five reasons: the trail was stopped, the S3 bucket policy no longer allows CloudTrail, the KMS key policy denies it, the bucket was deleted or renamed, or the events you expect were never in scope. Start with get-trail-status. IsLogging only says whether the trail is recording events; it says nothing about whether delivery works. Each delivery path has its own error field, so read the one that matches the symptom:

If logging is off, someone called StopLogging — CloudTrail Event history in the trail's home Region keeps 90 days of management events, so you can find who without enabling anything new. If logging is on but delivery errors show access denied, inspect the bucket policy: CloudTrail needs s3:GetBucketAcl on the bucket and s3:PutObject on the AWSLogs/ prefix, granted to the principal cloudtrail.amazonaws.com, with the bucket-owner-full-control ACL condition. If the trail uses SSE-KMS, the customer managed key's policy must also allow cloudtrail.amazonaws.com to call kms:GenerateDataKey* — tighten that key policy and delivery stops immediately even though the bucket policy is untouched. A deleted or renamed bucket produces a no-such-bucket delivery error; the trail keeps pointing at the old name until you update it.

Incomplete logs are usually a scope problem, not a delivery problem. Trails record management events by default; S3 object-level operations and Lambda invocations are data events and are only logged when you enable them through event selectors. If an investigation finds no record of an object being read, check the selectors before suspecting tampering.

Organization trails add two traps. First, the trail is owned by the management account, so the aws:SourceArn condition must be that trail's ARN (which contains the management account ID) and aws:SourceAccount must be the management account ID — even when the bucket lives in a separate log archive account. Building those values from the bucket owner's account ID breaks delivery for the whole organization. Second, member-account logs land under AWSLogs/<organization ID>/<member account ID>/, and by default member accounts cannot read them — only the bucket owner can. Member-account engineers see the trail in their console but cannot edit it or open its files; "missing" logs are usually this access model, not a delivery failure. To let a member read its own logs, add a bucket policy statement granting its principals s3:GetObject on its own account prefix only.

Here is the method applied to the most common Task 1.3 scenario. The security team hardened the central logging bucket's policy last Tuesday; today an analyst notices no new CloudTrail objects since exactly that change. Source: the trail status shows logging is still on, so events are being recorded — the source is healthy, and the gap is downstream. Delivery: LatestDeliveryError reports an access-denied error against the bucket, which localizes the fault precisely. Destination permissions: diff the bucket policy against its previous version. The hardening added an aws:SourceArn condition to the CloudTrail statements, but it referenced the wrong trail ARN — so the statements granting cloudtrail.amazonaws.com the s3:GetBucketAcl and s3:PutObject permissions now never match, and every delivery attempt is refused. This is the signature failure mode of well-intentioned confused-deputy hardening: aws:SourceArn and aws:SourceAccount conditions are correct practice, but one wrong account ID or trail name silently severs delivery. Fix and verify: correct the condition value, then confirm new objects arrive and the delivery error clears from the trail status. CloudTrail keeps retrying failed deliveries for up to 30 days, so a fix within that window backfills the gap — but treat the outage window as unverified and reconcile against CloudTrail Lake or Event history, which are unaffected by S3 delivery failures. The exam-relevant lesson: a trail that says it is logging can still be delivering nothing, and the delivery error string tells you which stage to inspect — never start by rebuilding the trail.

A VPC Flow Log that shows nothing has one of three problems: broken delivery permissions, a destination that refuses the write, or traffic that was never eligible for capture. When the destination is CloudWatch Logs, delivery runs through an IAM role that must trust vpc-flow-logs.amazonaws.com and allow logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents; if the role or its trust policy is edited, the flow log's delivery status reports the failure. When the destination is S3, there is no role — the log delivery service writes directly, and the bucket policy must allow delivery.logs.amazonaws.com. Read the records themselves before declaring a fault: NODATA means there was simply no traffic in the aggregation interval, while SKIPDATA means records were skipped because of a capacity constraint or internal error. And some absences are by design — flow logs never capture instance metadata traffic to 169.254.169.254, DHCP, or DNS queries to the Amazon-provided resolver, so do not chase those as delivery bugs.

CloudWatch Logs has two permission surfaces of its own. Services that deliver logs into a log group — Route 53 Resolver query logging, for example — require a log group resource policy naming the service principal; without it, delivery is refused. Downstream, subscription filters forward log events to Kinesis Data Streams, Firehose, or Lambda, and each target has a different permission chain: streams and Firehose need an IAM role that CloudWatch Logs assumes to put records, while Lambda needs a resource-based policy allowing the CloudWatch Logs service principal to invoke it. When logs land in the group but never reach the SIEM, the subscription filter's role or the target's policy is the broken link.

Cross-account streaming adds a third surface. A CloudWatch Logs destination in the receiving account wraps the Kinesis stream or Firehose stream, holds the role that CloudWatch Logs assumes to write into it, and carries a destination access policy that lists which sender accounts (or which organization, with aws:PrincipalOrgID) may subscribe. A sending account's PutSubscriptionFilter call fails with access denied when it is not in that policy. Separate the two permission layers when diagnosing: the access policy decides who may subscribe and is evaluated per sender, while the destination's role decides whether CloudWatch Logs can write to the stream (and use its KMS key) and is shared by every sender.

Lambda and API Gateway both write to CloudWatch Logs under an IAM role, but a different role in each case — and that is where most of their logging failures live. Lambda sends a function's logs using the function's execution role, which needs logs:CreateLogStream and logs:PutLogEvents (plus logs:CreateLogGroup if Lambda must create the group). By default logs go to /aws/lambda/<function name>; the function's logging configuration can point it at a custom, shared log group instead. A least-privilege statement scoped to the default group ARN stops matching the moment you change the destination, and the function keeps running while writing nothing. Lambda does not use a log group resource policy — the identity writing is the execution role, so the fix is to scope that role's statement to the new group.

The logging configuration also filters what is written. With the JSON log format, Lambda applies two separate levels: the application log level filters what your code emits through its logging library, and the system log level filters the platform's own records (start, end, report). Raise the application level to ERROR and every INFO line your code writes is dropped before it reaches CloudWatch Logs; changing the system level does not bring it back. Log-level filtering requires JSON format; the Text format has none.

API Gateway REST APIs write logs using a CloudWatch Logs role ARN set in the API Gateway account settings — a role that trusts apigateway.amazonaws.com, typically with the AmazonAPIGatewayPushToCloudWatchLogs managed policy. That setting is stored per Region: an account whose APIs log fine in one Region gets an error demanding the role ARN the first time someone enables logging in another. API Gateway creates the execution log group itself, so pre-creating the group or adding a log group resource policy does not help, and the integration's Lambda role is irrelevant. With the role in place, logging is a stage setting, and the two kinds answer different questions:

CloudFront has three logging options, and each fails in its own way. Standard logging (legacy) writes log files to an S3 bucket using bucket ACLs: the bucket must have ACLs enabled so CloudFront's log delivery account can be granted write access. Set the bucket's Object Ownership to Bucket owner enforced — a common baseline control — and ACLs are disabled, so legacy delivery stops. Standard logging (v2) delivers the same access logs through CloudWatch vended logs to S3, CloudWatch Logs or Firehose; for S3 it is authorized by a bucket policy letting delivery.logs.amazonaws.com write (and, for SSE-KMS, a key policy grant to the same principal), so it works with ACLs disabled. Its delivery source is configured through the CloudWatch Logs API in us-east-1, even when the bucket is elsewhere, and a Firehose destination must be in us-east-1. Both standard options deliver with a delay of minutes and are best effort.

Real-time logs send records to a Kinesis Data Streams stream within seconds, with a sampling rate and field list you choose, using an IAM role that CloudFront assumes to write to the stream. Choose them when detection must act on edge traffic as it happens. Requests served from the edge cache never reach the origin, so only edge-side sources see them: CloudFront's own logs, or the logs of an AWS WAF web ACL associated with the distribution, which inspects every viewer request before the cache answers. Origin-side sources such as VPC Flow Logs, load balancer logs or a WAF on the origin load balancer see only cache misses.

A Route 53 health check that reports unhealthy while the application works is almost always a network or placement problem, not an application one. Endpoint health checks are run by Route 53 health checkers on the internet, so the endpoint's security groups and network ACLs must allow their requests. Tightening a security group to a corporate range blocks them, the check fails, and DNS failover moves traffic away from a healthy endpoint. Allow the checkers without reopening to everyone by referencing the AWS-managed prefix list com.amazonaws.<Region>.route53-healthchecks (the ranges are also published as ROUTE53_HEALTHCHECKS in the AWS IP address ranges file). Do not confuse it with the CloudFront origin-facing prefix list, which covers a different service.

Because checkers come from the internet, endpoint checks cannot reach private IP addresses or internal load balancers, whatever the security group says. For a failover record in a private hosted zone, use a health check that monitors a CloudWatch alarm — for example on a load balancer's healthy host count — which needs no network path. A calculated health check only combines the status of child checks, so it inherits their reachability problems. Finally, Route 53 health check metrics are published to CloudWatch in us-east-1, so alarms on a health check's status must be created there; an alarm built in another Region finds no metric.

When the CloudWatch agent stops sending metrics or a log file, the cause is one of four things: the agent never loaded the configuration you edited, the configuration itself is wrong, the instance lacks permission, or the agent cannot reach the service endpoints. Loading: the agent runs from a configuration it translated when you last ran amazon-cloudwatch-agent-ctl -a fetch-config. Editing the JSON file and restarting the service (or stop/start) reuses the old translated configuration, so new collect_list entries are ignored and the agent's own log (/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log) never mentions them. Re-run fetch-config with -c file:<path> for a local file or -c ssm:<parameter> for Parameter Store; the agent loads whichever source you name, so a stale parameter or file reloads stale settings. Use -m ec2 on EC2 so the agent uses the instance role; -m onPremise expects a configured credentials profile and Region instead.

Configuration: under logs.logs_collected.files.collect_list, each entry needs the exact file_path and a target log group, and the user the agent runs as must be able to read the file. Permissions: the instance profile needs CloudWatchAgentServerPolicy (or equivalent cloudwatch:PutMetricData and CloudWatch Logs write permissions); a missing permission shows as access-denied errors in the agent log. Network: timeouts rather than access-denied errors point at reachability. Metrics go to the CloudWatch endpoint (monitoring) and logs to the CloudWatch Logs endpoint (logs), so instances in private subnets without a NAT gateway need an interface VPC endpoint for each, with private DNS enabled; add an ssm endpoint when the configuration comes from Parameter Store, and an ec2 endpoint when the configuration appends EC2 dimensions or tags (the agent calls ec2:DescribeTags).

GuardDuty stays silent for four reasons, and only one of them is "there was no threat." First, GuardDuty is a Regional, per-account service: the detector must be enabled in every Region of every account, and activity in a Region where the detector is disabled generates nothing at all. Attackers deliberately operate in unmonitored Regions, so an org-wide, all-Region enablement check is the first diagnostic step when findings seem thin. Second, suppression rules auto-archive matching findings — GuardDuty still generates them, but they go straight to the archive and never surface to your downstream consumers, so a suppression rule written too broadly (for example, matching an entire severity range or a whole CIDR block) mutes real detections invisibly. Review suppression rules whenever a finding type you expect simply never shows up. Third, trusted IP lists tell GuardDuty not to generate findings for traffic involving the listed addresses; an over-broad list — a corporate range that also contains a compromised host — blinds detection for exactly the traffic you care about. Fourth, feature-specific findings require their protection plan to be enabled, per account and per Region, and two S3 features are easy to confuse:

The same holds for the EKS, RDS, Lambda and Runtime Monitoring plans. In an organization, also confirm the member account is actually a GuardDuty member in that Region under the delegated administrator. Accounts created after the organization was set up become members automatically only if auto-enable is configured for new accounts in that Region; otherwise a new account has no detector there and produces no findings in either console; the delegated administrator's Accounts list shows it as not a member (not enabled). The diagnostic order mirrors the triage method: is the detector on (source), is anything archiving or excluding it (suppression, trusted IPs), and is the specific detection capability licensed on (protection plan)?

When AWS Security Hub CSPM shows fewer findings than reality, check five settings in order: standards, controls, AWS Config recording, integrations, and aggregation. (Security Hub CSPM is the name, since 2025, of the posture-management service that evaluates controls and ingests ASFF findings; the newer AWS Security Hub correlates signals from it and other services.) A security standard must be enabled before any of its controls evaluate, and individual controls can be disabled per account and Region — a disabled control produces no compliance findings, so a resource can drift out of compliance with nothing on the dashboard. Disabling noisy controls is legitimate tuning, but it is also the first thing to audit when a known-bad configuration shows no finding; a disabled control shows its status as Disabled. Most controls are evaluated by AWS Config rules, so they depend on the Config recorder recording the resource type they check. If AWS Config is not recording a resource type that an enabled control checks, Security Hub CSPM cannot evaluate it: the control receives WARNING findings, so its status shows Unknown, and the Config.1 control reports a FAILED finding for the recording gap, while controls for recorded types keep reporting normally. A fully stopped recorder affects every Config-based control at once. Read the control statuses precisely:

When the newer AWS Security Hub is enabled alongside Security Hub CSPM, CSPM uses its own service-linked recorder (AWSConfigurationRecorderForSecurityHubCSPM) that records every resource type its controls need, so exclusions in your own recorder no longer cause these gaps and Config.1 always passes. Next, integrations: Security Hub CSPM only receives findings from products whose integration is enabled. If the GuardDuty integration is off, GuardDuty findings exist in GuardDuty but never appear in Security Hub CSPM. Third-party tools send findings in AWS Security Finding Format (ASFF) through the batch-import API; findings that are not valid ASFF are rejected at ingest, so a partner product that "stopped reporting" may be sending malformed payloads rather than nothing.

Cross-Region gaps are their own class. Security Hub CSPM findings live in the Region where they were created; cross-Region aggregation replicates them to a single aggregation Region you designate, from linked Regions you specify. If the console looks empty, confirm you are in the aggregation Region, that the source Region is actually linked, and — in an organization — that you are looking at the delegated administrator account, which is the only place member-account findings roll up. A finding created in a Region outside the linked set will never replicate, which reads as "Security Hub CSPM lost my finding" but is purely configuration. Note that aggregation moves Security Hub CSPM findings only; it does not move the underlying service's events, which matters for the alerting failure covered next.

Second walkthrough, same method. A high-severity GuardDuty finding appeared in us-west-2 on Saturday; on-call was never paged. The alert pipeline is an EventBridge rule targeting an SNS topic, built in us-east-1. Source: the finding exists in the GuardDuty console in us-west-2, so detection worked — and it was not suppressed, since suppressed findings land in the archive. Consumer, part one — Region: GuardDuty emits finding events to the default event bus in the Region where the finding was generated. The only rule lives in us-east-1, so the us-west-2 event met no rule at all. This is the single most common alerting gap: detection is multi-Region, alerting is single-Region. The fix is to deploy the rule in every Region where detectors run, or forward events cross-Region to a central bus. Consumer, part two — pattern: even in the right Region, the rule must match the real finding JSON. Event patterns are exact: a pattern matching severity as the string "8" never matches the numeric 8 in the actual event, a wrong source or detail-type value matches nothing, and a disabled rule matches everything and does nothing. Test with GuardDuty's sample findings rather than waiting for a real one. Consumer, part three — target: the SNS topic's access policy must allow events.amazonaws.com to publish; a Lambda target needs a resource-based permission for EventBridge; a cross-account bus needs a bus policy. EventBridge authorizes SNS, SQS and Lambda targets through those resource policies, not through a role on the target — so a topic whose access policy lacks a statement for events.amazonaws.com rejects every EventBridge delivery, regardless of which other services can publish to it. Read the rule's metrics as a pair: MatchedEvents rising proves the pattern works; FailedInvocations rising alongside it proves the target refused delivery. Any of these failures is silent — nothing pages you about your paging pipeline, so alarm on FailedInvocations.

An alarm that never fires usually failed at the filter, the math, or the notification hop — in that order. Metric filters are literal and case-sensitive: a filter watching for "UnauthorizedOperation" will not match "unauthorizedOperation", and term-based patterns match whole terms, so verify the pattern against real log lines with the filter-testing tools before trusting it. Filters also apply only to log events ingested after the filter exists — they never backfill, so a freshly created filter proves nothing about last week. Alarm math comes next: a count-style metric alarmed on Average instead of Sum may never cross its threshold, a long period smooths a short burst below the line, and because a metric filter emits no datapoint at all when nothing matches, the missing-data treatment decides whether silence reads as OK, alarm, or insufficient data. The notification hop hides two classics. An email or HTTPS subscription stuck in pending-confirmation delivers nothing until confirmed — the alarm fires, SNS accepts the publish, the subscriber hears silence. And a topic encrypted with a KMS key silently rejects publishes from CloudWatch alarms or EventBridge unless the key policy grants the publishing service kms:GenerateDataKey* and kms:Decrypt; the AWS managed aws/sns key cannot be used for this because its policy is not editable, so encrypted alerting topics must use a customer managed key. Finally, AWS Config: if the configuration recorder is stopped, resource changes go unrecorded and rules stop evaluating changes, so compliance dashboards quietly freeze at their last state; if the delivery channel's bucket policy or KMS key breaks, history and snapshot delivery fail while the resource timeline keeps updating. When the delivery bucket uses SSE-KMS with a customer managed key and Config uses its service-linked role, Config writes as the config.amazonaws.com service principal, so the key policy must allow that principal kms:GenerateDataKey and kms:Decrypt; switching a central bucket's default encryption without that grant silently stops delivery from every account. Check recorder and delivery channel status whenever compliance data looks stale rather than wrong: a recorder problem stops configuration items, a delivery problem stops only the files.