Troubleshooting Security Monitoring, Logging, and Alerting on AWS

Triage every broken detection pipeline in the same order: source → delivery → destination permissions → consumer. First, confirm the event was generated at all — is the trail logging, is GuardDuty enabled in that Region, did the API call actually happen there? Second, check delivery: does the producing service still hold the permissions it needs to write, and is the delivery role or resource policy intact? Third, check the destination: does its bucket policy, log group resource policy, or KMS key policy still allow the writing principal, and does the destination still exist under the same name? Fourth, check the consumer: does the EventBridge pattern actually match the real event JSON, is the rule enabled and in the right Region, is the alarm on the right statistic, is the subscription confirmed and allowed to receive? Working in this order matters because later stages fail silently — a rule that matches nothing raises no error anywhere.

CloudTrail stops delivering for one of five reasons: the trail was stopped, the S3 bucket policy no longer allows CloudTrail, the KMS key policy denies it, the bucket was deleted or renamed, or the events you expect were never in scope. Start with the trail status: it reports whether the trail is logging and surfaces LatestDeliveryError. If logging is off, someone called StopLogging — check CloudTrail's own management events for who. If logging is on but delivery errors show access denied, inspect the bucket policy: CloudTrail needs s3:GetBucketAcl on the bucket and s3:PutObject on the AWSLogs/ prefix, granted to the principal cloudtrail.amazonaws.com, with the bucket-owner-full-control ACL condition. If the trail uses SSE-KMS, the customer managed key's policy must also allow cloudtrail.amazonaws.com to call kms:GenerateDataKey* — tighten that key policy and delivery stops immediately even though the bucket policy is untouched. A deleted or renamed bucket produces a no-such-bucket delivery error; the trail keeps pointing at the old name until you update it.

Incomplete logs are usually a scope problem, not a delivery problem. Trails record management events by default; S3 object-level operations and Lambda invocations are data events and are only logged when you enable them through event selectors. If an investigation finds no record of an object being read, check the selectors before suspecting tampering. Finally, with an organization trail, logs from every member account land in the central bucket under per-account prefixes — member-account engineers can see the trail exists but have no access to that bucket by default, so "missing" logs are often an access-model misunderstanding rather than a delivery failure.

Here is the method applied to the most common Task 1.3 scenario. The security team hardened the central logging bucket's policy last Tuesday; today an analyst notices no new CloudTrail objects since exactly that change. Source: the trail status shows logging is still on, so events are being recorded — the source is healthy, and the gap is downstream. Delivery: LatestDeliveryError reports an access-denied error against the bucket, which localizes the fault precisely. Destination permissions: diff the bucket policy against its previous version. The hardening added an aws:SourceArn condition to the CloudTrail statements, but it referenced the wrong trail ARN — so the statements granting cloudtrail.amazonaws.com the s3:GetBucketAcl and s3:PutObject permissions now never match, and every delivery attempt is refused. This is the signature failure mode of well-intentioned confused-deputy hardening: aws:SourceArn and aws:SourceAccount conditions are correct practice, but one wrong account ID or trail name silently severs delivery. Fix and verify: correct the condition value, then confirm new objects arrive and the delivery error clears from the trail status. CloudTrail retries failed deliveries for a limited period, so a prompt fix typically backfills the gap — but treat the outage window as unverified and reconcile against CloudTrail Lake or Event history, which are unaffected by S3 delivery failures. The exam-relevant lesson: a trail that says it is logging can still be delivering nothing, and the delivery error string tells you which stage to inspect — never start by rebuilding the trail.

A VPC Flow Log that shows nothing has one of three problems: broken delivery permissions, a destination that refuses the write, or traffic that was never eligible for capture. When the destination is CloudWatch Logs, delivery runs through an IAM role that must trust vpc-flow-logs.amazonaws.com and allow logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents; if the role or its trust policy is edited, the flow log's delivery status reports the failure. When the destination is S3, there is no role — the log delivery service writes directly, and the bucket policy must allow delivery.logs.amazonaws.com. Read the records themselves before declaring a fault: NODATA means there was simply no traffic in the aggregation interval, while SKIPDATA means records were skipped because of a capacity constraint or internal error. And some absences are by design — flow logs never capture instance metadata traffic to 169.254.169.254, DHCP, or DNS queries to the Amazon-provided resolver, so do not chase those as delivery bugs.

CloudWatch Logs has two permission surfaces of its own. Services that deliver logs into a log group — Route 53 Resolver query logging, for example — require a log group resource policy naming the service principal; without it, delivery is refused. Downstream, subscription filters forward log events to Kinesis Data Streams, Firehose, or Lambda, and each target has a different permission chain: streams and Firehose need an IAM role that CloudWatch Logs assumes to put records, while Lambda needs a resource-based policy allowing the CloudWatch Logs service principal to invoke it. When logs land in the group but never reach the SIEM, the subscription filter's role or the target's policy is the broken link.

GuardDuty stays silent for four reasons, and only one of them is "there was no threat." First, GuardDuty is a Regional, per-account service: the detector must be enabled in every Region of every account, and activity in a Region where the detector is disabled generates nothing at all. Attackers deliberately operate in unmonitored Regions, so an org-wide, all-Region enablement check is the first diagnostic step when findings seem thin. Second, suppression rules auto-archive matching findings — GuardDuty still generates them, but they go straight to the archive and never surface to your downstream consumers, so a suppression rule written too broadly (for example, matching an entire severity range or a whole CIDR block) mutes real detections invisibly. Review suppression rules whenever a finding type you expect simply never shows up. Third, trusted IP lists tell GuardDuty not to generate findings for traffic involving the listed addresses; an over-broad list — a corporate range that also contains a compromised host — blinds detection for exactly the traffic you care about. Fourth, feature-specific findings require their protection plan to be enabled: no S3 Protection means no S3-specific findings, no Malware Protection means no malware scan results, and the same holds for the EKS, RDS, and Lambda plans — each is enabled per account and per Region. In an organization, also confirm the member account is actually enabled under the delegated administrator, not merely invited. The diagnostic order mirrors the triage method: is the detector on (source), is anything archiving or excluding it (suppression, trusted IPs), and is the specific detection capability licensed on (protection plan)?

When Security Hub shows fewer findings than reality, check four settings in order: standards, controls, integrations, and aggregation. A security standard must be enabled before any of its controls evaluate, and individual controls can be disabled per account and Region — a disabled control produces no compliance findings, so a resource can drift out of compliance with nothing on the dashboard. Disabling noisy controls is legitimate tuning, but it is also the first thing to audit when a known-bad configuration shows no finding. Next, integrations: Security Hub only receives findings from products whose integration is enabled. If the GuardDuty integration is off, GuardDuty findings exist in GuardDuty but never appear in Security Hub. Third-party tools send findings in AWS Security Finding Format (ASFF) through the batch-import API; findings that are not valid ASFF are rejected at ingest, so a partner product that "stopped reporting" may be sending malformed payloads rather than nothing.

Cross-Region gaps are their own class. Security Hub findings live in the Region where they were created; cross-Region aggregation replicates them to a single aggregation Region you designate, from linked Regions you specify. If the console looks empty, confirm you are in the aggregation Region, that the source Region is actually linked, and — in an organization — that you are looking at the delegated administrator account, which is the only place member-account findings roll up. A finding created in a Region outside the linked set will never replicate, which reads as "Security Hub lost my finding" but is purely configuration. Note that aggregation moves Security Hub findings only; it does not move the underlying service's events, which matters for the alerting failure covered next.

Second walkthrough, same method. A high-severity GuardDuty finding appeared in us-west-2 on Saturday; on-call was never paged. The alert pipeline is an EventBridge rule targeting an SNS topic, built in us-east-1. Source: the finding exists in the GuardDuty console in us-west-2, so detection worked — and it was not suppressed, since suppressed findings land in the archive. Consumer, part one — Region: GuardDuty emits finding events to the default event bus in the Region where the finding was generated. The only rule lives in us-east-1, so the us-west-2 event met no rule at all. This is the single most common alerting gap: detection is multi-Region, alerting is single-Region. The fix is to deploy the rule in every Region where detectors run, or forward events cross-Region to a central bus. Consumer, part two — pattern: even in the right Region, the rule must match the real finding JSON. Event patterns are exact: a pattern matching severity as the string "8" never matches the numeric 8 in the actual event, a wrong source or detail-type value matches nothing, and a disabled rule matches everything and does nothing. Test with GuardDuty's sample findings rather than waiting for a real one. Consumer, part three — target: the SNS topic's access policy must allow events.amazonaws.com to publish; a Lambda target needs a resource-based permission for EventBridge; a cross-account bus needs a bus policy. Any of these fails silently — EventBridge records a failed invocation metric, but nothing pages you about your paging pipeline. Alarm on that metric.

An alarm that never fires usually failed at the filter, the math, or the notification hop — in that order. Metric filters are literal and case-sensitive: a filter watching for "UnauthorizedOperation" will not match "unauthorizedOperation", and term-based patterns match whole terms, so verify the pattern against real log lines with the filter-testing tools before trusting it. Filters also apply only to log events ingested after the filter exists — they never backfill, so a freshly created filter proves nothing about last week. Alarm math comes next: a count-style metric alarmed on Average instead of Sum may never cross its threshold, a long period smooths a short burst below the line, and because a metric filter emits no datapoint at all when nothing matches, the missing-data treatment decides whether silence reads as OK, alarm, or insufficient data. The notification hop hides two classics. An email or HTTPS subscription stuck in pending-confirmation delivers nothing until confirmed — the alarm fires, SNS accepts the publish, the subscriber hears silence. And a topic encrypted with a KMS key silently rejects publishes from CloudWatch alarms or EventBridge unless the key policy grants the publishing service kms:GenerateDataKey* and kms:Decrypt; the AWS managed aws/sns key cannot be used for this because its policy is not editable, so encrypted alerting topics must use a customer managed key. Finally, AWS Config: if the configuration recorder is stopped, resource changes go unrecorded and rules stop evaluating changes, so compliance dashboards quietly freeze at their last state; if the delivery channel's bucket policy or KMS key breaks, history and snapshot delivery fail. Check recorder and delivery channel status whenever compliance data looks stale rather than wrong.