AWS Network Troubleshooting: VPC Flow Logs and the Layered Diagnosis Method

Every VPC connectivity failure lives in one of five control layers, and checking them in a fixed order beats jumping around the console. Work through: route tables (does the source subnet have a route to the destination — and does the destination have a route back?), security groups, network ACLs, gateway and endpoint state (is the NAT gateway, peering connection, transit gateway attachment, or VPC endpoint actually present, up, and in the right place?), and finally DNS (is the client even resolving the name to the IP address you think it is?).

The distinction that powers the whole method: security groups are stateful — if a request is allowed in, the reply is automatically allowed out, regardless of outbound rules. Network ACLs are stateless — the reply is a brand-new packet that must match an explicit allow rule in the opposite direction, typically on the ephemeral port range 1024-65535.

That leads to return-path thinking: a TCP connection only works if the request is permitted by every control on the way there and the reply is permitted by every stateless control on the way back. When you audit a path, trace it twice — forward against the destination port, and backward against the ephemeral source port. Asymmetric failures, where the SYN arrives but the reply dies, look like timeouts to the application but are unmistakable in flow logs.

Know what network ACLs never filter, so you don't write rules for it or blame it: traffic to and from the Amazon Route 53 Resolver (the VPC+2 address), the instance metadata service, DHCP and the Amazon Time Sync Service is not evaluated by NACLs. A DNS failure is therefore never fixed by adding a NACL rule for port 53 to the Resolver — look at the VPC's DNS settings, Resolver rules or the client's resolver configuration instead.

Only after the five layers check out do you blame the operating system firewall or the application itself — which AWS network controls cannot see, and which VPC Reachability Analyzer will not evaluate either.

When an instance cannot reach the internet, the cause is almost always a missing route, a missing public address, or a NAT gateway that has no path out itself. Which rule applies depends on the subnet type and on which kind of NAT gateway you run.

Public subnet

• No IGW route: the subnet's route table must contain 0.0.0.0/0 pointing at the internet gateway. A subnet is only "public" because of this route.
• No public IP: the instance needs a public IPv4 address or an Elastic IP. A correct route with no public address still fails, because the internet gateway performs one-to-one NAT against that address.

Private subnet behind a NAT gateway

• Missing NAT route: the private subnet's route table needs 0.0.0.0/0 pointing at the NAT gateway, and no more-specific route sending that traffic elsewhere.
• Zonal NAT gateway in the wrong subnet: a (public) zonal NAT gateway is created in a subnet, and that subnet must be public — its route table must send 0.0.0.0/0 to the internet gateway. A zonal NAT gateway created in a subnet with only the local route is Available but has no path out, so everything behind it silently fails. The fix is a NAT gateway in a public subnet (you cannot move one), not an internet route added to a subnet that must stay private.

A zonal NAT gateway lives in one Availability Zone: if private subnets in AZ B route through a NAT gateway in AZ A, an AZ A failure removes egress for AZ B too, and you pay cross-AZ data charges in the meantime. The resilient zonal pattern is one NAT gateway per AZ, each serving its own AZ's subnets.

Regional NAT gateways (2025) change the placement rule. You create one per VPC without choosing a subnet; it expands into every AZ where you have workloads, so it needs no public subnet. AWS creates a dedicated route table for it with a preconfigured route to the internet gateway, and private subnets in every AZ can route 0.0.0.0/0 to the same NAT gateway ID. Regional NAT gateways support only public NAT — private NAT (to on-premises or overlapping networks) still needs a zonal NAT gateway.

Port exhaustion and idle connections

Each NAT gateway IPv4 address supports up to 55,000 concurrent connections to each unique destination (destination IP, port and protocol). Many clients hammering one popular endpoint exhaust that, and the CloudWatch metric ErrorPortAllocation climbs while new connections fail — existing ones keep working. Remedies, in AWS's order of convenience: associate secondary IPv4 addresses with the existing NAT gateway (each adds another 55,000; a zonal NAT gateway takes up to 8 addresses, a regional one up to 32 per AZ), split clients across more NAT gateways (one per AZ, or several private subnets each routed to a different NAT gateway), or cut the client's connection count. IdleTimeoutCount shows connections dropped for idleness: a NAT gateway times out a connection that has been idle for 350 seconds and answers later packets with a TCP RST, so long-lived quiet connections need application traffic or TCP keepalive below 350 seconds. PacketsDropCount counts packets the gateway itself dropped. Note too that NAT gateways pass only TCP, UDP and ICMP.

IPv6: egress-only internet gateway, not NAT

IPv6 addresses are globally unique, so there is no NAT for outbound-only IPv6. The egress-only internet gateway is the IPv6 answer: route ::/0 to it, and instances can initiate outbound IPv6 connections while inbound-initiated connections are blocked. A NAT gateway handles IPv4 egress; a ::/0 route to a NAT gateway or a plain internet gateway is the wrong tool when the requirement is "outbound IPv6 only".

When two instances in the same VPC cannot communicate, start with security groups — specifically with what the rules reference. A classic failure: the database security group allows MySQL from the "app" security group, but the instance actually connecting was launched with a different group, or the rule references the load balancer's group while the app tier connects to the database directly. Confirm the source's actual group membership, not what the architecture diagram says.

If security groups check out, walk the network ACL scenario, because it produces the exam's favorite trap. An app server at 10.0.1.25 opens a connection to a database at 10.0.2.40 on port 3306. The connection times out. Flow logs on the database's ENI show a pair of records:

2 123456789012 eni-0a1b2c3d 10.0.1.25 10.0.2.40 48512 3306 6 4 216 1719923400 1719923460 ACCEPT OK
2 123456789012 eni-0a1b2c3d 10.0.2.40 10.0.1.25 3306 48512 6 4 188 1719923400 1719923460 REJECT OK

Read the signature: the request was ACCEPTed and the reply was REJECTed on the same ENI. A security group can never produce this — it is stateful, so an accepted request implies an allowed reply. The only control that can accept a request and reject its reply is the stateless network ACL. Here, the database subnet's NACL has no outbound allow for ephemeral ports; the fix is an outbound rule allowing TCP 1024-65535 to the app subnet's CIDR.

Generalize it: between any two hosts, every NACL on the path needs allows in both directions covering both the destination port and the ephemeral return ports — four rule evaluations per NACL, not one.

Security group references across VPCs

Referencing a security group instead of a CIDR is only possible within reach of one Region:

• Same-Region VPC peering: you can reference a security group in the peer VPC; for a peer in another account, use the account-qualified form (account-id/sg-id).
• Inter-Region VPC peering: security group references are not supported — the rule must use the peer's CIDR range (or specific IP addresses).
• Transit gateway: security group referencing works between VPCs attached to the same transit gateway in the same Region once the feature is enabled on the transit gateway and its VPC attachments; it does not extend across transit gateway peering to another Region.

A rule that cannot match the traffic simply never allows it, so the symptom is an inbound REJECT in the destination's flow log even though "the app group is allowed".

Peering failures are usually route failures. VPC peering is non-transitive: if VPC A peers with B and B peers with C, A cannot reach C through B, no matter what routes you write. Expecting transitivity is a design error whose fix is another peering connection, a transit gateway, or PrivateLink — not more routes.

Within a working peering connection, routes must exist on both sides: VPC A's route tables need the B CIDR pointing at the pcx- peering connection, and VPC B's route tables need the A CIDR pointing at the same connection. Miss one side and you get an asymmetric black hole — SYNs arrive but replies have nowhere to go, and the client sees timeouts. Peering has no route propagation: every route is static and must be added by hand to each relevant route table. The peering connection's DNS resolution option does not create a path either; it only makes the peer's public DNS hostnames resolve to private IP addresses across the connection. Overlapping CIDRs are a harder stop: VPCs with overlapping ranges cannot be peered at all.

A transit gateway adds a second routing layer, and both must be right in both directions:

• VPC route tables need a route for the remote CIDR pointing at the transit gateway. VPC route tables do not learn transit gateway routes by propagation — these are static routes you add.
• Transit gateway route tables: each attachment is associated with at most one transit gateway route table (an unassociated attachment cannot route), which is the table used to route traffic arriving from that attachment. Routes get into a table by propagation from attachments or by static entries.
• The return path uses a different table. The reply is routed by the table associated with the destination's attachment. In a segmented design (for example, a spoke table and a shared-services table), the destination attachment must propagate into the source's table and the source attachment must propagate into the destination's table; one-way propagation produces timeouts that look like a security problem.

CloudWatch transit gateway metrics tell you which kind of routing gap you have: PacketDropCountNoRoute means no route in the associated transit gateway route table matched the destination (a missing association, propagation or static route), while PacketDropCountBlackhole means a route matched but it is a blackhole route — either a deliberate blackhole static route or one left behind when its attachment was deleted. Both prove the drop happened at the transit gateway, not inside a VPC.

PrivateLink as an alternative, and what each option costs

When one VPC only needs to reach a single service in another VPC, AWS PrivateLink (an interface endpoint to an endpoint service, usually fronted by a Network Load Balancer) avoids routing altogether: it works with overlapping CIDRs and exposes only that service, one-way. Cost is part of the choice:

VPC Reachability Analyzer answers "can traffic get from A to B, and if not, which component stops it?" It performs static configuration analysis of a path between a source and destination (instances, network interfaces, gateways, transit gateway attachments, peering connections and more) without sending any packets, and when the path is not reachable it names the blocking component — the specific security group, NACL rule, route table or missing route. Reach for it when the configuration spans many hops; use flow logs when you need to see what live traffic actually did. With AWS Organizations trusted access (and optionally a delegated administrator account) it can analyze paths whose resources sit in different member accounts.

Its blind spot is the host. Reachability Analyzer evaluates AWS configuration only, so it can report a path as reachable while the operating system firewall drops the packet or nothing is listening. The two host-level failures look different from the client:

Network Access Analyzer is a different tool with a similar name. It does not trace one path; it audits your whole network against a Network Access Scope you define (for example, "no path from the internet gateway to the database subnets") and reports every finding that violates it. Use Reachability Analyzer to troubleshoot a broken connection, Network Access Analyzer to prove there is no unintended access.

VPC Flow Logs record metadata about IP traffic at three capture levels — an entire VPC, a subnet, or a single ENI — and publish to CloudWatch Logs or Amazon S3. The default record format is a space-separated line: version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status. For troubleshooting you mostly read four of them: srcaddr and dstaddr, the ports, and action (ACCEPT or REJECT).

Three signatures answer most connectivity questions:

Remember what an ACCEPT does and does not mean. It records only that the security group and network ACL allowed the packet. A packet the guest operating system's firewall then drops is still logged as ACCEPT, and so is the TCP reset a host sends back when nothing is listening on the port — flow logs cannot see inside the instance.

The "no record" case has one caveat: flow logs deliberately exclude some traffic, including queries to the Amazon DNS resolver, instance metadata requests to 169.254.169.254, and DHCP. Absent DNS traffic is normal; an absent application flow is a finding. Also glance at log-status: NODATA means no traffic was seen in the aggregation window, while SKIPDATA means records were dropped — so your absence of evidence may not be evidence of absence.

Flow log records are aggregated: the maximum aggregation interval is 10 minutes (the default), and you can choose 1 minute when you need faster evidence. Records also take a few minutes to be delivered, so a fix tested seconds ago may not show yet.

Custom fields for translated and asymmetric flows

A custom log format adds fields the default line lacks. The most useful for troubleshooting: pkt-srcaddr and pkt-dstaddr (the original packet's addresses when an intermediate interface — a NAT gateway, a load balancer, a transit gateway attachment — sits in the flow), tcp-flags (SYN, SYN-ACK, FIN, RST, so you can see handshakes and resets), flow-direction (ingress or egress relative to the interface) and traffic-path (the egress path, such as through an internet gateway or a NAT gateway).

On a zonal NAT gateway's network interface every connection appears as two legs:

So to find which instance talked to an external address from the NAT gateway's log alone, filter the instance-to-NAT leg on pkt-dstaddr and read srcaddr; the translated leg never shows the instance, in any field.

Layered architectures need layered logs. Each log type isolates fault to one tier, and the skill the exam tests is picking the one that answers your actual question.

ELB access logs are the load-balancer-versus-target separator. A 504 with a target_processing_time of -1 means the target never responded before the timeout — a backend problem. A 502 points at a broken or malformed target response or a failed connection to the target. A 502 with target_status_code of - (no response) and no matching entry in the application's own log usually means the target closed a keep-alive connection while the load balancer was sending a request on it: the target's keep-alive timeout is shorter than the load balancer's idle timeout. The fix is on the target — set its keep-alive timeout longer than the load balancer's idle timeout; raising the load balancer's idle timeout widens the gap and makes it worse. When elb_status_code is a 5xx but target_status_code shows the target returned 200, the fault is in the load balancer tier, not the application. The log also records which target served each request, so you can spot one bad instance hiding behind an otherwise healthy fleet — then hand the health-check tuning to your reliability playbook.

AWS WAF web ACL logs are for false-positive hunting: when legitimate users report being blocked, filter for BLOCK actions and read which rule fired. When a rule inside a rule group (for example an AWS managed rule group) terminated the request, terminatingRuleId holds the rule group's identifier, and the specific rule inside it is in ruleGroupList (its terminating rule). Tune that one rule — for example override its action to Count and add a scope-down — rather than disabling the whole group. Other sources answer less: the ALB access log's actions_executed field shows waf (the web ACL acted) but not which rule, and the console's sampled requests show only a sample of recent traffic, not every request.

WAF logging has strict destination rules: the CloudWatch Logs log group, S3 bucket or Amazon Data Firehose stream must have a name starting with aws-waf-logs-. A CloudWatch Logs log group must be in the same account and Region as the web ACL — for a web ACL attached to CloudFront, that Region is US East (N. Virginia), us-east-1.

CloudFront standard logs record every viewer request (delivered to S3, CloudWatch Logs or Firehose); real-time logs stream selected fields to Kinesis Data Streams within seconds. Container logs matter as the last tier: once every network layer shows clean, the answer is in the application's own output (ECS with the awslogs driver or FireLens, EKS with a log agent such as Fluent Bit, into CloudWatch Logs).

A falling CloudFront cache hit ratio has three usual causes: a cache key that is too broad, TTLs that are effectively zero, and unnecessary invalidations. Diagnose before you tune.

Walkthrough. Origin load doubles and the distribution's cache hit rate reads 35%. Per request, the x-cache response header shows "Miss from cloudfront" even on repeated fetches of the same page. In the standard logs, x-edge-result-type is dominated by Miss — not Error — so the origin is healthy and this is purely a caching problem. The cache policy turns out to forward all query strings, and the site's marketing links append unique tracking parameters, so every visitor generates a unique cache key and nothing is ever reused. The application only varies its response on lang. Restricting the cache key to that one query string collapses thousands of one-off cache entries into a handful, and the hit ratio recovers.

The general rules:

• Cache key too broad: every header, cookie or query string in the cache policy becomes part of the cache key, so each variation is a separate cached object. Include only the values the origin actually uses to vary the response. If the origin needs a value (a tracking parameter for analytics, an auth header) but the response does not vary on it, put it in an origin request policy instead: values there are forwarded to the origin but are not part of the cache key.
• TTL zero: if the origin sends Cache-Control: no-cache, private, or max-age=0 and the minimum TTL is 0, CloudFront revalidates or refetches everything. CloudFront uses the origin's max-age clamped between the cache policy's minimum and maximum TTL, and the default TTL when the origin sends none — align the two deliberately. TTLs only control how long one cached object lives: raising the maximum TTL does nothing for a fragmented cache key (each variant is still cached separately) and nothing against invalidations, which remove objects whatever their TTL.
• Invalidations: invalidating on every deploy empties the cache repeatedly. Prefer versioned object names (app.v42.js) so new content gets a new URL and old entries simply expire.

Origin Shield adds a central caching layer in front of the origin, which cuts origin load when many edge locations miss on the same object. It does not repair a cache key that makes every request unique — those requests still miss at Origin Shield too.

Distinguish misses from errors: an x-edge-result-type of Error means the origin returned a 5xx or timed out — a reliability problem, mitigated by an origin group with origin failover, not by cache tuning.

Site-to-Site VPN problems split into two states. Tunnel down (the TunnelState metric reads 0): IKE negotiation is failing — a mismatched pre-shared key, a wrong customer gateway IP or device configuration, or an on-premises firewall blocking UDP 500/4500. Tunnel up but no traffic: the tunnel is fine and the routing is not. With static routing, the on-premises CIDRs must be entered on the VPN connection and routed in the VPC route tables; with BGP, route propagation must be enabled on the route table. And security groups still apply to VPN traffic — instances must explicitly allow the on-premises CIDR, which is easy to forget because the traffic feels internal. Direct Connect raises the same route-and-return questions: the virtual private gateway or Direct Connect gateway must advertise the right prefixes and the VPC route table must send on-premises CIDRs back the same way.

Private connectivity failures follow the endpoint type:

• Interface endpoint resolves to a public IP: private DNS is not enabled on the endpoint, so the service's public hostname bypasses it — and from a private subnet the connection then dies. Enable private DNS, with DNS support enabled on the VPC.
• Connection to an interface endpoint times out: the endpoint's security group must allow inbound 443 from the consumers.
• Explicit denies: the endpoint policy can reject specific actions or principals even when the network path is clean.
• Gateway endpoints (S3, DynamoDB) work through the route table: a subnet whose route table is not associated with the endpoint sends traffic out the NAT path instead — check associations before anything else.
• Cross-account PrivateLink: the endpoint sits in a pending-acceptance state until the service owner accepts it; nothing flows before that.

CloudWatch has three network monitoring services, and each watches a different stretch of the path. Pick by where the users or the far end are:

The distinction is tested through the scenario: probes to an on-premises IP over a VPN or Direct Connect link point to Network Synthetic Monitor; slow or lossy traffic between workloads inside AWS points to Network Flow Monitor; users in a particular city or on a particular ISP seeing problems points to Internet Monitor. All three publish metrics to CloudWatch, so you alarm on them like any other metric. Older material that calls the hybrid probe service "CloudWatch Network Monitor" means Network Synthetic Monitor.