SAP-C03 Network Architecture: Multi-VPC Connectivity, IPAM, Network Security, IPv6 and Global Routing

Pick the connectivity model from the shape of the problem: VPC peering for a small number of VPC pairs that need full two-way routing, a transit gateway when many VPCs, accounts and on-premises networks must route to each other under central control, AWS PrivateLink when consumers need to reach one service (not a whole network), and VPC sharing when many accounts should run resources in fewer, centrally owned VPCs.

At scale the standard pattern is a transit gateway per Region, created in a central network account and shared with workload accounts through AWS RAM, so each account keeps its own VPCs and attaches them. Regions are joined by transit gateway peering attachments, which do not propagate routes: you add static routes in each transit gateway route table for the other Region's CIDR ranges. This replaces hundreds of point-to-point peering connections with a handful of route tables the network team controls.

Peering is still the right answer when the problem is small and stable. Two VPCs with non-overlapping ranges and no other networks to reach get the cheapest private path from one peering connection, because a hub or PrivateLink adds hourly and per-GB charges that grow with the volume copied between them.

Connecting on-premises networks redundantly (Direct Connect and VPN failover) and hybrid DNS belong to the hybrid integration lesson; here the VPN or Direct Connect gateway appears only as one more transit gateway attachment to route.

A transit gateway segments traffic through two separate settings on each attachment: the route table it is associated with decides where traffic arriving from that attachment can go, and the route tables it propagates into decide which tables learn a route to that attachment. Each attachment is associated with exactly one route table but can propagate into many.

The common isolation design for production, development and shared services:

• A production route table, associated with every production VPC attachment, that receives propagations only from shared services and the VPN or Direct Connect attachment.
• A development route table built the same way, so production and development never learn each other's routes.
• A shared route table, associated with the shared-services and VPN attachments, that receives propagations from every VPC so responses can return.

With this design a new VPC gets the right connectivity simply by being associated with the correct table and propagating into the shared table, which is what makes segmentation operable at hundreds of VPCs.

Return traffic uses a different table

A transit gateway looks up each packet in the table associated with the attachment the packet entered from. A request from a new spoke is looked up in the spoke table; the response from shared services is looked up in the shared table. If the new spoke's attachment does not propagate into (or have a static route in) the shared table, requests arrive but responses are dropped. One-way traffic through a transit gateway almost always means a missing route in the return attachment's table.

Blackhole routes and the egress-VPC hairpin

A blackhole static route drops matching traffic. Transit gateways use longest-prefix match, so a blackhole for a broad summary range (the whole private block the company allocates from) still lets more specific propagated routes, such as those for shared services, win. This matters whenever a default route points at a central egress VPC: the egress VPC sends private-range traffic back to the transit gateway through a table that knows every VPC, so a packet from one isolated environment to an address in another can leave on the default route and re-enter through the egress VPC. A blackhole route for the summary range in each isolated environment's table drops that traffic at the first hop, while internet-bound traffic still follows the default route.

Network ACLs inside each VPC can block CIDRs too, but they must be edited per VPC and are easy to miss when a VPC is added, so they are a weak substitute for routing-based segmentation.

AWS PrivateLink exposes a service by putting it behind a Network Load Balancer (or a Gateway Load Balancer, for appliances), creating a VPC endpoint service for that load balancer, and letting each consumer create an interface VPC endpoint in its own subnets. The consumer reaches the service through private IP addresses in its own VPC, so the two VPCs can use overlapping CIDRs, and the consumer can reach only that service, never any other resource in the provider VPC.

Controls the provider holds:

• Allowed principals on the endpoint service permissions limit which AWS accounts, IAM roles or users can create an endpoint to it.
• Acceptance required adds a manual approval of each endpoint connection on top of that.
• Supported Regions (cross-Region PrivateLink): the provider adds other Regions to the endpoint service, and consumers in those Regions create interface endpoints locally that carry traffic to the service in its home Region. The provider deploys nothing in the consumer Region, and neither side builds peering or inter-Region networking. Extra load balancers in the consumer Region are not needed, and they would need inter-Region networking to reach the provider's targets, which is exactly what cross-Region PrivateLink avoids.

Availability Zones are matched by AZ ID

A consumer can create an endpoint only in Availability Zones where the provider's load balancer is enabled. Cross-zone load balancing spreads traffic across targets but does not add zones to the service. Zone names such as us-east-1a are mapped to physical zones differently in every account, so a provider whose NLB uses two zone names may overlap with only one of a consumer's zones. Compare AZ IDs (such as use1-az4) and enable the NLB in subnets whose AZ IDs match the consumers' zones; an NLB can have zones added after creation.

PrivateLink is the natural answer for SaaS providers with many customers: no route tables to coordinate, no CIDR planning across companies, and access that is one-directional by design.

VPC sharing lets a central network account own a VPC and share its subnets with other accounts in the same organization through AWS RAM. The two roles split cleanly:

That split suits a company whose network team must keep control of subnets, routing and network ACLs while application teams still own and pay for their resources in their own accounts. Any design that puts application resources into the network account itself gives up that per-account ownership, billing and blast-radius separation.

Centralizing interface endpoints

Interface endpoints are billed per endpoint, per AZ, per hour, so repeating the same set of endpoints in every spoke VPC multiplies the charge by the number of VPCs. Create them once in a shared-services VPC reachable over the transit gateway, then fix name resolution: an endpoint's private DNS option creates a hidden private hosted zone that only its own VPC can use. Turn private DNS off on the central endpoints, create a Route 53 private hosted zone per service name (for example kms.eu-west-1.amazonaws.com) with an alias record to the endpoint, and associate each zone with every spoke VPC (cross-account associations need an association authorization). Applications keep their default SDK endpoint names and need no code change.

Gateway endpoints stay in each VPC

Amazon S3 and DynamoDB also offer gateway endpoints: free, added as a route-table target, and usable only from the VPC they belong to. Traffic arriving over a transit gateway, peering or VPN cannot use them, so the lowest-cost private path to S3 for many VPCs is a gateway endpoint in every VPC (S3 interface endpoints are reachable from other networks, but they are billed like every interface endpoint). An endpoint policy controls what passes through the endpoint; one that allows S3 access only when aws:ResourceOrgID equals the company's organization ID stops instances from copying data to buckets owned by anyone else. The control has to sit on the path out of the network: a resource policy is attached to the bucket it protects, so it governs who reaches the company's buckets, not where data can be sent.

Amazon VPC IP Address Manager (IPAM) prevents overlapping and exhausted address space by allocating every VPC CIDR from managed pools and tracking which ranges are in use across the organization. Integrated with AWS Organizations (through a delegated administrator account), IPAM builds a hierarchy: a top-level pool, Regional pools beneath it, then environment or business-unit pools. Pools are shared with OUs or accounts through AWS RAM, so infrastructure-as-code templates request "a /22 from the production eu-west-1 pool" and receive a non-overlapping range with no manual approval.

• Allocation rules on a pool set minimum, maximum and default netmask lengths and required tags for allocations.
• Monitoring shows utilization and flags resources whose CIDRs are unmanaged or overlap.

Pools and allocation rules apply only when a request names a pool; anyone can still type a CIDR by hand. The preventive control is a service control policy on the workload OU that denies ec2:CreateVpc and ec2:AssociateVpcCidrBlock (which adds secondary CIDRs) unless the ec2:Ipv4IpamPoolId condition key matches an approved pool (ec2:Ipv6IpamPoolId does the same for IPv6). Monitoring and compliance rules report a stray CIDR only after the VPC exists, so they are detective controls.

Subnet tiers

Route each private subnet to the NAT gateway in its own Availability Zone, so an AZ failure does not take another AZ's egress with it and cross-AZ data charges are avoided.

Running out of addresses

A subnet's CIDR is fixed for its lifetime and a VPC's primary CIDR block cannot be changed. When subnets fill up, associate a secondary IPv4 CIDR block with the VPC (ideally allocated from IPAM), create new subnets from it in each AZ, and add them to the Auto Scaling group or service. This keeps the workload, its endpoints and its connections in the same VPC; anything that relocates the workload is a migration project. Size subnets generously at the start, because every interface endpoint, load balancer node and container task consumes addresses too.

Security groups filter traffic per network interface and are stateful allow lists; network ACLs filter traffic per subnet and are stateless, ordered lists of allow and deny rules. Use security groups as the main control and network ACLs as a subnet-wide guardrail or to block specific addresses.

Defence in depth for a database tier pairs both: a security group that allows the database port only from the application tier's security group, and a custom network ACL on the database subnets that allows the port inbound from the application subnets' CIDRs and ephemeral ports outbound back to them. The ACL keeps the subnet closed even if someone later adds a permissive security group rule. To block a handful of abusive IP ranges quickly, add network ACL deny rules numbered lower than the allow rule; a security group cannot express a deny. Network ACLs have a rule quota per direction (20 by default, adjustable), so long block lists belong in AWS WAF or Network Firewall instead.

Reading VPC Flow Logs

Flow logs record metadata for each flow (addresses, ports, protocol, bytes, ACCEPT or REJECT), not packet contents. Their pattern points to the culprit:

• Inbound ACCEPT followed by outbound REJECT for the response: the stateless network ACL is missing an outbound ephemeral-port rule (a security group would have allowed the response automatically).
• Inbound REJECT: either the security group or the network ACL denies the request.

When a security tool needs the packet payloads, for an intrusion detection system for example, use VPC Traffic Mirroring: it copies traffic from chosen network interfaces to a target such as an NLB in a security VPC, without agents on the source instances. Logging features in general record metadata about traffic, never the packets themselves.

Centralized inspection routes traffic from many VPCs through one inspection point attached to the transit gateway: AWS Network Firewall when an AWS managed firewall is acceptable, and a Gateway Load Balancer in front of third-party appliances when the company must use a vendor product.

Inspection VPC and appliance mode

In the classic design, spoke route tables on the transit gateway send traffic to an inspection VPC whose transit gateway subnets route to a firewall endpoint in the same AZ. Without extra configuration the transit gateway enters the inspection VPC through the attachment interface in the sender's AZ, so a flow between instances in different AZs leaves through one AZ's firewall and returns through the other's. Each stateful firewall sees only half of the conversation and drops it. Turning on appliance mode on the inspection VPC's attachment makes the transit gateway pick one interface per flow for its whole life, keeping the path symmetric. The setting matters on the attachment of the VPC that holds the stateful appliances, because that is where the transit gateway chooses which AZ interface a flow uses. Network Firewall can also be attached directly to a transit gateway as a transit gateway-attached firewall, which removes the inspection VPC; the symmetric-routing concern belongs to designs that use a VPC.

Central egress with domain filtering

Replacing per-VPC NAT gateways with a central egress VPC (Network Firewall endpoints in front of shared NAT gateways, spokes sending 0.0.0.0/0 to the transit gateway) gives one inspection and logging point. To allow HTTPS only to approved domains, use a Network Firewall stateful domain list rule group: it matches the server name carried in the TLS handshake (SNI) or the HTTP Host header. Connections that carry no approved name match no allowed domain and are dropped by an allow list, so a workload cannot get around it by connecting straight to an IP address. Filtering at the DNS layer acts only on lookups that go through the resolver, so it complements, but cannot replace, filtering on the traffic itself.

Gateway Load Balancer for third-party appliances

A Gateway Load Balancer distributes traffic across a fleet of appliances, removes unhealthy ones through health checks, keeps every packet of a flow on the same appliance, and forwards the original packets encapsulated in GENEVE so addresses are never translated. Load balancers that terminate connections or rewrite addresses cannot put a transparent, bump-in-the-wire appliance in the path, which is why the GWLB exists. The appliances live in a security VPC, the GWLB is exposed as an endpoint service, and each application VPC gets GWLB endpoints, which also lets other VPCs or accounts use the same fleet later.

Inbound inspection to a public load balancer uses ingress routing: place the GWLB endpoints in their own subnets, associate an edge route table with the internet gateway that sends traffic for the load balancer subnets to the endpoint in the same AZ, and route the load balancer subnets' 0.0.0.0/0 to that endpoint so responses return through the appliance. The endpoints need subnets of their own, because each subnet has one route table and the endpoint subnets and load balancer subnets need different routes.

AWS WAF protects HTTP and HTTPS applications at layer 7 by evaluating requests against a web ACL associated with a supported resource: Amazon CloudFront distributions, Application Load Balancers, Amazon API Gateway REST APIs, AWS AppSync GraphQL APIs, Amazon Cognito user pools and a few others. It cannot be attached to a Network Load Balancer; non-HTTP traffic is filtered with network ACLs, security groups or Network Firewall.

• AWS managed rule groups cover common attacks without writing rules, such as the SQL database rule group for SQL injection and the core rule set for common web exploits.
• Rate-based rules count requests per client IP (or another key) over a time window and block or challenge sources above the limit, which handles login floods from single addresses.
• IP set rules allow or block lists of addresses at layer 7.

All of this needs no application change. Attacks carried inside HTTP requests, such as injection in query strings, are a web ACL job: DDoS-focused services and packet-level firewalls work at other layers.

Organization-wide baselines with AWS Firewall Manager

When every internet-facing ALB and CloudFront distribution across hundreds of accounts must carry a baseline, including resources and accounts created later, use AWS Firewall Manager. It needs AWS Organizations, a designated Firewall Manager administrator account, and AWS Config enabled in the member accounts and Regions in scope. An AWS WAF policy with automatic remediation creates and associates a web ACL on every in-scope resource as it appears. CloudFront needs a global policy and ALBs need a policy in each Region that has them. The policy's first and last rule groups are fixed by the central team, and account owners can still add their own rules between them. What Firewall Manager adds is continuous remediation: a one-time deployment of web ACLs does not reach resources created afterwards, and preventive policies cannot require an association, because associating a web ACL is a separate API call.

Migrating an existing VPC to dual stack means adding IPv6 alongside IPv4 at every layer: CIDR blocks, routes, filtering, load balancers and DNS. IPv4 keeps working throughout, and existing subnets cannot be converted to IPv6-only; IPv6-only workloads go in new subnets.

1. Addresses: associate an IPv6 CIDR (Amazon-provided, from IPAM, or your own) with the VPC and an IPv6 prefix (commonly a /64) with each subnet.
2. Routes: associating a CIDR adds no default route. Public subnets need ::/0 to the internet gateway. VPC IPv6 addresses are globally unique and NAT gateways do not translate IPv6 to IPv6, so private subnets send ::/0 to an egress-only internet gateway, which allows outbound-started connections and their responses but refuses any connection started from the internet, even if a security group is too open.
3. Security groups: they receive an IPv6 allow-all outbound rule automatically only if their outbound rules were never modified; inbound IPv6 rules are always added by you.
4. Network ACLs: only an unmodified default network ACL gets IPv6 rules automatically. Every custom network ACL needs IPv6 rules that mirror its IPv4 rules, or IPv6 traffic is denied.
5. Load balancers and DNS: switch the load balancer to a dual-stack IP address type and publish an alias AAAA record as well as the A record.

Load balancer IP address types

Each DNS record type answers only its own query type, so IPv6-only clients need an AAAA record whatever the target supports. Targets can stay IPv4 in every case.

IPv6-only subnets reaching IPv4-only services

An IPv6-only workload reaches IPv4-only destinations through DNS64 plus NAT64. Enable DNS64 on the subnet so the Route 53 Resolver synthesizes AAAA answers inside the well-known prefix 64:ff9b::/96 for names that have only A records, then route 64:ff9b::/96 to a NAT gateway, which performs NAT64 and sends the traffic out over IPv4. Native IPv6 destinations still go through ::/0 to the egress-only internet gateway. When software uses an IPv4 literal instead of a name, DNS64 never sees it; embed the address in the prefix yourself (192.0.2.33 becomes 64:ff9b::c000:221, the four octets written in hexadecimal) and NAT64 handles it the same way.

Multi-Region traffic is steered either by DNS with Route 53 routing policies, or by anycast networking with AWS Global Accelerator; choose Global Accelerator when clients need fixed IP addresses, failover that does not wait for DNS caches, or non-HTTP traffic.

Health matters as much as the policy. Alias records to AWS resources use Evaluate Target Health; other records attach health checks, and an unhealthy record is skipped. Policies combine by aliasing one record set to another: a geolocation record for Europe can point at the European Region while the Default geolocation record aliases a set of latency records for the remaining Regions. Note the residency trap: if the Europe record is health-checked and fails, Route 53 falls back to the Default record and sends European users elsewhere, so a record that must never fail over carries no health evaluation.

Route 53 health checkers run on the internet, so they cannot reach endpoints that have only private IP addresses, even for records in a private hosted zone. For private resources, publish a CloudWatch metric that reflects health, create an alarm on it, and use a health check that monitors the CloudWatch alarm.

AWS Global Accelerator

A standard accelerator gives two static anycast IP addresses that clients enter at the nearest AWS edge location, then carries traffic over the AWS network to healthy endpoints (ALBs, NLBs, EC2 instances, Elastic IPs) in endpoint groups, one group per Region. Because the IPs never change, enterprise customers can allow-list them, and Regional failover happens in seconds inside the network instead of waiting for DNS TTLs in client resolvers. It handles TCP and UDP.

• Traffic dials set the percentage of traffic each endpoint group (Region) accepts; traffic above the dial goes to the next closest healthy group. Lowering one Region's dial shifts users to another Region without changing the IPs.
• Endpoint weights split traffic between endpoints inside one group.
• Client IP address preservation on ALB (and EC2) endpoints lets the application see the user's real address, which the ALB then passes in X-Forwarded-For. ALBs do not accept Proxy Protocol.

A custom routing accelerator maps accelerator ports deterministically to specific EC2 instance IPs and ports in VPC subnet endpoints, so an application such as a game matchmaker can send each user to an exact session server behind a few static IPs without public instance addresses. Edge caching services serve HTTP content; pinning a session to one exact server port is what custom routing is for.