High Availability: ELB and Route 53 Health Checks, Multi-AZ Design

The ALB-versus-NLB choice comes down to the layer at which the load balancer must make decisions: an Application Load Balancer reads HTTP requests, a Network Load Balancer forwards connections. Two further types exist for narrower jobs — the Gateway Load Balancer for appliance fleets and the legacy Classic Load Balancer, which new designs don't use.

Three combinations come up in operations work:

• Static IPs or PrivateLink in front of layer-7 routing. An NLB can forward to an ALB through an ALB-type target group. The NLB supplies the static IPs, or serves as the load balancer behind a VPC endpoint service (AWS PrivateLink accepts only an NLB or a GWLB, never an ALB directly), while the ALB keeps its path and host rules. PrivateLink consumers reach the service through interface endpoints in their own VPCs, so overlapping CIDR ranges on either side don't matter.
• Appliance insertion. A Gateway Load Balancer distributes traffic across a scalable fleet of virtual appliances, health-checks them and stops sending flows to failed ones. Route tables send traffic to a GWLB endpoint, so inspection is transparent to the applications on either side.
• Client IP preservation on an NLB changes only the source address a target sees (and therefore what the target's security group must allow). It plays no part in which target or zone receives a flow.

Every load balancer type places nodes in each enabled Availability Zone, which makes it the traffic-steering half of every Multi-AZ design in the rest of this lesson.

A single Application Load Balancer can serve many domains and services, because each listener evaluates an ordered list of rules and an HTTPS listener can hold several certificates.

• Rule order. Listener rules are evaluated from the lowest priority number to the highest, and the first rule that matches wins; the default action applies only when nothing matches. A broad rule such as /api/* with a lower number than a narrower /api/admin/* rule captures the narrower traffic too, so give the specific rule the lower number.
• Host-based routing. Rules can match the Host header, so shop.example.com and blog.example.com can reach different target groups through the same listener on port 443.
• Several certificates on one listener. An HTTPS listener has a default certificate plus a certificate list; the ALB uses SNI (the host name the client sends in the TLS handshake) to present the matching certificate. You don't need one load balancer or one port per domain.

Within a target group, the ALB's routing algorithm decides which healthy target gets each request: round robin (the default), least outstanding requests (useful when request cost varies a lot), or weighted random. The setting exists only on ALB target groups. An NLB has no such choice: it selects a target per flow with a flow hash of the connection's protocol, addresses and ports, so a long-lived connection stays on one target.

A target group is the unit you actually operate: a set of targets (instances, IP addresses, Lambda functions, or an ALB behind an NLB) plus the health-check and traffic settings that decide which of them receive traffic. One load balancer can route to many target groups, and one target group's health-check settings apply to every target in it — which is why a bad health-check setting takes out the whole group at once.

Cross-zone load balancing controls whether a load balancer node in one AZ can send traffic to targets in another.

With cross-zone off on an NLB, clients' DNS lookups spread traffic roughly evenly across the zones' nodes, so an AZ with fewer targets gives each of them more work: an AZ holding three targets while another holds nine leaves the three each carrying roughly three times the load of their peers. There are two fixes. Turn cross-zone on (which adds inter-AZ data charges), or keep the same number of targets in every enabled AZ — for example by letting a balanced Auto Scaling group place them — which costs nothing extra.

Deregistration delay (called connection draining on the Classic Load Balancer) is how a target leaves gracefully. A deregistered target enters the draining state: it gets no new requests, while in-flight requests have up to the configured delay to finish — 300 seconds by default, configurable from 0 to 3,600 seconds. When an Auto Scaling group scales in an instance attached to a target group, it waits for draining to finish before terminating it. Tune the value to the workload. Raise it above the longest legitimate request (long file transfers, streaming exports), or anything still running when the delay expires is cut off. Lower it to just above the longest request when every request finishes in seconds and deployments sit in draining for minutes for no reason.

ELB health checks are periodic probes that each load balancer node sends to every registered target, and their settings live on the target group:

• Protocol — HTTP or HTTPS for an ALB target group (ALB target groups have no TCP health check); TCP, HTTP or HTTPS for NLB and GWLB target groups.
• Port — traffic-port (the default) probes each target on the port it was registered with, so the check follows the application if targets are re-registered on a new port. An explicit port number overrides that: if the application moves to a new port but the check still names the old one explicitly, every target fails even though the application is serving.
• Path — a lightweight endpoint such as /health.
• Success codes (matcher) — 200 by default; an ALB accepts a list or range such as 200,302 or 200-399. Health checks don't follow redirects: a 3xx response is judged by its own status code against the matcher, so an application that redirects unauthenticated requests (for example to a sign-in page) fails a 200-only check while working normally. The cleaner fix is an unauthenticated health path that returns 200; the alternative is a matcher that also accepts the redirect status.
• Interval, timeout and thresholds — the unhealthy threshold is the number of consecutive failures before a target is marked unhealthy; the healthy threshold is the number of consecutive successes before it is trusted again. Detection time is roughly interval × threshold: with a 10-second interval and an unhealthy threshold of 3, a crashed target leaves rotation about 30 seconds later, and with a healthy threshold of 5 it returns about 50 seconds after recovery.

What happens to an unhealthy target: the load balancer stops routing new requests to it but keeps probing, and traffic returns once it passes the healthy threshold. The load balancer itself never terminates or replaces anything. Replacement is the Auto Scaling group's job, and only if the group's health check type includes ELB; with the default EC2 status checks only, an instance with a dead application but a running VM stays in the fleet, unhealthy, indefinitely.

Fail open. When every target in an ALB target group is unhealthy, the ALB routes requests to all of them anyway rather than refusing traffic — so a broken health check alone often causes little user impact. Fail-open ends as soon as enough targets are healthy again, and from then on only the healthy targets get traffic. If targets recover gradually, the first few to pass absorb the whole load and may be overwhelmed before the rest catch up. The target group health settings control this: the unhealthy state routing threshold (default: a minimum of 1 healthy target; can be set as a count or a percentage) defines how many healthy targets are needed before the load balancer stops failing open. A related DNS failover threshold decides when a zone's load balancer node is taken out of DNS.

When every target goes unhealthy at the same moment, in every AZ, the cause is a systemic mismatch between the health check and the application, not a problem with individual instances. Work this list:

1. Path. Did the new version move or drop the health endpoint? A check against /health that now returns 404 fails every target. Curl the path from inside the VPC and read the actual status code.
2. Port and protocol. Is the health check set to an explicit port the application no longer listens on, instead of traffic-port? Does the application now redirect HTTP to HTTPS (a 301 against a matcher of 200)?
3. Security groups. The instance security group must allow inbound traffic from the load balancer — best expressed by referencing the ALB's security group — on the health-check port. Security groups are stateful: replies to allowed inbound probes go out regardless of outbound rules, so editing outbound rules doesn't cause this. Rules on the client-facing side (the ALB's own inbound rules, or NACLs on the path from users) affect clients reaching the ALB, not target health.
4. The application itself. A deploy-wide fault — bad configuration, a database login that fails at startup — that makes the application answer 500 everywhere is a genuine all-unhealthy.

Contrast with some targets unhealthy: that is instance-level (one host's crashed process, a full disk) or zonal, so you investigate the specific instance or AZ, not the target-group configuration.

Errors the load balancer generates itself point at where the fault is:

Route 53 health checks decide whether DNS gives out an answer at all: when the health check associated with a record fails, Route 53 stops returning that record and answers with the next-best healthy record instead. There are three types:

• Endpoint health checks probe an IP address or domain name over HTTP, HTTPS or TCP from Route 53 health checkers in several locations around the world. Route 53 combines their reports into one status: the endpoint is healthy if more than 18% of the checkers report it healthy. HTTP and HTTPS checks can add string matching: Route 53 searches the first 5,120 bytes of the response body for a string you specify, which catches an application that returns 200 with an error page. Choose a phrase that appears only in good responses.
• Calculated health checks combine other health checks: the parent check is healthy while at least a threshold number of its child checks are healthy, which expresses rules such as "the application is up if at least two of three endpoints are up."
• CloudWatch alarm health checks take their state from a CloudWatch alarm. Use them for private resources: Route 53 health checkers run outside your VPC and can probe only publicly reachable endpoints, so a private instance is monitored through its metrics and an alarm instead.

Failover routing is the active-passive pattern: a PRIMARY record with a health check, and a SECONDARY record (often a static site or a standby stack) that Route 53 serves only while the primary is unhealthy. How the primary gets its health depends on the record type:

Because a CNAME can't sit at the zone apex, failover for example.com itself uses alias records with Evaluate Target Health (or an attached health check). A failover record that "never fails over" usually has neither a health check nor Evaluate Target Health on the primary.

Resilient environments use two layers of health checking, and each layer handles a different failure. Inside a Region, ELB is the layer. The standard highly available web tier is an Auto Scaling group (from a launch template, minimum of two instances) spanning two or more Availability Zones, registered with a load balancer whose nodes span the same AZs, with the group's health check type set to ELB. An instance fails → the load balancer stops routing to it after a few failed probes → the Auto Scaling group terminates and replaces it. An AZ fails → load balancer nodes in the surviving AZs keep serving, and the group launches capacity in the remaining zones. No DNS change is involved.

Two Auto Scaling timers interact with health checks. The health check grace period delays the group's first health evaluation of a new instance; if the application takes longer to bootstrap than the grace period, the group marks the instance unhealthy and replaces it before it is ready, over and over. Set the grace period longer than the real bootstrap time. The default instance warmup is different: it controls when a new instance's metrics start counting toward scaling decisions, and has nothing to do with health.

Across Regions, Route 53 is the layer. Failover records that alias each Region's load balancer, with Evaluate Target Health or a health check on each, move users to the secondary Region when the primary stops answering. DNS failover is slower: resolvers and clients keep using a cached answer until its TTL expires, so some users keep reaching the failed Region for a while after Route 53 switches. Keep TTLs low on failover records.

Each layer needs a different depth of health check:

Some failovers should be started by an operator, or should move traffic away from a zone that health checks still consider healthy. Amazon Application Recovery Controller (ARC) covers both cases:

Zonal shift is off by default on ALBs and NLBs: you enable the zonal shift integration (a load balancer attribute) before you can start a shift. Plan for the capacity: the remaining zones must absorb the shifted traffic. ARC's older readiness check feature is no longer open to new customers, so new designs use the features in the table.

Two related services are easy to confuse with ARC:

• AWS Global Accelerator gives an application two static anycast IP addresses (global, not per AZ) and routes users over the AWS network to the nearest healthy Regional endpoint group. A traffic dial per endpoint group sets the percentage of traffic that group receives, so you can drain a Region without waiting for DNS caches to expire. ALB endpoints receive TCP only; UDP needs NLB or EC2 endpoints.
• AWS Resilience Hub assesses an application against the RTO and RPO you define and recommends improvements, alarms and recovery procedures. It does not perform failover.

An RDS Multi-AZ DB instance deployment keeps a standby in a different Availability Zone, updated by synchronous replication: the primary doesn't acknowledge a write until the standby has it. When the primary or its AZ fails (or during some maintenance), RDS fails over automatically, typically in 60–120 seconds: it promotes the standby and points the DB endpoint's DNS name at it. The standby is never readable — it serves no queries and adds no capacity.

Two corrections to common assumptions. A read replica in another AZ does not make an instance Multi-AZ: replication is asynchronous and nothing promotes it automatically. To offload reporting from a Multi-AZ DB instance without changing its deployment type, add a read replica and point the reports at the replica's endpoint. And RDS Proxy read-only endpoints route only to reader instances of an Aurora cluster or an RDS Multi-AZ DB cluster, so they can't make a Multi-AZ DB instance deployment's standby readable. Multi-AZ DB clusters are available for RDS for MySQL and RDS for PostgreSQL.

The client side of a failover. Failover changes the IP address behind the endpoint name, so applications must connect by endpoint name (never a stored IP address) and must not cache DNS for long. RDS recommends a DNS cache TTL of less than 30 seconds for clients that cache. Java is the usual problem: the JVM caches lookups according to networkaddress.cache.ttl. Current JDKs default to about 30 seconds, but older JVMs running with a security manager cached successful lookups forever, and some applications set a long or infinite value explicitly; either way the application keeps connecting to the old primary until it restarts. Set that property to a short value.

The cache tier needs failover too, because losing a cache sends its full read load to the database. For Valkey or Redis OSS, a replication group with replicas in other AZs and Multi-AZ enabled promotes a replica automatically when the primary node fails. The application recovers only if it connects through the right endpoint:

A configuration endpoint exists only for cluster-mode-enabled (sharded) clusters. Memcached behaves differently: a node-based Memcached cluster has no replication and no failover, so losing a node loses that node's data. ElastiCache Serverless for Memcached does replicate data across AZs. When a cache must survive node or AZ loss, use a Valkey or Redis OSS replication group with Multi-AZ, or a serverless cache.

Static stability means an environment survives an Availability Zone failure without changing anything or launching anything at the moment of failure: the surviving capacity and network paths are already in place. Start by knowing what recovers by itself. Automatic: a load balancer stops routing to failed targets and AZs; an Auto Scaling group replaces failed instances; RDS Multi-AZ promotes its standby; a Valkey or Redis OSS replication group with Multi-AZ promotes a replica; zonal autoshift moves load balancer traffic. Operator action required: promoting an RDS read replica; repointing a DNS record that has no health check; launching capacity that wasn't pre-provisioned. If the recovery path includes a human step, the design isn't fault tolerant as described.

Capacity math. If launching instances during an AZ event can't be relied on, pre-provision enough that the surviving zones carry peak load alone: with N zones, run peak ÷ (N − 1) in each. For example, a tier that needs 8 instances at peak across three AZs runs 4 per AZ (12 in total), so losing one zone still leaves 8.

Then look for single-AZ dependencies, because one shared component can undo an otherwise Multi-AZ design. The classic trap is a single zonal NAT gateway: a standard (zonal) NAT gateway lives in one AZ, so if every private subnet routes through it and that AZ fails, healthy instances in the other AZs lose outbound internet access. There are two statically stable fixes:

The same logic applies to VPC interface endpoints: place an endpoint network interface in each AZ your workload uses. Deeper subnet and routing mechanics belong to the networking domain; here, the skill is finding the shared single-AZ component and removing it.