CCNA IPsec VPNs: Site-to-Site, Remote Access, ESP, AH, Tunnel and Transport Mode

IPsec is an open-standard framework of protocols that protects IP packets at Layer 3, so every application carried over IP (web, voice, file transfer, routing updates inside a tunnel) is protected without the applications knowing or changing. It is a framework rather than one algorithm: the peers negotiate which security protocol, cipher, hash, key-exchange group and authentication method to use, and older algorithms can be replaced without changing the framework.

IPsec provides four security services. Each comes from a different building block, which is why a configuration line can be read service by service.

Anti-replay is the service people forget. A replayed packet is an exact, unaltered copy, so it still decrypts and its integrity check still verifies; only the sequence number shows that the receiver has already accepted it. Keys themselves are created by Diffie-Hellman during IKE, never by the cipher or the HMAC.

In a transform set, each keyword names one of these blocks:

crypto ipsec transform-set TS-SALES esp-aes 192 esp-sha384-hmac
 mode tunnel

esp-aes 192 is the cipher (confidentiality); esp-sha384-hmac is the HMAC (integrity and origin authentication); mode tunnel is the encapsulation mode.

A site-to-site VPN joins whole networks through two gateways, and the hosts behind them are unaware of it; a remote-access VPN joins one user's device to the corporate network, with VPN client software on that device. Both send private traffic across the internet inside an encrypted tunnel; they differ in where the tunnel ends and whom it authenticates.

The deciding facts in a design question are usually where the users are and whether software can be installed on the hosts. Fixed sites full of hosts that cannot run a client call for site-to-site; users who move between networks they do not own call for remote access. A GRE tunnel by itself is not a VPN in the security sense: it encapsulates but neither encrypts nor checks integrity.

ESP (Encapsulating Security Payload, IP protocol 50) encrypts and authenticates packets; AH (Authentication Header, IP protocol 51) only authenticates them and leaves the contents readable. Both are carried directly over IP with a protocol number, not over TCP or UDP, so neither has a port.

Readable application data inside protocol 51 packets is therefore normal AH behaviour, not a fault. Whether a payload is encrypted is decided by the protocol, not by the mode: ESP encrypts in both tunnel and transport mode (the special null-encryption transform is an algorithm choice, not a mode), and AH encrypts in neither.

Every ESP or AH packet carries an SPI (Security Parameter Index) that tells the receiver which security association to use, and a sequence number for anti-replay.

Tunnel mode encapsulates the entire original packet, header included, behind a new outer IP header; transport mode keeps the original IP header and protects only what follows it. Tunnel mode is the default on Cisco routers and the normal choice between gateways; transport mode is used when the two hosts protecting the traffic are the same devices that sent it.

The ICV (integrity check value) comes last; the ESP trailer holds the padding and next-header field.

To hide both the data and who is talking, you need ESP in tunnel mode. AH in either mode encrypts nothing, so even AH tunnel mode leaves the inner addresses and payload readable; it only adds a new header and an integrity check. The mode decides encapsulation and size. It does not decide whether the packet is wrapped in UDP (NAT detection does that) or which destinations a remote client sends into the tunnel (split tunneling does that).

IKE (Internet Key Exchange) is the control protocol that authenticates the peers, agrees on algorithms and creates the keys; it runs over UDP port 500 (UDP 4500 once NAT is detected), and it never carries user data. The result of IKE is a set of security associations (SAs): agreements on algorithms, keys and lifetimes. ESP or AH then carry the data using those SAs.

IKEv1: two phases

• Phase 1 builds one bidirectional ISAKMP (IKE) SA, a secure channel for further negotiation. Main mode uses six messages: agree a policy, run Diffie-Hellman, then authenticate each other inside the new encrypted channel. Aggressive mode compresses this into three messages but exposes more of the exchange.
• Phase 2 (quick mode), three messages protected by the phase 1 SA, negotiates the IPsec SAs: the transform set (ESP or AH, cipher, HMAC, mode), the traffic to protect (the proxy identities from the crypto ACL) and optional PFS.

The phase 1 policy and the phase 2 transform set hold different parameters, and mixing them up is a common mistake.

Peers must find one ISAKMP policy where encryption, hash, authentication method and DH group all match; for the lifetime, Cisco accepts a peer whose proposal is equal to or shorter than its own and uses the shorter value. What to protect (the crypto ACL) and with whom (the peer address) are crypto map settings, not part of either policy.

crypto isakmp policy 10
 encryption aes 256
 hash sha256
 authentication pre-share
 group 14
 lifetime 86400
crypto isakmp key MySecret address 192.0.2.77
!
crypto map VPNMAP 10 ipsec-isakmp
 set peer 192.0.2.77
 set transform-set TS-SALES
 set pfs group14
 match address VPN-TRAFFIC
!
interface GigabitEthernet0/0/0
 crypto map VPNMAP

IKEv2: four messages

IKEv2 replaces main, aggressive and quick mode with two request/response exchanges. IKE_SA_INIT agrees algorithms, runs Diffie-Hellman and detects NAT; IKE_AUTH authenticates the peers and creates the first child (IPsec) SA. Four messages therefore produce both the IKE SA and the first IPsec SA; further child SAs and rekeys use CREATE_CHILD_SA. IKEv2 still builds an IKE SA first, still uses UDP 500 and 4500, and has NAT traversal built in. It is the version Cisco recommends for new VPNs.

show crypto isakmp sa shows how far IKEv1 phase 1 got, and show crypto ipsec sa shows the phase 2 IPsec SAs and their packet counters; together they locate a failure in phase 1, in phase 2 or in the data path. (For IKEv2 the equivalent is show crypto ikev2 sa, where a finished SA shows READY.)

QM_IDLE is reached only after a successful exchange, so it rules out a silent peer and a key mismatch. In the same way, phase 2 parameters (transform sets, crypto ACLs, PFS) are compared only in quick mode, after phase 1 has finished, so main mode states point to phase 1 settings, reachability or authentication.

IPsec SAs are one-way

An IPsec SA protects traffic in one direction only, so a working tunnel shows at least one inbound and one outbound ESP SA for each pair of protected networks, each with its own SPI. The receiver uses the SPI in each arriving packet to pick the inbound SA. The IKE SA, by contrast, is bidirectional and appears only in the ISAKMP output.

R1# show crypto ipsec sa
   local  ident (addr/mask/prot/port): (172.20.8.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (172.20.40.0/255.255.255.0/0/0)
   current_peer 192.0.2.77 port 500
    #pkts encaps: 3127, #pkts encrypt: 3127, #pkts digest: 3127
    #pkts decaps: 2984, #pkts decrypt: 2984, #pkts verify: 2984

Read the counters by direction. encaps counts packets this router encrypted and sent; decaps counts packets it received and decrypted. In a healthy tunnel, like the one above, both counters climb together. When one direction climbs and the other stays at zero, the SAs exist (IKE has finished both phases), so the fault is in the data path of the silent direction: either protected packets are lost on the way (for example a filter that admits IKE but not IP protocol 50), or the far router never encrypts its traffic because it routes it out an interface that has no crypto map. Negotiation faults such as a mismatched key or transform set show up earlier and differently: as a missing SA, not as one-sided counters.

A firewall between two IPsec peers must permit IKE on UDP 500 and the data protocol, which is ESP (IP protocol 50) when no NAT is in the path, or UDP 4500 when NAT traversal is in use. Because ESP and AH have no ports, a rule written for TCP or UDP port 50 or 51 matches nothing.

NAT traversal (NAT-T) exists because NAT and PAT devices translate by port, and raw ESP has none. During the first IKE exchange both peers send hashes of their addresses and ports; if what arrives does not match, a NAT device is in the path. The peers then move IKE to UDP 4500 and wrap every ESP packet in a UDP 4500 header, which the NAT device can translate like any other UDP flow. Keepalives hold the translation open. NAT-T is a standard extension for IKEv1 and part of IKEv2; Cisco IOS XE enables it by default. Without NAT, ESP stays as raw protocol 50 under both IKE versions.

AH cannot cross NAT, even with NAT-T: AH's integrity check includes the outer IP addresses, the NAT device rewrites them, and the receiver's check fails. IKE can still complete, so the symptom is a tunnel that comes up but passes no data, fixed by using ESP.

Configuring NAT itself, and the ACL syntax for these rules, are covered in the NAT and PAT and IPv4 ACL lessons.

Diffie-Hellman (DH) lets two peers derive the same shared secret by exchanging only public values, so the keys are never sent across the network; peer authentication (pre-shared keys or certificates) then proves that the other side really is the intended peer. DH creates keys but proves nothing about identity, which is why both are needed.

The DH group sets the strength of the exchange: group 14 (2048-bit) and the elliptic-curve groups 19, 20 and 21 are current choices, while groups 1, 2 and 5 are legacy. Both peers must agree on the group; the group says nothing about who the peer is.

Perfect forward secrecy

Without PFS, the keys of each new IPsec SA are derived from the keying material of the phase 1 (IKE) SA, however often the SAs rekey. With PFS (set pfs group14 in the crypto map), every IPsec SA, including each rekey, runs a fresh DH exchange, so exposing the keys of one SA does not help an attacker derive the keys of the SAs before or after it. Forward secrecy is a property of how keys are derived, so it comes only from that per-SA DH exchange. The cost is the extra DH computation at each rekey.

Pre-shared keys vs certificates

For certificate authentication each router needs a trustpoint for the CA, an identity certificate obtained by enrolling with that CA, and a correct clock (NTP), because each peer checks the other certificate's validity dates and a router with the wrong time rejects a valid certificate as expired or not yet valid. The certificate carries the peer's public key and the CA's signature vouches for the identity; the session keys still come from Diffie-Hellman.

GRE over IPsec carries multicast and routing-protocol traffic across an encrypted VPN: GRE (IP protocol 47) encapsulates any packet, including OSPF or EIGRP multicast hellos, into unicast GRE packets between the two routers, and IPsec then encrypts those GRE packets. A classic crypto-map tunnel protects only unicast traffic that matches its crypto ACL, so routing adjacencies cannot form across it: a crypto map has no mechanism for carrying multicast at all.

The two protocols complement each other: GRE alone carries multicast but encrypts nothing; IPsec alone encrypts but, with crypto maps, carries no multicast. A routed tunnel interface protected by IPsec, such as a GRE tunnel with tunnel protection ipsec profile or a static virtual tunnel interface (VTI), gives you an interface that routing protocols can run over.

Choose transport mode when the endpoints match

When the GRE tunnel's source and destination are the same two addresses as the IPsec peers, GRE has already added an outer IP header with those addresses. Tunnel mode would add a second, identical header for nothing, so transport mode is the efficient choice: the GRE header and everything inside it become the ESP payload and are still encrypted, and only the redundant header (and its bytes per packet) is saved. Protection is the same in either mode; the ESP trailer and ICV are present in both.

crypto ipsec transform-set TS-GRE esp-aes 256 esp-sha256-hmac
 mode transport
crypto ipsec profile GRE-PROT
 set transform-set TS-GRE
!
interface Tunnel0
 ip address 172.16.0.1 255.255.255.252
 tunnel source GigabitEthernet0/0/0
 tunnel destination 192.0.2.77
 tunnel protection ipsec profile GRE-PROT

Remote-access VPNs run over either IKEv2/IPsec or TLS with DTLS, and the choice decides which networks the client can connect from; split tunneling then decides which of the user's traffic goes through the tunnel. Cisco Secure Client supports both transports, terminating on a headend such as a Cisco Secure Firewall (ASA or Threat Defense) or an IOS XE router.

The DTLS fallback matters for voice and video. Normally the client sends data over DTLS on UDP 443, which suits real-time traffic. When UDP is unavailable, the client keeps the TLS session and carries everything over TCP instead. Bulk transfers barely notice, but real-time media suffers, because TCP retransmissions and head-of-line blocking delay and bunch up voice packets. Standard IKE/IPsec uses UDP 500 and 4500, so it cannot pass a filter that admits only web traffic; TLS on TCP 443 is the transport designed for that case.

Split tunneling vs full tunneling

With full tunneling (tunnel-all), the client sends every destination into the VPN, so the headend and the HQ security stack (proxy, firewall) see and can inspect all of the user's traffic, at the cost of HQ internet bandwidth. With split tunneling, the headend pushes a policy and the client itself decides per destination: listed corporate prefixes go into the tunnel, everything else goes straight out the local internet connection, unprotected by the VPN and never seen by the headend.

Hairpinning (U-turn) is different: it lets traffic that arrived through the tunnel leave again through the same outside interface of the headend, for example full-tunnel users browsing the internet via HQ. Hairpinned traffic still enters and leaves HQ, so it adds to the HQ link's load; bandwidth is saved only by keeping traffic out of the tunnel, which is what split tunneling does.