What is ESP?

Encapsulating Security Payload (ESP) is the IPsec protocol, IP protocol number 50, that encrypts and authenticates the contents of a tunneled IP packet. It is the workhorse of IPsec: while IKE negotiates the keys, ESP does the actual work of protecting every packet that crosses the tunnel, in Versa’s SD-WAN overlay and its SASE client connections alike.

Key takeaways

• ESP encrypts, authenticates, and integrity-protects tunneled IP packets.
• It runs as IP protocol number 50, distinct from AH and plain IKE.
• IKE negotiates the security association; ESP applies it packet by packet.
• Tunnel mode wraps the whole packet; transport mode wraps only the payload.
• Versa uses ESP for SD-WAN overlay paths and SASE client VPN tunnels.

On this page: Why it matters · How it works · Types · ESP & Versa · FAQ

Why is ESP important?

ESP is the part of IPsec that does the actual data protection. IKE handles peer authentication and key negotiation, but every packet crossing an IPsec tunnel is wrapped, encrypted, and authenticated by ESP. It operates below the application layer, so any protocol inside the tunnel is protected without modification – why ESP underlies site-to-site VPNs, SD-WAN overlays, and remote-access clients alike.

ESP also displaced its sibling protocol, Authentication Header (AH), as the default IPsec choice. AH authenticates a packet, including outer header fields, but provides no encryption – and because it covers those fields, address rewriting makes it impossible to carry across a NAT gateway. ESP authenticates only the payload, so it tolerates NAT far better – but not unaided: ESP is IP protocol 50 and has no ports for a NAT device to map, so in the general case tunnels rely on NAT-Traversal (NAT-T, RFC 3948), which detects a NAT on the path during IKE negotiation and encapsulates ESP inside UDP port 4500. That workable NAT story, versus AH’s impossible one, is one reason ESP became the near-universal choice.

What is ESP?

Structurally, ESP wraps the protected data between a header and a trailer. The header carries a Security Parameter Index (SPI), identifying which security association – and therefore which keys and algorithms – to use, plus a sequence number guarding against replay. The payload is encrypted with the negotiated cipher, and a trailer adds padding, the next-protocol field, and an optional Integrity Check Value for tamper detection.

The security association (SA) governing an ESP flow is negotiated by IKE beforehand: cipher, hash or AEAD mode, and key lifetime. ESP itself negotiates nothing – it applies whatever the SA specifies, which is why Versa’s VOS devices expose ESP as a distinct “IPsec transform,” separate from the IKE transform used for key exchange.

Types of ESP

ESP supports two encapsulation modes. Tunnel mode wraps an entire original IP packet inside a new outer packet, hiding the original addressing – used for virtually all site-to-site and remote-access VPNs. Transport mode encrypts only the payload, keeping the original header exposed, used mainly for host-to-host protection.

Cipher choice has evolved through generations. Early deployments paired 3DES or AES-CBC with a separate MD5 or SHA hash. Modern deployments prefer combined (AEAD) modes such as AES-GCM, encrypting and authenticating in a single, hardware-accelerated pass. A NULL-encryption option also exists – authentication only, no encryption. Versa’s VOS devices expose this full range: 3DES and AES in CBC, GCM, and CTR modes, paired with MD5, SHA-1, SHA-2, or XCBC hashing, plus NULL.

ESP & Versa

Versa uses ESP in two places. In the SD-WAN overlay, ESP encrypts the control- and data-plane traffic flowing over IPsec paths between Controller nodes and CPE devices – every overlay path is, underneath, an ESP-protected tunnel. In the Versa Secure Access Service, ESP protects the tunnel between a SASE client and its gateway: administrators set IKE and IPsec fields on separate tabs, choosing an IKE transform for key exchange and a distinct IPsec transform – cipher and hashing – plus a PFS group and an SA lifetime (28,800 seconds by default).

The SASE client can try a tunnel order of DTLS, IPsec, or TLS, so ESP-based IPsec is one of three transport options, configured identically for Secure SD-WAN and NGFW deployments. Tunnel authentication uses a CA certificate or preshared key.

Frequently asked questions

What is the difference between ESP and AH?

AH authenticates a packet, including outer header fields, but provides no encryption; ESP encrypts the payload and authenticates only what it protects. Because ESP does not cover the outer header, it can cross a NAT device – though as a portless IP protocol it generally needs NAT-Traversal (UDP 4500 encapsulation) to do so – while AH cannot be NATed at all. That is why nearly all modern IPsec deployments use ESP, and AH is rarely deployed.

What is the ESP header made of?

An ESP header carries a Security Parameter Index identifying the security association and a sequence number that prevents replay. It precedes the encrypted payload; a trailer follows with padding, a next-header field, and an optional integrity check value for authentication.

Does ESP always encrypt traffic?

Not necessarily – ESP supports a NULL-encryption option that authenticates a packet without encrypting it, useful for troubleshooting or where confidentiality is handled elsewhere. In production, including Versa deployments, ESP is almost always configured with a real cipher such as AES-GCM or AES-CBC.

What is the difference between ESP tunnel mode and transport mode?

Tunnel mode encapsulates an entire original IP packet inside a new encrypted one, hiding internal addressing – the default for site-to-site and remote-access VPNs. Transport mode encrypts only the payload and keeps the original header exposed, used mainly for direct host-to-host protection.

What negotiates the keys that ESP uses?

IKE (Internet Key Exchange) negotiates the security association ESP relies on – authenticating the peers and agreeing on ciphers, hashes, and key lifetimes. ESP itself performs no negotiation; it applies whatever the active security association specifies to every packet it protects.

More on ESP

If you want to learn more, these are sources we trust.

Related terms

If you’re reading about ESP, these are worth looking at next.