IPSec Protocol Framework
The preceding section discussed encryption, authentication, and integrity. This section explains how encryption, integrity, and authentication are applied to the IPSec protocol suite.
As mentioned, IPSec is a framework of open standards. IPSec spells out the messaging to secure the communications but relies on existing algorithms, such as DES and 3DES, to implement the encryption and authentication. The two main IPSec framework protocols are as follows:
AH—AH, shown in Figure 1-15, is the appropriate protocol when confidentiality is not required or permitted. It provides data authentication and integrity for IP packets passed between two systems. It is a means of verifying that any message passed from Router A to Router B was not modified during transit. It verifies that the data's origin was either Router A or Router B. AH does not provide data confidentiality (encryption) of packets. It does the following:
Ensures data integrity
Provides origin authentication (ensures that packets definitely came from the peer router)
Uses a keyed-hash mechanism
Does not provide confidentiality (no encryption)
Provides anti-replay protection
All text is transported in the clear.
)
Figure
1-15 AH
ESP—A security protocol may be used to provide confidentiality (encryption) and authentication. ESP, shown in Figure 1-16, provides confidentiality by performing encryption at the IP packet layer. IP packet encryption conceals the data payload and the identities of the ultimate source and destination. ESP provides authentication for the inner IP packet and ESP header. Authentication provides data origin authentication and data integrity. Although both encryption and authentication are optional in ESP, at a minimum, one of them must be selected. ESP provides
Data confidentiality (encryption)
Data integrity
Data origin authentication
Anti-replay protection
)
Figure
1-16 ESP
AH
Authentication is achieved by applying a keyed one-way hash function to the packet to create a hash or message digest. The hash is combined with the text and is transmitted. Changes in any part of the packet that occur during transit are detected by the receiver when it performs the same one-way hash function on the received packet and compares the value of the message digest the sender has supplied. The fact that the one-way hash also involves the use of a symmetric key between the two systems means that authenticity is guaranteed.
The AH function is applied to the entire datagram, except for any mutable IP header fields that change in transit (such as Time To Live [TTL] fields that are modified by the routers along the transmission path). AH works as follows:
The IP header and data payload are hashed.
The hash is used to build an AH header, which is inserted into the original packet.
The new packet is transmitted to the IPSec peer router.
The peer router hashes the IP header and data payload.
The peer router extracts the transmitted hash from the AH header.
The peer router compares the two hashes. The hashes must exactly match. Even if one bit is changed in the transmitted packet, the hash output on the received packet changes, and the AH header does not match.
AH supports the HMAC-MD5 and HMAC-SHA-1 algorithms. AH authentication and integrity are shown in Figure 1-17.
)
Figure
1-17 AH Authentication and Integrity
NOTE
AH is IP protocol 51. If you are using IPSec, ensure that any perimeter routers will pass IP protocol 51 as well as Internet Security Association and Key Management Protocol (ISAKMP)—that is, UDP port 500.
ESP
ESP, shown in Figure 1-18, provides confidentiality by encrypting the payload. It supports a variety of symmetric encryption algorithms. The default algorithm for IPSec is 56-bit DES. Cisco products also support the use of 3DES and AES for stronger encryption.
)
Figure
1-18 ESP Protocol
){kind=link}
ESP can be used alone or in combination with AH. ESP with AH also provides integrity and authentication of datagrams. First, the payload is encrypted. Next, the encrypted payload is sent through a hash algorithm—HMAC-MD5 or HMAC-SHA-1. The hash provides origin authentication and data integrity for the data payload.
Alternatively, ESP may also enforce anti-replay protection by requiring that a receiving host set the replay bit in the header to indicate that the packet has been seen.
Between two security gateways, the original payload is well protected, because the entire original IP datagram is encrypted. An ESP header and trailer are added to the encrypted payload. With ESP authentication, the encrypted IP datagram and the ESP header or trailer are included in the hashing process. Last, a new IP header is appended to the front of the authenticated payload. The new IP address is used to route the packet through the Internet.
When both ESP authentication and encryption are selected, encryption is performed before authentication. One reason for this order of processing is that it facilitates rapid detection and rejection of replayed or bogus packets by the receiving node. Before decrypting the packet, the receiver can authenticate inbound packets. By doing this, it can detect the problems and potentially reduce the impact of DoS attacks.
NOTE
ESP is IP protocol 50. If you are using IPSec, ensure that any perimeter routers pass IP protocol 50 as well as ISAKMP—that is, UDP port 500.
Modes of Operation
ESP and AH can be applied to IP packets in two different ways, or modes:
Transport mode
Tunnel mode
Transport Mode
Transport mode, shown in Figure 1-19, protects the packet's payload, higher-layer protocols, but leaves the original IP address in the clear. The original IP address is used to route the packet through the Internet. ESP transport mode is used between two hosts. Transport mode provides security to the higher-layer protocols only.
)
Figure
1-19 Transport Mode
Tunnel Mode
ESP tunnel mode, shown in Figure 1-20, is used when either end of the tunnel is a security gateway, a Concentrator, a VPN optimized router, or a PIX Firewall. Tunnel mode is used when the final destination is not a host, but a VPN gateway. The security gateway encrypts and authenticates the original IP packet. Next, a new IP header is appended to the front of the encrypted packet. The new outside IP address is used to route the packet through the Internet to the remote end security gateway. Tunnel mode provides security for the whole original IP packet.
)
Figure
1-20 Tunnel Mode