How IPSec Works
IPSec involves many component technologies and encryption methods. Yet IPSec's operation can be broken down into five main steps. The five steps are summarized as follows:
| Step 1 | Interesting traffic initiates the IPSec process—Traffic is deemed interesting when the IPSec security policy configured in the IPSec peers starts the IKE process. | |
| Step 2 | IKE phase one—IKE authenticates IPSec peers and negotiates IKE SAs during this phase, setting up a secure channel for negotiating IPSec SAs in phase two. | |
| Step 3 | IKE phase two—IKE negotiates IPSec SA parameters and sets up matching IPSec SAs in the peers. | |
| Step 4 | Data transfer—Data is transferred between IPSec peers based on the IPSec parameters and keys stored in the SA database. | |
| Step 5 | IPSec tunnel termination—IPSec SAs terminate through deletion or by timing out. |
This five-step process is shown in Figure 1-15.
){kind=link}
Figure 1-15 The Five Steps of IPSec
Step 1: Defining Interesting Traffic
Determining what type of traffic is deemed interesting is part of formulating a security policy for use of a VPN. The policy is then implemented in the configuration interface for each particular IPSec peer. For example, in Cisco routers and PIX Firewalls, access lists are used to determine the traffic to encrypt. The access lists are assigned to a crypto policy such that permit statements indicate that the selected traffic must be encrypted, and deny statements can be used to indicate that the selected traffic must be sent unencrypted. With the Cisco Secure VPN Client, you use menu windows to select connections to be secured by IPSec. When interesting traffic is generated or transits the IPSec client, the client initiates the next step in the process, negotiating an IKE phase one exchange.
Step 1 is shown in Figure 1-16.
){kind=link}
Figure 1-16 Defining Interesting Traffic
Step 2: IKE Phase One
The basic purpose of IKE phase one is to authenticate the IPSec peers and to set up a secure channel between the peers to enable IKE exchanges. IKE phase one performs the following functions:
Authenticates and protects the identities of the IPSec peers
Negotiates a matching IKE SA policy between peers to protect the IKE exchange
Performs an authenticated Diffie-Hellman exchange with the end result of having matching shared secret keys
Sets up a secure tunnel to negotiate IKE phase two parameters
IKE phase one occurs in two modes:
Main mode
Aggressive mode
Main Mode
Main mode has three two-way exchanges between the initiator and receiver.
First exchange—The algorithms and hashes used to secure the IKE communications are agreed upon in matching IKE SAs in each peer.
Second exchange—This exchange uses a Diffie-Hellman exchange to generate shared secret keying material used to generate shared secret keys and to pass nonces, which are random numbers sent to the other party, signed, and returned to prove their identity.
Third exchange—This exchange verifies the other side's identity. The identity value is the IPSec peer's IP address in encrypted form. The main outcome of main mode is matching IKE SAs between peers to provide a protected pipe for subsequent protected ISAKMP exchanges between the IKE peers. The IKE SA specifies values for the IKE exchange: the authentication method used, the encryption and hash algorithms, the Diffie-Hellman group used, the lifetime of the IKE SA in seconds or kilobytes, and the shared secret key values for the encryption algorithms. The IKE SA in each peer is bidirectional.
Aggressive Mode
In the aggressive mode, fewer exchanges are done and with fewer packets. In the first exchange, almost everything is squeezed into the proposed IKE SA values, the Diffie-Hellman public key, a nonce that the other party signs, and an identity packet, which can be used to verify the initiator's identity through a third party. The receiver sends everything back that is needed to complete the exchange. The only thing left is for the initiator to confirm the exchange. The weakness of using the aggressive mode is that both sides have exchanged information before there is a secure channel. Therefore, it is possible to sniff the wire and discover who formed the new SA. However, aggressive mode is faster than main mode.
Step 2 is shown in Figure 1-17.
){kind=link}
Figure 1-17 IKE Phase One
Step 3: IKE Phase Two
The purpose of IKE phase two is to negotiate IPSec SAs to set up the IPSec tunnel. IKE phase two performs the following functions:
Negotiates IPSec SA parameters protected by an existing IKE SA
Establishes IPSec security associations
Periodically renegotiates IPSec SAs to ensure security
Optionally performs an additional Diffie-Hellman exchange
IKE phase 2 has one mode, called quick mode. Quick mode occurs after IKE has established the secure tunnel in phase one. It negotiates a shared IPSec policy, derives shared secret keying material used for the IPSec security algorithms, and establishes IPSec SAs. Quick mode exchanges nonces that provide replay protection. The nonces are used to generate new shared secret key material and prevent replay attacks from generating bogus SAs.
Quick mode is also used to renegotiate a new IPSec SA when the IPSec SA lifetime expires. Base quick mode is used to refresh the keying material used to create the shared secret key based on the keying material derived from the Diffie-Hellman exchange in phase one.
Perfect Forward Secrecy
If perfect forward secrecy (PFS) is specified in the IPSec policy, a new Diffie-Hellman exchange is performed with each quick mode, providing keying material that has greater entropy (key material life) and thereby greater resistance to cryptographic attacks. Each Diffie-Hellman exchange requires large exponentiations, thereby increasing CPU use and exacting a performance cost.
Step 4: IPSec Encrypted Tunnel
After IKE phase two is complete and quick mode has established IPSec SAs, information is exchanged by an IPSec tunnel. Packets are encrypted and decrypted using the encryption specified in the IPSec SA. This IPSec encrypted tunnel can be seen in Figure 1-18.
){kind=link}
Figure 1-18 IPSec Encrypted Tunnel
Step 5: Tunnel Termination
IPSec SAs terminate through deletion or by timing out. An SA can time out when a specified number of seconds have elapsed or when a specified number of bytes have passed through the tunnel. When the SAs terminate, the keys are also discarded. When subsequent IPSec SAs are needed for a flow, IKE performs a new phase two and, if necessary, a new phase one negotiation. A successful negotiation results in new SAs and new keys. New SAs can be established before the existing SAs expire so that a given flow can continue uninterrupted. This can be seen in Figure 1-19.
){kind=link}
Figure 1-19 Tunnel Termination