The IP security (IPSec) is an Internet Engineering Task Force (IETF) standard suite of protocols between 2 communication points across the IP network that provide data authentication, integrity, and confidentiality. It also defines the encrypted, decrypted and authenticated packets. The protocols needed for secure key exchange and key management are defined in it.
Uses of IP Security –
IPsec can be used to do the following things:
IPsec can be used to do the following things:
- To encrypt application layer data.
- To provide security for routers sending routing data across the public internet.
- To provide authentication without encryption, like to authenticate that the data originates from a known sender.
- To protect network data by setting up circuits using IPsec tunneling in which all data is being sent between the two endpoints is encrypted, as with a Virtual Private Network(VPN) connection.
- Any scheme that is developed for providing network security needs to be implemented at some layer in protocol stack as depicted in the diagram below −
Layer Communication Protocols Security Protocols Application Layer HTTP FTP SMTP PGP. S/MIME, HTTPS Transport Layer TCP /UDP SSL, TLS, SSH Network Layer IP IPsec The popular framework developed for ensuring security at network layer is Internet Protocol Security (IPsec).
Virtual Private Network
Ideally, any institution would want its own private network for communication to ensure security. However, it may be very costly to establish and maintain such private network over geographically dispersed area. It would require to manage complex infrastructure of communication links, routers, DNS, etc.
IPsec provides an easy mechanism for implementing Virtual Private Network (VPN) for such institutions. VPN technology allows institution’s inter-office traffic to be sent over public Internet by encrypting traffic before entering the public Internet and logically separating it from other traffic.
The simplified working of VPN is shown in the following diagram −
IP Security Architecture

Overview of IPsec
IPsec is a framework/suite of protocols for providing security at the IP layer.
Origin
In early 1990s, Internet was used by few institutions, mostly for academic purposes. But in later decades, the growth of Internet became exponential due to expansion of network and several organizations using it for communication and other purposes.
With the massive growth of Internet, combined with the inherent security weaknesses of the TCP/IP protocol, the need was felt for a technology that can provide network security on the Internet. A report entitled "Security in the Internet Architecture” was issued by the Internet Architecture Board (IAB) in 1994. It identified the key areas for security mechanisms.
The IAB included authentication and encryption as essential security features in the IPv6, the next-generation IP. Fortunately, these security capabilities were defined such that they can be implemented with both the current IPv4 and futuristic IPv6.
Security framework, IPsec has been defined in several ‘Requests for comments’ (RFCs). Some RFCs specify some portions of the protocol, while others address the solution as a whole.
Operations Within IPsec
The IPsec suite can be considered to have two separate operations, when performed in unison, providing a complete set of security services. These two operations are IPsec Communication and Internet Key Exchange.
- IPsec Communication
- It is typically associated with standard IPsec functionality. It involves encapsulation, encryption, and hashing the IP datagrams and handling all packet processes.
- It is responsible for managing the communication according to the available Security Associations (SAs) established between communicating parties.
- It uses security protocols such as Authentication Header (AH) and Encapsulated SP (ESP).
- IPsec communication is not involved in the creation of keys or their management.
- IPsec communication operation itself is commonly referred to as IPsec.
- Internet Key Exchange (IKE)
- IKE is the automatic key management protocol used for IPsec.
- Technically, key management is not essential for IPsec communication and the keys can be manually managed. However, manual key management is not desirable for large networks.
- IKE is responsible for creation of keys for IPsec and providing authentication during key establishment process. Though, IPsec can be used for any other key management protocols, IKE is used by default.
- IKE can be used with already defined key management framework Internet Security Association Key Management Protocol (ISAKMP).
- ISAKMP is not IPsec specific, but provides the framework for creating SAs for any protocol.
This chapter mainly discusses the IPsec communication and associated protocol employed to achieve security.
IPsec Communication Modes
IPsec Communication has two modes of functioning; transport and tunnel modes. These modes can be used in combination or used individually depending upon the type of communication desired.
Transport Mode
- IPsec does not encapsulate a packet received from upper layer.
- The original IP header is maintained and the data is forwarded based on the original attributes set by the upper layer protocol.
- The following diagram shows the data flow in the protocol stack.

- The limitation of transport mode is that no gateway services can be provided. It is reserved for point-to-point communications as depicted in the following image.

Tunnel Mode
- This mode of IPsec provides encapsulation services along with other security services.
- In tunnel mode operations, the entire packet from upper layer is encapsulated before applying security protocol. New IP header is added.
- The following diagram shows the data flow in the protocol stack.

- Tunnel mode is typically associated with gateway activities. The encapsulation provides the ability to send several sessions through a single gateway.
- The typical tunnel mode communication is as depicted in the following diagram.

- As far as the endpoints are concerned, they have a direct transport layer connection. The datagram from one system forwarded to the gateway is encapsulated and then forwarded to the remote gateway. The remote associated gateway de-encapsulates the data and forwards it to the destination endpoint on the internal network.
- Using IPsec, the tunneling mode can be established between the gateway and individual end system as well.

IPsec Protocols
IPsec uses the security protocols to provide desired security services. These protocols are the heart of IPsec operations and everything else is designed to support these protocol in IPsec.
Security associations between the communicating entities are established and maintained by the security protocol used.
There are two security protocols defined by IPsec — Authentication Header (AH) and Encapsulating Security Payload (ESP).
Authentication Header
The AH protocol provides service of data integrity and origin authentication. It optionally caters for message replay resistance. However, it does not provide any form of confidentiality.
AH is a protocol that provides authentication of either all or part of the contents of a datagram by the addition of a header. The header is calculated based on the values in the datagram. What parts of the datagram are used for the calculation, and where to place the header, depends on the mode cooperation (tunnel or transport).
The operation of the AH protocol is surprisingly simple. It can be considered similar to the algorithms used to calculate checksums or perform CRC checks for error detection.
The concept behind AH is the same, except that instead of using a simple algorithm, AH uses special hashing algorithm and a secret key known only to the communicating parties. A security association between two devices is set up that specifies these particulars.
Encapsulation Security Protocol (ESP)
ESP provides security services such as confidentiality, integrity, origin authentication, and optional replay resistance. The set of services provided depends on options selected at the time of Security Association (SA) establishment.
In ESP, algorithms used for encryption and generating authenticator are determined by the attributes used to create the SA.
Security Associations in IPsec
Security Association (SA) is the foundation of an IPsec communication. The features of SA are −
- Before sending data, a virtual connection is established between the sending entity and the receiving entity, called “Security Association (SA)”.
- IPsec provides many options for performing network encryption and authentication. Each IPsec connection can provide encryption, integrity, authenticity, or all three services. When the security service is determined, the two IPsec peer entities must determine exactly which algorithms to use (for example, DES or 3DES for encryption; MD5 or SHA-1 for integrity). After deciding on the algorithms, the two devices must share session keys.
- SA is a set of above communication parameters that provides a relationship between two or more systems to build an IPsec session.
- SA is simple in nature and hence two SAs are required for bi-directional communications.
- SAs are identified by a Security Parameter Index (SPI) number that exists in the security protocol header.
- Both sending and receiving entities maintain state information about the SA. It is similar to TCP endpoints which also maintain state information. IPsec is connection-oriented like TCP.
Parameters of SA
Any SA is uniquely identified by the following three parameters −
- Security Parameters Index (SPI).
- It is a 32-bit value assigned to SA. It is used to distinguish among different SAs terminating at the same destination and using the same IPsec protocol.
- Every packet of IPsec carries a header containing SPI field. The SPI is provided to map the incoming packet to an SA.
- The SPI is a random number generated by the sender to identify the SA to the recipient.
- Destination IP Address − It can be IP address of end router.
- Security Protocol Identifier − It indicates whether the association is an AH or ESP SA.
Example of SA between two router involved in IPsec communication is shown in the following diagram.
