What is IPsec protocol, and how does IPsec VPN work?

Over the past few years, there has been a trend towards a transition to a remote work format. Along with it, there has been a growing demand for secure means of data exchange, where virtual private networks (VPNs) play an essential role. A VPN helps you stay anonymous while surfing the web and ensures the security of data transmitted over public networks. But have you ever wondered how this happens?

The basis of any VPN is its protocols, which protect data from cyber-attacks and surveillance. A VPN Protocol is a set of rules that govern how data is transmitted between your device and a VPN server. Among the many protocols, IPsec (Internet Protocol Security) stands out. It is considered one of the most reliable and widely used. In this article, we'll look at what IPsec VPN is, why it's so important to corporate networks, and how administrators can maintain network security and integrity using IPsec tunnels.

What is IPsec?

IPsec protocol (short for IP Security) is a set of network protocols that enables secure communications between devices over IP networks (groups of computers connected via their unique IP addresses). The protocol guarantees the confidentiality, integrity, and authenticity of data packets that are transmitted between these devices by authenticating and encrypting IP packets.

A data packet (IP packet) is a data structure that organizes information for transmission over a network. It consists of a header, payload, and trailer.

• IP Header is a section at the beginning of a packet that contains instructions for routing the packet to the correct destination.
• Payload is the actual information contained inside the packet.
• Trailer is additional data at the end of the payload that indicates the end of the data packet.

What is the use of IPsec?

Security protocols such as IPsec are fundamental because standard network transfer methods such as TCP (Transmission Control Protocol) or FTP (File Transfer Protocol) are not protected by encryption. IPsec protocol is ideal for maintaining your privacy on IP networks and is often used to create VPN connections using IPsec VPN. IPsec has become a security standard due to its strong encryption and authentication methods.

IPsec helps protect your device when using public networks. Moreover, it also protects network data by creating encrypted channels known as IPsec tunnels, which encrypt all information transmitted between two data points.

The protocol also protects against Man-in-the-Middle (MitM) attacks when an attacker intercepts and modifies data transmission, routing it through their computer. IPsec assigns each data packet a unique number and examines it to detect signs of packet reuse.

What protocols are used in IPsec?

IPsec is a protocol suite that consists of several individual protocols. These protocols are:

• Authentication header (AH). AH ensures that the data remains intact during transmission and is not tampered with. It also verifies the sender by using cryptographic methods such as HMAC (Hash-based message authentication code).
• Encapsulating security payload (ESP). ESP is responsible for the confidentiality, integrity and authentication of IP packets. It encrypts the contents of the packet to prevent it from being intercepted and ensures that the data remains unchanged during transmission.
• Security association (SA). SA is the basis of communication in IPsec. Each SA sets security parameters, such as IPsec encryption method, authentication method, and key management, for each VPN connection.
• Internet key exchange (IKE). Key management is necessary to exchange encrypted keys between IPsec devices securely. IKE phase helps negotiate and create shared keys for secure communications.

How does IPsec tunnel work?

Computers exchange data via an IPsec tunnel according to a specific algorithm. It can be divided into several stages.

• Traffic identification. All data sent over the network is broken up into smaller pieces called packets. Packets contain both a payload and headers so that computers receiving the packets know what to do with them. IPsec adds authentication and encryption information, and trailers after the main payload. After the device receives the packet, it compares it with the configured IPsec policy to determine whether the packet should be forwarded through the IPsec tunnel. The traffic that needs to be sent through the IPsec tunnel is called “interesting” traffic.
• Authentication. Devices then negotiate “security associations” (SA) that determine how to transmit data via IPsec tunnel securely. This includes encryption and authentication methods, as well as keys for transmitting data.
• Key exchange. After identifying the “interesting traffic”, the local network device negotiates an SA with other computers. At this point, devices begin key exchange via the IKE protocol to establish secure communication and exchange keys for data transfer.
• Encryption. After establishing a secure connection, the data is transmitted through an IPsec tunnel. A security authentication header or encapsulation security payload is used to encrypt and authenticate the data to ensure its confidentiality and integrity. The encryption mechanism prevents data being intercepted during transmission. The authentication mechanism ensures the integrity of the data and prevents tampering during transmission through the IPsec tunnel.
• Data transfer. In this phase, the IPsec sender uses an encryption algorithm and key to encapsulate the original data to encrypt the IP packet. The sender and recipient then use the same algorithm and authentication key to process the encrypted packets to obtain the integrity check value (ICV). If the ICVs received at both ends of the IPsec tunnel are the same, the receiver decrypts the packet. If they are different, this means that the data has been tampered with, and the recipient discards the packet.
• Tunnel shutdown. Once the communication between the two parties is complete, the IPsec tunnel is disabled to save resources.

What are IPsec modes?

IPsec protocol can operate in two different modes with varying degrees of security.

In Tunnel mode, IPsec encrypts all the information in the data packet, including the payload and the header, and adds a new header to it. This allows you to change routing and successfully transfer data. IPsec tunnel mode is suitable for data transmission on public networks.

In Transport mode, IPsec, on the other hand, transforms the payload of the data packet, leaving the IP header unchanged. This means that the routing cannot be changed. This IPsec mode is typically used for direct communication between two computers on a reliable network.

What are IPsec ports?

The IPsec tunnel uses different ports and protocols to create secure channels for data transmission. Each port performs specific tasks:

• UDP port 500. UDP port 500 is used for the IKE protocol and is responsible for establishing secure communication channels and device authentication. It operates in basic and aggressive modes to exchange data.
• UDP port 4500. This port is used for NAT (Network Address Translation) traversal traffic in IPsec VPN. NAT is a mechanism in TCP/IP networks that can translate IP addresses of transit packets. Thanks to port 4500, IPsec traffic passes through NAT devices without modification.
• Protocol No. 50. Adds additional headers to IP packets to protect data.
• Protocol No. 51. Adds cryptographic checksums to IP packets to detect changes.

Thus, UDP ports 500 and 4500 are used for communication and NAT traversal, while protocols 50 and 51 provide data security. These methods work together and use IPsec to create a secure IPsec VPN tunnel.

What is IPsec VPN?

VPN services use special rules to encrypt data that is transferred between devices. IPsec VPN is a type of VPN that uses these rules to create an encrypted IPsec tunnel over the Internet.

First, the data is placed in a special IPsec packet, which is then encrypted. Immediately after, this packet is sent to a special VPN server, where it is decrypted and sent to its final destination.

Types of IPsec VPN

Depending on their installation, IPsec VPNs can be divided into two main types.

• Site-to-site VPNs. This type of VPN connects various networks in different locations, allowing secure communication between them. It is usually used to connect branches, server centers or remote offices to a central network.

• Remote access VPNs. Remote access VPN allows individual users to securely connect to a corporate network from remote locations. These users access network resources, applications and services as if they were directly in the office.

Pros and cons of IPsec VPN

There are a number of advantages and disadvantages to using IPsec VPN. Let's take a closer look at them.

Pros of IPsec VPN

• Works at the network level. IPsec operates at the network level, not the application level, which means that data is encrypted at the sender and decrypted only at the recipient, providing a high level of security.
• Security. The AH and ESP protocols provide a high level of security, and the system's flexibility allows the use of different encryption algorithms, making data hacking quite difficult.
• Versatile IPsec VPN can protect various data types and support many operating systems and routers.

Cons of IPsec VPN

• Complexity of configuration. More complex to set up and manage compared to alternative protocols.
• Compatibility issues. Failure to comply with IPsec standards may cause compatibility issues, especially when connecting to other networks.
• Reduced network performance. IPsec VPN may reduce performance due to high CPU usage, especially when transmitting small data packets.

IPsec VPN or SSL VPN: which one is better?

Let's compare IPsec VPN with an alternative protocol — SSL VPN (Secure Sockets Layer).

IPsec VPN works at the network layer and protects all traffic between IP points. In contrast, SSL VPN works at the application layer and protects traffic only between the browser and the server, using the TLS protocol to encrypt HTTPS traffic.

TLS encryption ensures that anyone who tries to intercept a message cannot find out usernames, passwords, or other sensitive data.

However, because this encryption protects the communication between the browser and the server, it does not encrypt other data that could be used to track a user online, such as IP addresses, physical locations, and operating systems.

ISPs, corporations, and criminals can access and use this information against you. To avoid such risks, it is advisable to choose an IPsec VPN.

How to connect IPsec VPN on different devices

The L2TP/IPsec protocol is built into most modern platforms by default, including Windows and macOS. This is convenient because you don't need to install additional software, and you can be sure of its security.

Let's look at how to set up L2TP/IPsec VPN on Windows 10 and macOS using the built-in capabilities of the operating system.

Windows 10

Step 1. Open the taskbar and click on the "Network" icon → "Network settings" → "VPN" → "Add VPN connection".

Step 2. Set up your VPN connection:

• Select "Built-in VPN provider for Windows"
• Set the connection name
• Enter the domain name or IP address of your VPN server
• Select "Layer 2 Tunneling Protocol with IPsec (L2TP/IPsec)"
• Select "Username and Password" for Login Type

Step 3. Enter your username and password if required. Click "Save" to complete the VPN setup.

Step 4. Click the network icon on the taskbar. Select your VPN connection from the list and click "Connect". You can now use the IPsec VPN.

To disconnect from the IPsec VPN, click the network icon again, select your VPN connection, and click"Disconnect".

You can also check your network properties by opening the "Network Settings" window and selecting "Change adapter settings". There, you will find your VPN connection and can learn more about it.

macOS

Step 1. Open "System Preferences" → "Network".

Step 2. Click + and select VPN from the list. For VPN type, select "L2TP over IPsec".

Step 3. Enter the tunnel's name in the "Service name" field in the opened window and click "Create".

Step 4. Click "Authentication Settings" and enter your VPN password.

Step 5. Click "Advanced" and select "Send all traffic through VPN connection" to protect all traffic.

Step 6. Click "Connect" to connect to the VPN server.

Congratulations! Your IPsec VPN client is now configured on macOS.

Why AdGuard VPN?

Exclusive protocol

AdGuard VPN has a unique proprietary protocol that combines high speed and security.

Exclusion lists

Users can choose which apps and sites the VPN will work on.

Selecting a DNS server

Users can select a DNS server for enhanced security and traffic control, including ad blocking and malware protection.

Quick locations

Users can choose servers based on ping to optimize connection speed. AdGuard VPN constantly updates the list of available locations.

Compatibility with AdGuard Ad Blocker

AdGuard VPN apps are available for Windows, Mac, Android, and iOS, and extensions are available for all major browsers. In mobile versions, AdGuard VPN and AdGuard Ad Blocker work in Integrated mode, which does not require any configuration.

QUIC support

The QUIC protocol improves connection quality in challenging conditions like mobile Internet or public Wi-Fi networks.

Kill Switch

Automatically disconnects from the Internet when the VPN connection is lost, ensuring the safety of personal data on unreliable networks.

App exclusions

Like website exclusions, this option lets you choose which apps will work over the VPN, increasing online security and traffic control.

Wrapping up

IPsec VPN is an indispensable tool when you need to establish a secure network connection. With time-tested technology and ongoing development, the IPsec protocol suite is the ideal security solution for enterprise and public networks.