How to troubleshoot IPsec VPN misconfigurations

It's difficult to imagine the IT world without virtual private networks (VPNs). Many inexpensive small home and office (SOHO) routers allow you to set up a VPN server with just a few clicks and support hardware acceleration for encryption algorithms. But if you've ever configured an Internet Protocol Security (IPsec)-based VPN, you know how much trouble configuring consistent encryption algorithms can bring. Generally, there shouldn't be a problem if you configure both the VPN client and the server to use the same software versions.

[ Get the guide to installing applications on Linux. ]

This article demonstrates some troubleshooting steps for VPN misconfigurations using RouterOS as the example platform. While RouterOS v7 with WireGuard is still actively developed, many network devices don't support WireGuard as an industry standard.

IPsec will remain relevant for a long time, and it's important to be able to troubleshoot it.

A common VPN misconfiguration

Consider one of the most common cases. There's a router configured with a VPN server, and the latest update has been released. The update might fix a critical CVE, add new features, or improve performance. But even a planned update can become a nightmare if you're not prepared.

The example described in this article uses MikroTik's RouterOS v7.х. with Fedora 35 as the client. Other clients can have similar problems, which are solved by configuring the server because many clients are unconfigurable.

In Fedora 35, two RPM packages implement IPsec. One is Libreswan, installed in the base system. The other is Strongswan, which can be substituted for Libreswan. Libreswan doesn't have modp1024/DH2 support, so updating it (or installing the operating system with the default Libreswan client) will likely result in an inoperative VPN client.

This problem can be solved by replacing the Libreswan packages with Strongswan. This workaround helps until the next update.

Set up your environment

Before you can work through this tutorial, you need to configure the server and client.

Configure the server

For this demonstration, I intentionally configured the server to be inoperable because Fedora and RouterOS vendors use a fixed configuration on the latest software version by default.

/interface l2tp-server export show-sensitive
/interface l2tp-server server set default-profile=default enabled=yes ipsec-secret=12345678900987654321 use-ipsec=yes
/ppp export show-sensitive
/ppp secret add name=test1 password=test1
/ppp profile set *0 local-address=192.168.88.1 remote-address=default-dhcp
/ip ipsec export show-sensitive
/ip ipsec profile set [ find default=yes ] dh-group=modp1024 dpd-maximum-failures=10 enc-algorithm=aes-256,aes-192,aes-128
/ip ipsec proposal set [ find default=yes ] auth-algorithms=sha256 pfs-group=none
/system logging add topics=debug 

Configure the client

This example uses Fedora 35 running a 5.16.9 kernel and Libreswan version 4.6-1.

First, activate verbose logging in SELinux for IPsec:

$ semanage permissive -a ipsec_t

[ Improve your skills managing and using SELinux. ]

Next, create a connection:

$ nmcli c add con-name test1 type vpn \
vpn-type l2tp vpn.data 'gateway = 192.168.88.1, \
ipsec-enabled = yes, machine-auth-type = psk, \
user = test1, user-auth-type = password'

Start troubleshooting

After the environment is ready, you can start to solve problems.

First, activate the connection:

$ nmcli c up test1 --ask

After entering the password and pre-shared key (PSK), the connection completes with a res=failed error. View the logs with journalctl:

$ journalctl  -af _TRANSPORT=audit
...
CRYPTO_IKE_SA pid=8392 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:ipsec_t:s0 msg='op=start direction=initiator conn-name="dde76c3a-87c4-4ca9-b628-63e88dceca52" connstate=3 ike-version=1 auth=PRESHARED_KEY cipher=none ksize=0 integ=none prf=none pfs=none  raddr=192.168.88.1 exe="/usr/libexec/ipsec/pluto" hostname=? addr=192.168.88.207 terminal=? res=failed'
[...]

Executing journalctl -u ipsec shows the supported encryption algorithms, hash algorithms, Platform Firmware Resilience (PFR) technology, and so on:

$ journalctl -u ipsec
loading secrets from "/etc/ipsec.d/ipsec.nm-l2tp.secrets"
...
ike (phase1) algorithm values: AES_CBC_256-HMAC_SHA2_256-MODP2048, AES_CBC_256-HMAC_SHA2_256-MODP1536, AES_CBC_256-HMAC_SHA1-MODP2048, AES_CBC_256-HMAC_SHA1-MODP1536, AES_CBC_256-HMAC_SHA1-DH20, AES_CBC_128-HMAC_SHA1-DH19, 3DES_CBC-HMAC_SHA1-MODP2048
from whack: got --esp=aes256-sha1,aes128-sha1,3des-sha1
ESP/AH string values: AES_CBC_256-HMAC_SHA1_96, AES_CBC_128-HMAC_SHA1_96, 3DES_CBC-HMAC_SHA1_96
...

There are two phases to create a connection between two peers:

• Phase 1 is the Internet Key Exchange (IKEv1/IKEv2).
• Phase 2 is the Authentication Header (AH) or Encapsulating Security Payload (ESP).

AH is deprecated, so it's necessary to use ESP.

[ Cheat sheet: Get a list of Linux utilities and commands for managing servers and networks. ]

What happens in Phase 1?

Phase 1 creates a secure tunnel that can be used in Phase 2. During Phase 1, two hosts negotiate the identification method, encryption algorithms, hash algorithms, and Diffie-Hellman (DH) groups. Also, they identify each other.

Phase 1 can work in two modes: aggressive and main. The first mode can successfully be completed after an exchange of three unencrypted packets. The second one occurs after six. Initially, the sender and the receiver negotiate parameters for setting up IKE Security Association (SA). Then they establish a secret key using DH key exchange. Finally, they exchange identity information and authenticate each other.

The key negotiated in Phase 1 allows IKE peers to communicate securely in Phase 2. The Internet Security Association and Key Management Protocol (ISAKMP) is the negotiation protocol that lets two hosts agree on building an IPsec SA. When Phase 1 completes successfully, Phase 2 begins.

What happens in Phase 2?

In Phase 2, the participants negotiate the IPsec SAs for encrypting and authenticating the ensuing exchanges of user data. Each peer performs key computing and generates keys for IPsec SA encryption and authentication. That's why each IPsec SA is guaranteed to use a unique key for subsequent data transfer encryption and authentication. In this phase, the message is encrypted by an encryption algorithm negotiated in Phase 1. This is also called Quick Mode.

Phase 2 can complete only after Phase 1 because all packets are encrypted. This complicates debugging at this stage. If Phase 2 completes successfully, an IPsec tunnel is created.

[ Network getting out of control? Check out Network automation for everyone, a complimentary book from Red Hat. ]

Check the server logs

In the previous logs, you can see a list of some encryption algorithms, hash algorithms, and DHs in Phase 1. They weren't set anywhere during server and client configuration.

In Phase 2, it's proposed to use _HMAC_SHA1_96 hashes (such as AES_CBC_256-HMAC_SHA1_96, AES_CBC_128-HMAC_SHA1_96, and 3DES_CBC-HMAC_SHA1_96).

They have not been configured anywhere, either. This is significant because when the client and server establish a connection, they try to negotiate some parameters supported by each side. If they can't negotiate them, the VPN connection won't be established.

If you look at the server logs, you can see the following entries:

...
ipsec,debug proposal #0 len=256
ipsec,debug begin.
...
ipsec,debug transform #0 len=36
ipsec,debug type=Life Type, flag=0x8000, lorv=seconds
ipsec,debug type=Life Duration, flag=0x8000, lorv=28800
ipsec,debug type=Encryption Algorithm, flag=0x8000, lorv=AES-CBC
ipsec,debug,packet encryption(aes)
ipsec,debug type=Hash Algorithm, flag=0x8000, lorv=4
ipsec,debug hash(sha2_256)
ipsec,debug type=Authentication Method, flag=0x8000, lorv=pre-shared key
ipsec,debug type=Group Description, flag=0x8000, lorv=2048-bit MODP group
ipsec,debug dh(modp2048)
...

According to the algorithms list, the client proposed this to the server during the IKE stage. The next Local (server-side) server looks for an appropriate option from the following list:

[...]
ipsec,debug -compare proposal #1: Local:Peer
ipsec,debug (lifetime = 86400:28800)
ipsec,debug (lifebyte = 0:0)
ipsec,debug enctype = AES-CBC:AES-CBC
ipsec,debug (encklen = 256:256)
ipsec,debug hashtype = SHA:4
ipsec,debug authmethod = pre-shared key:pre-shared key
ipsec,debug dh_group = 1024-bit MODP group:2048-bit MODP group
[...]

This continues until all options have been compared.

If you prefer to provision resources using the cloud provider's default GUI (also known as ClickOps), you'll have to wait longer for a successful connection because comparing all the suggested Peer options with all Local options takes some time.

The logs on the server side confirm the negotiation failed:

[...]
  ipsec,error no suitable proposal found.
  ipsec,error 192.168.88.207 failed to get valid proposal.
  ipsec,error 192.168.88.207 phase1 negotiation failed.
[...]

This failure suggests a problem:

• In the latest versions, Libreswan deprecated modp1024/DH2, but this DH group was once used in RouterOS by default.
• If a client is used as is, then the IKE/ESP values might be incorrect by default.

This is why ESP fails without IKE.

Fix the problem

The first thing to do is to add or replace a supported DH group on both peers. The MikroTik wiki's section about configuring Proposal shows some potential confusion:

Proposal information that will be sent by IKE daemons to establish SAs for certain policy.

Without knowing how the IPsec protocol works, the Proposal tab can be confused with the Profile tab. The first one refers to ESP (Phase 2), and the other one to IKE (Phase 1):

/ip ipsec profile set default dh-group=modp1024,modp2048

Reconnect and look at the client logs:

#1: IKE SA established {auth=PRESHARED_KEY cipher=AES_CBC_256 integ=HMAC_SHA1 group=MODP2048}
DPD: dpd_init() called on ISAKMP SA
IKE SA stage was completed successfully. Algorithms and DH are negotiated. Audit log informs about it additionally:
CRYPTO_IKE_SA pid=13619 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:ipsec_t:s0 msg='op=start direction=initiator conn-name="63d57ce1-1e4c-459e-8851-e693c6ae7a17" connstate=1 ike-version=1 auth=PRESHARED_KEY cipher=aes ksize=256 integ=sha1 prf=sha1 pfs=MODP2048  raddr=192.168.88.1 exe="/usr/libexec/ipsec/pluto" hostname=? addr=192.168.88.207 terminal=? res=success'
...

Now, cipher, ksize, integ, prf, and pfs have been defined, unlike in the logs at the beginning of debugging. On the server side, IKE SA completed successfully:

...
ipsec,debug authmethod = pre-shared key:pre-shared key
ipsec,debug dh_group = 2048-bit MODP group:2048-bit MODP group
ipsec,debug -an acceptable proposal found-
ipsec,debug dh(modp2048)
ipsec,debug -agreed on pre-shared key auth-
...
ipsec,info ISAKMP-SA established 192.168.88.1[500]-192.168.88.207[500] spi:054565abedbe9592:584e50c6c2b1db04
...

Phase 2 follows Phase 1:

...
child state #2: UNDEFINED(ignore) => QUICK_I1(established CHILD SA)
#2: initiating Quick Mode IKEv1+PSK+ENCRYPT+PFS+UP+IKE_FRAG_ALLOW+ESN_NO+ESN_YES {using isakmp#1 msgid:ea5eec40 proposal=AES_CBC_256-HMAC_SHA1_96, AES_CBC_128-HMAC_SHA1_96, 3DES_CBC-HMAC_SHA1_96 pfsgroup=MODP2048}
...
#2: sent Quick Mode request
...
#1: ignoring informational payload NO_PROPOSAL_CHOSEN, msgid=00000000, length=12
...

In Phase 2, Quick Mode completes with a message log NO_PROPOSAL_CHOSEN.

After initiating Quick Mode you can see that Peer (client-side) proposes algorithms and the PFS group that was declined by Local:

...
ipsec,debug peer's single bundle:
ipsec,debug  (proto_id=ESP spisize=4 spi=021b8170 spi_p=00000000 encmode=Transport reqid=0:0)
ipsec,debug   (trns_id=AES-CBC encklen=256 authtype=hmac-sha1)
ipsec,debug   (trns_id=AES-CBC encklen=128 authtype=hmac-sha1)
ipsec,debug   (trns_id=3DES encklen=0 authtype=hmac-sha1)
ipsec,debug my single bundle:
ipsec,debug  (proto_id=ESP spisize=4 spi=00000000 spi_p=00000000 encmode=Transport reqid=96:96)
ipsec,debug   (trns_id=AES-CBC encklen=256 authtype=hmac-sha256)
ipsec,debug   (trns_id=AES-CBC encklen=192 authtype=hmac-sha256)
ipsec,debug   (trns_id=AES-CBC encklen=128 authtype=hmac-sha256)
ipsec,debug not matched
ipsec,error no suitable proposal found.
ipsec,error 192.168.88.207 failed to pre-process ph2 packet.
...

In the logs, the ESP hash algorithms did not match. If SHA1 is added to the Proposal section on the server side, all the phases will complete successfully.

From Libreswan's IPsec configuration and connections documentation:

ike: IKE encryption/authentication algorithm to be used for the connection (phase 1 aka ISAKMP SA). The format is "cipher-hash;modpgroup, cipher-hash;modpgroup, ..." Any left out option will be filled in with all allowed default options.Multiple proposals are separated by a comma.

esp: Specifies the algorithms that will be offered/accepted for a Child SA negotiation. If not specified, a secure set of defaults will be used. Sets are separated using commas and pluses.

If some of the options in the section are not defined, Libreswan will use the Diffie-Hellman group values by default. However, you should check the configuration files:

$ sudo grep -E 'include|esp|ike' /etc/ipsec.conf
ikev2=insist
include /etc/crypto-policies/back-ends/libreswan.config
include /etc/ipsec.d/*.conf

$ sudo grep -E 'include|esp|ike' /etc/crypto-policies/back-ends/libreswan.config
...
ike=aes_gcm256-sha2_512+sha2_256-dh19+dh14+dh31+dh20+dh21+dh15+dh16+dh18,chacha20_poly1305-sha2_512+sha2_256-dh19+dh14+dh31+dh20+dh21+dh15+dh16+dh18,aes256-sha2_512+sha2_256-dh19+dh14+dh31+dh20+dh21+dh15+dh16+dh18,aes_gcm128-sha2_512+sha2_256-dh19+dh14+dh31+dh20+dh21+dh15+dh16+dh18,aes128-sha2_256-dh19+dh14+dh31+dh20+dh21+dh15+dh16+dh18
esp=aes_gcm256,chacha20_poly1305,aes256-sha2_512+sha1+sha2_256,aes_gcm128,aes128-sha1+sha2_256

At first glance, it seems that the settings were found. But if you compare them with the Local and Peer logs in Phase 1, there will be no match. There are settings for more secure connections proposed in configuration files. The value for option IKEv2 implies the use of the second version of the protocol.

That is why the settings from the configuration files can be replaced at each stage.

[Cheat sheet: Old Linux commands and their modern replacements ]

The NetworkManage-l2tp plugin is the next component participating in the process of establishing a connection. The configuration files are generated in /var/run/nm-l2tp-ID-CONNECTION/:

$ sudo grep -E 'include|esp|ike' /var/run/nm-l2tp-a7e327a8-8f5c-4a2f-b116-bb695fd9e760/
 
ike=aes256-sha2_256-modp2048,aes256-sha2_256-modp1536,aes256-sha1-modp2048,aes256-sha1-modp1536,aes256-sha1-ecp_384,aes128-sha1-ecp_256,3des-sha1-modp2048 esp=aes256-sha1,aes128-sha1,3des-sha1
ikev2=no

Now you have everything you need. If you look at the IPsec IKEv1 weak legacy algorithms and backward compatibility documentation for the nm-l2tp plugin, you can find which algorithms are used by default. Also, the logs confirm this. The plugins' configuration files have priority.

To establish a connection, it is necessary to add the ipsec-esp option to the connection setting:

$ nmcli c modify test1 vpn.data ipsec-esp=aes256-sha256,gateway=192.168.88.1,ipsec-enabled=yes,machine-auth-type=psk,user=test1,user-auth-type=password

$ nmcli c up test1

The connection has been established successfully, and IPsec audit logs have the success status:

audit[11793]: CRYPTO_IPSEC_SA pid=11793 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:ipsec_t:s0 msg='op=start conn-name="63d57ce1-1e4c-459e-8851-e693c6ae7a17" connstate=2, satype=ipsec-esp samode=transport cipher=AES ksize=256 integ=HMAC_SHA2_256 in-spi=19716967(0x19716967) out-spi=105637207(0x105637207) in-ipcomp=0(0x00000000) out-ipcomp=0(0x0000000... exe="/usr/libexec/ipsec/pluto" hostname=? addr=192.168.88.207 terminal=? res=success'
...

ESP settings are different from the default ones:

...
from whack: got --esp=aes256-sha256
ESP/AH string values: AES_CBC_256-HMAC_SHA2_256_128

Quick Mode completes successfully:

...
#2: initiating Quick Mode IKEv1+PSK+ENCRYPT+PFS+UP+IKE_FRAG_ALLOW+ESN_NO+ESN_YES {using isakmp#1 msgid:f7f0d4cd proposal=AES_CBC_256-HMAC_SHA2_256_128 pfsgroup=MODP2048}
...
#2: sent Quick Mode request
...
#2: IPsec SA established transport mode {ESP=>0x0af7d490 <0x1ef38071 xfrm=AES_CBC_256-HMAC_SHA2_256_128 DPD=passive}
...

In the logs on the server side, Phase 2 completes with matched status:

...
ipsec,debug peer's single bundle:
ipsec,debug   (trns_id=AES-CBC encklen=256 authtype=hmac-sha256)
ipsec,debug my single bundle:
ipsec,debug   (trns_id=AES-CBC encklen=256 authtype=hmac-sha256)
ipsec,debug   (trns_id=AES-CBC encklen=192 authtype=hmac-sha256)
ipsec,debug   (trns_id=AES-CBC encklen=128 authtype=hmac-sha256)
ipsec,debug matched
...

[ Want to test your sysadmin skills? Take a skills assessment today. ]

Key takeaways

Although many users choose OpenVPN and WireGuard, IPsec still takes the lead in the world of tunnel building. However, its debugging can be problematic, especially if you don't know all the details.

This article shows a few problems VPN users and administrators can face. I hope this article will help make IPsec debugging easier.