MTU woes in IPsec tunnels and how you can fix it

Today I ran into a problem with IPsec Xauth PSK and the built-in Android VPN client (Android 4.1.2), resulting in some sites (such as not loading through the VPN tunnel. Turns out I was dealing with MTU issues. When the Android VPN is started, it sets the MTU to 1500 on the tun0 interface:

$ ip link show tun0
33: tun0: <POINTOPOINT,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500 link/none

Looking at the Android Source, it appears someone must have forgotten to take care of IPsec Xauth PSK. With PPTP and L2TP based VPNs, the MTU is reduced to 1400 (line 758 – 778).

In comparison:

  • strongSwan Android client: MTU 1400
  • OS X / iOS 7 built-in IPsec client: MTU 1280 (for what it’s worth, 1280 is also the minimum IPv6 packet size and thus the MTU minimum required to make IPv6 work)
  • Windows 7 built-in IPsec client: MTU 1400
  • Cisco VPN client: MTU 1300

Among the tested clients, only the connection through the Android VPN client was causing the issue with stalling websites. In a nutshell, I was able to fix it with the following on the VPN server:

$ iptables -t mangle -A FORWARD -o eth0 \
 -p tcp -m tcp --tcp-flags SYN,RST SYN \
 -m tcpmss --mss 1361:1536 \
 -j TCPMSS --set-mss 1360

$ echo 1 >/proc/sys/net/ipv4/ip_no_pmtu_disc

And here is why.

IP fragmentation – The original problem.

On the VPN server side, we have the interface set to a standard Ethernet MTU 1500. In the scenario with the Android client, the MTU along the entire path is 1500. This leaves room for up to 1460 bytes of data payload per packet (also referred to as the maximum segment size MSS).

Keep in mind that IPsec in tunnel mode adds an ESP header and an additional IP header for tunneling the packet (usually with an additional size of around 70-80 bytes). When a packet is nearly the size of the MTU and when you tack on this encapsulation overhead, it is likely to exceed the MTU of the outbound link.  That’s where IP fragmentation kicks in – which could lead to performance degradation of your VPN tunnel. Or worse…

Path MTU discovery (PMTUD). A failed solution.

To avoid IP fragmentation, many TCP/IP stacks have path MTU discovery (PMTUD) implemented. To tell you right away: It doesn’t work for me, and it’s not going to work for you either. PMTUD attempts to discover the largest IP datagram that may be sent without fragmentation through an IP path. Instead of fragmenting a too-large IP packet, the VPN server is told (through the Don’t Fragment (DF) flag in the IP header of the sender) to discard the packet and reply with an ICMP fragmentation required (type3, subtype 4) message.

Recap: The sender is the website that you try to load on your VPN client.

When the sender receives this ICMP packet, it learns to use a smaller MTU for packets sent to our VPN server. In theory. In reality, many websites (senders like stupidly implement ICMP filters that break PMTUD functionality. And that’s where hell breaks loose.  The sender is expecting an acknowledgement for the original packet from our server, but since the packet was discarded, the acknowledgment never comes. Time goes by, then the sender repeats sending the too-large packet. The result? The VPN sender discards the packet, again, sends yet another ICMP message, and so on. Meanwhile our client on the other end of the VPN tunnel cannot tell what’s happening and is desperately waiting for some data. Everything appears to be stalled – a state which is also referred to as a black hole connection.

Because PMTUD doesn’t always work on the Internet, the use of it only makes sense in a site-to-site VPN, where basically one operator maintains the networks and is able to enable the forwarding of the “fragmentation required” ICMP (type 3, subtype 4) notifications in all firewalls in between.

MSS size reduction. A working solution.

So we can rule out PMTUD, but there is another way to assure that our VPN connection is working properly.

$ iptables -t mangle -A FORWARD -o eth0 \
	-p tcp -m tcp --tcp-flags SYN,RST SYN \
	-m tcpmss --mss 1361:1536 \
	-j TCPMSS --set-mss 1360

This iptables rule reduces the size of the allowed packet size by reducing the MSS of TCP SYN packets. The –set-mss value explicitly sets the MSS to 1360, which is a customary size for IPsec IPv4 interfaces. The –mss option is used to match only those MSS that are between 1361 and 1536 bytes (we don’t want to rewrite the MSS of a VPN client that uses a smaller MSS). The result in a tcpdump:

05:01:56.795798 IP > Flags [S], seq 2621580326, win 14600, options [mss 1460,sackOK,TS val 56614 ecr 0,nop,wscale 6], length 0
05:01:56.795865 IP > Flags [S], seq 2621580326, win 14600, options [mss 1360,sackOK,TS val 56614 ecr 0,nop,wscale 6], length 0
05:01:56.802695 IP > Flags [S.], seq 3057551576, ack 2621580327, win 14480, options [mss 1460,sackOK,TS val 1410945796 ecr 56614,nop,wscale 8], length 0

In line 1, you see the request from our Android client through the tunnel to Notice the MSS is 1460. In the second line, it is our VPN host initiating the TCP handshake with the external site. Only now  the MSS value is rewritten to 1360, thanks to our iptables rule. As a consequence, the TCP connection will use the lower MSS of the two end points, which is 1360. Voila!

You can go further and restrict the iptables rule to rewrite only those packets that are forwarded from our VPN client:

$ iptables -t mangle -A FORWARD -o eth0 \
	-p tcp -m tcp --tcp-flags SYN,RST SYN \
	-s \
	-m tcpmss --mss 1361:1536 \
	-j TCPMSS --set-mss 1360

This assumes that the address pool of your virtual IPs is (see rightsourceip in /etc/ipsec.conf).

There is also an option to determine the MSS size dynamically (“MSS clamping”, via the –clamp-mss-to-pmtu option), but it wouldn’t fix IPsec for clients that set their MTU too high (like in the Android example).

The MSS iptables rule doesn’t work with UDP applications. UDP is a connectionless protocol; hence there is no way to negotiate a MSS during the handshake. The only solution to guarantee that UDP works is to disable the Don’t Fragment (DF) bit in the IP header of the sender. This will allow our VPN server to fragment any UDP packet, if necessary. In Linux, you do it like this:

$ echo 1 >/proc/sys/net/ipv4/ip_no_pmtu_disc

The VPN server will fragment outgoing UDP packets that exceed the interface MTU, which may not be as great for performance and reliability reasons, but at least it doesn’t break the tunnel connection. Fortunately, most folks don’t use UDP for anything much greater than DNS.

9 thoughts on “MTU woes in IPsec tunnels and how you can fix it

  1. Thanks for this article!

    The “mss size reduction” with iptables works for me using the following iptables rules:
    iptables -A FORWARD -s -p tcp –tcp-flags SYN,RST SYN -j TCPMSS –clamp-mss-to-pmtu
    iptables -A FORWARD -d -p tcp –tcp-flags SYN,RST SYN -j TCPMSS –clamp-mss-to-pmtu

    I use strongswan in a site2site scenario.

    • Thanks for this article! and thanks for your tips!!

      My scenario :
      vpn pptp client on linux vm and client on pod ( container on openshift )
      We route all vpn traffic on linux vm

      with your rules the communication is ok!!!

  2. Great article, very helpful. Two comments:

    “Fortunately, most folks don’t use UDP for anything much greater than DNS.”

    – Actually I was looking into an MTU issue on an L2L IPSEC which needs to enable a video call as well (not my original intention, but an executive scope creep)

    “in reality, many websites (senders like stupidly implement ICMP filters”

    – Agreed, they block ICMP altogether instead of filtering out ICMP attack mechanisms such as smurf attacks. However, in all fairness, you can be smurfed if you allow ICMP type 3. Although one can argue ICMP is the least effective attack protocol, protection against well-known (D)DOS attempts should not be referred to as ‘stupidly’

    That all being said, great article with good insight. Thank you! Forwarded to colleagues for awareness.

  3. Thanks man.

    This really helped me out. I was having trouble reaching google services (Play store etc..) and using my webrowser (Android 5.02) over my IPsec Xauth PSK tunnel to my home VPN server (Strongswan running on Ubuntu)

    All other sites were reachable.

    I soon as I added your code to my iptables firewall, I had instant access to the playstore and

    Once again Thanks.

  4. “Fortunately, most folks don’t use UDP for anything much greater than DNS.”

    …except for most every media streaming protocol.

Leave a Reply

Your email address will not be published.