Community discussions

MikroTik App
 
en1gm4
Member Candidate
Member Candidate
Topic Author
Posts: 123
Joined: Sun Oct 02, 2016 6:27 pm
Location: UK

understanding and fixing MTU/MSS/PMTU with IPsec

Sun Jan 13, 2019 2:55 pm

We have a in issue with our office connection to AWS via an IPSEC tunnel in that anything session oriented (http, ssh) will not work properly,
We discovered however that reducing the MTU on the ethernet interface on one of the computers in the office to 1400 appears to solve the problem and both SSH and HTTP work fine.
I am presuming that changing to 1400 lowers the MTU to the lowest in the path and messages are being correctly passed to web servers (SYN?) on AWS to make that work... but at 1500 there is something not working in the path as the web servers are not ever "hearing" that 1500 bytes is too big
From this it appears that PMTU is not working and devices on AWS are not reducing packet sizes correctly.

It is a relatively simple setup
Office connected to internet via PPPoE DSL (RB4011)
CHR on AWS connecting directly to AWS internet connection and creating an IPsec tunnel back to the office (ESP)

I've used ping with DF set to ascertain that the largest ping between the office and our AWS vpc is 1378. Add 28 bytes for IP and ICMP should make the MTU 1406. This traffic goes over the ipsec vpn
Pinging from the same host in the office to 8.8.8.8 gives a 1452 ping + 28 = 1480 MTU which is consistent with the MTU setting on the office router (4011) connected via ADSL modems to our ISP (Plusnet in the UK). Internet traffic works fine.
I get the same maximum ping size the other way (from AWS vpc to the office)

I'm not clear on what to configure to fix this ... despite much reaching on MTU. MSS, clamping, SYN etc. (as well as lots of posts about VPN's causing problems like this)
Lowering the MTU on the Ethernet to the modem or on the CHR does not seem right as it is working fine for all our web traffic in what is pretty much a standard configuration.
It would seem that focusing on the the IPsec tunnel and ensuring that both ends know that MTU/MSS would be sensible but I cannot find any setting in the actual tunnel config.
That seems to leave using a mangle ? But how?

I keep seeing something like this mentioned... but what PMTU size is it clamping to? Will it reduce mss on traffic that is already working fine to the internet rather than over the tunnel? on some of the traffic will go to AWS. Most just goes to the internet directly
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes \
protocol=tcp tcp-flags=sy
sorry for the newbie questions... I have tried to do my homework first.. honest.
 
idlemind
Forum Guru
Forum Guru
Posts: 1146
Joined: Fri Mar 24, 2017 11:15 pm
Location: USA

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Jan 14, 2019 12:07 am

You have two points of concern actually. Traffic inside of the VPN and traffic outside of the tunnel.

You'll want to make sure you are allowing the ICMP too big and fragmentation needed messages on input and forward (outside of tunnel and inside of tunnel).

MSS clamping is technically not required if MTU and path MTU discovery is working correctly. Relying on MSS clamping only corrects TCP traffic.

I'm on mobile right now so I can't check the policy based VPN options around these features. I usually use tunnel interfaces with IPSEC securing the tunnel over tunnel mode IPSEC but I'm weird. I can try to update more later.
Last edited by idlemind on Thu Oct 10, 2019 6:15 pm, edited 1 time in total.
 
en1gm4
Member Candidate
Member Candidate
Topic Author
Posts: 123
Joined: Sun Oct 02, 2016 6:27 pm
Location: UK

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Jan 14, 2019 12:53 am

Thanks.
At the moment all traffic outside of the tunnel (to the internet from the office or from our vpc) works fine (although it may be worth checking if things are getting fragmented thet shouldn't)
I'll have a look at what might be blocking MTU discovery. I think all ICMP are allowed between any devices in our internal 10.x.x.x network. Any other known suspects? It's not something I know much about.
I'd certainly prefer that MTU discovery was automatic.. given the issue is only on our vpn it does not seem likely that is any other device.... But I'll also double check for any possible aws security group issue.
 
en1gm4
Member Candidate
Member Candidate
Topic Author
Posts: 123
Joined: Sun Oct 02, 2016 6:27 pm
Location: UK

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Jan 14, 2019 12:16 pm

for the record, altering the MTU on the ethernet interface of our AWS instance to the same value worked out using ping testing (1406) fixes the problem so it seems clear that PMUD is not working

this doc helped
https://community.cisco.com/t5/collabor ... -p/3115561

... but I have not yet worked out how to track down where the ICMP messages that control this are not working
Presumably I could also use the router to change the MSS manually instead of changing MTU in AWS??
(changing that MTU lowers performance for access coming from the internet and even more so for internal to AWS where Amazon us using 9000 byte frames)
 
en1gm4
Member Candidate
Member Candidate
Topic Author
Posts: 123
Joined: Sun Oct 02, 2016 6:27 pm
Location: UK

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Jan 14, 2019 6:16 pm

I feel like I am in a conversation with myself here but doing it (briefly) anyway in hope it will help someone in future.
Adding a mangle to rewrite the mss on syn packets going from our office to our AWS VPC seems to have done the trick. The VPC hosts then see a 1364 mss which is small enough to cross the ipsec tunnel without fragmentation.

Figuring out why ICMP is not allowing this to be automatic is a task for another day
 
Trackboy
Member Candidate
Member Candidate
Posts: 215
Joined: Mon Oct 31, 2011 11:19 am
Location: Hungary

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Thu Jan 17, 2019 2:55 pm

Hello guys! Sorry for my english first.....
I have got similar problem or i think so. I have got a Mikrotik IKEv2 road warrior VPN with RSA authentication. I have got PPPoE connection at home. MTU and MRU is 1480.
When i connect with Strongswan client on Linux Mint, i experience that, ping is fine, there is not packet loss at all, but the browsing is not working well.
I tried with WIN 10, but that is ok, working everything fine, so this is really weird for me.
 
en1gm4
Member Candidate
Member Candidate
Topic Author
Posts: 123
Joined: Sun Oct 02, 2016 6:27 pm
Location: UK

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Fri Jan 18, 2019 4:00 pm

Trackboy,
the first thing i did, which really helped, was to use Ping with the DF flag set to discover the actual MTU that gets through.
perhaps try that and see if you are getting the same max packet size in Windows and Linux? The options are slightly different in linux and windows
there are lots of sites and videos that describe how to do this.
once i knew the max MTU, i tried a few hundred/thousand pings at that size just to be sure there was no other issue causing packet loss... and I shortened the interval to push as many pings as quickly as i could
 
Trackboy
Member Candidate
Member Candidate
Posts: 215
Joined: Mon Oct 31, 2011 11:19 am
Location: Hungary

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Tue Jan 22, 2019 10:48 am

Trackboy,
the first thing i did, which really helped, was to use Ping with the DF flag set to discover the actual MTU that gets through.
perhaps try that and see if you are getting the same max packet size in Windows and Linux? The options are slightly different in linux and windows
there are lots of sites and videos that describe how to do this.
once i knew the max MTU, i tried a few hundred/thousand pings at that size just to be sure there was no other issue causing packet loss... and I shortened the interval to push as many pings as quickly as i could
Thank you very much for your answer. I will give it a try as soon as i can.
 
Trackboy
Member Candidate
Member Candidate
Posts: 215
Joined: Mon Oct 31, 2011 11:19 am
Location: Hungary

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Tue Jan 22, 2019 3:06 pm

I put the firewall mangle rule, bit this one is not solved my problem completely, when i type the speedtest.net, sometimes working, sometimes not:

ip firewall mangle add chain=forward action=change-mss log=yes new-mss=clamp-to-pmtu protocol=tcp tcp-flags=syn out-interface=ISP_PPPoE passthrough=yes disabled=no

I made some test too: when i ping my router the max MTU is 1452, but it should be good, because my PPPoE connection MTU and MRU is 1480 right ? IP+ICMP header is 28 bytes.
 
ahtoh
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Fri Jan 25, 2013 3:10 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Thu Oct 10, 2019 6:07 pm

I'm having the same problem with IPSEC/IKEv2 client on Mikrotik
Windows 10 client works fine, but that's because win10 creates a separate interface with MTU set to 1400
Mikrotik does not create a PPP interface for IPSEC tunnels, thus leaving MTU unchanged.
I know there is a mangle rule to clamp the MSS value but it is not perfect and only works for TCP, I'd want to lower the MTU for the tunnel somehow to 1400, is there a way to do this on Mikrotik?
 
msatter
Forum Guru
Forum Guru
Posts: 2937
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Fri Oct 11, 2019 12:09 am

I am afraid that you have to lower than 1400. I can send 1500 over PPPoE and I can use 1398 and 1382 through the IKEv2 tunnel. I have to set this value hard because clamp to pmtu is not working for me.
 
ahtoh
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Fri Jan 25, 2013 3:10 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Fri Oct 11, 2019 4:20 am

How do I do that?
There is no any option to set the MTU for the IPSEC tunnel in Mikrotik
 
msatter
Forum Guru
Forum Guru
Posts: 2937
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Fri Oct 11, 2019 1:58 pm

Not in the IPSEC section.

With IKEv2 you use connection marking or source addresses and those are used in the Mangle line to target traffic in the tunnel.

I can't copy from the terminal in the Android APP so I have seach a computer to able to give you the lineI use for that.

It should not be needed to use such a line but traffic is not reaching the client on so it can adapt.
 
msatter
Forum Guru
Forum Guru
Posts: 2937
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Fri Oct 11, 2019 4:01 pm

My line:
add action=change-mss chain=forward connection-mark=!no-mark dst-port=!993,8291 log-prefix=MSS new-mss=1382 passthrough=yes protocol=tcp tcp-flags=syn tcp-mss=!0-1382
I apply a change-mss in the forward chain. My connections are marked so I use !no-mark (not marked connections and the the "!" means not) and exclude TCP ports 993 and 8291 why are not IKEv2 traffic. My new MTU is set to 1382 and I have only to check packet larger than 1382. Only the packets sent at the beginning of a transaction have to be checked and that is why there is a tcp-flags=syn.

Outdated for IKEv2 connections. See: viewtopic.php?f=2&t=154449&p=763404#p763404
Last edited by msatter on Sat Oct 24, 2020 3:31 pm, edited 2 times in total.
 
ahtoh
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Fri Jan 25, 2013 3:10 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Sat Oct 24, 2020 11:14 am

I lost my previous config and trying to set up Ipsec tunnel again
I use ipsec mode config with src address list "vpn"
here is my mangle rule, but it does not seem to work, the sites are loaded slowly and not fully
what am I missing here?
/ip firewall mangle
add action=change-mss chain=forward new-mss=1382 passthrough=yes protocol=tcp src-address-list=vpn tcp-flags=syn tcp-mss=!0-1382
 
msatter
Forum Guru
Forum Guru
Posts: 2937
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Sat Oct 24, 2020 3:37 pm

See: viewtopic.php?f=2&t=161967#p824619

Your line should work but maybe 1382 is still to big for your connection. Try again with 1200 and the work your eay up.

Or try the better sollution. for IKEv2.
 
ahtoh
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Fri Jan 25, 2013 3:10 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Sat Oct 24, 2020 8:42 pm

yes I'm using IKEv2
my setup is the same as described here (with option 1 only) https://wiki.mikrotik.com/wiki/IKEv2_EA ... d_RouterOS

I added this but it did not help
/ip ipsec policy
move *ffffff destination=0
add action=none dst-address=192.168.11.0/24 src-address=0.0.0.0/0 place-before=1
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 10898
Joined: Mon Dec 04, 2017 9:19 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Sat Oct 24, 2020 9:03 pm

Can you explain what is this "move *ffffff destination=0" doing?
I don't know what was @msatter's setup in which he has moved the last policy (*ffffffff) to the beginning (top) of the list, but in your case, I don't understand the place-before=1. The action=none policy exempting packets for the LAN subnet from being handled by the subsequent policies must be placed before (above) the policy template from which the actual policy for the IP address assigned using mode-config is created.

Post the output of /ip ipsec policy print while the tunnel is up. Replace the public IP by a.b.c.d before posting.
 
msatter
Forum Guru
Forum Guru
Posts: 2937
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Sat Oct 24, 2020 9:16 pm

Have you done the click/tap post preview test on this site?

@Sindy *ffffff is the .id of the default. I do not know if that is needed anymore. This because of auto sort that implemented not that long ago.

But then auto sort could be taking care of that now and that would make it much much simpler.
Last edited by msatter on Sat Oct 24, 2020 9:19 pm, edited 1 time in total.
 
ahtoh
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Fri Jan 25, 2013 3:10 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Sat Oct 24, 2020 9:18 pm

this is how it looks when the tunnel is up
the first line is something that comes with default config, I did not touch that
Capture.PNG
You do not have the required permissions to view the files attached to this post.
 
ahtoh
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Fri Jan 25, 2013 3:10 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Sat Oct 24, 2020 9:46 pm

if I add the mss mangle rule it starts to work, but again, every time I navigate on this forum (including preview page), there is a 2-3 seconds delay before it opens a page
Some sites like speedtest.net take 10 or 20 seconds "connecting" before it starts loading
 
msatter
Forum Guru
Forum Guru
Posts: 2937
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Sat Oct 24, 2020 10:35 pm

Can you post what Sindy ask you to post? Looking at the picture I don't see a ptoblem but often a picture does not show all. An export will.

Update: I am now behind a Winbox and to me only the line in IPsec-Policy works and disabling it and enabling a MSS change line in Mangle does not work for me....anymore, strange.
 
ahtoh
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Fri Jan 25, 2013 3:10 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Sun Oct 25, 2020 3:50 am

I thought the screenshot would be better as this is not very readable
Flags: T - template, B - backup, X - disabled, D - dynamic, I - invalid, A - active, * - default 
 #      PEER          TUNNEL SRC-ADDRESS                                         DST-ADDRESS                                         PROTOCOL   ACTION  LEVEL    PH2-COUNT
 0 T X*                      ::/0                                                ::/0                                                all       
 1                           0.0.0.0/0                                           192.168.11.0/24                                     all        none   
 2 T                         ::/0                                                ::/0                                                all       
 3   DA  ike2-rw-cl... yes    10.15.1.1/32                                        0.0.0.0/0                                           all        encrypt unique           1
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 10898
Joined: Mon Dec 04, 2017 9:19 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Sun Oct 25, 2020 10:02 am

My fault, I forgot the modifier detail in the print command. The problem with screenshots is that in many cases they show less than a commandline print detail; in this particular case (ipsec policy list) the screenshot shows almost everything (only the proposal and ipsec transport protocol are missing there).

However, it shows that the action=none policy is located before the dynamically generated one, which means that the issue must be a different one in your case. Since an action=change-mss rule in mangle resolves the situation, the issue must be related to MTU in some way; since even with that rule in place it takes some sites 10-20 s to open, there must be something in addition.

In this situation, I would use /tool sniffer on the LAN interface as the first step, and if it was not enough to identify the issue, I'd use action=log or action=sniff-tzsp rules to see how the packets heading towards the IPsec tunnel look like before IPsec policy grabs them.

But first, post the full configuration export (see my automatic signature below regarding anonymisation hints), maybe something can be seen there.
 
ahtoh
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Fri Jan 25, 2013 3:10 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Oct 26, 2020 3:10 am

Here is the full export.
Firewall rules are all default I believe
# oct/25/2020 20:02:05 by RouterOS 6.47.6
# software id = 4QAA-IY6H
#
# model = RBSXTsqG-5acD
# serial number = XXX
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
add authentication-types=wpa2-psk eap-methods="" management-protection=\
    allowed mode=dynamic-keys name=WIFI supplicant-identity="" \
    wpa2-pre-shared-key=xxx
/interface wireless
set [ find default-name=wlan1 ] band=5ghz-a/n/ac channel-width=\
    20/40/80mhz-XXXX country="united states3" disabled=no frequency=auto \
    installation=any security-profile=WIFI ssid=WIFI \
    station-roaming=enabled wireless-protocol=802.11 wmm-support=enabled
/ip ipsec mode-config
add name=ike2-rw responder=no src-address-list=vpn
/ip ipsec policy group
add name=ike2-rw
/ip ipsec profile
add enc-algorithm=aes-256,aes-128 name=ike2-rw
/ip ipsec peer
add address=vpn.myserver.com disabled=yes \
    exchange-mode=ike2 name=ike2-rw-client profile=ike2-rw
/ip ipsec proposal
add name=ike2-rw pfs-group=none
/ip pool
add name=default-dhcp ranges=192.168.11.30-192.168.11.254
/ip dhcp-server
add address-pool=default-dhcp disabled=no interface=ether1 lease-time=1d \
    name=defconf
/user group
set full policy="local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,pas\
    sword,web,sniff,sensitive,api,romon,dude,tikapp"
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=ether1 list=LAN
add comment=defconf interface=wlan1 list=WAN
/ip address
add address=192.168.11.1/24 interface=ether1 network=192.168.11.0
/ip dhcp-client
add comment=defconf disabled=no interface=wlan1
add add-default-route=no interface=ether1
/ip dhcp-server network
add address=192.168.11.0/24 gateway=192.168.11.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan
/ip firewall address-list
add address=192.168.11.46 list=vpn
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=change-mss chain=forward disabled=yes new-mss=1380 passthrough=yes \
    protocol=tcp src-address-list=vpn tcp-flags=syn tcp-mss=!0-1380
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
/ip ipsec identity
add auth-method=eap certificate="" eap-methods=eap-mschapv2 generate-policy=\
    port-strict mode-config=ike2-rw password=xxx peer=ike2-rw-client \
    policy-template-group=ike2-rw username=xxx
/ip ipsec policy
set 0 disabled=yes
add action=none dst-address=192.168.11.0/24 src-address=0.0.0.0/0
add group=ike2-rw proposal=ike2-rw template=yes
/system clock
set time-zone-name=America/Chicago
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 10898
Joined: Mon Dec 04, 2017 9:19 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Oct 26, 2020 12:46 pm

The action=none policy addresses the case when a packet sent by the LAN client is too large to fit to the WAN if encapsulated into IPsec transport packet and needs to be made smaller (by sending a smaller portion of the TCP buffer). So it addresses an issue with local client's PMTU discovery on the path towards the server.

But as your action=change-mss rule matches on src-address-list=vpn, it adjusts the MSS value only in the SYN packets sent by the LAN client, which indicate to the remote server that the packets received by the LAN client must not exceed that size. In another words, there is an issue with remote server's PMTU discovery on the path towards the client in your LAN.

Sniffing at LAN should show that with that mangle rule disabled, the initial SYN, SYN+ACK, ACK exchange and maybe several subsequent small packets will get through, but then a long silence will follow because the large packets sent by the server won't ever get through the IPsec tunnel to you (or maybe even reach the remote end of the IPsec tunnel if the MTU bottleneck is between that point and the server); with that rule enabled, the session will continue normally even after the initial exchange of small enough packets, as the server will be sending packets with sufficiently small segment size and therefore their overall size will fit into the bottleneck's MTU.

So try reducing the MSS to an even smaller value (e.g. 1280) in the mangle rule, to see whether such modification changes anything about the delays in page download you've experienced. But it is quite likely that at a certain point in time, you'll have to talk to your VPN provider about this, as there is nothing you could do at your end to make the PMTU work automatically.

The only drawback of a static MSS adjustment is that you'll splice the TCP stream into more smaller packets even for servers for which it'll actually not be necessary.
 
ahtoh
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Fri Jan 25, 2013 3:10 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Oct 26, 2020 9:24 pm

I tried to reduce it to 1200 but still the same
Windows VPN client works just fine. It simply creates a new tunnel interface with MTU of 1400 and everything works great.
So it's not something with my provider, it's Mikrotik VPN implementation that does not work.
I'm behind two NATs here, not sure if that might somehow be related to the issue.
 
msatter
Forum Guru
Forum Guru
Posts: 2937
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Oct 26, 2020 10:35 pm

I did not see a way to reduce the MTU except for the SYNC. NAT is not a problem because a tunnel is used. UDP/4500.

Despite my IKEv2 is eorking great and MSS is never triggered I have sometimes problems retrieving TLS certificates when browsing.
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 10898
Joined: Mon Dec 04, 2017 9:19 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Oct 26, 2020 10:39 pm

Windows VPN client works just fine. It simply creates a new tunnel interface with MTU of 1400 and everything works great.
I forgot about this point, you've mentioned it already.

I'm behind two NATs here, not sure if that might somehow be related to the issue.
The only difference is between "no NAT" and "one or more NATs". Double NAT is no worse than a single one.

So it's not something with my provider, it's Mikrotik VPN implementation that does not work.
Sounds like that, if not for the fact that I've never seen issues like that with any IKEv2 VPN provider I've ever dealt with, and other people around here also seem to only have the issue with the client=>server direction. But it's true I haven't tested this with 6.47.6 anywhere yet. So it would require a test setup, where you would be connecting via the VPN to a known remote server sending known traffic, to find out what's the issue by sniffing the traffic sent and received by the server as well as the Mikrotik's WAN and LAN, to see at which point the MTU notification breaks.

At the moment I can only imagine the provider to ignore the don't fragment flag in the packets received from the server and Mikrotik not reassembling fragmented IPsec transport packets properly, whereas the Windows' IPsec stack would reassemble them. This should be relatively easy to check without any special arrangement - just disable the rule, sniff to file on the WAN for the VPN server address alone (not for UDP protocol and/or port as this would prevent fragments from being captured) and open one of the problematic sites. Then, open the file in Wireshark and look for fragmented packets (display filter (ip.flags.mf == 1) || (ip.frag_offset > 0) will show you all fragments in the file).
 
ahtoh
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Fri Jan 25, 2013 3:10 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Oct 26, 2020 11:56 pm

I think this goes beyond my level of troubleshooting.
I'll just report this to Mikrotik and leave this to them

p.s. I wonder if the issue is because my WAN interface is wireless which is less common and have not been probably tested thoroughly
 
ahtoh
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Fri Jan 25, 2013 3:10 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Fri Oct 30, 2020 7:37 am

Mikrotik support quickly identified the issue and suggested me to add these firewall rules:

/ip firewall filter add chain=forward action=accept src-address-list=vpn dst-address-list=!vpn place-before=6
/ip firewall filter add chain=forward action=accept src-address-list=!vpn dst-address-list=vpn place-before=6

I don't know why but it worked!

I also identified that the maximum mss value in my case is 1402 instead of 1382 suggested here
I dont know how to properly calculate it though, I just keep increasing it until it stopped working
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 10898
Joined: Mon Dec 04, 2017 9:19 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Fri Oct 30, 2020 10:13 am

I don't know why but it worked!
I have concentrated so much on the MTU part that I've missed that you have the action=fasttrack-connection rule in filter. Once a connection gets fasttracked, most of its packets bypass firewall handling and also IPsec policy matching. The reason why these connections do not stop working completely is that only most packets, not all, of the fasttracked connection actually take the fast path. Those which don't are sufficient for TCP sessions to survive thanks to retransmissions.

So the rules suggested by support prevent the packets which need to be handled by IPsec from ever hitting the action=fasttrack-connection rule.

I also identified that the maximum mss value in my case is 1402 instead of 1382 suggested here
I dont know how to properly calculate it though, I just keep increasing it until it stopped working
From this I understand that the action=change-mss rule is still necessary, so the issue with PMTU discovery from the server to the client still exists, correct? As for the proper calculation, the try and fail method is faster even if you know what exactly to include into the calculation.
 
ahtoh
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Fri Jan 25, 2013 3:10 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Fri Oct 30, 2020 7:21 pm

yes the mss rule is still necessary
on the contrary, enabling or disabling the additional ipsec policy "action=none dst-address=192.168.11.0/24 src-address=0.0.0.0/0" had no visible effect
 
agrisvv
just joined
Posts: 23
Joined: Sun Oct 14, 2012 6:36 pm
Location: LATVIA

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Fri Dec 11, 2020 10:23 am

My comments about MTU:
most of clients(android,ios,windows) work because they create new interface with MTU 1400.
But linux strongswan and mikrotik itself dose not have new interface with MTU 1400.

So far for my tests i used this websites bugzilla_mozilla_org and git_kolab_org witch dose not worked..
for strongwan linux helps this rule or client interface with 1400 mtu:
add action=change-mss chain=forward comment=\
    "vpn msu change" new-mss=1364 passthrough=yes protocol=tcp src-address=10.2.5.0/24 \
    tcp-flags=syn tcp-mss=1365-65535
My tested websites woked with 1380 msu. can somone explain what to use better 1300 mikrotik manual mentioned as example / 1380 this topic example / 1364 AWS ipsec mentioned somewhere.
 
jochi3017
just joined
Posts: 1
Joined: Wed Jul 27, 2022 6:33 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Wed Jul 27, 2022 6:50 pm

Hello, I have problems with the vpn tunnel from site to site

I have a Mikrotik Router with ROUTEROS v6.48, I have a local network, I have a client-to-site L2tp Server that works perfectly.

But I have spent a lot of time trying to create a tunnel between my main office and a branch, I manage to make the tunnel, I manage to ping from router to router and from router to the local networks at both ends,
but what I do not achieve is that both networks of different sites communicate. I have tried adding firewall rules and still nothing. I need help from an expert with this

I repeat I have an L2TP server that windows clients and others can connect to correctly, my problem lies with the point-to-point connection between the networks!
 
emunt6
Member Candidate
Member Candidate
Posts: 112
Joined: Fri Feb 02, 2018 7:00 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Sun Jul 31, 2022 12:56 am

HI!

Maybe some of you will be help:
Under Linux the following solves the PMTU problem:

/etc/sysctl.conf
--------------------------------------------------------
net.ipv4.ip_no_pmtu_disc = 1
--------------------------------------------------------

iptables -t filter -A INPUT --fragment -j ACCEPT
iptables -t mangle -o eth1 -p tcp -m tcp --sport:1-65535 -d <remote-subnet> -j TCPMSS --clamp-mss-to-pmtu
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 10898
Joined: Mon Dec 04, 2017 9:19 pm

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Sun Jul 31, 2022 7:44 am

I need help from an expert with this
Any expert needs to see the configuration exports to be able to help. Plus you should create a dedicated topic - from your description, your issue seems to be unrelated to the subject of this topic.

Who is online

Users browsing this forum: AndreKR, CGGXANNX and 57 guests