Hi, I have problems with branch office, I've replaced 1100AHx4 with CCR2116 (7.17.2) and reconfigured VPN, new one is EoIP with IPSec. And now I have troubles with vCenter connecting to the branch office ESXi (two servers). There are timeouts and disconnections. I've tried to put them to same VLAN as vCenter (added this VLAN to the bridge) to make sure it isn't a firewall issue but no luck.
I've found on Broadcome site someone had similar problem and helped lower MTU on VPN (https://community.broadcom.com/vmware-c ... to-vcenter) so I've tried 1200, 1300 and 1500 and it seems that on one ESXi it helped and working better with MTU 1200 but the other one isn't communicating.
OLD and fully working configuration was IPsec and above EoIP with routed networks. NEW configuration is direct EoIP with turned on IPsec and EoIP interface added to bridge with VLANs. Networks are still routed but now I can bridge some VLANs directly if I want to.
What else should I try? Thanks in advance for your help!

Connecting L2 networks with EoIP tunnels having an MTU below 1500 will almost certainly lead to erratic behaviour as there is no IP path MTU detection within the L2 broadcast domain. The encapsulated traffic will generate fragmented traffic on the WAN connection, some ISPs may not handle this well leading to delayed and/or out-of-order packet delivery leading to poor performance.

Do you really need L2 / ethernet bridged connectivity between sites? L3 / IP routed connectivity with the tunnel MTU optimised to fit encapsulated traffic, tunnel and encryption overhead exactly into the maximial size of a WAN packet, together with MSS clamping is likely to provide better performance.

Thank you for your reply. It's possible to change tunnel to IPsec but i liked the easy way establishing EoIP with IPSec. Problems made only vCenter, everything else worked fine. I've tried lower MTU on vCenter ethernet and it helped. ATM I ended up with EoIP outside bridge just routed network, it should work better.
Here is more information about EoIP in bridge and packet fragmentation: viewtopic.php?t=172376

As you do not need L2 connectivity then IPIP with IPsec or GRE with IPsec have less overhead than EoIP with IPsec (this is proprietary to Mikrotik and is based on GRE).

Working out the MTU can be tricky. I tend to use L2TP/IPsec with AES-CBC and SHA-256 which has an optimal MTU of 1420 when both WAN have an MTU of 1500, or 1404 with if the remote end has a WAN MTU of 1492 (e.g. for PPPoE) and/or is behind NAT.

Thank you for your help, I'll prepare new connection and I'll swap it. It's much better but it's still buggy.L2 would be better, sometime it's handy to have remote VLANs but mostly I need a stable connection without such problems. Even manually configured IPsec and EoIP worked better than EoIP with turned on IPsec.

EoIP is indeed handy at times, e.g. for diagnosing issues with the hotspot service where you really need to be connected to the L2 network remotely. For these occasional connections setting up an EoIP tunnel with an MTU of 1500 within the permanent L2TP/IPsec tunnel (used for managment of remote devices, RADIUS, etc.) is fine as the fragmentation is unseen by the WAN providers, the regular L2TP/IPsec TCP traffic is generally unfragmented as MSS clamping ensures the bulk of the packets exactly fit within the L2TP/IPsec tunnel MTU.

The forgotten protocol BCP ?
SSTP, L2TP and PPTP all 3 work wit the BCP (Bridge control protocol) that allows for larger MTU than 1500, (eg 1600) what allows to have the full MTU 1500 in an inner tunnel, by combining multiple streams with smaller packets for one packet of the (outer) tunnel.

Documentation/example is for PPP-BCP only, but it works on PPTP-BCP and SSTP-BCP, and should work for L2TP-BCP.
I forwarded Netflix streaming, being undetected for varying MTU, with an MTU 1500, over SSTP-BCP and further used PPTP-BCP internally for faster processing.

If you want to test: The spanning-tree-protocol (STP) must be enabled, for the tunnel-bridge to be created !
https://wiki.mikrotik.com/Manual:BCP_br ... _bridging)

BCP indeed does work but it does not interwork with bridges on which vlan-filtering is enabled (unless something has changed recently). But alone it doesn't help with MTU size - you need to use it in conjunction with another "forgotten protocol", MLPPP. The beauty of MLPPP is that it slices the payload packets into multiple transport ones small enough that they wouldn't have to be fragmented. So for the same size of the payload packet, you get two non-fragmented transport packets instead of two fragments of a single transport packet, so you don't get hit by dropping of non-first fragments that happen far too often. Of course, the packet rate is still almost double as compared to restricting the size of the payload packets by setting a lower MTU on the payload interface.

So when I need L2 tunneling over the internet, I use L2TP without BCP but with MLPPP to carry VXLAN transport packets as L2TP payload, and provide enough extra space so that the VXLAN interface MTU could be 1500 bytes. So far no issues. Just VXLAN has its own unique behaviors you have to get used to.

I did query support regarding BCP with VLAN-aware bridges some years ago and was told they had no immediate plans to fix it.

However, changes are afoot with the recent major bridge VLAN rework. I've not had chance to look into as yet, but in the 7.17 changelog:
*) bridge - correctly display PPP interfaces in VLAN menu;
*) ppp - added support for bridge-port-pvid configuration via ppp profile;
*) ppp - added support for bridge-port-trusted configuration via ppp profile;

BCP indeed does work but it does not interwork with bridges on which vlan-filtering is enabled (unless something has changed recently). But alone it doesn't help with MTU size - you need to use it in conjunction with another "forgotten protocol", MLPPP. The beauty of MLPPP is that it slices the payload packets into multiple transport ones small enough that they wouldn't have to be fragmented. So for the same size of the payload packet, you get two non-fragmented transport packets instead of two fragments of a single transport packet, so you don't get hit by dropping of non-first fragments that happen far too often. Of course, the packet rate is still almost double as compared to restricting the size of the payload packets by setting a lower MTU on the payload interface.

So when I need L2 tunneling over the internet, I use L2TP without BCP but with MLPPP to carry VXLAN transport packets as L2TP payload, and provide enough extra space so that the VXLAN interface MTU could be 1500 bytes. So far no issues. Just VXLAN has its own unique behaviors you have to get used to.

Yes indeed. I also prefer two similar sized packets, than the largest possible and then just the remaining bytes. The only thing I needed to set was the MMRU as far as I remember. (MRRU is for the MLPPP protocol ?) I only used the automaticly created bridge as endpoint for this tunnel, just as illustrated in that wiki document. This document was the only thing I looked at, so in my mind MLPPP and BCP where connected. VLAN and all other things were defined at the usual bridge. It was a temporary fix to patch internet access from the broken Starlink connection, with the Starlink connection of another network.

I have one more question. I've established IPIP tunnel above IPsec, Actual MTU is 1402. Should I manually set it to 1500 or leave it? ATM I didn't swap the traffic from EoIP so I can prepare. Thanks for your reply.

Just set IPIP tunnel's property mtu to 1500 ... it'll handle the necessary fragmentation and defragmentation (on the receiver side) just fine.

Thank you.
Everytime I asked about MTU in tunnel, it's recommended to set MTU to 1500. So why it isn't default settings with possibility to change it when I need to?

Because it is the best thing to do from one perspective and the worst one from another. And there is no ideal solution for all cases due to the number of clueless network administrators out there.

TCP is designed to automatically adjust the packet size to the lowest MTU on the path between the client and the server. By doing that, it uses the bandwidth and routing power most efficiently, because it chops the data into pieces of the optimal size already at source. If you force the MTU to 1500 and the actual lowest MTU of the path is lower than that, each packet that exceeds that path MTU has to be fragmented, which has two unpleasant consequences:

• further down the path, it needs to be treated as two separate packets, so the packet rate that loads the routers doubles and the payload-to-overhead ratio becomes slightly worse as each fragment has its own IP header (and possibly also the Ethernet header),
• if some router down the path drops non-first fragments (which happens surprisingly often, partially because people misconfigure their firewall rules), the complete packet cannot be reassembled at the destination, which means it is effectively dropped as a whole.

So if you force the MTU to 1500 on a tunnel interface, you double the packet rate and add extra overhead already at your own router (because the payload packet has 1500 and thus, together with the tunnel overhead, it exceeds the 1500 of the outgoing interface so it needs to be fragmented right there), but the connection works even if Path MTU Detection is broken on the path between the client and the server for some reason (most often due to clueless admins blocking ICMP completely for "security by obscurity" reasons). But it only works if the non-first fragments aren't dropped. So if you are "lucky", you fix connections to those servers where PMTUD is broken on the path, but break connections to servers where non-first fragments are dropped on the path.

So let's continue. I've reconfigured connection. First it creates IPSec and above is created IPIP tunnel and routed networks. Basically It's the same configuration like before. And vCenter can't make stable connection to ESXi. After lowering MTU it works.
Here are differences:
- old routers were 1100AHx4 and 4011
- they had older ROS - 7.6
- above IPSec there was EoIP tunnel
- new routers are CCR2004 and CCR2116 with ROS 7.18.2 (same behavior since 7.16)

vCenter is the only one loosing connections. I can even ping with MTU more than 1500 and it fragments correctly. HTTPS services works correctly. Do you have any idea what else should I check or what else could be wrong?

I've not had chance to look into as yet.

@tdw

Three point basket.

vlan-filtering with BCP works well.
Just tested on v718.
Not sure how come i missed that at change log

Do you have any idea what else should I check or what else could be wrong?

You can sniff at both end devices of the tunnel, and you should see packets to arrive to one of them that are larger than the IPIP tunnel MTU so the device cannot forward them, so it sends ICMP "fragmentation needed, MTU xyz" back to their sender, but the sender either doesn't receive these ICMP notification or ignores them. The root cause may be incorrect firewall rules or TCP stack issues. Wireshark will help you analyse the contents of the ICMP packets in detail, but it is basically enough to see them in the sniff.

I don't know. Packet fragmentation and defragmentation works correctly. If I set on vCenter MTU 1500 and send ping -s 1472 -4 -M do remote_esx_IP, it returns answer. Firewall rules are very simple, some input, some forward. Wrong rule should cause packet drop for all kind of communication but it works.

EDIT: If I set MTU 1500 it works but eg I set virtual machine Power on, it starts but on vCenter it shows Power on virtual machine and 100% instead of Completed. There isn't correct response. If I have wrong firewall rules I won't be able to do anything with ESXi. If I set MTU 1300 on vCenter it works without any problem. And this happened when a swaped Mikrotiks.

EDIT2: It won't show Web concole too. It was very good point to try highest working MTU and it's 1402. Anyway I tried log ICMP packets on Microtics and ESXi (vCenter can't show ICMP packets in tcpdump for unknown reason) and there wasn't single note about fragmented packets anywhere.

What I suspect and suggest is that the PMTUD (Path MTU Discovery) for the affected TCP traffic fails. You cannot discover this by trying with ICMP (ping). If tcpdump on ESXi doesn't show ICMP, run the sniffer on the Mikrotik as I have suggested initially. But if the tcpdump on ESXi shows 1500-byte or even larger TCP packets to be repeatedly sent to the router for a destination behind the tunnel, it is a sufficient indication of such an issue.

TCP normally sets the do-not-fragment flag on all packets it sends, because it is more efficient to slice the payload into more smaller packets than to fragment every big packet. If the router receives a packet with a do-not-fragment flag set, it obeys and drops it rather than forwarding it fragmented, and it notifies the sender about doing that. Upon receiving the notification, the sender takes less unacknowledged payload bytes from the buffer and sends them again, with do-not-fragment still set. This way, the segment size (the amount of payload bytes that can be sent in a single packet) gradually lowers down to the smallest MTU on the path to the destination. This process is called PMTUD.

If the sender does not receive that notification, or chooses to ignore it, it keeps hammering with packets of the same size with do-not-fragment flag set until the session fails for exhaustion of the retransmission timeout.

Yet another common way to address this is to use action=change-mss mangle rules in forward chain to modify the MSS (Maximum Segment Size) value in the SYN packets the TCP client and server use to inform each other about the amount of data per packet their input buffers can handle. However, some TCP stacks "know better" and ignore the received MSS value, so this method also does not resolve 100 % of cases.

If I remember correctly ...
I think between two distant mikrotiks , you can make a secure ipsec , then put a secure eoip tunnel inside of the ipsec.
And get something that has double encryption on packets as it passes through the public internet and might offer some larger MTU options.