The root cause of your issue is most likely a broken PMTUD (Path MTU Discovery) on the path between your router(s) and those servers in the internet. The MTU of a bridge always gets automatically adjusted to the one of that member port of the bridge whose MTU is the lowest, to prevent the router itself from attempting to send larger packets.

So for any connection that passes through this bridge, either PMTUD must work properly, or you have to use a bandaid to substitute the PMTUD. The bandaid is a pair of rules in chain forward of /ip firewall mangle, such as

/ip firewall mangle
add chain=forward in-interface=kokiri-eoip-br protocol=tcp tcp-flags=syn tcp-mss=1361-65535 action=change-mss new-mss=1360
add chain=forward out-interface=kokiri-eoip-br protocol=tcp tcp-flags=syn tcp-mss=1361-65535 action=change-mss new-mss=1360

If 1360 turns out to be still too high, start from 1280 and try finding the largest working value iteratively.

The above solution is just one of the possible ones. A much better one would be to use an L3 tunnel instead, but I figure you have strong reasons to use an L2 one.

For the price of doubling the packet rate, you can either force the MTU of the EoIP tunnel to 1500, which means that the EoIP transport packets will be fragmented and thus there is a risk of losing them as some networks drop non-first fragments, or use L2TP with MLPPP and BCP, which chops already the payload packets during encapsulation so the transport ones do not get fragmented.

The first approach (the two mangle rules) cannot actually break anything unless you force a lower MSS than 1280, so you can safely try that even during the rush hours.

As far as I'm aware devices in the same L2 network are unlikely to do PMTUD for link-local packets, they expect the network to support whatever the interface MTU is set to so you should set the EoIP MTU to 1500.

Obviously this will lead to fragmentation as 1500 + EoIP encapsulation + IPsec encapsulation > WAN MTU, I've found it is better to ensure the fragmentation isn't seen across the internet as ISPs sometimes don't handle fragments very well.

L3 routing and MSS clamping would be a better solution unless you really need L2 connectivity between sites.

You can use those mangle rules using a slightly more complicated way. Use a bridge filter rule to assign a packet mark (for simplicity, EoIP) to frames entering the bridge via the EoIP interface, and use mangle/prerouting rules to add source IP addresses of packets bearing this packet mark to an address list (also named EoIP), with address-list-timeout like 1h or so. Then replace the in-interface and out-interface matching in the change-mss rules by src-address-list=EoIP and dst-address-list=EoIP, respectively.

Is there any other way to lower the MTU below 1500 on an EoIP interface and not affect everyone else on the bridge even though they aren't even sending traffic over the EoIP interface.

There isn't. From the point of view of the IP stack, which deals with the MTU, the bridge (as in "the virtual interface of the router connected to the virtual port of the virtual switch") is the IP interface. So if the bridge interface has an MTU of 1500 and you give it a packet that has more than the EoIP's MTU, the bridge (as in "the virtual switch") will forward that packet to the EoIP port but EoIP will drop it without sending an ICMP notification back because bridges (as in "virtual switches") do not generate ICMP.

So the MTU of the bridge interface automatically decreases to the MTU of the member port whose MTU is the smallest of all; if you let EoIP set it automatically, it takes the MTU of the interface through which it sends the transport packets and subtracts the size of the headers from it. So the bridge accommodates to that, making the router send "fragmentation needed" if the packet to be routed via the bridge interface is bigger than that.

If you force the EoIP MTU to 1500, it simply means that EoIP will split the transport packets, carrying payload packets larger than the limit, into fragments, and even if the resulting doubling of the packet rate doesn't bother you, you may hit the issue of some networks randomly dropping non-first fragments of packets. A workaround to this is to use L2TP in BCP (bridge) mode with MLPPP, which splits the payload packets before loading them into the transport ones, and the transport ones are never fragmented. The issue of double packet rate remains, though.

is there a better more modern way to connect remote sites across the internet instead of EoIP where MTU sizing is not an issue and may also take less CPU resources and have better throughput?

The best way is to avoid L2 tunneling completely, so the MTU issues are handled at routing level as they should. Without restricting the payload MTU, any tunneling (even an L3 one) always causes (almost) doubling of packet rate simply because any kind of tunneling adds overhead to the payload packets when forming the transport ones, so the excess bytes overflow to a fragment. "Almost" because some payload packets are smaller then the tunnel MTU.

So if you do need the L2 tunneling across the internet, L2TP with BCP and MLPPP is still your best current bet as its transport packets are not fragmented. As for CPU load, it doesn't matter much whether you use EoIP, L2TP, or VxLAN. And if you use encryption, the process of encapsulating L2 frames into L3 transport packets is a negligible part of the CPU load compared to the encrytion and decryption.

I would guess you have somehow managed to engage the L3 and L2 tunnel in parallel, but it's just a wild guess. I'd have to see the complete configuration exports from both ends of the tunnel as well as the addresses you used for the ping (both source and destination). Don't forget to anonymize the exports while maintaining integrity of address prefixes if any public or global addresses are used in the configuration.

Is any local-address and remote-address specified on the /ppp secret row at the server side? If yes, what happens if you remove them?

Should there be a mac address associated with this interface somewhere? Also should arp=proxy-arp be set on the bridges and the admin-mac address at both ends?

No and no. The bridges the BCP tunnels become dynamic ports of already have some MAC addresses; whether these are assigned automatically or using admin-mac has no impact on the operation of the BCP. Of course the bridges geting interconnected must not have identical MAC addresses, but this requirement is nothing BCP-specific.

arp=proxy-arp makes sense when you connect individual Windows machines as L3 (non-BCP) L2TP clients, as Windows automatically add a route via the tunnel to the whole A (/8), B (/12), or C (/24) class subnet their individually assigned address fits into. So placing the pool for these clients into the LAN subnet allows you to avoid additional configuration on Windows side, but you must make the local devices in the LAN send the packets for the Windows clients to the router, which is achieved by means of proxy-arp that makes the router respond with its own MAC address to ARP requests regarding the IP addresses of currently active L2TP clients.

The reason why I was asking whether you have assigned IP addresses to the tunnel endpoints was to find out whether that may be the reason why the ping reports duplicates. As said it was just a wild guess.

I can only imagine multi-path propagation between the two devices to cause such an effect, but if it was due to a plain L2 loop, it should either cause much more harm than just duplicating ICMP echo requests or responses if no loop prevention mechanism is in use, or it should only last until the loop prevention mechanism cuts the loop, so not longer than a few seconds. And it makes no sense that this would be different for the EoIP case and for the BCP case if the two are used, one at a time, to interconnect the same bridges.

The only mechanism known to me that intentionally sends the same packet through two distinct paths is bonding in broadcast mode, but you haven't mentioned anything like that. But you haven't mentioned a hundred other things about your configuration either, because they are presumably unrelated to the issue.

In fact, the problem with these mysteries is that they are always caused by something you assume to be unrelated. That's why it only helps to post the complete configuration export, not just those few bits you assume to be relevant.

If it was bonding in broadcast mode, it is still strange that it would only affect BCP and not EoIP.

Another strange thing is that the duplication seems to only happen in one direction, otherwise you should get 4 response packets to a single echo request packet (unless the ICMP stack remembers the serial number of the last request and doesn't respond duplicates - I'd have to investigate a bit to be sure but it seems to me it would be against the purpose of ICMP echo).

At this point I would use sniffing at both devices, matching only on the IP address of the remote ping participant and icmp, to see what exactly happens there - from where does the ping request come in, which interfaces it passes through, and at which device it actually takes two outgoing interfaces.

/interface bridge filter
add action=mark-packet chain=input in-interface=eoip1 new-packet-mark=eoip1
/ip firewall mangle
add action=add-src-to-address-list address-list=eoip1 address-list-timeout=10m chain=prerouting packet-mark=eoip1
Of course these rules must be blended properly into your existing firewall.

use-ip-firewall=yes is a special setting of bridge whose primary purpose is to allow insertion of queues (for QoS) to the bridging path; for any other purposes it just causes unpleasant surprises. The fact that the ip firewall does not see the packet marks assigned by bridge filter may be one of them. So unless you had some other reason to set it to yes, revert it to no. If you have set it to yes for some other reason, there is no need to use bridge filter at all because with use-ip-firewall=yes, you can match on in-bridge-interface in IP firewall rules.

The bridge filter has its own input,output, and forward chains; forward is used for frames being bridged between external ports of the same bridge, input/output are related to frames being bridged from/to an external port to/from the internal port of the bridge (the one that is connected to the internal IP interface). Therefore all traffic that is routed is an input/output traffic from the perspective of the bridge firewall, although most of it is normally a forward one from the perspective of the ip firewall.

You only need to collect the IP addresses of devices connected via the EoIP tunnel at the router, because only there you can adjust the TCP MSS value in mangle. So they would be useless to collect them on a device that only bridges the traffic between EoIP and a physical port.

I have tested the rules I've posted upon the request of @nichky (of course with use-ip-firewall=no), so I know they work.

So try with use-ip-firewall set to no, and if it still doesn't work, show me the export of /ip firewall and of /interface bridge filter, and maybe explain what I may have missed. If both devices interconnected using EoIP were routers, there would be no need to use EoIP for the interconnection, so I assume you do need an L2 interconnection of the LANs at both sites.

• the netfilter (firewall) does not allow to change MSS in prerouting and input chains, and
• the two Mikrotiks most likely normally do not talk to each other using the addresses from the subnet that lives inside the EoIP tunnel.

So I assume what happens is that when you spawn the internal speedtest between the two Tiks after a long silence, the address of the server one is not on the eoip1 address list of the client one when the latter sends its SYN packet, so the MSS cannot get adjusted at the client because the server's address is not on the address list yet (it only appears there once the SYN,ACK from he server arrives) nor can it be adjusted at the server because MSS adjustment of received packets is simply not possible.

If this assumption is correct, it should actually not depend on the direction of the failing test but on its order of execution, i.e. the first test after long silence should fail regardless the direction.

Verify this assumption first; if it turns out not to be the reason, sniffing into a file at both EoIP tunnels simultaneously while doing the failing test should give you some hints on what is going on. There is a lot of space for doubt because if I remember correctly, there is a "control" session that spawns the actual test session, so the control communication should add the remotes' addresses to the local address lists before the actual test session starts, but maybe you use some other address of the server than the one in the subnet hosted in the EoIP tunnel when telling the bandwidth test client where to connect?

If someone properly reports that to Mikrotik, there sure will. But that's just my experience-based opinion, and I have no power over Mikrotik R&D priorities. Some fixes have taken years to happen.

Are you sure nothing has changed in the configuration between the last known good state and the occurrence of the issue? From the description I didn't understand that there was a change of behavior, let alone related to ROS version - the fact that the bridge adjusts its own MTU to the smallest one among the member ports' ones has been there since Day 1 of bridge implementation.