max-MTU Question

DeDMorozzzz · Fri Mar 31, 2023 8:16 am

Hello
Why setting maximum supported MTU is not used as a default setup?
I'm talking only about my own network, not regarding connection to any other networks, to avoid MTU problem that occurs in case of using vlan\Vxlan\EoIP\VPN?
Considering, that all devises of a subject-network support enlarged MTU?
Lets consider using MTU=2000 as a default setup

pe1chl · Fri Mar 31, 2023 11:06 am

Industry standard MTU is 1500. When you want something else, you will have to configure it.

rextended · Fri Mar 31, 2023 11:08 am

Industry standard MTU is 1500. When you want something else, you will have to configure it.

+10k

(Without considering that, if you don't configure the internal network correctly, you will create nothing but outgoing problems...)

DeDMorozzzz · Fri Mar 31, 2023 4:56 pm

Industry standard MTU is 1500. When you want something else, you will have to configure it.
+10k

(Without considering that, if you don't configure the internal network correctly, you will create nothing but outgoing problems...)

What problems, I wander?)
I have an EoIP links for PPPoE, sometimes over vlans, Im using mikrotik devises 99%. What can go wrong?
In the beginning i've said -"Considering, that all devises of a subject-network support enlarged MTU"

rextended · Fri Mar 31, 2023 5:11 pm

if you don't understand what @pe1chl wrote to you,
in short, it has to be like this and it has to stay like this,
any other consideration is useless.

If you need a different value, you can change it.
Simple and precise.

Amm0 · Fri Mar 31, 2023 5:18 pm

Why setting maximum supported MTU is not used as a default setup?
[...]
Lets consider using MTU=2000 as a default setup

You asking about "not used as a default setup" – not what may be best in your environment.

Max MTU on the internet is 1500. https://www.rfc-editor.org/rfc/rfc894

DeDMorozzzz · Fri Mar 31, 2023 5:36 pm

I'm not going to change any outgoing interface mtu, IT IS MENTIONED THAT:
1 All devices support enlarged MTU (All are mikrotik)
2 MTU's that are going to be changed ARE INSIDE of the controlled network
the question was - "why not just change MTU INSIDE THE NETWORK" to avoid possible problems with fragmentation if anything will change and everything should be done once again (added qinq\VxLAN\VPN\EoIP)
All your answers are not concerning the question.

pe1chl · Fri Mar 31, 2023 5:39 pm

It is not even clear what your question is, let alone what problem you have.
RouterOS is flexible. Configure it as you like!

rextended · Fri Mar 31, 2023 5:46 pm

All your answers are not concerning the question.

Ah, no? and what is "the question"?

Topic tite: "max-MTU Question"
Is the question?

Why setting maximum supported MTU is not used as a default setup?

Read reply on post #2, is the perfect answer.

I'm talking only about my own network, not regarding connection to any other networks, to avoid MTU problem that occurs in case of using vlan\Vxlan\EoIP\VPN?

If it is a question,
read reply on post #2, is the perfect answer.

Considering, that all devises of a subject-network support enlarged MTU?

Read reply on post #2, is the perfect answer.

Lets consider using MTU=2000 as a default setup

Read reply on post #2, is the perfect answer.

Since 3 people didn't understand anything else from what you have wrote, then enlighten us by writing better what the question would be...

DarkNate · Fri Mar 31, 2023 6:41 pm

Max L3 MTU should default to 1500 to minimise idiots from sending jumbo frames to the public internet.

However, for L2 MTU, there's no reason for it to not be maxed out.

Even if device A<>Devic B have 9000<>9216 L2 MTU, it doesn't break anything if L3 MTU is equivalent on both example 1500<>1500 or 8000<>8000.

For networks that I have been hired to design, fix or clean up, we always enable jumbo frames intra-AS. So L2 is maxed on all equipment on all vendors. L3 MTU intra-AS is set to 9000 wherever possible.

If some old hardware supports only 3000 MTU, we use 3000 L3 MTU for that particular link between device A and B, where B only does 3000.

PMTUD does its job and correctly sends packets/frames in correct size based on the path. We've never had any fragmentation in the networks I deployed large MTU on.

I learnt how to MTU correctly with jumbo frames from this article:
viewtopic.php?t=176358

I've also deployed jumbo frames for WAN/Internet when upstream supports it, but generally I try to avoid it as when they use MPLS, and they are stupid, the failover path is below 1400 MTU, so the jumbo frames on the WAN port then breaks PMTUD. So for WAN/Internet, I would use 1500, unless you use custom script to check MTU every 10 seconds and adjust interface MTU based on results.

DeDMorozzzz · Fri Mar 31, 2023 7:38 pm

DarkNate thank you
Everyone else - get lost please

Amm0 · Fri Mar 31, 2023 7:57 pm

You're in the "Beginner Basics" forum. And no "beginner" should be messing with MTU. Defaults are fine.

I think you should learn to write your questions more clearly – you'd get better answers. No need to be rude.

DarkNate · Fri Mar 31, 2023 8:43 pm

You're in the "Beginner Basics" forum. And no "beginner" should be messing with MTU. Defaults are fine.

MTU is a primary school level computer networking concept. It is “basics” in every sense of the word. One needs to have grown up illiterate without formal education to think otherwise.

If we mean PMTUD, then sure that's more advanced.

Amm0 · Fri Mar 31, 2023 8:53 pm

You're in the "Beginner Basics" forum. And no "beginner" should be messing with MTU. Defaults are fine.
MTU is a primary school level computer networking concept.

Yeah I assumed a toddler, in @DarkNate tautology.

DeDMorozzzz · Sat Apr 01, 2023 4:01 am

You're in the "Beginner Basics" forum. And no "beginner" should be messing with MTU. Defaults are fine.

I think you should learn to write your questions more clearly – you'd get better answers. No need to be rude.

You are sooooo funny trying to look important and advanced
That is why you think, that everyone is a beginner. In the first post i've mentioned that I just try to understand why everyone calculates MTU instead of just setting 2000 for example. I've wrote about vlans,Vxlans, EoIPs maybe because I work with it?
It was just a question - "why not?"
It is in the "Beginner Basics" not because I'm a beginner, It is hear because It fits fine here. What other forum fits better? - RouterOS general discussion? Forwarding Protocols?
So please get lost and do something useful

DarkNate thanks ones again

pe1chl · Sat Apr 01, 2023 10:12 am

Can you please explain again what your question is and what answer you are expecting?
Because even after re-reading it, that is not at all clear to me.
Default MTU is 1500. That is not going to change. But when you want 2000, you can set that.
Why is that not OK for you?

DeDMorozzzz · Sat Apr 01, 2023 1:55 pm

Can you please explain again what your question is and what answer you are expecting?
Because even after re-reading it, that is not at all clear to me.
Default MTU is 1500. That is not going to change. But when you want 2000, you can set that.
Why is that not OK for you?

The point is - Why calculating and even think of it at all, if there is a simple way? Why this approach isn't used as a default way of handling MTU?
It can bring a lot of problems if you are using any VPN, or vlan, why not set it to maximum as DarkNate does and just forget about it?

mkx · Sat Apr 01, 2023 2:31 pm

One of reasons: different devices support different maximum L2 packet sizes. So different devices will have to use different L3 MTU. But that L2 maximum packet size is true also for switches which may be in path between L3 devices (and switches, being L2 devices, don't do ICMP). Next: PMTU only works between L3 devices and ICMP reply only gets back from device which first successfully receives large packet and can't forward it because egress interface has smaller (L3!) MTU. And the whole process doesn't work inside same L2 network, in such network L3 MTU on all devices has to be the same (device can't send back ICMP for too large packet if it can't receive it first).

And tgen, 1500 bytes has been industry standard since ethernet was concieved in late 1970ies. Industry standards don't get changed at a whim of one user.
So back to answer #2 above.

rextended · Sat Apr 01, 2023 2:36 pm

If you write VPN, since I doubt you mean VPN for Local LAN, obviously L3 MTU size can't be more than 1500, for what already DarkNate wrote:

Max L3 MTU should default to 1500 to minimise idiots from sending jumbo frames to the public internet.

At this point is completely useless have 2000 (or 9000) as L2 MTU on WAN interface.

Since default MTU of 1500 is valid only for L3 layer (usually MikroTik devices have 1580 or 1588 as L2 default MTU)...........

Not all peripherals have the option to have 9000 or even 2000, it depends on the model and brands...
So it is obvious that by buying a MikroTik device that supports 9000 on L2, it is not said that then all the other peripherals will line up,
so it's up to you to configure them, changing the defaults from 158x on L2 to the maximum supported,
AND THEN correctly set the MTUs of the L3 which obviously have to be calculated.
So the calculations are always there, and the MTU defaults for L3 are always 1500, and for L2 it's usually 1500 + L3 headers

If your connection on ISP is, for example, by pppoe, if your ISP do not do something to have a full MTU vs Internet of 1500, you usually have less (ex. 1492)
And you can not do anything about that.
If you use VPN (or others) on that pppoe, you have to calc the correct VPN L3 MTU size (1492 - VPN overhead) or you do fragmentation.
Etc. etc. etc.

So, still valid reply on post #2 (and also what DarkNate wrote)

pe1chl · Sat Apr 01, 2023 4:12 pm

MTU of 9000 is introduced only with 1Gbps interfaces, and is called "jumbo frames".
It is only practical to use on a very limited local network like SAN or NAS network between fixed hosts and storage, usually in a data center scenario.
You cannot mix the same MTU on different hosts on the same network, because the hosts using the default 1500 byte MTU will normally not be able to RECEIVE a packet of 9000 bytes from another host.
So a local network with MTU much larger than 1500 is normally not practical, and even the margin above 1500 can be quite thin. The L2MTU can be like 1508 or 1512.

Amm0 · Sat Apr 01, 2023 5:37 pm

DarkNate isn't wrong, and gives decent advice – if you want to set L2MTU to max that's fine and leave headroom for whatever in future. But still don't think it being higher is going to have any effect from the defaults values, unless you also change change the L3MTU higher too.

And default L2MTU in recent v7 does allow for full frame 1500 L3 MTU...as L2MTU is 1568 or higher for most ethernet things. At L2MTU = 1568, that's enough for VXLAN over VLAN-enabled ethernet without any changes from defaults. No arithmetic math required, which seems to be goal?

DarkNate suggest same as everyone else:

Max L3 MTU should default to 1500 to minimise idiots from sending jumbo frames to the public internet.

So if you're sure that none of the "internal network" with a higher L3MTU will traverse the internet, then sure set a higher L3MTU on internal, non-internet hosts. As @pe1chl notes jumbo frames have a use, but typically:

only practical to use on a very limited local network like SAN or NAS network between fixed hosts and storage, usually in a data center scenario.

pe1chl · Sat Apr 01, 2023 5:42 pm

DarkNate isn't wrong, and gives decent advice – if you want to set L2MTU to max that's fine and leave headroom for whatever in future.

I'm not so sure about that. When this setting was only cosmetical it probably would not be there.
I guess this setting is used to allocate receive (and maybe transmit) buffers for the device driver, and setting it needlessly high will just use more memory and maybe be slower.
And of course it will bring no immediate benefit. You can just as well increase it when required.

DarkNate · Sun Apr 02, 2023 8:56 am

Higher number on L2 doesn't increase memory usage.

But if you set varying L2 MTU profiles, then it will affect the number of possible profiles loaded into memory. Each ASIC has limited amount of capacity for storing MTU profiles.

Hence, max it all out on all ports to create a single (or two) MTU profile-only, on the ASIC.

DarkNate · Sun Apr 02, 2023 8:58 am

only practical to use on a very limited local network like SAN or NAS network between fixed hosts and storage, usually in a data center scenario.

This is clearly written by someone who's an expert.

Jumbo frames benefits ISPs, Telecom, IXPs and carriers wherever possible, whoever supported. Enable 9000 L3 MTU and of course maxed L2 MTU.

Just remember to ask for jumbo frame VLAN from your IXP provider and also confirm with your upstream transit provider if they do jumbo frames and ask for the value.

he.net does jumbo frames out of the box.

mkx · Sun Apr 02, 2023 11:55 am

Code: Select all
only practical to use on a very limited local network like SAN or NAS network between fixed hosts and storage, usually in a data center scenario.
This is clearly written by someone who's an expert.

Jumbo frames benefits ISPs, Telecom, IXPs and carriers wherever possible, whoever supported. Enable 9000 L3 MTU and of course maxed L2 MTU.

The way I see it there are two (distinct) use cases where frame sizes larger than standard 1500 bytes is desirable:

transit and backbone networks
These typically involve adding some kind of overhead necessary for efficient traffic separation, such as VLAN, MPLS, GTP, PPPoE, etc overhead. If MTU in those networks is kept low, then L3 MTU may have to be decreased, leading to necessity to fragment packets. And as we all know, fragmentation causes both higher L3 overhead due to additional packet headers as well as higher CPU load on routers performing fragmentation, any firewalls beyond that point and receiver.
In order to avoid need for fragmentation in transit/backbone, there's the famous PMTUD with ICMP Fragmentation Needed, meaning that sender effectively performs fragmentation, which then avoids CPU overhead in transit/backbone nodes but still causes increased L3 overhead (less than fragmentation in transit/backbone which in worst case doubles the overhead while reduction of MTU increases overhead by some percent depending on actual PMTU).
datacenter LANs
It's important to remember why jumbo frames came along in data centers. It was a feature of FDDI networks (only later jumbo frames became optional normality in ethernet) and it was there to make processing overhead lower. Because those were the days of 286-class PCs and older (and much slower by today's standards) mainframes whichndid not really have processing power for real-time processing of packet headers.
In short: to reduce PPS.
This problem is much smaller (or even non-existant) these days and benefits of having jumbo frames are much less. Today it's fairly easy to saturate most (but fastest) interdace rates regardless of MTU used using average hardware. Which means that drawbacks of using jumbo frames can overshadow benefits.

It's clear that @DarkNate is talking mainly about case #1 above while @pe1chl is talking about case #2 above. It's not entirely clear which case covers OP's use case, but if I have to guess I'd choose #2. Based on discussion from his side I doubt OP knows what he wants to achieve and probably he's not aware of problems he'd get into with increasing the MTU (specially if that would be made a vendor standard which is what he's proposing if I correctly understand initial post of this thread). Again: there are a few very good reasons to stick to industry standard settings and it's clearly advanced topic to decide when to deviate from that standard. Yes, technical act of changing the value is indeed "beginner basic".

DarkNate · Sun Apr 02, 2023 12:14 pm

I work with both ISPs/Telcom networking and DC networking.

Everywhere I go, intra-AS it's all 9K MTU on L3 and maxed on L2 on each network devices. If Device A<>Device B is less than 9k, I simply configure the max L3 MTU for that particular interconnect. PMTUD takes care of the rest.

While not all customers/hosts are able to utilise 9K MTU, this allows future-proofing of the network for use such as VXLAN/EVPN encap/decap, and also customers can create WireGuard//IPSec tunnels between different end-points via the same backbone no problem with 8000+ MTU inside the tunnels.

The biggest problem with MTU/Jumbo frames config and deployment is lack of fundamentals understanding, concept and experience of doing jumbo frames everywhere you go.

The second problem is poor PMTUD config in the network backbone, it's less common, but still fairly present out there.

mkx · Sun Apr 02, 2023 12:44 pm

And then we come back to the fact that blindly changing MTU, without understanding consequences in each particular case very well, can cause big problems...

You're in the "Beginner Basics" forum. And no "beginner" should be messing with MTU. Defaults are fine.
MTU is a primary school level computer networking concept.

DarkNate · Sun Apr 02, 2023 12:55 pm

[moderator]
removed big part of post
[/moderator]
Otherwise, simply quit tech, and move to arts and humanities, you don't need fundamental understanding of MTU or BGP over there, your “feelings” are enough to get by.

Larsa · Sun Apr 02, 2023 1:34 pm

One might say you strongly remind me of once upon a time a talented but (in)infamous Northern European network specialist who acctively took part to build the first commercial IP networks in Europe. At first he refused to accept dial-up internet but was later ditched due to customer demand.

He later disappeared to the Big network company and hated all other providers. Since you're hanging around here I conclude it can't be the same guy but unfortunately you have the same condescending attitude. However, his was due to ASD (autism spectrum disorder) and once you got to know him he was a pretty decent bloke.

DarkNate · Sun Apr 02, 2023 2:08 pm

One might say you strongly remind me of once upon a time a talented but (in)infamous Northern European network specialist who acctively took part to build the first commercial IP networks in Europe. At first he refused to accept dial-up internet but was later ditched due to customer demand.

He later disappeared to the Big network company and hated all other providers. Since you're hanging around here I conclude it can't be the same guy but unfortunately you have the same condescending attitude. However, his was due to ASD (autism spectrum disorder) and once you got to know him he was a pretty decent bloke.

I'm not him and never heard of this. Either way, I certainly don't “hate”, hate is a strong word, and it requires energy to hate. You could say I have a strong dislike for stupidity – You can't blame me for faulting stupidity.

And I don't see the connection between him being against dial-up (at that time) and me. I prefer modern day networking approaches end-to-end. I ain't about to sit here and say MPLS is king, where VXLAN/EVPN can do many of the things already that MPLS was once required such as L2 VPN/VRFs etc.

Sun Apr 02, 2023 2:12 pm

According to warning and

It's not my job to cure mental illness aka stupidity/low IQ.

you deserved a week of vacation to calm down.

DeDMorozzzz · Mon Apr 03, 2023 9:23 am

Thank you all for answering.
DarkNate, at least 50% of total "Thanking" goes to you.

mkx, I'm talking about a mostly wireless network.
I'm dealing with PPPoE over EoIP planning to use OSPF to add L3-bonuses into WLan.
Maybe use MPLS\VPLS but not sure about that.
The way I see it now - deal with MTU and set PPPoE MTU to 1500 instead of 1480 (Is that a worthy effort? )

rextended · Mon Apr 03, 2023 10:17 am

As already written here:
viewtopic.php?p=993942#p993639

You can't go to 1500 inside pppoe, unless your ISP allows you to.
Regardless of the settings you can set on the device, if the provider doesn't allow you an MTU of 1500, you can't have it.
Sure, you can handwrite 1500, but still you'll have problems because you don't actually have a MTU of that value on the ISP side.

DeDMorozzzz · Mon Apr 03, 2023 10:36 am

As already written here:
viewtopic.php?p=993942#p993639

You can't go to 1500 inside pppoe, unless your ISP allows you to.
Regardless of the settings you can set on the device, if the provider doesn't allow you an MTU of 1500, you can't have it.
Sure, you can handwrite 1500, but still you'll have problems because you don't actually have a MTU of that value on the ISP side.

As I have said, All devices are in my own network so the PPPoE-Server is

rextended · Mon Apr 03, 2023 12:07 pm

It's not clear at all what you're talking about and how your network is made, if you want advice for the inside, you'd better be more specific.

If the ultimate goal of networking is NOT to get out on the Internet, anything can be done
(and you can have specific suggestions if you let us understand how the network is made).

However, if the ultimate goal of the network is to get devices out to the Internet, the MTU must necessarily be 1500 or less, again depending on the ISP.

If you make PPPoE connections, through EoIP tunnels, you have to calculate all the necessary overhead, otherwise there is no real MTU of 1500.

For example, physical interface MTU 1500, EoIP MTU (without fragmentation) must be 1500 - 42 = 1458
PPPoE MTU (without fragmentation) must be 1458 - 8 = 1450.
Therefore, to have a link that does not generate more traffic than it needs, the MTU must be 1450.

Instead, for have a real MTU of 1500 from device vs. internet,
MTU on physical interface must be at least 1550 for have EoIP without fragmentation of 1550- 42 = 1508
and real PPPoE MTU of 1508 - 8 = 1500.

mkx · Mon Apr 03, 2023 12:09 pm

mkx, I'm talking about a mostly wireless network.

Layer 1 (physical medium) doesn't matter, it's still two distinct use cases. If you're building your own (wireless) backbone network, then it's case #1 from my post ... and yes, you can increase L2 MTU as far as you want/need ... given that used hardware lets you.

But that doesn't mean that changing default MTU size in ROS is feasible. You'd always have problems. Default means that L3 interface on CPE, facing your backbone, would have MTU of 2k. And your backbone creates some overhead (EOIP, PPPoE, ... you name it), so you'd still have to set up your backbone nodes with larger MTU to avoid fragmentation on your nodes. Since you seem to know what you're doing, it's sensible to expect that you'll set up your backbone the way you want/need and leave MT defaults at values safe to use by everybody, even those who don't know squat about networking.
So back to post #2 above ...

Larsa · Mon Apr 03, 2023 12:57 pm

In addition to what mkx said, perhaps this might shed some more light on the subject using a couple of examples.

https://www.packetstreams.net/2018/07/t ... 3-mtu.html
viewtopic.php?t=131909#p648935

pe1chl · Mon Apr 03, 2023 1:48 pm

I still don't understand the use case for setting MTU to 2000.
1512, that I can understand. Or maybe 1600 as a "set and forget" case for all common encapsulations.
But I would not know any encapsulation protocol that has 500 bytes of overhead and requires MTU 2000 to transport the de-facto standard 1500 byte MTU.
Then there is MTU 9000 (jumbo frames), with the application I already described.
As others have written, you cannot have >1500 byte MTU on the internet traffic. That will not work.

DeDMorozzzz · Mon Apr 03, 2023 3:57 pm

I still don't understand the use case for setting MTU to 2000.
1512, that I can understand. Or maybe 1600 as a "set and forget" case for all common encapsulations.
But I would not know any encapsulation protocol that has 500 bytes of overhead and requires MTU 2000 to transport the de-facto standard 1500 byte MTU.
Then there is MTU 9000 (jumbo frames), with the application I already described.
As others have written, you cannot have >1500 byte MTU on the internet traffic. That will not work.

Will 1512 be enough to set PPPoE MTU to 1500, send it through EoIP and provided by another network vlan (they have max-L2MTU=9000) to PPPoE-Server?
If PPPoE MTU=1500, haw can a larger L3 MTU of backbone network do harm of any kind?
I've got the point that L2 MTU can be set to maximum

rextended · Mon Apr 03, 2023 4:01 pm

No, if you have read my post, must be al least 1550
viewtopic.php?t=194990#p993951

EoIP add 42 and PPoE add 8, so 42 + 8 = +50

EoIP can pass 1508 PPPoE (for have user's LAN MTU 1500) frame FRAGMENTING it on two frames.
Approx example: for send one fragmented packet of 1508, must transmit two packet, one that transmort 1458, and the other that transport the remaining 50,
But for each packet are needed 42 bytes to be sended over EoIP... In total for send one fragmented packet of 1508, send two packet:
one of 1500 (42 + 1458 data) and another of 92 (42 + 50 of data), without count other 20 bytes for guard interval between two packets, and another 4 bytes for FCS.
So, for transmit 1508, fragmenting the packet, are needed ~1616 and more than the double of the time,
one high percentage than instead increase L2 MTU and EoIP MTU (without fragmentation)...

mkx · Mon Apr 03, 2023 5:49 pm

Or another view ... from local LAN PC IP stack towards backbone:

normal IP packets, size including IP headers: 1500
PPPoE adds 8 bytes overhead:                    +8
EOIP adds 42 bytes of overhead:                +42
--------------------------------------------------------------
total:                                        1550

So:

local LAN (ethernet) interface will have MTU of 1500 (can have larger L2-MTU).
PPPoE gateway will have 1500 byte MTU on LAN interface (and matching L2-MTU). Will have at least 1508 byte L2 MTU on interface carrying PPPoE traffic.
PPPoE interface (/interface/pppoe-client/) will then have max-mtu=1500 and hopefully PPPoE server will grant that wish.
EOIP gateway will have at least 1508 byte L2 MTU on interface carrying PPPoE and at least 1550 byte MTU on IP interface carrying EOIP traffic. Remember, EOIP encapsulates payload into proper IP packets. Then EOIP gateway has to have L2 MTU at least 1550 bytes on interface which carries EOIP traffic. Upstream router has to match L3 MTU (of 1550 bytes as per example).

In case where items #2 and #3 are stacked in same device (e.g. Mikrotik router), then EOIP interface is the one which has to have L2 MTU of at least 1508 bytes. And network interface, which carries EOIP traffic (towards remote end of EOIP tunnel, e.g. ether1 if it's used to connect towards remote end), has to have MTU of at least 1550 bytes.

Larsa · Mon Apr 03, 2023 6:04 pm

I think this article sums up all the relevant parts pretty well: "Networkworld - MTU size issues, fragmentation, and jumbo frames"

rextended · Mon Apr 03, 2023 10:36 pm

@DeDMorozzzz
Why you do not use VLAN instead of EoIP (not knowing why you use EoIP, this a legit question)
If you use VLAN, you do not have to do any on MTU, just set on VLAN the MTU of 1508 and can transparently support pppoe with 1500 MTU...

VLAN use 4 byte on L2 MTU + 8 extra for PPPoE, usually L2 MTU supported from various vendor is max 1518 and 1512 do not cause any problem.

DeDMorozzzz · Tue Apr 04, 2023 11:10 am

@DeDMorozzzz
Why you do not use VLAN instead of EoIP (not knowing why you use EoIP, this a legit question)
If you use VLAN, you do not have to do any on MTU, just set on VLAN the MTU of 1508 and can transparently support pppoe with 1500 MTU...

VLAN use 4 byte on L2 MTU + 8 extra for PPPoE, usually L2 MTU supported from various vendor is max 1518 and 1512 do not cause any problem.

The main reason as I see it now is to make an L3 routed network instead of L2 switched
The greater part of this network is a wireless network, sometimes stretched through several wireless hops. Wireless links are not that stable, sometimes it causes drops and packets needs to be transmitted all the way ones more. L3 network (as I understand it) will transmit the packet only through the link that has dropped a packet
Also the owner of another network (that provides a vlan) doesn't let doing QinQ, so it is L3 linked now. But the main reason is a packet retransmission through several links (L2 variant) in the case of a packet loss

pe1chl · Tue Apr 04, 2023 11:14 am

EoIP will not retransmit packets! A lost packet is simply not delivered, it is up to the endpoint to transmit it again.

rextended · Tue Apr 04, 2023 11:23 am

What write @pe1chl is correct, but you can have routed network also with VLAN... (and still use EoIP just on unsupported QinQ part)

L2 is on switching, and not "hubbing" and the packet go directly to the needed destination, is not broadcasted to all devices like is one old "hub"...

pe1chl · Tue Apr 04, 2023 12:05 pm

Also note that MikroTik has no routing protocol available to make an imperfect WiFi network working without problems.
The existing routing protocols are mainly intended for wired links that either are down or work 100% correctly.
A link with packet loss is used roughly the same way as a link without. There is no algorithm that routes around packet loss.
(one could hope that there is some feedback from S/N ratio and packet loss into a suitable routing protocol, but there isn't)

So, when your wireless links are not that stable, your network will not be that stable. Either you improve the links or you live with it.

DeDMorozzzz · Tue Apr 04, 2023 12:48 pm

EoIP will not retransmit packets! A lost packet is simply not delivered, it is up to the endpoint to transmit it again.

but if the network is L3 (based on OSPF), than means the endpoint is just the AP\Bridge point isn't it?

pe1chl · Tue Apr 04, 2023 2:53 pm

You seem to think that with L3 there will be re-transmission across a different path but that isn't any more the case than with L2.
E.g. a L2 spanning-tree setup would behave the same way as a L3 routed network: any packet lost on any of the links will mean the packet has to be re-transmitted from the source.
And OSPF (or BGP, or spanning-tree for that matter) will not re-route based on "too much packet loss". It will only re-route on total failure of the link.

rextended · Tue Apr 04, 2023 3:06 pm

[…] the packet has to be re-transmitted from the source. […]

And it is the final destination device that must re-request the lost packets,
provided that the protocol used by the application allows it... (for example UDP VoIP/SIP no... One download via HTTP(S) yes)

DeDMorozzzz · Tue Apr 04, 2023 3:39 pm

You seem to think that with L3 there will be re-transmission across a different path but that isn't any more the case than with L2.

No mi opinion is
Lets think of it as row of devises A-B-C-D-E
A- is a PPPoE-server, B,C and D are wireless devises E- is a mikrotik wireless-CPE
I think that with L3, there is a buffer in every hop and if a packet is lost at the D-E distance, it will be retransmitted only via this final hop (as every device is a router and it is an IP-connection)
In case of L2 a lost packet will have to be send through the A-B-C-D-E devices
That is my concern

pe1chl · Tue Apr 04, 2023 3:56 pm

Ok, but that is not true.
In IP networks, the network layer (L3) is datagram. There is no re-transmission at L3.
There can only be re-transmission at L4, or in some link layers at L2.

rextended · Tue Apr 04, 2023 5:10 pm

wlan try to retransmit "full frame" packet till 7 times, on default, but regardless it's L2 or L3+ content.
It's hardware retry (note: it's hardware, not L2 or L3+).

DeDMorozzzz · Tue Apr 04, 2023 5:27 pm

Ok, but that is not true.
In IP networks, the network layer (L3) is datagram. There is no re-transmission at L3.
There can only be re-transmission at L4, or in some link layers at L2.

L3 IP is a datagram? So were packets are?

DeDMorozzzz · Tue Apr 04, 2023 5:28 pm

wlan try to retransmit "full frame" packet till 7 times, on default, but regardless it's L2 or L3+ content.
It's hardware retry (note: it's hardware, not L2 or L3+).

Thank you I didn't know that

mkx · Tue Apr 04, 2023 7:15 pm

L3 IP is a datagram? So were packets are?

Datagram is a generic term ... and is more or less the same as packet.

pe1chl · Tue Apr 04, 2023 7:32 pm

Ok, but that is not true.
In IP networks, the network layer (L3) is datagram. There is no re-transmission at L3.
There can only be re-transmission at L4, or in some link layers at L2.
L3 IP is a datagram? So were packets are?

Datagram means that the delivery of the packets is best-effort. The packet will not be re-tried at L3 when the link layer drops it.

Amm0 · Tue Apr 04, 2023 8:31 pm

All true. But the crux here is TCP (which is correctly Layer 4 in ISO model, not "L3")... TCP does retry & importantly very likely the majority of the underlying traffic.

But easy to believe a "flaky" (errors/drop/etc) PtP link you cannot control would essentially cause "shitty" TCP performance.

e.g. if errors are happening at Layer 2 (ethernet/wireless), the last thing you want to is for Layer 3 packets (IP) to get fragmented on top... which could happen if MTU is too high... And, even MTU was correct, any Layer 2 corrected errors/re-transmitted that typically add latency. Either one of those problems is going slow down TCP.

So the general idea to "smooth" the "flaky" internal links by using some tunneling? If so, just thinking... if you really cannot control the properties of the PtP links to prevent errors/drop/etc that causing the desire to tunnel... I'm not sure you escape some homework to quantify the "shity-ness" of the transport and its max MTU before making decisions on tunneling/MTU. e.g. if they are wireless links, typically you'd have ~2300 MTU available...but if you can't set MTU on the wireless hardware, it's MTU is going to "win" if it's the standard 1500.

DeDMorozzzz · Wed Apr 05, 2023 4:29 am

So the general idea to "smooth" the "flaky" internal links by using some tunneling?

The tunneling is for PPPoE but also it does network segmentation as a side-effect

The question for all - Don't you think, that my goal of all that is a good way of improving network quality? I'm talking about increasing PPPoE's MTU to 1500. now its 1480 (MTU of a backbone network is already raised to feat 1480)
The point of changing the MTU inside the network is to make it equal to the MTU outside the network.

nichky · Wed Apr 05, 2023 6:39 am

very useful website provided by @Amm0

https://baturin.org/tools/encapcalc/

rextended · Wed Apr 05, 2023 11:01 am

Don't you think, that my goal of all that is a good way of improving network quality?

O-B-V-I-O-U-S-L-Y

But I have already wrote the method to be used, my clients all have MTU 1500 between Internet and LAN by using VLAN & PPPoE
viewtopic.php?t=194990#p994098

The point of changing the MTU inside the network is to make it equal to the MTU outside the network.

Is a very great choice.

I understand can you can not do QinQ on some parts of your net because you transit inside another provider network that you do not control,
but on you parts VLANs can be used, and also you can use EoIP (or other tunnel type) just in the part where is not usable the VLAN.

DeDMorozzzz · Thu Apr 06, 2023 8:42 am

Don't you think, that my goal of all that is a good way of improving network quality?
O-B-V-I-O-U-S-L-Y
but on you parts VLANs can be used, and also you can use EoIP (or other tunnel type) just in the part where is not usable the VLAN.

Thank you, I guess you are right.
I'll study the links above to get ready to change the net

nichky · Thu Apr 06, 2023 9:19 am

put the full config here, maybe someone will modificate that

DeDMorozzzz · Sat Apr 08, 2023 5:26 am

put the full config here, maybe someone will modificate that

Do you mean export-compact, or graphical scheme?
I haven't carefully read the links provided by guys above yet, but at least I'm going to compare CPU-usage of EoIP and VLAN.
VLAN- overhead is low, it's not a proprietary.
Thank you for being envolved.

nichky · Sat Apr 08, 2023 7:04 am

CPU should be fine,
just export

Buckeye · Sun Apr 09, 2023 7:10 am

When responding to another thread, I noticed that my new RB5009 has defconf value for L2MTU set to 1514, which to me seems too low.

Screen shots here.

Am I misinterpreting something?

Buckeye · Sun Apr 09, 2023 8:39 am

According to warning and
It's not my job to cure mental illness aka stupidity/low IQ.
you deserved a week of vacation to calm down.

DarkNate is polite compared to some poster's I knew from comp.os.vms. Specifically Carl J Lydick. He was extremely knowledgeable and helpful, but extremely intolerant of posters that did not meet his standards.

pe1chl · Sun Apr 09, 2023 11:58 am

When responding to another thread, I noticed that my new RB5009 has defconf value for L2MTU set to 1514, which to me seems too low.

Screen shots here.

Am I misinterpreting something?

What is your problem? It is high enough for the normal use cases, and when you have a special use case (like MPLS or jumbo frames) you can increase it up to the Max L2 MTU.

Buckeye · Sun Apr 09, 2023 7:18 pm

What is your problem? It is high enough for the normal use cases, and when you have a special use case (like MPLS or jumbo frames) you can increase it up to the Max L2 MTU.

I don't consider vlans a special use case.

It just seemed extremely odd to me that the RB5009 had such a low value compared to my hEX S (which only supports MAX-L2MTU of 2026, but its default L2MTU is set to 1596)

Also the RB5009 doesn't seem to adhere to what this previous post said:

And default L2MTU in recent v7 does allow for full frame 1500 L3 MTU...as L2MTU is 1568 or higher for most ethernet things. At L2MTU = 1568, that's enough for VXLAN over VLAN-enabled ethernet without any changes from defaults. No arithmetic math required, which seems to be goal?

I am running v7.8 on both the hEX S and the RB5009.

Amm0 · Sun Apr 09, 2023 7:25 pm

And default L2MTU in recent v7 does allow for full frame 1500 L3 MTU...as L2MTU is 1568 or higher for most ethernet things. At L2MTU = 1568, that's enough for VXLAN over VLAN-enabled ethernet without any changes from defaults. No arithmetic math required, which seems to be goal?
I am running v7.8 on both the hEX S and the RB5009.

My RB5009 is the PoE version, maybe difference.

But I'm guessing the default value that's populated for L2MTU depends perhaps on what version a netinstall/reset-configuration was been done. But just "upgrade" RouterOS version would keep the existing L2MTU & so it may be using an "old default". But dunno. I can say the the default L2MTU has varied over the years, but seen the L2MTU=1568 a few times now.

Buckeye · Sun Apr 09, 2023 8:09 pm

For what it's worth, this RB5009 was purchased in Mar 2023 and was delivered with v7.6 (even though it could be down-graded to factory-software: 7.4.1)

I partitioned to 4 partitions, copied to part1 and upgraded to 7.7, then copied to part2 and upgraded to 7.8.

I am really surprised that more info about partitions isn't advertised. It is a really nice feature, but the documentation sucks. It doesn't even have a mention of the activate command. It also isn't clear (to me) what restore-config-from and save-config-to actually do. Is it more like a backup/restore or an export/import? (there is a difference in what those will "copy")

[demo@RB5009-1-P2] > system/routerboard/print
routerboard: yes
model: RB5009UG+S+
serial-number: ************
firmware-type: 70x0
factory-firmware: 7.6
current-firmware: 7.7
upgrade-firmware: 7.8
[demo@RB5009-1-P2] > partitions/print
Flags: A - ACTIVE; R - RUNNING
Columns: NAME, FALLBACK-TO, VERSION, SIZE
# NAME FALLBACK-TO VERSION SIZE
0 part0 next RouterOS v7.6 Oct/17/2022 10:55:40 256MiB
1 part1 next RouterOS v7.7 Jan/12/2023 07:35:45 256MiB
2 AR part2 next RouterOS v7.8 Feb/24/2023 09:03:00 256MiB
3 part3 next RouterOS v7.6 Oct/17/2022 10:55:40 256MiB
[demo@RB5009-1-P2] >

Let me upgrade the routerboard firmware and see if anything changes.

[demo@RB5009-1-P2] > system/routerboard/print
routerboard: yes
model: RB5009UG+S+
serial-number: ************
firmware-type: 70x0
factory-firmware: 7.6
current-firmware: 7.8
upgrade-firmware: 7.8
[demo@RB5009-1-P2] > partitions/print
Flags: A - ACTIVE; R - RUNNING
Columns: NAME, FALLBACK-TO, VERSION, SIZE
# NAME FALLBACK-TO VERSION SIZE
0 part0 next RouterOS v7.6 Oct/17/2022 10:55:40 256MiB
1 part1 next RouterOS v7.7 Jan/12/2023 07:35:45 256MiB
2 AR part2 next RouterOS v7.8 Feb/24/2023 09:03:00 256MiB
3 part3 next RouterOS v7.6 Oct/17/2022 10:55:40 256MiB
[demo@RB5009-1-P2] > interface/print
Flags: R - RUNNING; S - SLAVE
Columns: NAME, TYPE, ACTUAL-MTU, L2MTU, MAX-L2MTU, MAC-ADDRESS
# NAME TYPE ACTUAL-MTU L2MTU MAX-L2MTU MAC-ADDRESS
0 R ether1 ether 1500 1514 9796 48:A9:8A:75:DC:15
1 S ether2 ether 1500 1514 9796 48:A9:8A:75:DC:16
2 S ether3 ether 1500 1514 9796 48:A9:8A:75:DC:17
3 S ether4 ether 1500 1514 9796 48:A9:8A:75:DC:18
4 S ether5 ether 1500 1514 9796 48:A9:8A:75:DC:19
5 S ether6 ether 1500 1514 9796 48:A9:8A:75:DC:1A
6 S ether7 ether 1500 1514 9796 48:A9:8A:75:DC:1B
7 S ether8 ether 1500 1514 9796 48:A9:8A:75:DC:1C
8 S sfp-sfpplus1 ether 1500 1514 9796 48:A9:8A:75:DC:1D
;;; defconf
9 R bridge bridge 1500 1514 48:A9:8A:75:DC:16
[demo@RB5009-1-P2] > interface/bridge/print
Flags: X - disabled, R - running
0 R ;;; defconf
name="bridge" mtu=auto actual-mtu=1500 l2mtu=1514 arp=enabled
arp-timeout=auto mac-address=48:A9:8A:75:DC:16 protocol-mode=rstp
fast-forward=yes igmp-snooping=no auto-mac=no admin-mac=48:A9:8A:75:DC:16
ageing-time=5m priority=0x8000 max-message-age=20s forward-delay=15s
transmit-hold-count=6 vlan-filtering=yes ether-type=0x8100 pvid=1
frame-types=admit-all ingress-filtering=yes dhcp-snooping=no
[demo@RB5009-1-P2] > system/resource/print
uptime: 9m30s
version: 7.8 (stable)
build-time: Feb/24/2023 09:03:00
factory-software: 7.4.1
free-memory: 831.7MiB
total-memory: 1024.0MiB
cpu: ARM64
cpu-count: 4
cpu-frequency: 350MHz
cpu-load: 0%
free-hdd-space: 227.0MiB
total-hdd-space: 256.0MiB
write-sect-since-reboot: 119
write-sect-total: 29126
bad-blocks: 0%
architecture-name: arm64
board-name: RB5009UG+S+
platform: MikroTik
[demo@RB5009-1-P2] >

Updating routerboard firmware did not make a difference.

mkx · Sun Apr 09, 2023 8:39 pm

Does /interface export shiw command which is setting l2mtu to the low value? If it's shown, then the default is different in running ROS version. If it's not shown, then it's default value. Could be that this value was (erronous?) default in factory-installed ROS, but upgrades then don't override it with new defaults.

Re partitions: restore-config-from and backup-config-to transfers config between partitions and I'd bet it's similar to backup/restore (not export/import). copy transfers also ROS.

pe1chl · Sun Apr 09, 2023 8:40 pm

What is your problem? It is high enough for the normal use cases, and when you have a special use case (like MPLS or jumbo frames) you can increase it up to the Max L2 MTU.
I don't consider vlans a special use case.

That is why I did not mention them. VLANs are in many cases catered for automatically (i.e. you do not need to fiddle with any parameter).

It just seemed extremely odd to me that the RB5009 had such a low value compared to my hEX S (which only supports MAX-L2MTU of 2026, but its default L2MTU is set to 1596)

It is not clear what the L2-MTU parameter is necessary for, but I think the whole reason for having it is controlling the buffer allocation in the low level driver. And that may not be the same in all drivers, it may depend on the origin of the driver.
I think that some of the drivers will allocate memory for receive buffers based on L2-MTU, i.e. they keep a stash of buffers large enough to hold a worst-case received packet.
You need to set L2 MTU large enough to accommodate the largest possible packet, but there is no advantage in setting it higher, and there actually may be a disadvantage because more memory is allocated for receive buffers, and new receive buffers allocated from kernel memory are larger and thus result in worse memory pool fragmentation.
Of course this is the kind of thing that you will normally not notice at all. When you are so obsessed with L2 MTU, just set it at the max allowed value so you sleep better. Remember it is a settable parameter! Everyone can set it to his/her liking. That is the beauty of RouterOS: you still have a lot of control at low level, there isn't any "wizard" calulating these values for you and potentially causing unexpected problems. It is all up to you.

DarkNate · Sun Apr 09, 2023 9:01 pm

DarkNate is polite compared to some poster's I knew from comp.os.vms. Specifically Carl J Lydick. He was extremely knowledgeable and helpful, but extremely intolerant of posters that did not meet his standards.

I'm certainly not the “rudest” person to have walked the earth, there are people far worse than me who do far worse things than just argue on a forum, I don't see mods or law enforcement doing squat in that area. But sure, mods may sleep better at night thinking kicking me out makes the world a better place.

Regardless OP, you already got clear answer from me on MTU, you already know what you need to do in your network for jumbo frames, this topic should be marked as solved.

As for “defaults”, it'll likely never change, not only MikroTik, it's the same on Juniper, Cisco, Huawei, Arista.

Mon Apr 10, 2023 12:35 am

It did not take DarkNate more than few hours to call someone an idiot just after he had got ability to post again so not only moderators could sleep better.

Buckeye · Mon Apr 10, 2023 1:41 am

Does /interface export show command which is setting l2mtu to the low value? If it's shown, then the default is different in running ROS version. If it's not shown, then it's default value. Could be that this value was (erronous?) default in factory-installed ROS, but upgrades then don't override it with new defaults.

I think we need a sample size greater than one to know much. @ammo has stated his PoE version did now show l2mtu of 1514, but it is also possible it had a differenet factory installed version.

without verbose, neither my RB760iGS (which came with factory-software: 6.46.4 and factory-firmware: 6.46.4 and it was never netinstalled (upgrading to 7.2? was one of the first things I did). So if anything, it seems the larger l2mtu would have been dragged from v6, but neither the hEX S or the RB5009 exports without verbose have any mention of l2mtu. I would have expected that if a default changed, that the upgrade may "override" the new default, but I would also expect the override to show up in the non-verbose export (since it would no longer match the new default). As is the case for the things that show up in the hEX S non-verbose export. Untill I saw this, I forgot I never cleaned up the limit to 10M on the hEX ether5 (when I was doing some testing of negotiation).

From the RB760iGS hEX S (both /interface/ethernet/export verbose and /interface/ethernet/export)

[demo@RB760iGS-1] > interface/ethernet/export verbose
# apr/09/2023 18:11:10 by RouterOS 7.8
# software id = ****-****
#
# model = RB760iGS
# serial number = ************
/interface ethernet
set [ find default-name=ether4 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled \
    arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1596 \
    loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=DC:2C:6E:7B:10:F4 mtu=\
    1500 name=eth4-BR-SW_U10_T241 orig-mac-address=DC:2C:6E:7B:10:F4 rx-flow-control=off speed=1Gbps tx-flow-control=\
    off
set [ find default-name=ether1 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled \
    arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1596 \
    loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=DC:2C:6E:7B:10:F1 mtu=\
    1500 name=ether1-WAN orig-mac-address=DC:2C:6E:7B:10:F1 rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether2 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled \
    arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1596 \
    loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=DC:2C:6E:7B:10:F2 mtu=\
    1500 name=ether2-BR-SW-Base-U1 orig-mac-address=DC:2C:6E:7B:10:F2 rx-flow-control=off speed=1Gbps tx-flow-control=\
    off
set [ find default-name=ether3 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled \
    arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1596 \
    loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=DC:2C:6E:7B:10:F3 mtu=\
    1500 name=ether3-BR-SW-U241 orig-mac-address=DC:2C:6E:7B:10:F3 rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether5 ] advertise=10M-half,10M-full arp=enabled arp-timeout=auto auto-negotiation=yes \
    bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1596 loop-protect=default \
    loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=DC:2C:6E:7B:10:F5 mtu=1500 name=\
    ether5-off_bridge_wrk orig-mac-address=DC:2C:6E:7B:10:F5 poe-lldp-enabled=no poe-out=auto-on poe-priority=10 \
    power-cycle-interval=none !power-cycle-ping-address power-cycle-ping-enabled=no !power-cycle-ping-timeout \
    rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=sfp1 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled \
    arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1596 \
    loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=DC:2C:6E:7B:10:F6 mtu=\
    1500 name=sfp1 orig-mac-address=DC:2C:6E:7B:10:F6 rx-flow-control=off sfp-shutdown-temperature=95C speed=1Gbps \
    tx-flow-control=off
/interface ethernet switch
set 0 !cpu-flow-control mirror-source=none mirror-target=none name=switch1
/interface ethernet switch port
set 0 !egress-rate !ingress-rate
set 1 !egress-rate !ingress-rate
set 2 !egress-rate !ingress-rate
set 3 !egress-rate !ingress-rate
set 4 !egress-rate !ingress-rate
set 5 !egress-rate !ingress-rate
[demo@RB760iGS-1] > interface/ethernet/export
# apr/09/2023 18:11:15 by RouterOS 7.8
# software id = ****-****
#
# model = RB760iGS
# serial number = ************
/interface ethernet
set [ find default-name=ether4 ] name=eth4-BR-SW_U10_T241
set [ find default-name=ether1 ] name=ether1-WAN
set [ find default-name=ether2 ] name=ether2-BR-SW-Base-U1
set [ find default-name=ether3 ] name=ether3-BR-SW-U241
set [ find default-name=ether5 ] advertise=10M-half,10M-full name=ether5-off_bridge_wrk
[demo@RB760iGS-1] >

From the RB5009UG+S+ (both /interface/ethernet/export verbose and /interface/ethernet/export)

[demo@RB5009-1-P2] > /interface/ethernet/export verbose 
# apr/09/2023 18:07:01 by RouterOS 7.8
# software id = ****-****
#
# model = RB5009UG+S+
# serial number = ***********
/interface ethernet
set [ find default-name=ether1 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full,2500M-full arp=\
    enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1514 \
    loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=48:A9:8A:75:DC:15 mtu=\
    1500 name=ether1 orig-mac-address=48:A9:8A:75:DC:15 rx-flow-control=off speed=2.5Gbps tx-flow-control=off
set [ find default-name=ether2 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled \
    arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1514 \
    loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=48:A9:8A:75:DC:16 mtu=\
    1500 name=ether2 orig-mac-address=48:A9:8A:75:DC:16 rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether3 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled \
    arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1514 \
    loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=48:A9:8A:75:DC:17 mtu=\
    1500 name=ether3 orig-mac-address=48:A9:8A:75:DC:17 rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether4 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled \
    arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1514 \
    loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=48:A9:8A:75:DC:18 mtu=\
    1500 name=ether4 orig-mac-address=48:A9:8A:75:DC:18 rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether5 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled \
    arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1514 \
    loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=48:A9:8A:75:DC:19 mtu=\
    1500 name=ether5 orig-mac-address=48:A9:8A:75:DC:19 rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether6 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled \
    arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1514 \
    loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=48:A9:8A:75:DC:1A mtu=\
    1500 name=ether6 orig-mac-address=48:A9:8A:75:DC:1A rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether7 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled \
    arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1514 \
    loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=48:A9:8A:75:DC:1B mtu=\
    1500 name=ether7 orig-mac-address=48:A9:8A:75:DC:1B rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether8 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled \
    arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1514 \
    loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=48:A9:8A:75:DC:1C mtu=\
    1500 name=ether8 orig-mac-address=48:A9:8A:75:DC:1C rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=sfp-sfpplus1 ] advertise="" arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=\
    unlimited/unlimited disabled=no full-duplex=yes l2mtu=1514 loop-protect=default loop-protect-disable-time=5m \
    loop-protect-send-interval=5s mac-address=48:A9:8A:75:DC:1D mtu=1500 name=sfp-sfpplus1 orig-mac-address=\
    48:A9:8A:75:DC:1D rx-flow-control=off sfp-rate-select=high sfp-shutdown-temperature=95C speed=10Gbps \
    tx-flow-control=off
/interface ethernet switch
set 0 cpu-flow-control=yes mirror-egress-target=none name=switch1
/interface ethernet switch port
set 0 !egress-rate !ingress-rate mirror-egress=no mirror-ingress=no mirror-ingress-target=none
set 1 !egress-rate !ingress-rate mirror-egress=no mirror-ingress=no mirror-ingress-target=none
set 2 !egress-rate !ingress-rate mirror-egress=no mirror-ingress=no mirror-ingress-target=none
set 3 !egress-rate !ingress-rate mirror-egress=no mirror-ingress=no mirror-ingress-target=none
set 4 !egress-rate !ingress-rate mirror-egress=no mirror-ingress=no mirror-ingress-target=none
set 5 !egress-rate !ingress-rate mirror-egress=no mirror-ingress=no mirror-ingress-target=none
set 6 !egress-rate !ingress-rate mirror-egress=no mirror-ingress=no mirror-ingress-target=none
set 7 !egress-rate !ingress-rate mirror-egress=no mirror-ingress=no mirror-ingress-target=none
set 8 !egress-rate !ingress-rate mirror-egress=no mirror-ingress=no mirror-ingress-target=none
set 9 !egress-rate !ingress-rate mirror-egress=no mirror-ingress=no mirror-ingress-target=none
[demo@RB5009-1-P2] > /interface/ethernet/export
# apr/09/2023 18:07:06 by RouterOS 7.8
# software id = ****-****
#
# model = RB5009UG+S+
# serial number = ***********
[demo@RB5009-1-P2] >

rextended · Mon Apr 10, 2023 2:35 am

@Buckeye, about default MTU on RB5009:

viewtopic.php?p=995459#p995459

Buckeye · Mon Apr 10, 2023 3:21 am

When responding to another thread, I noticed that my new RB5009 has defconf value for L2MTU set to 1514, which to me seems too low.

Screen shots here.

Am I misinterpreting something?
What is your problem? It is high enough for the normal use cases, and when you have a special use case (like MPLS or jumbo frames) you can increase it up to the Max L2 MTU.

I was incorrectly interpreting l2mtu as if it was full frame mtu.

DeDMorozzzz · Mon Apr 10, 2023 9:34 am

this topic should be marked as solved.

It really should.
Should I mark in anyway, or it's a moderator's job?

mozerd · Mon Apr 10, 2023 11:04 am

It really should.
Should I mark in anyway, or it's a moderator's job?

DeDMorozzzz YOU should mark it as solved because you are the initiator ....

BTW, DarkNate is a very knowledgeable person and I 4 1 admire his many contributions regardless
...

anav · Mon Apr 10, 2023 3:09 pm

To be more accurate, the initiator is the only one that can classify it solved. ( choose the most appropriate post and there is a green mark with a checkbox in the upper right hand corner and when you hover over states something like 'accept this answer'

DeDMorozzzz · Mon Apr 10, 2023 4:09 pm

To be more accurate, the initiator is the only one that can classify it solved. ( choose the most appropriate post and there is a green mark with a checkbox in the upper right hand corner and when you hover over states something like 'accept this answer'

Thank you, not all forums are the same or it just my experience

Frederick88 · Tue Apr 11, 2023 5:25 am

This thread has been an interesting read leading me to look into MTU...

PMTUD does its job and correctly sends packets/frames in correct size based on the path. We've never had any fragmentation in the networks I deployed large MTU on.

This has me wondering, if I change my computers NIC to Jumbo MTU 9000, and visit https://www.speedguide.net/analyzer.php - results say my MTU is 9000.. however, since router's WAN port (RB4011 ether1) is set to default MTU 1500, why doesn't PMTUD negotiate an MTU of 1500 with the computer's NIC, for traffic that's headed out router's ether1 (set with MTU 1500)?

Secondary question - if using a CRS3xx, with hardware VLAN'ing enabled, does this still count towards L3 MTU, or L2 MTU since it doesn't have to go through CPU routing?

mkx · Tue Apr 11, 2023 8:11 am

Secondary question - if using a CRS3xx, with hardware VLAN'ing enabled, does this still count towards L3 MTU, or L2 MTU since it doesn't have to go through CPU routing?

Normal (L3 MTU) only applies when device communicates over L3 (IP). If device is used as switch, then it only communicates on L2 and L2MTU applies. If device is used for routing, then yes, L3 MTU does matter. IIRC it's been explained that if connection gets offloaded to HW, device doesn't act correctly in PMTUD procedure (i.e. it doesn't respond with ICMP size exceeded to oversize packet) because that would have to involve drvice's CPU and HW offloading is done not to involve CPU. This is how it was done in some older v7 and I don't remember seeing change log entry which would say otherwise since then. Or there was some such change log entry but it didn't click to me.

anav · Tue Apr 11, 2023 2:01 pm

I learned, dont mess with MTU unless you know what you are doing and why!
Default MTU works in most cases, and dont mess with ICMP--> let it ping, let it ping........catchy tune.
I'm still on the fence on if there is any relationship between L2 MTU and L3 MTU and that is how confusing this thread has been.

pe1chl · Tue Apr 11, 2023 2:05 pm

This thread has been an interesting read leading me to look into MTU...

PMTUD does its job and correctly sends packets/frames in correct size based on the path. We've never had any fragmentation in the networks I deployed large MTU on.
This has me wondering, if I change my computers NIC to Jumbo MTU 9000, and visit https://www.speedguide.net/analyzer.php - results say my MTU is 9000.. however, since router's WAN port (RB4011 ether1) is set to default MTU 1500, why doesn't PMTUD negotiate an MTU of 1500 with the computer's NIC, for traffic that's headed out router's ether1 (set with MTU 1500)?

That site probably uses the MSS inside a TCP SYN to tell the MTU on your network.
When you do not configure MSS clamping in your router, that will likely cause issues.

Buckeye · Tue Apr 11, 2023 8:43 pm

This thread has been an interesting read leading me to look into MTU...

These MTU values are all for different PDU (protocol data units) depending on what layer is involved. It is similar to Russian Nesting Dolls, where when you open the outer layer there is a smaller container within. I.e. the "inner dimentions" of the outer containter must be large enough to contain the "outer dimentions" of any container within. Or another analogy, the shipping box must be bigger than any box it contains.

This is worth a read. https://datatracker.ietf.org/doc/html/d ... section-10 as it discusses issues with jumbo frames (one is that you must verify that all devices in your LAN are capable of dealing with the larger frames. For example, if you have any MT7621 based devices like RB750Gr3 (hEX) or RB760iGS (hEX S), they have hardware limits of 2K frames.

Amm0 · Tue Apr 11, 2023 9:22 pm

This thread has been an interesting read leading me to look into MTU...
It is similar to Russian Nesting Dolls, where when you open the outer layer there is a smaller container within. I.e. the "inner dimentions" of the outer containter must be large enough to contain the "outer dimentions" of any container within. Or another analogy, the shipping box must be bigger than any box it contains.

Well, things generally will just fragment if MTU is wrong...so not quite right. These oversized boxes and "fabergé packets" should get chopped off and glued back together... And depending on what's inside will determine how well that "glue" works.

pe1chl · Tue Apr 11, 2023 9:48 pm

Unfortunately, at some point in time someone decided that fragmentation is bad and that it is better to inform the sender that his packets are too big.
To implement that in the already existing network, it was not possible to send a "probe" message that would be examined at every router in the network and would get changed to set the value of the MTU at that point if it was lower than what was already in the message.
So instead, use was made of two other features of IP that already were part of the standard and thus assumed to be implemented everywhere:
1. there is a bit in the IP header that says "don't fragment", i.e. this particular packet is not to be fragmented
2. there is an ICMP message "size exceeded" that is to be returned to the sender whenever a router cannot fragment a packet (e.g. due to the "don't fragment" bit).

The "size exceeded" message at that time was extended to (optionally) return the actual MTU back to the sender. But that is not part of the original standard, so not all routers might do it. The message may be returned without mention of the actual MTU, one can only guess that that is smaller than the size of the sent packet.

So, to determine the MTU of the entire path (called PMTUD for "Path MTU Discovery") is an iterative process of sending ever smaller packets with "don't fragment" until they make it to the destination. Not very efficient, you don't want to do that every time.
But also, this means:
1. most traffic is now sent with "don't fragment", just to make this working, and the actual fragmentation/reassembly mechanism is mostly out of operation
2. we are completely dependent on the return of the ICMP message back to the sender. any clueless system admin that (e.g. after reading Steve Gibson's misinformation) is dropping all ICMP packets, will seriously foul up the network for anyone not able to transport 1500 byte sized packets end-to-end. And even more so for those who think that locally using jumbo frames (MTU 9000) and then connecting to internet via a 1500 byte MTU interface is going to work out well.

Another trick that works better than PMTUD, but only for TCP, is "MSS clamping". Again this requires implementation on all routers in the path that may present a smaller outgoing MTU than incoming MTU, but in some cases (e.g. having a PPPoE connection with MTU 1492) that could be only your own router. So it can work.
This is based on the fact that for TCP the endpoints actually negotiate the MTU to be used by TCP during the session setup. This is the MSS field of the SYN and ACK SYN packets. It is not actually the MTU, but the maximum segment sent in a TCP session, which when smaller than a single packet will actually limit the size of that packet.
For this, the MSS should be set to a value of MTU - size of IP header - size of TCP header. Typically that will be MTU-40.

Amm0 · Tue Apr 11, 2023 10:10 pm

Another trick that works better than PMTUD, but only for TCP, is "MSS clamping".

Could be wrong here...
But doesn't the PPP "profile" already default to doing TCP MSS adjustment for PPPoE/etc? Maybe that's not working...

But certainly not all "tunnel types" use the "PPP profiles", like EoIP, VXLAN, LTE, likely others....so in those case might want adjust TCP MSS yourself, based on the math.

Buckeye · Tue Apr 11, 2023 10:15 pm

MTU, MSS and PMTUD issues - PacketLife’s Path MTU Discovery blog post, (not so very) short but very good video TCP MSS clamping – what is it and why do we need it? by Ivan Pepelnjak

If you have time and tenacity, read Cisco's Resolve IPv4 Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPsec for a more thorough discussion of the issues.

pe1chl · Tue Apr 11, 2023 11:42 pm

After such a summary of info about "MSS clamping" I always like to add that I actually invented it

Or at least, I thought of it and implemented it in a version of KA9Q NET (a networking package for MS-DOS) in August of 1995.
I have never seen any indication of another manufacturer implementing it before that.

Frederick88 · Wed Apr 12, 2023 2:43 am

Does MSS Clamping cause much overhead on the router - or rather, does it diminish your WAN latency in any way, and negate any potential advantages of using larger MTU within the LAN to begin with?

Buckeye · Wed Apr 12, 2023 5:19 am

Does MSS Clamping cause much overhead on the router - or rather, does it diminish your WAN latency in any way, and negate any potential advantages of using larger MTU within the LAN to begin with?

I assume you meant "increase you WAN latency"?

The primary purpose of mss clamping is to avoid the need for fragmenting the packet once it has been sent. In other words, it "pre-fragments" the data into the size that can make it through all the paths to the destination. That way routers along the way don't have to waste resources fragmenting the packet while it is in transit.

MSS is computed per tcp connection, not per packet.

Frederick88 · Wed Apr 12, 2023 5:31 am

yes sorry, increase latency - diminish was a poor choice of word.
.

MSS is computed per tcp connection, not per packet.

so once tcp connection has been established and MSS calculated/computed, is there still ongoing compute each time it "pre-fragments" the ongoing packets within the established tcp connection?

Amm0 · Wed Apr 12, 2023 5:45 am

By "it 'prefragments"... more like the Mikrotik is a MITM modifying the initiator's request to suggest a lower TCP packet size, so the recipient server will use Mikrotik-modified MSS value.

Side note any TCP MSS mangle rule should also use "tcp-mss=", otherwise it will trigger per SYN packet – no need to modify if it's already lower.

/ip firewall mangle
add action=change-mss chain=postrouting comment="adjust TCP MSS" new-mss=1372 \
    out-interface=lte1 passthrough=yes protocol=tcp tcp-flags=syn tcp-mss=1373-65535

Note: The TCP MSS will be right if WAN L3MTU is 1500, so only comes up if actual MTU of path is lower. And TCP MSS rule does NOT help with UDP packets.

Buckeye · Wed Apr 12, 2023 7:32 am

Consider the case where everything on your LAN has an MTU of 1500, but you have a pppoe WAN connection that takes 8 bytes of overhead for pppoe encapsulation and makes the MTU of the pppoe interface 1492 instead of 1500.

You PC thinks it can receive 1500 byte packets because the MTU of the ethernet is 1500, and so mss will be set to 1460 (the 1500 - 20 bytes for ip header, and another 20 bytes for the TCP header). So it tells the other side it (the other side) is allowed to send 1460 bytes of data in each tcp packet during the tcp 3 way handshake. But your router sees this and knows that it will be going through the pppoe link, which can only allow 1492 bytes, so the router re-writes the 1460 as 1452 (8 bytes less), and that is what the partner at the other side of the tcp connection receives (so even though your PC is ready to receive a 1500 byte packet (with 1460 byte tcp payload), it "thinks" your PC can only receive 1452 bytes of payload, and the other side will use a smaller "boxes" that hold only 1492 bytes (payload plus ip header and tcp header) instead of the standard 1500 byte "boxes" when it is sending the data, so it will always fit into what the pppoe interface can receive. Likewise, if the other side tells your PC that it can receive 1500 (mss 1460), when it is coming through the router, the router also modifies that to change the MSS to 1452, so your PC won't try to send 1500 byte packets through the pppoe interface. Both host in the tcp connection think the restriction was from the other side, they can't tell that it was actually modified by the router.

It is like making sure that only 11 foot high trucks are allowed on the road leading to the 11' 8'' bridge. For a visual see this video Perfect peel at the 11foot8+8 bridge

pe1chl · Wed Apr 12, 2023 10:54 am

Does MSS Clamping cause much overhead on the router - or rather, does it diminish your WAN latency in any way, and negate any potential advantages of using larger MTU within the LAN to begin with?

It depends. When you are using "fasttrack" it increases the overhead, because you will need to turn that OFF at least for TCP SYN packets, which means at least another rule before the fasttrack.
For me, it is not important because I normally do not use "fasttrack" at all. My router setups are normally that complicated that I just don't want to be bothered by the extra configuration considerations I would have to make to keep it working OK.
But when you use fasttrack, completely disabling it of course causes a lot more overhead. Not really in the latency, but surely in the CPU load. And when CPU load nears 100%, there will be a loss in throughput.
If it is important, that depends on the model of router and the speed of your internet connection. For me with a RB4011 and a 180Mbps connection, it does not matter at all.
But when you have a RB2011 and are trying to get as close as possible to what a 500Mbps connection offers, it matters a lot!

That is why MikroTik offers a more efficient but less flexible MSS clamping service as part of some interface configs and as part of "PPP profiles".
So when you are concerned about efficiency, you can use that instead. But it is often reported that it does not work. I don't know, I never use it.

pe1chl · Wed Apr 12, 2023 11:31 am

And TCP MSS rule does NOT help with UDP packets.

That is true, but the issue usually does not manifest itself with UDP packets.
Most protocols that use UDP have request/response packet sizes that fit well into the usual "a-bit-less-than-1500" byte MTU caused by fitting PPPoE or a VPN into 1500 byte MTU encapsulation.
And when they are larger, which could e.g. be the case for DNS with EDNS option on a query with DNSSEC, they are often sent without the "don't fragment" bit, meaning that they will just be fragmented and re-assembled and arrive at the other end OK with slightly more overhead in the router.
As these special cases (a full-size UDP packet being fragmented) usually are quite rare, that does not really matter.

mkx · Wed Apr 12, 2023 2:48 pm

Does MSS Clamping cause much overhead on the router - or rather, does it diminish your WAN latency in any way, and negate any potential advantages of using larger MTU within the LAN to begin with?
It depends. When you are using "fasttrack" it increases the overhead, because you will need to turn that OFF at least for TCP SYN packets, which means at least another rule before the fasttrack.

Is it? Default config has fasttrack rule which includes "connection-state=established,related" ... and if I understand things correct, TCP connection only reaches state of established after the initial 3-way handshake completes ... and SYN flag is present only on initial two packets (one in each direction).

pe1chl · Wed Apr 12, 2023 3:03 pm

Ok, the initial SYN packet for sure will have "new" state, but I'm not so sure that the ACK SYN has that as well, it could be "established" state already (which formally begins only after another outgoing ACK).
But as mentioned, I never bother with "fasttrack" because I always have so many things that may be affected by it, and I rather buy a faster router than messing with that.
(saves me from having to complain that there is no IPv6 fasttrack too, so I can concentrate all my complaints on BGP and BFD

)

rextended · Wed Apr 12, 2023 3:46 pm

all my complaints on BGP and BFD

Are you related to @anav ???
I have Zero Trust for BFD unless 7.23.5.....

anav · Wed Apr 12, 2023 4:39 pm

NO relation, I couldnt give a rats buttocks about BFD, BFG or any B#Atch.......
Zerotrust cloudflare tunnel Options package is speaking my language though!!

But going back to Ammoss comment. Where its stated that any mangle rule should include "tcp-mss="
/ip firewall mangle
add action=change-mss chain=postrouting comment="adjust TCP MSS" new-mss=1372 \
out-interface=lte1 passthrough=yes protocol=tcp tcp-flags=syn tcp-mss=1373-65535

For Wirguard when there are mTU issues this is recommended......
/ip firewall mangle
add action=change-mss chain=forward comment="Clamp MSS to PMTU for Outgoing packets" new-mss=clamp-to-pmtu out-interface=wireguard1 passthrough=yes protocol=tcp tcp-flags=syn

Just to confirm thats still good.......... as there is no tcp-mss= Entry ??

Frederick88 · Wed Apr 12, 2023 4:43 pm

so long story short - don't fuck with larger MTU sizes within a network that at some point might access the internet.

Something like a network between |Server| and |Shared Storage/MAS|, could easily have jumbo frames on, providing |Server| and |Shared Storage| use separate dedicated NICS with MTU1500 for management and internet access (EG: cloud base portal access, updates, etc..)...

Whereas within home flat environment of laptop to switch to laptop to NAS, setting higher MTU may benefit while addressing everything in LAN, but once it goes through router to WAN, data will either be fragmented, or your router's gonna need MSS Clamping rules for negotiating accepted MSS size on both ends of sender and receiver...

Is this summary fair, or is this sleeping pill kicking in harder than I realised... though I'm typing super quick, which is interesting considering how drowsy I am...

rextended · Wed Apr 12, 2023 5:01 pm

@anav
If wireguard is used, and your ISP is like me than provide one L3 MTU of 1500 instead the shitty 1480 or 1492,
IPv4 wireguard add his own overhead of 60 bytes, leaving only one unfragmented MTU of 1440
IPv6 wireguard add his own overhead of 80 bytes, leaving only one unfragmented MTU of 1420

So, for avoid fragmentation, any packet must be 1440/1420 or less, respectively MSS must be (1500 - 40) - 60 = 1400 or for IPv6 (1500-60) - 80 = 1360

If you have PPPoE and shitty ISP, if your MTU is 1492 or less, you must subtract also 8 bytes overhead, is why you write 1372?

If instead you have VLAN or DHCP client, use 1400 for IPv4 and 1360 for IPv6.

But the connection is made from two ends, ignoring bottlenecks in the middle, the minimum MTU/MSS between the two connections should be used in the calculation.

I hope I didn't make a mistake or forgot something.

Amm0 · Wed Apr 12, 2023 5:13 pm

so long story short - don't fuck with larger MTU sizes within a network that at some point might access the internet.

That right IMO. Just add if your WAN/internet connection has a lower MTU than 1500... you may want to do special treatment of MTU.

[...]
But going back to Ammoss comment. Where its stated that any mangle rule should include "tcp-mss="
[...]
For Wirguard when there are mTU issues this is recommended......
/ip firewall mangle
add action=change-mss chain=forward comment="Clamp MSS to PMTU for Outgoing packets" new-mss=clamp-to-pmtu out-interface=wireguard1 passthrough=yes protocol=tcp tcp-flags=syn

Just to confirm thats still good.......... as there is no tcp-mss= Entry ??

The initial MSS value is created by a device on the network, not the router. Router will only change a MSS. So it's possible, perhaps unlikely uses a lower MSS initially. So without the filter for "big MSS values", you increasing may be increasing MSS beyond what the client requested.

The "new-mss=clamp-to-mss" option uses the MTU of the interface to set its value. But it's not "smart", it just uses the associated interface's MTU to set MSS value automatically, based on the interface MTU. As noted, this rule only triggers upon a new TCP connections, not all packets. The tcp-mss=13xx-65xxx filter reduces when it triggers even further to adjusting it only if it's "wrong".

So only suggestion to avoid interfering in that case. And also re-writing packets that don't need to be re-written seem cleaner. But don't think it's critical to include a tcp-mss filter string.

That is why MikroTik offers a more efficient but less flexible MSS clamping service as part of some interface configs and as part of "PPP profiles".
So when you are concerned about efficiency, you can use that instead. But it is often reported that it does not work. I don't know, I never use it.

It used to work in V6. But I recall PPP profile's MSS adjustment also using "new-mss=clamp-to-mtu" as the action... So the clamp-to-mtu depend on if the [L3] MTU was set correctly. And if MTU wasn't adjusted on the interface explicitly lower, then PPP profile setting likely do nothing given an impression of not working.

anav · Wed Apr 12, 2023 5:17 pm

yeah nothing is clear, it sems your saying that the default settings of 1420 by MT for wg is useless and should be set to 1500??

rextended · Wed Apr 12, 2023 5:24 pm

Do not mix MTU and MSS:
1420 is the default L3 MTU, ready for Wireguard connected by IPv6 (regardless what is transport internally) = MSS 1380.
If you do not use IPv6 for connect Wireguard tunnel, you can set it to 1440 and increase the MSS to 1400.
(and on both cases are supposed 1500 as L3 MTU on WAN side)

If the WAN connection is provided by PPPoE, if you do not have one L3 MTU of 1500 from your ISP,
must be subtracted 8 from all values,
so if you get the connection by PPPoE that have L3 MTU/MRU of 1492, and you use IPv4 only Wireguard link,
you have to set 1432 on Wireguard L3 MTU and 1392 on clamp tcp-mss

But count what is lower suported MTU/MSS between all peers.

pe1chl · Wed Apr 12, 2023 5:37 pm

so long story short - don't fuck with larger MTU sizes within a network that at some point might access the internet.

Something like a network between |Server| and |Shared Storage/MAS|, could easily have jumbo frames on, providing |Server| and |Shared Storage| use separate dedicated NICS with MTU1500 for management and internet access (EG: cloud base portal access, updates, etc..)...

I wrote that already in reply #20, but now in reply #104+ the discussion is still ongoing...

rextended · Wed Apr 12, 2023 5:51 pm

Actually I think you have already condensed everything in post #2, it couldn't have been written better...

Industry standard MTU is 1500. When you want something else, you will have to configure it.

just one little addon from post #3:

(Without considering that, if you don't configure the internal network correctly, you will create nothing but outgoing problems...)

anav · Wed Apr 12, 2023 5:52 pm

That is because the whole thread is a cluster duck of competing ideas without any context. I havent learned a thing, but then again, unless something affects my config, I dont care and so far, MTU and MSS and the like have been good to me. The only time I needed to mess with anything was to move WG MTU at both client and peer MT router to 1500 so that a remote user could use my internet to reach and work with different sites that needed logins. Make no sense to me, but it works. Otherwise leaving it at the default 1420 works great.

denisun · Sun Jul 23, 2023 5:59 pm

Do not mix MTU and MSS:
1420 is the default L3 MTU, ready for Wireguard connected by IPv6 (regardless what is transport internally) = MSS 1380.
If you do not use IPv6 for connect Wireguard tunnel, you can set it to 1440 and increase the MSS to 1400.
(and on both cases are supposed 1500 as L3 MTU on WAN side)

If the WAN connection is provided by PPPoE, if you do not have one L3 MTU of 1500 from your ISP,
must be subtracted 8 from all values,
so if you get the connection by PPPoE that have L3 MTU/MRU of 1492, and you use IPv4 only Wireguard link,
you have to set 1432 on Wireguard L3 MTU and 1392 on clamp tcp-mss

But count what is lower suported MTU/MSS between all peers.

My router have pppoe connection to internet with mtu 1492.
I have a wireguard server in router with ipv4 only access.

According to the above:
1. Should the mtu in wireguard server be
1492 - 60 = 1432
or
1492 - 60 - 8 = 1424 ?

2. Do i need to use clamping for outgoing wireguard traffic in router?
With range mss value (1432 or 1424 - 40) or clamp-to-ptmtu choise?
I have enable in pppoe profile the change mss option.

3. I have fasttrack enabled.
Should the change mss work in this case?

Amm0 · Sun Jul 23, 2023 7:50 pm

For a IPv4 path, WG MTU is 1432. If IPv6, WG MTU is 1412 — for PPPoE. If WAN is 1500, then WG MTU be 1420 for IPv6 and 1440 for IPv4.

Another way to look at is WG is 40 bytes, including UDP header. If the path between the peers is IPv4, that add 20 bytes, PPPoE adds another 8 bytes:
https://baturin.org/tools/encapcalc/?pr ... ,WireGuard

Most cases PMTUD works (e.g. unless some decides to block ping someplace), so TCP MSS mangle is many times superfluous. And the worse case without a MSS adjustment, is some fragmentation; but a MSS mangle rule always add some overhead and potential side-effects (or at least more stuff to consider)... I think you'd have to try it both ways to see its effects (e.g. with MSS adjustment mangle and without one).

Also you should consider the MTU of the remote peers that connect. The actual WG MTU over a path is lowest of the peers. Say you have only one remote peer that always connect via LTE, that have an even lower MTU than your PPPoE, and it might be better to match lower MTU of the path as that avoid the need for PMTUD to work.

denisun · Sun Jul 23, 2023 8:46 pm

For a IPv4 path, WG MTU is 1432. If IPv6, WG MTU is 1412 — for PPPoE. If WAN is 1500, then WG MTU be 1420 for IPv6 and 1440 for IPv4.

Another way to look at is WG is 40 bytes, including UDP header. If the path between the peers is IPv4, that add 20 bytes, PPPoE adds another 8 bytes:
https://baturin.org/tools/encapcalc/?pr ... ,WireGuard

Most cases PMTUD works (e.g. unless some decides to block ping someplace), so TCP MSS mangle is many times superfluous. And the worse case without a MSS adjustment, is some fragmentation; but a MSS mangle rule always add some overhead and potential side-effects (or at least more stuff to consider)... I think you'd have to try it both ways to see its effects (e.g. with MSS adjustment mangle and without one).

Also you should consider the MTU of the remote peers that connect. The actual WG MTU over a path is lowest of the peers. Say you have only one remote peer that always connect via LTE, that have an even lower MTU than your PPPoE, and it might be better to match lower MTU of the path as that avoid the need for PMTUD to work.

Thank you from your answer.
My router have ipv4 and ipv6 access.
I have two LTE clients.
One have ipv4 only access and other have both ipv4 and ipv6.
The path that you're talking about, concern the way that client connect to router or the access that wireguard give to client.
i.e. the wireguard give to clients only ipv4 access
In this case i should set the mtu in server
wg mtu: 1492 - 60 - 8 = 1424
or
wg mtu: 1492 - 80 - 8 = 1404

I test it and i have problems with some site when i set mtu < 1420.
I used clamping for outgoing only traffic.
The counters increase (thats work?) but i have the same problem.

I tried to set the same mtu in the client side but nothing change.