MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

roadracer96 · Mon Apr 18, 2022 7:07 pm

I did open a ticket for it but thought I would post here for comment:

MLAG synchronizes the LACP system-id of the secondary nodes ports with the LACP system-id of the primary node bonding interface that has the same MLAG id assigned to it.

Problem 1. If you reboot the primary node, the secondary node reverts its LACP system-IDs back to the chassis default. This causes an unnecessary forwarding flap/outage of 3-4 seconds (tested using 1sec LACP timers at least) when the primary node reboots. The same flap occurs when the primary node comes back, restoring MLAG peering and the secondary resets its LACP system-IDs to match the primary peer.

Problem 2. Somewhat related as it helps get you to a solution. You run different LACP system-IDs on every bonding port by default. This is technically wrong. The LACP system-ID should be the same for the entire chassis (And as such, the same for all MLAG peers). It bears no relevance to the underlying MAC address of the physical port. IT is used only for signalling. Only the LACP port ID should differ between different bonds.

Problem 3. Similar to problem 1, when the primary node reboots, the RSTP bridge ID reverts back to default. In an MCLAG environment with RSTP, both peers should always carry the same bridge ID at all times because they are by the very definition of MCLAG, the same physical port/device.

Solution: When the MLAG pair comes up for the first time and one device becomes primary, the MAC should be synchronized to the secondary peer and become immutable. This way, if the primary node reboots for an upgrade, the network never experiences any major re-convergence ( From both RSTP and LACP point of view). The port-ID of the primary device should be the MLAG ID. The port-ID of the secondary device should be the MLAG ID Plus some fixed value.. say.. Add 128 for each port on the local switch, add 1024 for being a secondary device. This insures that all LACP port IDs are unique across all ports in the bundle (Even across different switches). The LACP system ID would be the same across all devices.. All of these will be immutable. The LACP port KEY should generally just be the MLAG ID and be the same across both devices.

Enhanced solution (More preferred for those keen on MLAG): Allow a static LACP system ID to be set and a static RSTP bridge ID to be set. The RSTP bridge ID should not affect the MAC address of the actual bridge itself, each devices MAC should be unique for L3 purposes. Also let the user specify which device is node0 and which device is node1.

In both solutions, only MLAG bonds should be affected. the bond used for the ICL port and other non-MLAG bonds should follow traditional logic.

IT should also be noted that I observed inconsistent behavior with the RSTP ID and bridge mac of the secondary when the primary is rebooted if you dont have a static admin-mac set on the bridge.

smyers119 · Tue Apr 19, 2022 12:58 am

Problem 2. Somewhat related as it helps get you to a solution. You run different LACP system-IDs on every bonding port by default. This is technically wrong. The LACP system-ID should be the same for the entire chassis (And as such, the same for all MLAG peers). It bears no relevance to the underlying MAC address of the physical port. IT is used only for signalling. Only the LACP port ID should differ between different bonds.

What is your source of this "technically wrong" Because from what I see in the technical standard IEEE 802.1AX-2008, there should be a locally unique identifier

roadracer96 · Tue Apr 19, 2022 5:07 am

What is your source of this "technically wrong" Because from what I see in the technical standard IEEE 802.1AX-2008, there should be a locally unique identifier

That is correct. Locally unique means the SYSTEM ID is unique to the local L2 device. In the case of MLAG, the System is really 2 devices.

The only 2 devices I have to compare it to at the moment are Juniper, Extreme. Both of which use the chassis mac as the LACP system-id. Im 99 percent sure Cisco does the same thing.

It doesnt really hurt anything making it different, but in the end, why would you want to make life harder for yourself?

Tue Apr 19, 2022 5:01 pm

Thanks for sharing these details!

Your observations are correct. When a primary node is rebooted, the secondary node will revert its LACP system-ID and RSTP ID to the local values. This can cause some packet loss during LACP renegotiation, especially when the LACP rate is set to 30 seconds.

However, this design helps with issues when both MLAG nodes are up and running, but cannot communicate with each other (e.g. peer port is down). If both MLAG nodes would still use the same LACP system-ID and RSTP ID, it can cause a split-brain situation. The regular LACP client would think that everything is working correctly and it can send packets to any of the MLAG node due to transmit hash policy. But it can cause a packet loss if the destination MAC is located on the other MLAG node and the peer port is still unavailable.

Do you have any comments on how other vendors are dealing with this problem? We might consider improving our implementation or adding new configuration settings, but would like to hear other opinions as well.

smyers119 · Tue Apr 19, 2022 5:48 pm

What is your source of this "technically wrong" Because from what I see in the technical standard IEEE 802.1AX-2008, there should be a locally unique identifier
That is correct. Locally unique means the SYSTEM ID is unique to the local L2 device. In the case of MLAG, the System is really 2 devices.

The only 2 devices I have to compare it to at the moment are Juniper, Extreme. Both of which use the chassis mac as the LACP system-id. Im 99 percent sure Cisco does the same thing.

It doesnt really hurt anything making it different, but in the end, why would you want to make life harder for yourself?

No the system id is globally unique, then each LAG aggregation group also has a locally unique identifier. I guess I wasn't clear about that. Please refer to the technical documents for more specifics. The system id is created with a combination of the priority and mac address.

roadracer96 · Tue Apr 19, 2022 7:32 pm

Thanks for sharing these details!

Your observations are correct. When a primary node is rebooted, the secondary node will revert its LACP system-ID and RSTP ID to the local values. This can cause some packet loss during LACP renegotiation, especially when the LACP rate is set to 30 seconds.

However, this design helps with issues when both MLAG nodes are up and running, but cannot communicate with each other (e.g. peer port is down). If both MLAG nodes would still use the same LACP system-ID and RSTP ID, it can cause a split-brain situation. The regular LACP client would think that everything is working correctly and it can send packets to any of the MLAG node due to transmit hash policy. But it can cause a packet loss if the destination MAC is located on the other MLAG node and the peer port is still unavailable.

Do you have any comments on how other vendors are dealing with this problem? We might consider improving our implementation or adding new configuration settings, but would like to hear other opinions as well.

How exactly would they be reachable if the upstream device is running lacp? Upstream won't bring up 2 paths in an aggregate group with different system IDs, so the point is moot.

Mlag peers should always be able to communicate with each other. The ICL port itself should always be a lag. Juniper and extreme both allow you to have a backup heartbeat path via a gig copper and if the icl goes down the secondary node will bow out and send out of sync to the lacp peers.

In the unlikely case that the icl is down and the backup heartbeat link is down, and both switches are rebooted i wouldn't know what other vendors would do. Having dealt with literally hundreds upon hundreds of mlag devices I have honestly never had a failure like that.

I do remember testing juniper and upon reboot there is a timer delay before it will attempt to bring up icl. The icl is brought up and if after another timer expires a peer hasn't been established then the device will become the active primary unit and I *think* in that case it doesn't use the user defined lacp system id and reverts back to the chassis mac until the icl comes up.

In the end. A split brain situation from a single device failing is easy. Both were up so the cluster was armed. If device 1 stays up when device 2 goes down, it should be authoritative in using the defined system id. In a proper mlag setup, the link between the 2 devices should be highly redundant and have a backup. So for it to fail, 2 dataplane icl links would need to be down AND optionally a 3rd backup non data-plane link.

The case where both devices reboot and only 1 comes back up would be the oddball case. I'd argue that using a timer to wait for the cluster to form before assuming the peer is unreachable would be best. Like 120-180 seconds (maybe take the longest possible reboot time of a device and triple it. Like the case of a software upgrade which might take 40-60 seconds).

roadracer96 · Tue Apr 19, 2022 7:35 pm

No the system id is globally unique, then each LAG aggregation group also has a locally unique identifier. I guess I wasn't clear about that. Please refer to the technical documents for more specifics. The system id is created with a combination of the priority and mac address.

Yes. One system id that is the same for all lags. The lag key is unique across all defined lags in a chassis (or pair in the case of mlag). The lag port id should be unique across all ports inside of a particular lag.

roadracer96 · Fri Apr 22, 2022 5:45 am

Thanks for sharing these details!

Your observations are correct. When a primary node is rebooted, the secondary node will revert its LACP system-ID and RSTP ID to the local values. This can cause some packet loss during LACP renegotiation, especially when the LACP rate is set to 30 seconds.

However, this design helps with issues when both MLAG nodes are up and running, but cannot communicate with each other (e.g. peer port is down). If both MLAG nodes would still use the same LACP system-ID and RSTP ID, it can cause a split-brain situation. The regular LACP client would think that everything is working correctly and it can send packets to any of the MLAG node due to transmit hash policy. But it can cause a packet loss if the destination MAC is located on the other MLAG node and the peer port is still unavailable.

Do you have any comments on how other vendors are dealing with this problem? We might consider improving our implementation or adding new configuration settings, but would like to hear other opinions as well.

Any reply? Nobody has responded to my case yet either.

Hyperlight · Mon Nov 28, 2022 6:27 am

I ran into this issue on my recent tests of this feature. Its a deal breaker for me, as the outage time for LACP renegotiation + Spanning tree convergence is too long to be acceptable when the MAC address changes and then changes back again.

I would say the most common instance of needing to take a switch down is for ROS software upgrades. Given that the switch is only going to be down for a few mins I would suggest Mikrotik implement what Cisco has done and introduce a MAC-Address persistent timer. See below for documentation. This would prevent the flapping and re-covergence during routine maintenance but provide a failback in the unlikely case of the split-brain (at least when the timer expires).

https://www.cisco.com/c/en/us/td/docs/s ... 143BE510BC

*Note that Cisco has a MAC-persistence time of 0 which is forever that can be enabled, with the appropriate warnings.

roadracer96 · Sat Feb 25, 2023 5:17 am

Ive had these switches for over a year now.. I spent time with support explaining how it should work and they were receptive.. a year later.. nothing.. I have paper weights.. The latest version of code drops 75% of traffic in MCLAG... Ridiculous product. Something every other vendor has had figured out for 10+ years..

emunt6 · Fri Mar 03, 2023 2:00 am

Hi!
As a workaround, can you use the "static-LAG = bonding+balance-rr" ?
/interface bonding add name=bond0 slaves=ether1,ether2 mode=balance-rr

mischa01101 · Wed Dec 06, 2023 8:19 am

Ive had these switches for over a year now.. I spent time with support explaining how it should work and they were receptive.. a year later.. nothing.. I have paper weights.. The latest version of code drops 75% of traffic in MCLAG... Ridiculous product. Something every other vendor has had figured out for 10+ years..

same here - we've had so many issues with MLAG from Mikrotik so far, that it caused much more trouble than it should solve giving high-availability. At this point we have a ticket open when specific packets generate a loop on the peerlink melting down the network with 80GBit/s. After more then 20days no answer yet from support.

This feature should have never been released with the poor quality of their implementation. Really a shame and we will move away from Mikrotik for all mlag setups in the future, that is for sure.

tafkamax · Thu Dec 07, 2023 10:54 am

I am looking into using CRS518 switches in a MLAG pair for Ceph backend traffic.

It is really unfortunate that MLAG is implemented poorly, hope they will fix this issue.

spippan · Tue Dec 12, 2023 1:23 am

Thanks for sharing these details!
...

any news on that?

unfortunately advanced switching isn't in mikrotik's strong pocket i reckon
access layer - yeah, might work
aggregation layer - meh might maybe
core layer - nah, no way at all with L2 bugs reported/read in the forum and the stagnated (like BGP) improvement
a bummer. especially considering that mikrotik really packs some powerful switch chips/switch ASICs and the software messes it up

thefriendlyguy · Tue Jan 09, 2024 12:50 pm

Hi there!
I am experiencing the same bug with 2x CRS317-1G-16S+.
Upon reset of the primary things go "south".
Both are on 7.11.2... i saw there are more recent releases but according to the changelog there was no change regarding mlag.
In testing i saw "*) bridge - added MLAG support for MSTP bridges;" but i am not too keen on using testing software.
I would also love to see some improvements when it comes to MLAG.

Kind regards

robertpenz · Fri Feb 16, 2024 1:41 pm

Do you have any comments on how other vendors are dealing with this problem? We might consider improving our implementation or adding new configuration settings, but would like to hear other opinions as well.

I'm running MLAG setups for over 15 years and all vendors I used so far do following.

ISC must always be redundant - no single ISC Link .. normally thats done with LACP - all other setups are not supported by the vendor --> that solves the problem you described why you revert to local lacp mac.
You need to specify a lacp-mac (the same on both), if you don't do that you run in the above described problems also with other vendors - specially if you need to replace the unit which was used for the lacp-mac. You won't have a problem if one switch fails, but if you boot up with the replacment unit. so always set it.
The switches have a delay (30sec on the Extreme Switch I just checked) after boot from the documentation: "There are cases where MLAG ports comes up quicker than ISC ports after a switch reboot causing traffic loss during this time gap. This command allows you to configure a time delay for MLAG ports providing enough time for ISC ports/neighborship of other Layer 3 protocols to come up." Default is 30sec
There is a less than 1sec loop if a port comes up, normally no problem, but you can set a link-up-isolation to guard against that - never hat issues in my network's so far. From the documentation: "Under certain circumstances, a temporary (less than a second) loop condition exists when an MLAG port becomes operational, but before the remote MLAG peer installs the ISC blocking filter. MLAG linkup isolation addresses this condition by preventing any flood traffic (broadcast, unknown, unicast, etc.) received on a just operational MLAG port from being forwarded to ISC ports until the remote MLAG peer installs the ISC blocking filter." Default: off

Running with these settings with various vendors over the years .. I never ever had a problem with a split brain (min 2 links ISC) or that if the "master" switch goes down a traffic interruption. What I had about 10 years ago was a problem that if the failed switch got replaced, that after booting it started to fast to forward and accept traffic. We had a 6sec seconds downtime in the core - which is a problem if the storage and everything runs over it! This was then fixed by the vendor with the delay described above, and never had that problem since then - and I'm running currently about 40 non Mikrotik MLAG clusters.

With your implementation, you will have massive problems with Systems like a Netapp storage .. if the LACP MAC changes there .. the link goes done for min 30sec, I've seen up to 2min before it forwards traffic again over that link - that kills all vm that wait for a time if the NFS storage is not available but 2 min is too long.

An example MLAG config from an EXOS 100Gbit Switch in production.

configure mlag ports convergence-control conserve-access-lists 
configure mlag ports reload-delay 30
configure mlag ports reload-interval none
enable mlag port reload-delay
configure mlag ports link-up-isolation off
create mlag peer "otherswitchname" 
configure mlag peer "otherswitchname" ipaddress 192.168.30.102 vr VR-Default 
configure mlag peer "otherswitchname" alternate ipaddress none
configure mlag peer "otherswitchname" interval 1000 
configure mlag peer "otherswitchname" lacp-mac xx:xx:xx:xx:xx:xx
configure mlag peer "otherswitchname" authentication none
enable mlag port 9:3 peer "otherswitchname" id 9003 
enable mlag port 9:4 peer "otherswitchname" id 9004 
enable mlag port 10:1 peer "otherswitchname" id 10001 
enable mlag port 10:2 peer "otherswitchname" id 10002 
....

MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots

Re: MLAG Issue - MLAG functionality flaps LACP system-id of secondary when primary reboots