Hi,

I have 2xCRS326-24S+2Q+ running ROSv7.2.1. I have MLAG running over the 2x40gig ports (in a LACP) and then I have a LACP to a server over port 23 on both switches and another LACP to a HP switch over port 24 (this is only a single link atm since I have limited 10gig in the lab). Connected to the HP switch is an ESX server with dual 10gig.

As it stands, the LACP is working. If I disable the port on one switch traffic switches over to the ICCP link and out the other port - brilliant. My question is - how do I configure the switches to support transmit load balancing over the ICCP. I have tried all the policies and none seem to make it work

Thanks

AFAIK, the ICCP makes the Peer-Link insanely expensive.

Thus no Load-Balancing will occur. Any Frame arriving at the Switch will directly leave it via the MLAG-Member.
It is extremely undesirable to re-route it through the Peer-Link.

AFAIK, the ICCP makes the Peer-Link insanely expensive.

Thus no Load-Balancing will occur. Any Frame arriving at the Switch will directly leave it via the MLAG-Member.
It is extremely undesirable to re-route it through the Peer-Link.

I suspected something like this, except that some other vendors that support MLAG does this (Juniper and HP, not sure about the rest) - would be nice IMHO

If I understand the question correctly you would like traffic coming from an orphan port (port connected to only one chassis) to be load balanced to both a local member of an MLAG and a remote member of the same MLAG. I have never seen any vendor support this. Cisco’s VSS (iOS/iOS-XR) and vPC (NXOS) rig the LAG/port-channel selection to prefer members on the same chassis as the ingress traffic under all circumstances and will only use the inter-chassis link as a last resort. I would be surprised if the other major vendors supported it either.

This is by design. MLAG designs are intended to be symmetrical and should have no orphan ports (MLAGs with members only on one chassis) except in a degraded state. I have seen and had to maintain some unfortunate MLAG designs where the inter-chassis link needs to take a lot of bandwidth to accommodate permanently orphaned ports. Without exception, they have all ended in suffering and lamentation.

If this limitation is only a result of the parts you have for testing in the lab, that’s one thing, but I cannot recommend enough that anything that needs to connect to an MLAG in your final design should have at least one link to each chassis under normal operation.

I suspected something like this, except that some other vendors that support MLAG does this (Juniper and HP, not sure about the rest) - would be nice IMHO

Interesting … do you have a reference to the Juniper docs for the knob that enables this? I haven’t had to configure it on Juniper and am curious what they actually let you do.

If I understand the question correctly you would like traffic coming from an orphan port (port connected to only one chassis) to be load balanced to both a local member of an MLAG and a remote member of the same MLAG. I have never seen any vendor support this. Cisco’s VSS (iOS/iOS-XR) and vPC (NXOS) rig the LAG/port-channel selection to prefer members on the same chassis as the ingress traffic under all circumstances and will only use the inter-chassis link as a last resort. I would be surprised if the other major vendors supported it either.

This is by design. MLAG designs are intended to be symmetrical and should have no orphan ports (MLAGs with members only on one chassis) except in a degraded state. I have seen and had to maintain some unfortunate MLAG designs where the inter-chassis link needs to take a lot of bandwidth to accommodate permanently orphaned ports. Without exception, they have all ended in suffering and lamentation.

If this limitation is only a result of the parts you have for testing in the lab, that’s one thing, but I cannot recommend enough that anything that needs to connect to an MLAG in your final design should have at least one link to each chassis under normal operation.

The orphaned LACP is "lacp24" the one I want load balanced is "lacp23" - they are 2 different LACPs. Putting the question differently - if I have traffic come in on a single port, how do I get it to load balance across a MLAG ?

I suspected something like this, except that some other vendors that support MLAG does this (Juniper and HP, not sure about the rest) - would be nice IMHO

Interesting … do you have a reference to the Juniper docs for the knob that enables this? I haven’t had to configure it on Juniper and am curious what they actually let you do.

I'll see if I can dig up some old configs...this is going back like 5-6 years - backups hey ?

If this limitation is only a result of the parts you have for testing in the lab, that’s one thing, but I cannot recommend enough that anything that needs to connect to an MLAG in your final design should have at least one link to each chassis under normal operation.

Kinda, in production these will be uplinked properly (to a Cisco stack )

The orphaned LACP is "lacp24" the one I want load balanced is "lacp23" - they are 2 different LACPs. Putting the question differently - if I have traffic come in on a single port, how do I get it to load balance across a MLAG ?

Pardon the hasty tablet sketch … this type of thing?

The orphaned LACP is "lacp24" the one I want load balanced is "lacp23" - they are 2 different LACPs. Putting the question differently - if I have traffic come in on a single port, how do I get it to load balance across a MLAG ?

Pardon the hasty tablet sketch … this type of thing?


F9013D19-729A-4907-9094-7EC6D95AC844.jpeg

Yeah - that "will work" - host2 is an HP switch that has an ESX connected to it - the traffic is coming from 2 different subnet's, so l3+4 should load balance it pretty well. I will also mention that if I disable the port on chassis 1 the traffic is load balanced across the ICCP LACP

Gotcha. You get all of your LACP load-sharing properties back by just adding another link to the currently orphaned MLAG 24. It’s just a LAG from the perspective of “Host 2” (which happens to be an HP switch which then has yet another standard LAG to your server).

Is there a reason you can’t do that for the final production build?

Gotcha. You get all of your LACP load-sharing properties back by just adding another link to the currently orphaned MLAG 24. It’s just a LAG from the perspective of “Host 2” (which happens to be an HP switch which then has yet another standard LAG to your server).

Is there a reason you can’t do that for the final production build?

I don't understand how LACP24's state effects LACP23 ?

The final prod build will have LACP24 is a 2 port active state

I don't understand how LACP24's state effects LACP23 ?

You’re correct in that there is no direct relationship between the two on the Mikrotiks. In fact, unless you have more than one MLAG member on a given chassis, it doesn’t even apply any LACP load-sharing algorithm at all. This is paradoxically the preferred behavior so that your load-sharing works well end-to-end.

If the goal is to get some utilization out of both links to both Host 1 and Host 2, you actually want the Mikrotik boxes to not do any load-sharing of their own. Host 1 (whether they’re a host or a switch or whatever is applying the load-sharing logic) will pick a link to send on and then the Mikrotik will simply send it down the local link to Host 2. The reverse is true when Host 2 returns traffic: it picks a link and the Mikrotik will send it down the local link back to Host 1.

The only purpose of the interchassis link is to provide *some* connectivity in the degraded state where one or more of these links goes away.

The big picture is that a) you get load sharing across all four north-south links in the diagram without the Mikrotiks ever having to think about it, b) inter-chassis bandwidth is not a concern, and c) the end-to-end load-sharing result is basically identical to if you had Host 1 and Host 2 directly connected with two cables in a traditional LACP.

The final prod build will have LACP24 is a 2 port active state

Perfect, that will give you what you want

I don't understand how LACP24's state effects LACP23 ?

You’re correct in that there is no direct relationship between the two on the Mikrotiks. In fact, unless you have more than one MLAG member on a given chassis, it doesn’t even apply any LACP load-sharing algorithm at all. This is paradoxically the preferred behavior so that your load-sharing works well end-to-end.

If the goal is to get some utilization out of both links to both Host 1 and Host 2, you actually want the Mikrotik boxes to not do any load-sharing of their own. Host 1 (whether they’re a host or a switch or whatever is applying the load-sharing logic) will pick a link to send on and then the Mikrotik will simply send it down the local link to Host 2. The reverse is true when Host 2 returns traffic: it picks a link and the Mikrotik will send it down the local link back to Host 1.

The only purpose of the interchassis link is to provide *some* connectivity in the degraded state where one or more of these links goes away.

The big picture is that a) you get load sharing across all four north-south links in the diagram without the Mikrotiks ever having to think about it, b) inter-chassis bandwidth is not a concern, and c) the end-to-end load-sharing result is basically identical to if you had Host 1 and Host 2 directly connected with two cables in a traditional LACP.

The final prod build will have LACP24 is a 2 port active state

Perfect, that will give you what you want

Cool - bit of a bummer but I guess that is what "proper" stacking is for - If you have 2 hosts connected to 2 x Mikrotik "MLAG LACP" and they are passing 5gig of traffic, then a single 10gig connected host tries to pass more than 5gig of traffic you will run into port utilisation .

In our setup we will connect single port hosts to the Ciscos and then LACP them into the Mikrotik's.

Thanks for the time !

Cool - bit of a bummer but I guess that is what "proper" stacking is for

Yeah, MLAG and “stacking” really do have completely separate design goals. Because they both share some of the same capabilities, I understand and regularly see a lot of confusion.

The real goal of a stacking configuration is to simplify configuration at the edge. In order for a bunch of small boxes to act like a big box, you need enough bandwidth between the small boxes to pull off the lie without sacrificing performance. This is where Cisco’s big high-bandwidth bus connectors on the back come in. Everyone else that reuses other transmission media for stacking just makes the very reasonable gamble that you won’t need as much inter-chassis bandwidth out at the edge. For this reason, you don’t tend to see stacking technologies deployed in datacenters unless the budget is tight and the requirements are light.

The goal of an MLAG configuration is to provide layer-2/top-of-rack high availability for servers without spanning tree reconvergence. It is simpler and has more restrictions precisely because it permits more scalability. I checked what model you mentioned you were using (CRS326-24S+2Q) and those are a great example. You have 240Gbit/sec of bandwidth (not including the 80Gbit/sec from the 40GbE interfaces) per switch. A properly-designed, symmetric MLAG will permit you to connect 24 dual-connected 10GbE hosts (or downstream switches) that in the ideal case can collectively push a hypothetical 480Gbit/sec to each other without putting a single packet on the 2x40GbE interconnect. Try finding hardware with 480Gbit/sec of stacking interconnect [1] to pull that off.

In our setup we will connect single port hosts to the Ciscos and then LACP them into the Mikrotik's.

Perfect, that should work well.

Thanks for the time !

No problem! I’m drawn like a moth to the flame by MLAG questions because I have worked on far too many networks that have pushed Cisco’s implementations of it to beyond their mortal limits and paid the price for it.


[1] Cisco’s brand new flagship campus LAN edge switches (Catalyst 9300) support 480Gbit/sec of stack interconnect [2] but they’re the current cutting edge and they cost more than Mikrotik’s flagship router … each.
[2] https://www.cisco.com/c/en/us/products/ ... 41468.html