Community discussions

MikroTik App
  4. RouterOS
  5. General

MLAG: when primary is shutdown LACP stops to work

Post Reply
 
benoitc
Member Candidate
Member Candidate
Topic Author
Posts: 170
Joined: Wed Jul 08, 2020 12:33 am

MLAG: when primary is shutdown LACP stops to work

Sat Mar 26, 2022 5:16 pm

I am testing the MLAG feature and I have an issue when primary switch is down. When I launch a reboot, the MLAG is put on pause on the second switchj annd all slave sconnectiosn stops to work.
Capture d’écran 2022-03-26 à 15.54.07.png


For this exampel I have put upstream routers and one freebsd machine in LACP. I have 2 VLAN setup for these machines: 102 & the default 1. These vlans have been tagged for the control port:

What do I miss ?

Configuration on switch 2 (primary)
Code: Select all
[user@SW-2] /interface/bridge/mlag> print
     bridge: bridge
  peer-port: sfp-sfpplus3
[user@SW-2] /interface/bridge/port> print
Flags: H - HW-OFFLOAD
Columns: INTERFACE, BRIDGE, HW, PVID, PRIORITY, PATH-COST, INTERNAL-PATH-COST, HOR
IZON
#   INTERFACE     BRIDGE  HW   PVID  PRIORITY  PATH-COST  IN  HORIZON
0 H sfp-sfpplus3  bridge  yes    99  0x80             10  10  none   
1 H client-bond1  bridge  yes     1  0x80             10  10  none   
2 H upstream-r1   bridge  yes     1  0x80             10  10  none   
3 H upstream-r2   bridge  yes     1  0x80             10  10  none   
[user@SW-2] /interface/bridge/vlan> print
Flags: D - DYNAMIC
Columns: BRIDGE, VLAN-IDS, CURRENT-TAGGED, CURRENT-UNTAGGED
#   BRIDGE  VLAN-IDS  CURRENT-TAGGED  CURRENT-UNTAGGED
0   bridge         1  sfp-sfpplus3    bridge          
                                      client-bond1    
                                      upstream-r2     
                                      upstream-r1     
1   bridge       102  client-bond1                    
                      sfp-sfpplus3                    
                      upstream-r2                     
                      upstream-r1                    
2 D bridge        99                  sfp-sfpplus3  
                      
[user@SW-2] /interface/bonding> print
Flags: X - disabled; R - running 
 0  R name="client-bond1" mtu=1500 mac-address=48:8F:5A:1A:28:C9 arp=enabled 
      arp-timeout=auto slaves=sfp-sfpplus16 mode=802.3ad primary=none 
      link-monitoring=mii arp-interval=100ms arp-ip-targets="" 
      mii-interval=100ms down-delay=0ms up-delay=0ms lacp-rate=30secs 
      transmit-hash-policy=layer-2-and-3 min-links=0 mlag-id=10 

 1  R name="upstream-r1" mtu=1500 mac-address=48:8F:5A:1A:28:BA arp=enabled 
      arp-timeout=auto slaves=sfp-sfpplus1 mode=802.3ad primary=none 
      link-monitoring=mii arp-interval=100ms arp-ip-targets="" 
      mii-interval=100ms down-delay=0ms up-delay=0ms lacp-rate=30secs 
      transmit-hash-policy=layer-2-and-3 min-links=0 mlag-id=100 

 2  R name="upstream-r2" mtu=1500 mac-address=48:8F:5A:1A:28:BB arp=enabled 
      arp-timeout=auto slaves=sfp-sfpplus2 mode=802.3ad primary=none 
      link-monitoring=mii arp-interval=100ms arp-ip-targets="" 
      mii-interval=100ms down-delay=0ms up-delay=0ms lacp-rate=30secs 
      transmit-hash-policy=layer-2-and-3 min-links=0 mlag-id=101

Configuration on switch 1 (secondary)
Code: Select all
[user@SW-1] /interface/bridge/mlag> print
     bridge: bridge
  peer-port: sfp-sfpplus3

[user@SW-1] /interface/bridge/port> print
Flags: H - HW-OFFLOAD
Columns: INTERFACE, BRIDGE, HW, PVID, PRIORITY, PATH-COST, INTERNAL-PATH-COST, HOR
IZON
#   INTERFACE     BRIDGE  HW   PVID  PRIORITY  PATH-COST  IN  HORIZON
0 H sfp-sfpplus3  bridge  yes    99  0x80             10  10  none   
1 H client-bond1  bridge  yes     1  0x80             10  10  none   
2 H upstream-r1   bridge  yes     1  0x80             10  10  none   
3 H upstream-r2   bridge  yes     1  0x80             10  10  none   


[user@SW-1] /interface/bridge/vlan> print
Flags: D - DYNAMIC
Columns: BRIDGE, VLAN-IDS, CURRENT-TAGGED, CURRENT-UNTAGGED
#   BRIDGE  VLAN-IDS  CURRENT-TAGGED  CURRENT-UNTAGGED
0   bridge       102  upstream-r2                     
                      client-bond1                    
                      upstream-r1                     
                      sfp-sfpplus3                    
1   bridge         1  sfp-sfpplus3    bridge          
                                      upstream-r2     
                                      client-bond1    
                                      upstream-r1     
2 D bridge        99                  sfp-sfpplus3   

[user@SW-1] /interface/bonding> print
Flags: X - disabled; R - running 
 0  R name="client-bond1" mtu=1500 mac-address=C4:AD:34:F4:7C:45 arp=enabled 
      arp-timeout=auto slaves=sfp-sfpplus16 mode=802.3ad primary=none 
      link-monitoring=mii arp-interval=100ms arp-ip-targets="" 
      mii-interval=100ms down-delay=0ms up-delay=0ms lacp-rate=30secs 
      transmit-hash-policy=layer-2-and-3 min-links=0 mlag-id=10 

 1  R name="upstream-r1" mtu=1500 mac-address=C4:AD:34:F4:7C:36 arp=enabled 
      arp-timeout=auto slaves=sfp-sfpplus1 mode=802.3ad primary=none 
      link-monitoring=mii arp-interval=100ms arp-ip-targets="" 
      mii-interval=100ms down-delay=0ms up-delay=0ms lacp-rate=30secs 
      transmit-hash-policy=layer-2-and-3 min-links=0 mlag-id=100 

 2  R name="upstream-r2" mtu=1500 mac-address=C4:AD:34:F4:7C:37 arp=enabled 
      arp-timeout=auto slaves=sfp-sfpplus2 mode=802.3ad primary=none 
      link-monitoring=mii arp-interval=100ms arp-ip-targets="" 
      mii-interval=100ms down-delay=0ms up-delay=0ms lacp-rate=30secs 
      transmit-hash-policy=layer-2-and-3 min-links=0 mlag-id=101
You do not have the required permissions to view the files attached to this post.
Top
 
benoitc
Member Candidate
Member Candidate
Topic Author
Posts: 170
Joined: Wed Jul 08, 2020 12:33 am

Re: MLAG: when primary is shutdown LACP stops to work

Sat Mar 26, 2022 6:59 pm

this seems to be related to freebsd asthe interface is automatically up in the routers. But can someone confirm the behaviour is expected (putting on pause the peer port status) ?

so to summarize:

* shutdown/rebooting secondary switch is done without any issue, network stays up
* disconnecting a cable is also working OK
* removing routers works as expected
* shutdown primary router introduces a delay before the connection is up again. I don't why exactly. Maybe a bug in freebsd 13 or a setting to change ?
Top
 
adcre
newbie
Posts: 28
Joined: Fri Dec 10, 2021 4:18 pm

Re: MLAG: when primary is shutdown LACP stops to work

Mon Mar 28, 2022 12:49 am

Can you post your interface configuration?

I think maybe is related to the vlans and you have same problem that op has on this post:
viewtopic.php?p=917617&hilit=mlag#p903356

Best Regards,

Diego
Top
 
benoitc
Member Candidate
Member Candidate
Topic Author
Posts: 170
Joined: Wed Jul 08, 2020 12:33 am

Re: MLAG: when primary is shutdown LACP stops to work

Tue Mar 29, 2022 3:26 pm

Can you post your interface configuration?

I think maybe is related to the vlans and you have same problem that op has on this post:
viewtopic.php?p=917617&hilit=mlag#p903356

Best Regards,

Diego
yeah I think i handled it but I wouldn't be against of another pair of eyes :))

Here are the relevant parts:
/interface bonding
add lacp-rate=1sec mlag-id=10 mode=802.3ad name=client-bond1 slaves=\
sfp-sfpplus16 transmit-hash-policy=layer-2-and-3
add mlag-id=200 mode=802.3ad name=client-bond2 slaves=sfp-sfpplus15 \
transmit-hash-policy=layer-2-and-3
add mlag-id=102 mode=802.3ad name=home-bond slaves=sfp-sfpplus5 \
transmit-hash-policy=layer-2-and-3
add mlag-id=100 mode=802.3ad name=upstream-r1 slaves=sfp-sfpplus1 \
transmit-hash-policy=layer-2-and-3
add mlag-id=101 mode=802.3ad name=upstream-r2 slaves=sfp-sfpplus2 \
transmit-hash-policy=layer-2-and-3
set bridge=bridge peer-port=sfp-sfpplus3
/interface bridge port
add bridge=bridge interface=sfp-sfpplus3 pvid=99
add bridge=bridge interface=client-bond1
add bridge=bridge interface=upstream-r1
add bridge=bridge interface=upstream-r2
add bridge=bridge interface=home-bond
add bridge=bridge interface=client-bond2
interface bridge vlan
add bridge=bridge tagged=\
sfp-sfpplus3,client-bond1,upstream-r1,upstream-r2,home-bond,client-bond2 \
vlan-ids=102
add bridge=bridge tagged=sfp-sfpplus3 vlan-ids=1
add bridge=bridge tagged=\
upstream-r1,upstream-r2,client-bond1,home-bond,client-bond2 vlan-ids=99
add bridge=bridge tagged=home-bond vlan-ids=101
add bridge=bridge tagged=home-bond vlan-ids=2
add bridge=bridge tagged=home-bond vlan-ids=3
add bridge=bridge tagged=home-bond vlan-ids=4
Top
 
EdPa
MikroTik Support
MikroTik Support
Posts: 346
Joined: Fri Sep 15, 2017 10:05 am
Location: Riga
Contact:
Contact EdPa

Re: MLAG: when primary is shutdown LACP stops to work

Tue Mar 29, 2022 4:14 pm

Hi benoitc,
* shutdown primary router introduces a delay before the connection is up again. I don't why exactly. Maybe a bug in freebsd 13 or a setting to change ?
How long is the introduced delay? Could it take up to 30 seconds?

When both MLAG switches are up, the secondary switch will change the LACP system-id and it will use the same system-id as the primary. This allows connecting regular LACP clients to two different physical switches while the client thinks it is connected to the same switch. You can monitor the "lacp-system-id" using the "/interface/bonding monitor" command and see how it changes after connecting/disconnecting the primary switch.

When the MLAG connection is lost, both switches will start to work independently. For the primary switch, it does not change anything, because the same LACP system-id is used in both scenarios. However, when the secondary switch loses the MLAG connection, it will change the LACP system-id to local values and perhaps this is the reason why connection delay and packet loss might appear?

Do you know if FreeBSD supports a 1-second LACP rate? In RouterOS, you can set it using "lacp-rate=1sec". Perhaps using a quicker LACP rate will decrease the introduced delay?

Last, you are using the pvid=99 on peer-port, while using the same VLAN in "/interface bridge vlan" menu. Try setting the peer-port to some unused untagged VLAN, e.g. pvid=999.
Top
 
adcre
newbie
Posts: 28
Joined: Fri Dec 10, 2021 4:18 pm

Re: MLAG: when primary is shutdown LACP stops to work

Tue Mar 29, 2022 11:39 pm

@EdPa i think is using vlan99 as the example shown in https://help.mikrotik.com/docs/display/ ... Quicksetup

@benoitc can you check if you have vlan filtering on the bridge?

Best Regards,

Diego
Top
 
benoitc
Member Candidate
Member Candidate
Topic Author
Posts: 170
Joined: Wed Jul 08, 2020 12:33 am

Re: MLAG: when primary is shutdown LACP stops to work

Wed Mar 30, 2022 2:11 pm

Hi benoitc,
* shutdown primary router introduces a delay before the connection is up again. I don't why exactly. Maybe a bug in freebsd 13 or a setting to change ?
How long is the introduced delay? Could it take up to 30 seconds?

When both MLAG switches are up, the secondary switch will change the LACP system-id and it will use the same system-id as the primary. This allows connecting regular LACP clients to two different physical switches while the client thinks it is connected to the same switch. You can monitor the "lacp-system-id" using the "/interface/bonding monitor" command and see how it changes after connecting/disconnecting the primary switch.

When the MLAG connection is lost, both switches will start to work independently. For the primary switch, it does not change anything, because the same LACP system-id is used in both scenarios. However, when the secondary switch loses the MLAG connection, it will change the LACP system-id to local values and perhaps this is the reason why connection delay and packet loss might appear?

Do you know if FreeBSD supports a 1-second LACP rate? In RouterOS, you can set it using "lacp-rate=1sec". Perhaps using a quicker LACP rate will decrease the introduced delay?

Last, you are using the pvid=99 on peer-port, while using the same VLAN in "/interface bridge vlan" menu. Try setting the peer-port to some unused untagged VLAN, e.g. pvid=999.

That may be around 30s yes; I will recheck. I noticed also there are some delay on the server due to bird esetting the routes on changes. I need to retest. For the rate, I did't found yet an option for it but stillooking. I will report soon .

As for the vlan it's only usedd for ctl. I have misread the doc and tagged the other devices for it. This is fixed now :) Thanks for the hilight :)
Top
 
benoitc
Member Candidate
Member Candidate
Topic Author
Posts: 170
Joined: Wed Jul 08, 2020 12:33 am

Re: MLAG: when primary is shutdown LACP stops to work

Wed Mar 30, 2022 2:12 pm

@EdPa i think is using vlan99 as the example shown in https://help.mikrotik.com/docs/display/ ... Quicksetup

@benoitc can you check if you have vlan filtering on the bridge?

Best Regards,

Diego
yes it has filtering enabled. . I will do a new test session later today , I will report it son :)
Top
 
kocifi
just joined
Posts: 1
Joined: Tue May 16, 2023 8:10 pm

Re: MLAG: when primary is shutdown LACP stops to work

Tue May 16, 2023 8:13 pm

@EdPa i think is using vlan99 as the example shown in https://help.mikrotik.com/docs/display/ ... Quicksetup

@benoitc can you check if you have vlan filtering on the bridge?

Best Regards,

Diego
yes it has filtering enabled. . I will do a new test session later today , I will report it son :)
@benoitc Were you able to resolve it?
Top
Post Reply

Who is online

Users browsing this forum: benonet, dogenzenji, elbob2002, jaclaz, wrkq and 79 guests
  4. RouterOS
  5. General