RSTP Problem with Bridge VLAN Filtering

t04s · Wed May 11, 2022 8:32 pm

RouterOS: v7.2.2
Router: CCR1036-12G-4S
---

Hi,

Hopefully someone can help. We've recently been having RSTP problems with one of our switches, where a failover link has been getting disabled. From what I read it looks like the problem could be that we don't have VLAN filtering on our bridges which seems to be problematic with some vendors of switches. See the basic network outline below.

The core setup is basic. There are three Netgear xs716 switches uplinked by trunks to the Mikrotik (from each switch to eth10, eth11 and eth12) carrying four VLANs 10, 11, 12 and 13. There is also a trunk uplink to our firewall (eth01). Two of the core switches are linked together for redundancy.

VLAN interfaces;

#   NAME                       MTU  ARP      VLAN-ID  INTERFACE
 0 R vlan10-management-eth01  1500  enabled      10  eth01-lan
 1 R vlan10-management-eth10  1500  enabled      10  eth10-lan
 2 R vlan10-management-eth11  1500  enabled      10  eth11-lan
 3 R vlan10-management-eth12  1500  enabled      10  eth12-lan
 4 R vlan11-voip-eth01        1500  enabled      11  eth01-lan
 5 R vlan11-voip-eth10        1500  enabled      11  eth10-lan
 6 R vlan11-voip-eth11        1500  enabled      11  eth11-lan
 7 R vlan11-voip-eth12        1500  enabled      11  eth12-lan
 8 R vlan12-data-eth01        1500  enabled      12  eth01-lan
 9 R vlan12-data-eth10        1500  enabled      12  eth10-lan
10 R vlan12-data-eth11        1500  enabled      12  eth11-lan
11 R vlan12-data-eth12        1500  enabled      12  eth12-lan
12 R vlan13-dmz-eth01         1500  enabled      13  eth01-lan
13 R vlan13-dmz-eth10         1500  enabled      13  eth10-lan
14 R vlan13-dmz-eth11         1500  enabled      13  eth11-lan
15 R vlan13-dmz-eth12         1500  enabled      13  eth12-lan

IP subnets;

# ADDRESS            NETWORK       INTERFACE              
;;; Management
0 172.22.10.1/24    172.22.10.0  br01-vlan10-management
;;; Data
1 172.22.11.1/24    172.22.11.0  br03-vlan11-data      
;;; VoIP
2 172.22.12.1/24    172.22.12.0  br02-vlan12-voip      
;;; DMZ
3 172.22.13.1/24    172.22.13.0  br04-vlan13-dmz

Bridges;

#   NAME                     MTU   ACTUAL-MTU  L2MTU
0 R br01-vlan10-management  auto        1500   1576
1 R br02-vlan11-voip        auto        1500   1576
2 R br03-vlan12-data        auto        1500   1576
3 R br04-vlan13-dmz         auto        1500   1576

Bridge ports;

#   INTERFACE                 BRIDGE                   HW   PVID  PRIORITY  PATH-COST  IN  HORIZON
 0   vlan10-management-eth12  br01-vlan10-management        10  0x80             10  10  none   
 1   vlan11-voip-eth12        br02-vlan11-voip              11  0x80             10  10  none   
 2   vlan12-data-eth12        br03-vlan12-data              12  0x80             10  10  none   
 3   vlan10-management-eth11  br01-vlan10-management        10  0x80             10  10  none   
 4   vlan11-voip-eth11        br02-vlan11-voip              11  0x80             10  10  none   
 5   vlan12-data-eth11        br03-vlan12-data              12  0x80             10  10  none   
 6   vlan10-management-eth10  br01-vlan10-management        10  0x80             10  10  none   
 7   vlan11-voip-eth10        br02-vlan11-voip              11  0x80             10  10  none   
 8   vlan12-data-eth10        br03-vlan12-data              12  0x80             10  10  none   
9   vlan13-dmz-eth10         br04-vlan13-dmz               13  0x80             10  10  none   
10   vlan13-dmz-eth11         br04-vlan13-dmz               13  0x80             10  10  none   
11   vlan13-dmz-eth12         br04-vlan13-dmz               13  0x80             10  10  none   
12   vlan10-management-eth01  br01-vlan10-management        10  0x80             10  10  none   
13   vlan11-voip-eth01        br02-vlan11-voip              11  0x80             10  10  none   
14   vlan12-data-eth01        br03-vlan12-data              12  0x80             10  10  none   
15   vlan13-dmz-eth01         br04-vlan13-dmz               13  0x80             10  10  none

We configured things this way because we also need inter VLAN routing. We find that everything functions fine. In fact it was fine for months but recently RSTP on one of the Netgears keeps setting one of our switch trunk ports to the Mikrotik to D-Disable and we have to manually re-enable it. It seems this could be an incompatibility with RSTP on the Mikrotik. We haven't had Bridge VLAN filtering enabled previously so the bridge VLAN table hasn't dynamically shown tagged and untagged traffic. Once we enabled it though we saw that there is untagged traffic as per;

# BRIDGE                   VLAN-IDS  CURRENT-TAGGED           CURRENT-UNTAGGED        
0 br01-vlan10-management       10  br01-vlan10-management  vlan10-management-eth12
                                                              vlan10-management-eth11
                                                              vlan10-management-eth10
                                                              vlan10-management-eth01
1 br02-vlan11-voip             11  br02-vlan11-voip        vlan11-voip-eth12      
                                                              vlan11-voip-eth11      
                                                              vlan11-voip-eth10      
                                                              vlan11-voip-eth01      
2 br03-vlan12-data             12  br03-vlan12-data        vlan12-data-eth12      
                                                              vlan12-data-eth11      
                                                              vlan12-data-eth10      
                                                              vlan12-data-eth01      
3 br04-vlan13-dmz              13  br04-vlan13-dmz         vlan13-dmz-eth10       
                                                              vlan13-dmz-eth11       
                                                              vlan13-dmz-eth12       
                                                              vlan13-dmz-eth01

But as this is a trunk uplink from the switches we'd expected to see everything tagged. If we only accept tagged frames things stop working. What do we have wrong here? And should bridge VLAN filtering be expected to resolve our RSTP issues?

Thanks,
t04s

tdw · Wed May 11, 2022 9:27 pm

If a Mikrotik bridge is set to protocol-mode=none it is not 802.1D compliant as the so-called 'slow protocols' which include STP, LACP, etc. are forwarded between ports which is likely the cause of your problem. This is actually useful in some scenarios, but not here.

As you are using the 'old style' of VLAN setup with a bridge for each VLAN ID and /interface vlan instances to tag and untag between the physical port and bridge you cannot change the protocol mode to anything else as the BPDUs from the bridge are tagged via the /interface vlan instances before leaving the physical ports, see https://help.mikrotik.com/docs/display/ ... ridgedVLAN and https://help.mikrotik.com/docs/display/ ... interfaces

You could enable spanning tree on the bridges and set the ports to edge=yes which filters out the BPDUs, or use bridge filters to block them.

The best method would be to use a single vlan-aware bridge, something along the lines of

/interface bridge
add name=bridge vlan-filtering=yes
/interface bridge port
add bridge=bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether1
add bridge=bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether10
add bridge=bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether11
add bridge=bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether12
/interface bridge vlan
add bridge=bridge tagged=bridge,ether1,ether10,ether11,ether12 vlan-ids=10
add bridge=bridge tagged=bridge,ether1,ether10,ether11,ether12 vlan-ids=11
add bridge=bridge tagged=bridge,ether1,ether10,ether11,ether12 vlan-ids=12
add bridge=bridge tagged=bridge,ether1,ether10,ether11,ether12 vlan-ids=13
/interface vlan
add interface=bridge name=bridge-vlan10-management vlan-id=10
add interface=bridge name=bridge-vlan11-voip vlan-id=11
add interface=bridge name=bridge-vlan12-data vlan-id=12
add interface=bridge name=bridge-vlan13-dmz vlan-id=13
/ip address
add address=172.22.10.1/24 interface=bridge-vlan10-management
add address=172.22.11.1/24 interface=bridge-vlan11-voip
add address=172.22.12.1/24 interface=bridge-vlan12-data
add address=172.22.13.1/24 interface=bridge-vlan13-dmz

t04s · Thu May 12, 2022 1:55 am

Thanks for the pointers. What's odd is that RSTP is already enabled on the bridges and bridge ports are set to auto which are being dynamically detected as edge=yes.

But yes, it seems like a reconfiguration to a single bridge is needed. I did try this and had some issue but will give it another go.

Thanks,
t04s

mkx · Thu May 12, 2022 8:34 am

What's odd is that RSTP is already enabled on the bridges and bridge ports are set to auto which are being dynamically detected as edge=yes.

Having some sort of xSTP enabled on bridge by default most of times doesn't hurt and can sometimes save admin's butt. And RSTP seems to be most widely usable variant. However it's not 100% universally best setting.
And using bridge per VLAN is obsolete since quite a few years ago (since 6.41 introduced vlan-filtering on bridge).

So if one is configuring their device in a non-default / non-intended way (which ROS gladly allows), the one has to understand all the details in order to get it right, safety belts built in ROS are very limited.

t04s · Thu May 12, 2022 10:44 am

Understood. Will try reconfiguring the bridge VLAN setup as per best practice.

Thanks,
t04s

anav · Thu May 12, 2022 4:02 pm

After fixing the vlans, did you try MSTP vice RSTP?

t04s · Thu May 12, 2022 7:54 pm

This is working a treat! Appreciate the help.

Had to also update DHCP relay to use the new VLAN interfaces. So far so good.

There is one final thing however; I use routing marks in the mangle table to mark outbound (public IPs) per VLAN so I can pick it up in the routing rules and send to the correct gateway for each VLAN - 172.22.10.2, 172.22.11.2 and so on. But since upgrading to RouterOS /ip route in WebFig seems to be missing lots of routes, and clicking Add New does nothing. Also rules seem like they should have been moved to /routing rule but those are empty.

Yet see the output of /ip route print detail;

1  As   dst-address=0.0.0.0/0 routing-table=VLAN12-Outbound pref-src=172.22.12.1
         gateway=172.22.12.2 immediate-gw=172.22.12.2%vlan12-data 
         check-gateway=arp distance=1 scope=30 target-scope=10 
         suppress-hw-offload=no 

 2  As   dst-address=0.0.0.0/0 routing-table=VLAN10-outbound pref-src=172.22.10.1
         gateway=172.22.10.2 immediate-gw=172.22.10.2%vlan10-management 
         check-gateway=arp distance=1 scope=30 target-scope=10 
         suppress-hw-offload=no 

 3  As   dst-address=0.0.0.0/0 routing-table=VLAN11-Outbound pref-src=172.22.11.1
         gateway=172.22.11.2 immediate-gw=172.22.11.2%vlan11-voip 
         check-gateway=arp distance=1 scope=30 target-scope=10 
         suppress-hw-offload=no 

 4  As   dst-address=0.0.0.0/0 routing-table=VLAN13-Outbound pref-src=172.22.13.1
         gateway=172.22.13.2 immediate-gw=172.22.13.2%vlan13-dmz 
         check-gateway=arp distance=1 scope=30 target-scope=10 
         suppress-hw-offload=no

So static routes are there and working. It's also not an immediate issue as this is all fine from the CLI, so no issue to manage but does anyone know if this is a bug?

Thanks,
t04s

tdw · Thu May 12, 2022 8:36 pm

Had to also update DHCP relay to use the new VLAN interfaces. So far so good.

You would, but the full configuration wasn't provided so any other use if interfaces was unknown.

I use routing marks in the mangle table to mark outbound (public IPs) per VLAN so I can pick it up in the routing rules and send to the correct gateway

Why? Just specify the specific public IP addresses in a src-nat rule for each subnet https://wiki.mikrotik.com/wiki/Manual:I ... ic_address

t04s · Thu May 12, 2022 9:16 pm

You would, but the full configuration wasn't provided so any other use if interfaces was unknown.

Of course, was merely pointing out I remembered I needed to do that. Not really relevant.

The reason for the rules is this isn't the NAT device, that's the uplinked firewall. The trunk uplink is to have firewall policies per VLAN or floating rules across VLANs.

Thanks,
t04s

Fri May 13, 2022 4:25 am

check this

https://help.mikrotik.com/docs/display/ ... linterface

t04s · Wed May 25, 2022 7:05 pm

Sadly this didn't resolve it. I'd thought this had cracked it. The network had seemed more responsive but about four days later both uplinks to the Mikrotik from the 10G Netgears went into the D-Disable state, which was a worse case than before and brings down the network as one of those operates as the redundant path.

From the Netgear XS716T documentation it looks highly likely this is due to BPDU flooding. It does say that D-Disable is nearly always caused when a port "receives more than 15 BPDUs in a 3-second interval".

The bad thing about these devices is that the port stays disabled, rather than automatically coming back online once network conditions allow. I tried setting bridge priorities in such as way that one of these was the root bridge rather than the Mikrotik, which was sub-optimal anyway as a slower path but that hasn't helped. Still, ~4 days it goes down.

After fixing the vlans, did you try MSTP vice RSTP?

If the view is MSTP will help I'll happily try that. Just didn't think it was necessary on a simple network with only a handful of VLANs.

Does anyone have any advice on the best way to go about diagnosing this?

Thanks,
t04s

anav · Wed May 25, 2022 7:25 pm

Post your complete config..........
/export file=anynameyouwish

t04s · Wed May 25, 2022 9:24 pm

Which other parts do you want me to post?

Here is the detailed and updated config since the OP;

/interface

Flags: D - dynamic; X - disabled, R - running; S - slave; P - passthrough 
 0  RS  name="eth01-lan" default-name="ether1" type="ether" mtu=1500 actual-mtu=1500 l2mtu=1580 
        max-l2mtu=10222 mac-address=48:8F:5A:D3:73:24 ifname="eth4" ifindex=10 id=5 
        last-link-up-time=may/16/2022 07:14:02 link-downs=0 

 3  RS  name="eth10-lan" default-name="ether10" type="ether" mtu=1500 actual-mtu=1500 l2mtu=1580 
        max-l2mtu=10222 mac-address=48:8F:5A:D3:73:2D ifname="eth13" ifindex=19 id=14 
        last-link-up-time=may/16/2022 07:14:02 link-downs=0 

 4  RS  name="eth11-lan" default-name="ether11" type="ether" mtu=1500 actual-mtu=1500 l2mtu=1580 
        max-l2mtu=10222 mac-address=48:8F:5A:D3:73:2E ifname="eth14" ifindex=20 id=15 
        last-link-down-time=may/25/2022 18:42:49 last-link-up-time=may/25/2022 18:43:57 link-downs=4 

 5  RS  name="eth12-lan" default-name="ether12" type="ether" mtu=1500 actual-mtu=1500 l2mtu=1580 
        max-l2mtu=10222 mac-address=48:8F:5A:D3:73:2F ifname="eth15" ifindex=21 id=16 
        last-link-down-time=may/20/2022 14:26:18 last-link-up-time=may/25/2022 17:00:09 link-downs=8 

16  R   name="br01" type="bridge" mtu=auto actual-mtu=1500 l2mtu=1580 mac-address=48:8F:5A:D3:73:24 
        ifname="br4" ifindex=65 id=52 last-link-down-time=may/16/2022 18:03:34 
        last-link-up-time=may/16/2022 18:03:34 link-downs=8 

17  R   name="vlan10-management" type="vlan" mtu=1500 actual-mtu=1500 l2mtu=1576 
        mac-address=48:8F:5A:D3:73:24 ifname="vlan53" ifindex=66 id=53 
        last-link-down-time=may/16/2022 18:03:34 last-link-up-time=may/16/2022 18:03:34 link-downs=8 

18  R   name="vlan11-voip" type="vlan" mtu=1500 actual-mtu=1500 l2mtu=1576 
        mac-address=48:8F:5A:D3:73:24 ifname="vlan54" ifindex=67 id=54 
        last-link-down-time=may/16/2022 18:03:34 last-link-up-time=may/16/2022 18:03:34 link-downs=8 

19  R   name="vlan12-data" type="vlan" mtu=1500 actual-mtu=1500 l2mtu=1576 
        mac-address=48:8F:5A:D3:73:24 ifname="vlan57" ifindex=69 id=57 
        last-link-down-time=may/16/2022 18:03:34 last-link-up-time=may/16/2022 18:03:34 link-downs=10 

20  R   name="vlan13-dmz" type="vlan" mtu=1500 actual-mtu=1500 l2mtu=1576 mac-address=48:8F:5A:D3:73:2>
        ifname="vlan56" ifindex=68 id=56 last-link-down-time=may/16/2022 18:03:34 
        last-link-up-time=may/16/2022 18:03:34 link-downs=8

/ip/address

Flags: X - disabled, I - invalid, D - dynamic 
 0   ;;; Management
     address=172.22.10.1/24 network=172.22.10.0 interface=vlan10-management 
     actual-interface=vlan10-management 

 1   ;;; Data
     address=172.22.12.1/24 network=172.22.12.0 interface=vlan12-data actual-interface=vlan12-data 

 2   ;;; VoIP
     address=172.22.11.1/24 network=172.22.11.0 interface=vlan11-voip actual-interface=vlan11-voip 

 3 I ;;; Native
     address=172.18.99.1/24 network=172.18.99.0 interface=*18 actual-interface=*18 

 4   ;;; DMZ
     address=172.22.13.1/24 network=172.22.13.0 interface=vlan13-dmz actual-interface=vlan13-dmz 

 5   address=172.22.10.254/24 network=172.22.10.0 interface=eth02-lan actual-interface=eth02-lan

/interface/bridge

Flags: X - disabled, R - running 
 0 R name="br01" mtu=auto actual-mtu=1500 l2mtu=1580 arp=enabled arp-timeout=auto 
     mac-address=48:8F:5A:D3:73:24 protocol-mode=rstp fast-forward=yes igmp-snooping=no auto-mac=yes 
     ageing-time=5m priority=0x2000 max-message-age=20s forward-delay=15s transmit-hold-count=6 
     vlan-filtering=yes ether-type=0x8100 pvid=1 frame-types=admit-only-vlan-tagged 
     ingress-filtering=yes dhcp-snooping=no

/interface/bridge/port

Flags: X - disabled, I - inactive; D - dynamic; H - hw-offload 
 0 I   interface=eth03-lan bridge=*18 priority=0x80 path-cost=10 internal-path-cost=10 edge=auto 
       point-to-point=auto learn=auto horizon=none auto-isolate=no restricted-role=no 
       restricted-tcn=no pvid=10 frame-types=admit-all ingress-filtering=yes 
       unknown-unicast-flood=yes unknown-multicast-flood=yes broadcast-flood=yes tag-stacking=no 
       bpdu-guard=no trusted=no multicast-router=temporary-query fast-leave=no 

 1     interface=eth01-lan bridge=br01 priority=0x80 path-cost=10 internal-path-cost=10 edge=auto 
       point-to-point=auto learn=auto horizon=none hw=yes auto-isolate=no restricted-role=no 
       restricted-tcn=no pvid=1 frame-types=admit-only-vlan-tagged ingress-filtering=yes 
       unknown-unicast-flood=yes unknown-multicast-flood=yes broadcast-flood=yes tag-stacking=no 
       bpdu-guard=no trusted=no multicast-router=temporary-query fast-leave=no 

 2     interface=eth10-lan bridge=br01 priority=0x30 path-cost=10 internal-path-cost=10 edge=auto 
       point-to-point=auto learn=auto horizon=none hw=yes auto-isolate=no restricted-role=no 
       restricted-tcn=no pvid=1 frame-types=admit-only-vlan-tagged ingress-filtering=yes 
       unknown-unicast-flood=yes unknown-multicast-flood=yes broadcast-flood=yes tag-stacking=no 
       bpdu-guard=no trusted=no multicast-router=temporary-query fast-leave=no 

 3     interface=eth11-lan bridge=br01 priority=0x20 path-cost=10 internal-path-cost=10 edge=auto 
       point-to-point=auto learn=auto horizon=none hw=yes auto-isolate=no restricted-role=no 
       restricted-tcn=no pvid=1 frame-types=admit-only-vlan-tagged ingress-filtering=yes 
       unknown-unicast-flood=yes unknown-multicast-flood=yes broadcast-flood=yes tag-stacking=no 
       bpdu-guard=no trusted=no multicast-router=temporary-query fast-leave=no 

 4     interface=eth12-lan bridge=br01 priority=0x10 path-cost=10 internal-path-cost=10 edge=auto 
       point-to-point=auto learn=auto horizon=none hw=yes auto-isolate=no restricted-role=no 
       restricted-tcn=no pvid=1 frame-types=admit-only-vlan-tagged ingress-filtering=yes 
       unknown-unicast-flood=yes unknown-multicast-flood=yes broadcast-flood=yes tag-stacking=no 
       bpdu-guard=no trusted=no multicast-router=temporary-query fast-leave=no

/interface/bridge/vlan

Flags: X - disabled, D - dynamic 
 0   bridge=br01 vlan-ids=10 tagged=br01,eth01-lan,eth10-lan,eth11-lan,eth12-lan untagged="" 
     current-tagged=br01,eth01-lan,eth10-lan,eth12-lan,eth11-lan current-untagged="" 

 1   bridge=br01 vlan-ids=11 tagged=br01,eth01-lan,eth10-lan,eth11-lan,eth12-lan untagged="" 
     current-tagged=br01,eth01-lan,eth10-lan,eth12-lan,eth11-lan current-untagged="" 

 2   bridge=br01 vlan-ids=12 tagged=br01,eth01-lan,eth10-lan,eth11-lan,eth12-lan untagged="" 
     current-tagged=br01,eth01-lan,eth10-lan,eth12-lan,eth11-lan current-untagged="" 

 3   bridge=br01 vlan-ids=13 tagged=br01,eth01-lan,eth10-lan,eth11-lan,eth12-lan untagged="" 
     current-tagged=br01,eth01-lan,eth10-lan,eth12-lan,eth11-lan current-untagged=""

If there is anything else, please let me know.

Thanks,
t04s

RSTP Problem with Bridge VLAN Filtering

RSTP Problem with Bridge VLAN Filtering

Re: RSTP Problem with Bridge VLAN Filtering

Re: RSTP Problem with Bridge VLAN Filtering

Re: RSTP Problem with Bridge VLAN Filtering

Re: RSTP Problem with Bridge VLAN Filtering

Re: RSTP Problem with Bridge VLAN Filtering

Re: RSTP Problem with Bridge VLAN Filtering

Re: RSTP Problem with Bridge VLAN Filtering

Re: RSTP Problem with Bridge VLAN Filtering

Re: RSTP Problem with Bridge VLAN Filtering

Re: RSTP Problem with Bridge VLAN Filtering

Re: RSTP Problem with Bridge VLAN Filtering

Re: RSTP Problem with Bridge VLAN Filtering