Hi,

Looking at the block diagram for the RB1100AHx4:



I can see that Ether 1-5 and Ether6-10 and Ether 11-13 are on their own switch chips respectively.

After noting that:

Bridge HW vlan-filtering was added in the RouterOS 7.1rc1 (for RTL8367) and 7.1rc5 (for MT7621) versions. The switch does not support other ether-type 0x88a8 or 0x9100 (only 0x8100 is supported) and no tag-stacking. Using these features will disable HW offload.

I am trying to setup VLANs in such a way that I can utilise all ports without compromising going through the CPU if at all possible.

Would I have to create three seperate bridges, with each group of ether ports assigned to each bridge? (e.g. Ether 1-5 on vlan_bridge1, Ether 6-11 to vlan_bridge2, Ether 12-13 to vlan_bridge3)

I understand that in order to bypass, we can use a cat6 jumper between ports physically to keep VLAN filtering in hardware. However, how would I configure this aspect? Would you set a single port on each group and configure each as trunks which you then physically connect with an ethernet cable?

I note that specifically for the RB1100AHx4, in the switch menu there doesn't seem to be any obivous way to do anything with VLANs so I assume that we have to use the bridging way?



Also, if it is indeed the way to create separate bridges, since it isn't possible to assign a single Interface/VLAN to multiple bridges, how would you carry out this aspect? Would you create multiple VLANs with different names but the same VLAN ID and assign them to each individual bridge?

e,g,

Management VLAN name MGMT_1 with ID 99 assigned to vlan_bridge1
Management VLAN name MGMT_2 with ID 99 assigned to vlan_bridge2
Management VLAN name MGMT_3 with ID 99 assigned to vlan_bridge3

Perhaps I'm overcomplicating this but any guidance would be greatly appreciated. Many thanks in advance.

It's been said (by MT personnel) that it should be fine to have one bridge per switch chip and those bridges would then be HW offloaded. I don't have any device having multiple switch chips at hand, so I can't check if this is true. You can try yourself ... if bridge ports are actually HW offloaded, then they have 'H' flag displayed in /interface/bridge/port/print.

As to pass traffice between bridges: yes, you can use short ethernet jumper cables (cat5e would do as these are 1Gbps ports) and configure corresponding ports as trunk ports for involved VLANs. The other possibility is to use internal switch-CPU interconnect ... there are a few pro's to that and a con.

Pro:

• interconnects are 2.5Gbps while ports are 1Gbps
• one doesn't loose 4 ports for interconnects (one bridge would loose 2 ports for connecting the other two bridges)

Con:

• CPU gets loaded due to traffic. If CPU is busy with other tasks, e.g. routing or firewalling, then this would reduce overall performance
• broadcast/multicast traffic is sent out to all switch chips even if certain switch chip doesn't have any ports members of a particular VLAN
  there's no way of configuring VLANs on individual switch-CPU interconnects so CPU floods broadcasts to all interconnects. This "bug" consumes some of interconnect capacity, possibly slightly increases CPU load as it has to copy same data to multiple interconnects, but doesn't cause any foul play on the ether ports as switch chips will appropriately filter traffic on egress.
• one has to take care about which port group is connected which device ... so that most of traffic is contained within same switch chip

(the last Con is actually a Con in both scenarios).

If you decide to go with switch-CPU interconnects, then there are a few things to follow:

• use single bridge to cover all ports
• make bridge interface tagged member of any VLAN that needs to pass between different switch chips ... even if IP layer of ROS doesn't interact with those VLANs.
  The reason for this is a bug in ROS ... the way the switch-CPU interconnect is configured doesn't take into account multi-switch devices where inter-switch traffic has to pass CPU. HW offload config was programmed with single-switch devices in mind and on those the only traffic passing switch-CPU interconnect is traffic for IP layer of device.

Yes, you're right, on RTL8367 the (HW offloaded) bridge is the only way to deal with VLAN tags ... in v6 (which doesn't have HW offload suppot for RTL switch chips) it has to be done in software (the configuration, though, is identical).


Using same VLAN on different bridges: if you still prefer to stick with multi-bridge design, then I can think of a single possibility: make all brdiges tagged member of involved VLAN (VID 99 in your case), create appropriate VLAN interfaces, one per bridge ... so far it's as you already envisioned. Now, create another bridge, add those VLAN interfaces to it (and make sure all of these interfaces have hw=no set ... you don't want this bridge to get accidentally HW offloaded instead of one of those 3 you want HW offloaded). And then use interface, implicitly associated to this bridge, as your management interface.
This kind of bridge is (still) legitimate, it was the only way of doing VLAN-aware bridging on devices without switch chip (e.g. CCR1xxx) before ROS v6.42 (which brought us vlan-aware bridge).

Thank you very much for your reply. I just want to try out the way using short jumper cables... in the interest of learning actually.

It's been said (by MT personnel) that it should be fine to have one bridge per switch chip and those bridges would then be HW offloaded. I don't have any device having multiple switch chips at hand, so I can't check if this is true. You can try yourself ... if bridge ports are actually HW offloaded, then they have 'H' flag displayed in /interface/bridge/port/print.

I've created 2 bridges:



And as you can see, the H flag is on to indicate hardware offloading. (It's still hardware offloaded though if I use a single bridge necompassing all switch ports though).

As to pass traffice between bridges: yes, you can use short ethernet jumper cables (cat5e would do as these are 1Gbps ports) and configure corresponding ports as trunk ports for involved VLANs. The other possibility is to use internal switch-CPU interconnect ... there are a few pro's to that and a con.

Pro:

• interconnects are 2.5Gbps while ports are 1Gbps
• one doesn't loose 4 ports for interconnects (one bridge would loose 2 ports for connecting the other two bridges)

Con:

• CPU gets loaded due to traffic. If CPU is busy with other tasks, e.g. routing or firewalling, then this would reduce overall performance
• broadcast/multicast traffic is sent out to all switch chips even if certain switch chip doesn't have any ports members of a particular VLAN
  there's no way of configuring VLANs on individual switch-CPU interconnects so CPU floods broadcasts to all interconnects. This "bug" consumes some of interconnect capacity, possibly slightly increases CPU load as it has to copy same data to multiple interconnects, but doesn't cause any foul play on the ether ports as switch chips will appropriately filter traffic on egress.
• one has to take care about which port group is connected which device ... so that most of traffic is contained within same switch chip

(the last Con is actually a Con in both scenarios).

If you decide to go with switch-CPU interconnects, then there are a few things to follow:

• use single bridge to cover all ports
• make bridge interface tagged member of any VLAN that needs to pass between different switch chips ... even if IP layer of ROS doesn't interact with those VLANs.
  The reason for this is a bug in ROS ... the way the switch-CPU interconnect is configured doesn't take into account multi-switch devices where inter-switch traffic has to pass CPU. HW offload config was programmed with single-switch devices in mind and on those the only traffic passing switch-CPU interconnect is traffic for IP layer of device.

Yes, you're right, on RTL8367 the (HW offloaded) bridge is the only way to deal with VLAN tags ... in v6 (which doesn't have HW offload suppot for RTL switch chips) it has to be done in software (the configuration, though, is identical).


Using same VLAN on different bridges: if you still prefer to stick with multi-bridge design, then I can think of a single possibility: make all brdiges tagged member of involved VLAN (VID 99 in your case), create appropriate VLAN interfaces, one per bridge ... so far it's as you already envisioned. Now, create another bridge, add those VLAN interfaces to it (and make sure all of these interfaces have hw=no set ... you don't want this bridge to get accidentally HW offloaded instead of one of those 3 you want HW offloaded). And then use interface, implicitly associated to this bridge, as your management interface.
This kind of bridge is (still) legitimate, it was the only way of doing VLAN-aware bridging on devices without switch chip (e.g. CCR1xxx) before ROS v6.42 (which brought us vlan-aware bridge).

However, I'm a little stuck now and unsure as to how to proceed with two(or more bridges) as I cannot assign the same VLAN (/interface/vlan) to more than one bridge. I can of course create another VLAN with the same VLAN ID, but then how to deal with assigning IP addresses to the VLAN interfaces? I cannot assign the same IP address to e.g. MGMT_VLAN_1 (with id 99, assigned to vlan_bridge1) & MGMT_VLAN_2 (with id 99, assigned to vlan_bridge2) or can I? Or is it so that I have to make two lots of everything VLAN related..

If it's possible to create a basic example with say managment VLAN with VLAN ID 99 and two other VLANs with ID of 90 and 80 maybe I can understand it better.

Re single bridge: it should be HW offloaded as well ... with the gotcha about switch-CPU interconnect.

Re multi-bridge, single VLAN: something like this:

And the above example assumes that the trunk link (physical one) doesn't carry VLAN 99. If it does, then there's no need for the bridge-99 exercise, just anchor the vlan interface to one of (offloaded) bridges.
Regarding other VLANs, it depends on how you want to distribute VLANs over different bridges.

IMO this (patch cable) exercise brings more trouble than it's worth ... because single bridge only costs some CPU cycles but makes management so much easier.

Thank you for the example.
I'm anticipating this kind of setup:

Ether 1: trunk link to managed switch
Ether 2: trunk link to managed switch
Ether 3: trunk link to managed switch
Ether 4: trunk link to managed switch

Ether 5 connected via patch to Ether 6

Ether 7: untagged VLAN 80
Ether 8: untagged VLAN 80
Ether 9: untagged VLAN 90
Ether 10: trunk link to managed switch

Ether 11, 12, 13 will be for incoming services

You're right in that it's harder to manage but it's bugging me knowing that there is unfulfilled performance left on the table so to speak..

In the setup you outlined the third bridge (spanning ether 11-13) is not needed as those ports will likely be used in stand-alone manner.

Which leaves you with two bridges. And if you go with patch cable (trading some CPU cycles for complex setup), you actually should refrain from using one of bridges for communications between ROS and VLANs. Because:

1. the ether5-ether6 interconnect will be trunk and if you create a VLAN bridge as per my instructions in previous post, that would be a loop (for that particular VLAN, but if xSTP will be enabled - other than MSTP - then bridge will detect a loop for all VLANs).
2. that would use switch-CPU interconnects which is what you're trying to avoid

Alternatively, if you want to spread ROS traffic to both interconnects, you can use both bridges to host VLAN interfaces, but only one bridge for any of VLANs. Mind that all traffic between router and particular VLAN will use one switch-CPU interconnect (and then optionally traverse also patch cable if end device is on the "wrong" bridge) while the traffic would be passing the correct switch-CPU interconnect if thete was a single bridge (which would direct traffic towards the correct switch).



BUT: if you remove the connecting patch cable, you may loose in-band management access ... if you happen to connect management machine to one switch while management VLAN interface is connected to bridge running on top of the other switch. And much of traffic over other VLANs will follow. Which isn't likely to happen when using internal interconnects.

.... it's bugging me knowing that there is unfulfilled performance left on the table so to speak..

There are cases where each of approaches are clearly better than the other one and it really depends on traffic patterns.

If you can manage to contain bulk of traffic within respective port groups (switch chips), then both scenarios are equally well performing.

If the traffic is passing between the two port groups within same VLANs (i.e. traffic is switched/bridged and not routed) and the cumulative traffic between both port groups is below 1Gbps (each direction), then the scenario involving patch cable is better because traffic still doesn't pass CPU. However, this patch cable, being limited to 1Gbps, can become a bottleneck.

If lots of traffic is routed (either between VLANs or towards WAN), then lots of traffic will have to pass switch-CPU interconnects and will have to be processed by CPU. In case of traffic between "wrong" port group and router, traffic will have to pass patch cable. Which again can become a bottleneck.

The more I'm thinking about the setup, the more I'm convinced that patch cable scenario is the worse of the two.

i was considering doing similar to avoid CPU, but after reading this thread, i’m going to stay with the one bridge.

good read, thanks.

The more I delved into this, the more that I'm not entirely sure that it can actually be done. There just seemed too many strange ways (even in Mikrotik's world) of configuring things that I'd most likely forget even with documentation of it all.

I ended up using CPU-software interconnects and generally sticking to using the first group of ports to separate switches. The 2nd group are for lite or seldom used devices only.

I have had a very difficult time with this, things weren't working as expected even though I'm sure that the configuration is correct. I used RouterOS 7.8, fresh NetInstall.

For some reason, I was having difficulty getting addresses for any particular VLAN but it would sometimes work on reboot. It was definitely something to do with hardware offloading because if I switched off hardware offloading it would work comelptely perfectly. I had remembered to add the vlan_bridge to the taggeed ports too.

In the end, I completely wiped the bridge and VLAN configuration out again, re-did it and things seems to be working a bit more reliably.

Well.... I'm already get answer on this topic about "client connect each other on different chip" now.

Removed post.

Thank you.

I use an RB1100AHx4 for a few servers in a DC and noticed that the switch-CPU interconnects that are supposed to be HW-accelerated use the CPU a lot. Between the switches at full 1 Gbit rate, a single TCP connection through a VLAN uses around 17% CPU. The bridge is added to that VLAN but the VLAN doesn't have an interface, IP or other services, just passing packets. Is that normal? ROS 7.11.2.

The interconnect between both switch chips traverses CPU so it can't be HW accelerated (CPU has to shift all the bits). True HW acceleration would be if both switch chips would interconnect directly (i.e. you'd have switch port named e.g. switch1-switch2 just like you have switch1-cpu). You can make such a HW-accelerated connection yourself (use short UTP patch cord to connect ether5 and ether6), but you loose two ports.

I see. I got the impression that the CPU itself provides some kind of switch functionality that helps traversing packets at the rated 2.5 Gbps speed. But it seems that it's just a software bridge between the actual switch chips.

The RB1011 keeps amazing me for such an old piece of equipment.
With two of these one can setup an fully independent network with two WANs.
(using VRRF, the built-in relay bypass capability, and the redundant power supplies)
No other router in the MT arsenal can do this!!

MKX so to get it straight, you can bridge the specfic switch chips (two bridges) and both will be HW offloaded ( assumption I know ),
and you jumper the two switches by trunking the last port and then the first port of the two sides ( could be any two I know )

This is the most efficient way of connecting the vlans to see each other I am assuming??

From performance point of view ... yes. I guess (I don't have a RB1100 nor RB4011 to test) that both bridges would be HW offloaded, specially so if one would take care to "enslave" correct set of ports.
Passive wire instead of power hungry CPU. The only difference is 1Gbps (wire) vs. 2.5Gbps (CPU if it handles the load).
But, as mentioned before, using patch cord one looses 2 ports. If that's not a problem ....

The [RB1100AHx4] keeps amazing me for such an old piece of equipment.

I've been a fan of these for a while and use them at some sites. In fact, it's my usual test box for beta/rc's. Since it's ARM and has SATA support, you get all the V7 features (ZT, Containers, ROSE, BTH, etc.).
Only 32-bit ARM, so containers are bit trickier since there are less images available than for ARM64. And the RB5009 etc will run rings around in terms of routing performance, but it can handle a incoming fiber and a few VPNs no problem.

This is the most efficient way of connecting the vlans to see each other I am assuming??

The easiest is to pretend like it's three switches. Keep all the LAN/VLANs on one of the 5 port switches is the best plan. WAN traffic needs to CPU, so nothing to off-load if that's where VLAN is going. So use another 5 port group goes to WAN(s). Leave the last port group for management or 2nd WAN.

If you need more the 5 ports for VLAN/LAN... than yeah patching the port groups with ethernet cable free the CPU, at the expense of two ports... so you make 8 port (10-2) hardware switch using copper to connect port1-5 and port6-10. Or you can separate the VLANs across switch chips differently so that VLAN that only go internet, not other VLAN, are on a separate switch is another approach.

A related discussing about the ports is here: viewtopic.php?t=198922&hilit=RB1100AHx4