Hello
according to docs here:
https://v.ip6.io/archive/helpmt/docs/di ... tures.html

"CRS3xx, CRS5xx series switches, and CCR2116, CCR2216 routers support bridge hardware offloading with bonding interfaces. Only 802.3ad and balance-xor bonding modes are hardware offloaded, other bonding modes will use the CPU's resources. The built-in switch chip will always use Layer2+Layer3+Layer4 for a transmit hash policy, changing the transmit hash policy manually will have no effect. See more details on CRS3xx, CRS5xx, CCR2116, CCR2216 switch chip features."

To create an hardware offloaded bonding (802.3ad) I need to put into a bridge.
I have a question: I have a CRS317 where I use the l3-hw forwarding.
I have created a bond interface (802.3ad layer 2+3). Since I didnt put inside a bridge, I assume that is not hardware offloaded?

I assume that we can use the hardware offloaded bonding, only between mikrotik devices, since the layer2+3+4 is not a standard, right?
"This algorithm is not fully 802.3ad compliant. A
single TCP or UDP conversation containing both
fragmented and unfragmented packets will see packets
striped across two interfaces. This may result in out
of order delivery. Most traffic types will not meet
this criteria, as TCP rarely fragments traffic, and
most UDP traffic is not involved in extended
conversations. Other implementations of 802.3ad may
or may not tolerate this noncompliance."


Question #02: I have a 2116 and I created two bond (802.3ad layer 2+3) on sfpplus1+2 and 3+4.
I assume that they are not hardware offloaded then.


I need to create one bridge for each bond, including in that bridge the bond + the port that partecipate in bonding? This is the only way to have it hardware offloaded? And I assume only betweekn mikrotik devices? thank you

Thank you

layer2+3+4 is OK as long as you don't have fragmentation on your network (which you should never have these days due to poor support of fragments on the internet). The router is stateless, so if it sees a fragment then it has no idea what the original packet's TCP/UDP header was so it can no longer hash it properly. Keep in mind that if you mix it with non-MT hardware, the flows in the reverse direction will not be balanced the same.

Keep in mind that if you mix it with non-MT hardware, the flows in the reverse direction will not be balanced the same.

Indeed. But it has good side as well: if bond partners agree on bond mode (e,g. 802.3ad), then bond will work even if each partner uses completely different transmit hash policy ... receiving side will gladly accept frames received via any of bond links. Which in turn indeed often turns out as non-symetrical traffic over bond links in different directions.

Thank you all for your input.
I have made some test on heavy used links:

Basically I have created a bridge on the 2116 and put inside the bond interface.
It works flawlessy, no CPU usage change.
The H appears in the interface "bonding" in the bridge list.


The same test did with a CRS317
I have created the bridge containing the bond interface towards the 2116.
It appears the H in the interface in the bridge menu.
I have restarted the l3-hw offloading and everything works as expected.

I have tried to add a second bridge, with a second bond, the bond interface cannot get offloaded.

Only one bridge can be offloaded.

Yes I know.
so how can I offload multiple bonds?
I think about putting differents bond in the same bridge, then use different PVID and the same horizon
may be correct?

if you want to use bonding with offloading you must desing and configure the equipment more like a Switch than a router

Now this is possible because since the introduction of CCR 2116 and 2216 we have a powerfull router which includes a integrated switching ASIC, aditionally this ASIC can realize some Layer 3 operations

IF you want to use multiple bonded interfaces offloaded you must use a single hardware offloaded bridge

Bonding Guide for newer switches Like CRS 3xx 5xx and integrated switch on CCR 2116/2216

https://help.mikrotik.com/docs/display/ ... es-Bonding

i think one key detail is that slave interfaces participating on bonding must be retired or disabled as ports of the bridge, the bonding interface will integrate to the bridge as a port replacing this slave interfaces

we must abandon some old bridging techniques that we are dragging from Routeros 5.x and ROuteros 6.x before 6.41 when Bridge Vlan Filtering was introduced and is the base of the operation of CRX 3xx and later switching solutions by MikroTik

Thank you chechito for your detailed answer.

I have read carefully the docs and last night I made some tests.

On the 317 I configured a single bridge, and I put inside only the "bond" interfaces.
Only one get offloaded, the second bond no.
I have set different PVID to avoid broadcast/multicast traffic go from a bond to another.
It doesnt seem to work.

I think that eventually the CRS 317 that is just doing l3-hw routing... no frw rules, nothing, just plain l3-hw routing between interfaces can manage with 1% cpu load all the bond in "software".
I preferred to use the hardware solution but I need to test more.

What do you suggest other than putting different PVID (if this choice is correct) ?

... can manage with 1% cpu load all the bond in "software".

Block diagram of CRS317 says that switch-CPU interconnect is mere 3Gbps ... so if CPU won't glow, this interconnect will definitely severely limit your bond performance.

nice that info about switch-CPU interconnect capacity is now available

... can manage with 1% cpu load all the bond in "software".

Block diagram of CRS317 says that switch-CPU interconnect is mere 3Gbps ... so if CPU won't glow, this interconnect will definitely severely limit your bond performance.

Hello, sincerely I dont understand. Well, I understand your point but you may forget that I am using the L3-hw offload, and everything is working on the switch chip level, in hardware.
I have already configured 3 bonds (2x10G)x3
uplink
wholesale1
to my network

since everything is in L3-HW offload, I shouldnt never hit the limit of cpu!

In the evening I already do:
uplink to core: 9Gbit/sec
wholesale1: 3gbit/sec
to my network 6Gbit/sec:

I have 1% cpu since everything is in l3-hw.. so I dont understand how can I hit the 3gbit limit.

Hello, sincerely I dont understand.

I reacted to the all the bond in "software" part ... the software runs on CPU hence my consideration. But I may have misunderstood your statement.

I'm rather guessing that the 'H' indication on bonds might be buggy ... you have 3 bonds, all of them operating at multiple Gbps, and there doesn't seem to be any bottleneck (neither CPU nor interconnect), so I'm guessing that all bonds are actually HW offloaded.

Hello.
I have made further tests.

the CRS317 with l3-hw offload enabled, since it is just a plain switch, keeps the cpu at 2-3-4 % max, even if I do bond in the standard way without puttin it into a bridge.

I have tried doing the full hardware configuration, putting the three bonds in the bridge, everyone with H sign.
The solution is using the port isolation integrated in the switch chip and it works flawlessy, if I enable the horizon, it removes the H offload.
I keep the ip conf and the firewall conf in the input chain to protect the device and it works.
no issue at all.

Pratically no difference between bond in the bridge with the H enabled, or without, since I have l3-hw active.