L3 HW Offload support on RB5009

azzurro · Fri Apr 08, 2022 12:53 am

Hi,

i was wondering whether the RB5009 will get HW offloading support for L3 at some point. It is a bit funny imho, that I'm thinking about moving L3 to my CRS326 while having "The ultimate heavy-duty home lab router" just to get 10G wire speed. Well, at least that's going to save me one 10G port (which currently goes to the RB5009), if I'd be doing routing directly on the CRS326...

Thanks for any info in advance.

Fri Apr 08, 2022 2:29 am

How does L3HW on a router make sense? An L3 smart switch, yes, but a proper router like the 5009? The whole point is to make routing decisions, which is why they have fairly fast CPUs.

There’s nothing wrong with coupling a router to a dedicated switch. Let the router route, and the switch switch.

azzurro · Fri Apr 08, 2022 2:34 am

it seems that I'm failing to understand the concepts.
I want to let the router route (RB5009). Aren't routing decisions possible with L3 HW offloading? How are the 10 Gb/s advertised throughput achievable with a RB5009? I'm only getting around 1,5 Gb/s to around 5 Gb/s between two VLANs, depending on how many parallel threads I start (all tested with iperf3 without any special parameters).

I thought L3 HW means that the device can do all the routing (and routing decisions) in hardware and therefore reaches wire speed at L3. So switches like the CRS326 are "dumbed down" when doing L3 HW? What are the limitations in terms of features compared to a RB5009 which is using its CPU?

I'm a bit lost.

Thanks!

Fri Apr 08, 2022 3:34 am

Aren't routing decisions possible with L3 HW offloading?

Some routing decisions are, yes, but not all. Note the frequent lack of "HW" in the second column.

Even in the simple cases like IPv4 unicast, either someone has to configure the router with a static route, or it has to be provided by other means: implicit by VPN configuration, dynamic by OSPF, etc. Until someone tells the router what to route, it can't hardware-offload it. Even then, you have the "gateway=" case further down that list showing how it can require CPU intervention.

L3HW routing is better used for cases like inter-VLAN routing, where some outside agency applies VLAN tags, turning it into what is basically a switching problem for a switch chip that knows how to pair 802.1q tags with routing rules.

the 10 Gb/s advertised throughput

That's conditional. Are you doing full-size packets with no firewall rules and no queues?

The test results show 2.6 Gbit/sec for a typical IP firewall setup with an average of 512 byte packets, for instance.

I'm only getting around 1,5 Gb/s to around 5 Gb/s between two VLANs

Inter-VLAN routing is a pretty weak form of "routing." It's basically fixed, which is why it can be hardware-offloaded.

When I say "proper router," I think of IP firewall rules, VPNs, queueing, and such. These all require CPU intervention. IP firewall rules and queueing do because they're handled by the underlying Linux kernel. VPNs do because they're implemented either in the kernel or in userspace, so all they can hardware-offload is the crypto, and not always even then.

depending on how many parallel threads I start (all tested with iperf3 without any special parameters).

Between which two ports? There's only one 10G port on the box, and one 2.5G. Everything else is gigabit.

If you were relying on full-duplex on the lone 10G port, with one direction feeding the other, how is that realistic? You don't buy a router to route from one port back to the same port. It'll bottleneck to 2.5 or 1G regardless, in real-world cases, even with the ideal L3HW feature set.

I thought L3 HW means that the device can do all the routing (and routing decisions) in hardware

Nope, not even in the best CRS cases. See the first link.

switches like the CRS326 are "dumbed down" when doing L3 HW?

If by that you mean they don't have every RouterOS feature hardware-offloaded, then yes.

StubArea51 · Fri Apr 08, 2022 7:05 am

it seems that I'm failing to understand the concepts.
I want to let the router route (RB5009). Aren't routing decisions possible with L3 HW offloading?

Yes, dynamic and static routing works on platforms where hw offloading is enabled - here is an example of CRS317 running as an L3 switch w/ iBGP Route Reflection.

[zuul@sw-core-01.jan1.us.ipa] > routing/route/print where active && hw-offloaded && bgp
Flags: A - ACTIVE; b, y - COPY; H - HW-OFFLOADED
Columns: DST-ADDRESS, GATEWAY, AFI, DISTANCE, SCOPE, TARGET-SCOPE, IMMEDIATE-GW
    DST-ADDRESS      GATEWAY        AFI  DISTANCE  SCOPE  TARGET-SCOPE  IMMEDIATE-GW                                     
AbH 0.0.0.0/0        100.127.32.2   ip4       200     40            30  100.126.32.82%vlan3013-to-rtr-edge-02.jan1.us.ipa
AbH 10.255.44.0/22   100.126.32.66  ip4        20     40            10  100.126.32.66%ether3                             
AbH 10.255.228.0/24  100.126.32.66  ip4        20     40            10  100.126.32.66%ether3

Once the 5009 has hw-offload enabled, you'll be able to do L3 in hardware and save the CPU for other tasks which is what we do in the new CCR routers that have ASICs + CPU like the CCR2116 and the CCR2216 (both of which i'm testing right now).

StubArea51 · Fri Apr 08, 2022 7:38 am

How does L3HW on a router make sense? An L3 smart switch, yes, but a proper router like the 5009? The whole point is to make routing decisions, which is why they have fairly fast CPUs.

There’s nothing wrong with coupling a router to a dedicated switch. Let the router route, and the switch switch.

This is a bit misleading. While there is nothing wrong with using a dedicated switch and a router depending on the use case, the idea of letting "a router route and a switch switch" is a bit archaic in modern network engineering. I first learned this phrase in the mid 1990s and it's not really been true for a long time because we can offload complex L3 and MPLS tasks into hardware even in inexpensive equipment.

When we build service provider or data center networks, we want everything we can possibly get offloaded into an ASIC and then let the CPU pick up as needed. The difference between a "router" and a "switch" at L3 is mainly in packet header rewrites and the lines between the two terms are very blurry today.

This is why the routes are flagged at the time of FIB insertion with or without H, the router can immediately determine the packet flow via ASIC or CPU and can switch between the two without issue - which is a design that's been around for a couple of decades.

Also, if you notice, MikroTik made a distinct change to the design of the new CCR2116 and CCR2216 CCR routers and included an ASIC as well as a powerful CPU to help divide responsibilities in the router so that data plane operations can mostly be handled by the Marvell chip and processor intensive tasks like loading full BGP tables, setting localpref, communities, etc can be handled by the CPU. This brings the design of MikroTik's routers closer to Juniper MX, CIsco ASR9K, Noia 7750, etc

MikroTik already has IPv4 unicast offloaded which covers the vast majority of traffic needed and I've been testing IPv6 and MPLS hw-offload. Both of which are much closer to being done. We can get 10 GIg of MPLS/VPLS traffic in a CCR2116 over LDPv6 using GUA with only 4% CPU load.

The only thing that seems to be a bit hit or miss is route recursion and I think that probably needs more development to get fully baked.

azzurro · Fri Apr 08, 2022 10:33 am

L3HW routing is better used for cases like inter-VLAN routing,

That's what I'm trying to do here.

the 10 Gb/s advertised throughput

That's conditional. Are you doing full-size packets with no firewall rules and no queues?

afaik, iperf is sending as large packets as possible, so yes, full-size packets. I have no queues and only a few simple fw rules:

/ip firewall address-list
add address=192.168.0.0/16 list=LAN-NETWORKS
add address=172.16.0.0/12 list=LAN-NETWORKS
add address=10.0.0.0/8 list=LAN-NETWORKS
add address=192.168.25.11 list=PUBLISHED-SERVERS
add address=192.168.25.6 list=PUBLISHED-SERVERS
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid log-prefix=drop1
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid log-prefix=drop2
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new dst-address-list=PUBLISHED-SERVERS in-interface=ether1 log-prefix=bla new-connection-mark=\
out-fgt passthrough=yes src-address-list=!LAN-NETWORKS
add action=mark-routing chain=prerouting connection-mark=out-fgt connection-state=established,related dst-address-list=!LAN-NETWORKS in-interface=bridge log-prefix=blubb \
new-routing-mark=public-service-return-path passthrough=yes src-address-list=PUBLISHED-SERVERS

I'm only getting around 1,5 Gb/s to around 5 Gb/s between two VLANs

Inter-VLAN routing is a pretty weak form of "routing." It's basically fixed, which is why it can be hardware-offloaded.

When I say "proper router," I think of IP firewall rules, VPNs, queueing, and such. These all require CPU intervention. IP firewall rules and queueing do because they're handled by the underlying Linux kernel. VPNs do because they're implemented either in the kernel or in userspace, so all they can hardware-offload is the crypto, and not always even then.

Yes, that's why I wanted to HW-offload it but I can't enable it on the RB5009... Why not HW-offload what's possible and let the CPU handle what remains?

depending on how many parallel threads I start (all tested with iperf3 without any special parameters).

Between which two ports? There's only one 10G port on the box, and one 2.5G. Everything else is gigabit.

If you were relying on full-duplex on the lone 10G port, with one direction feeding the other, how is that realistic? You don't buy a router to route from one port back to the same port. It'll bottleneck to 2.5 or 1G regardless, in real-world cases, even with the ideal L3HW feature set.

I was using the single 10G port. Why shouldn't that be realistic? Do I need better, enterprise-class hardware for that? Because we do that all the time at work and we get full speed with our routers (Cisco stuff). Also, on my Fortigate firewall, I can route 1 Gb/s reliably on a single physical 1G interface, even with firewall rules. But it has ASICs, to be fair. But still, relying on a single 1G interface: why not?!

Once the 5009 has hw-offload enabled

you mean once RouterOS has support for hw-offloading on the 5009? Because I wasn't able to enable L3HW on it when I tried...

StubArea51 · Fri Apr 08, 2022 2:12 pm

you mean once RouterOS has support for hw-offloading on the 5009? Because I wasn't able to enable L3HW on it when I tried...

Yes, exactly. Once hw-offload is enabled in ROS, I would expect IPv4 unicast at a minimum to be offloaded for static routes, directly connected routes and RIP, OSPF, BGP with directly connected gateways. It seems likely based on the ASIC reference that queues and some fw rules can be offloaded as well.

Here is the link to the Marvell chip architecture for the 5009.

https://www.marvell.com/content/dam/mar ... -brief.pdf

azzurro · Fri Apr 08, 2022 2:14 pm

you mean once RouterOS has support for hw-offloading on the 5009? Because I wasn't able to enable L3HW on it when I tried...
Yes, exactly. Once hw-offload is enabled in ROS, I would expect IPv4 unicast at a minimum to be offloaded for static routes, directly connected routes and RIP, OSPF, BGP with directly connected gateways.

Here is the link to the Marvell chip architecture for the 5009.

https://www.marvell.com/content/dam/mar ... -brief.pdf

ok thanks, so are you confident that this will come as a feature for RB5009 or do you have that information from a reliable source or is this just a plain assumption? Just so I can tell what my chances for L3HW on this device are...

Fri Apr 08, 2022 3:17 pm

Azzurro

I would say your chances of seeing L3HW support on RB5009 and CCR2004-16G-2S+ are very high, say 85%

Mikrotik wouldn't spend a dime on hardware they were not going to use

. Hell they even removed the damn beeper from new products to save a few cents ! (I will forever grieve said beeper)

glow · Fri Apr 08, 2022 5:37 pm

Azzurro

I would say your chances of seeing L3HW support on RB5009 and CCR2004-16G-2S+ are very high, say 85%

Mikrotik wouldn't spend a dime on hardware they were not going to use . Hell they even removed the damn beeper from new products to save a few cents ! (I will forever grieve said beeper)

I don't have that much hope on the CCR2004 lineup, since none of those include switch ASICs for the SFP+ ports. The original model's "PIPE" doesn't seem to have any HW offload capabilities of use and the newer models have switch ASICs, but only for the 1GbE RJ45 ports.

However, the CCR2116 and CCR2216 have very powerful switch ASICs for all main ports!

That being said, I could be very wrong and the Annapurna Labs CPU might have hw offloads.

Fri Apr 08, 2022 6:43 pm

While I agree in principle that the lines between routers and switches are continually blurring, and the hardware seems to have the raw capabilities needed, we have to live in the present.

MikroTik has a lot on their plate at the moment. I hesitate to put a timeframe on it, but there have been outright bugs that have taken many months to be addressed; you have only to look at the moaning in the 7.2 release thread to see that ROS 7 still needs stabilizing. How long for adding new code?

Since MikroTik doesn’t pre-announce features, you’ll have to roll the bones. It might pay off quickly to add one of the lower end CRS3xx switches and rearchitect the network with classic separation of concerns.

Or, the feature could drop tomorrow.

Fri Apr 08, 2022 6:46 pm

CCR2116 and CCR2216 already have full L3HW support.
CCR2004 cannot have L3HW because its Marvell 88E6191 switch chip physically does not have L3 capabilities.
RB5009 uses Marvell 88E6393 with very limited L3HW options. Maybe one day, we will implement L3HW support for RB5009, but it is not worth it, in my opinion (which may differ from MikroTik's official). For example, CCR2216 can offload up to 120K IPv4 routes + 64K connected hosts + 4608 FastTrack connections. RB5009, on the other hand, can offload 256 - not K but entries! And those 256 entries are shared by routes, hosts, connections, and ACL rules (/in/eth/sw/rule). While those 256 L3HW entries may help reach pretty numbers in speed tests, I highly doubt its real-life usage. It is not enough to offload Inter-VLAN routing even between two /24 subnets.

Please don't get me wrong - RB5009 is a great product for the price, and its quad-core CPU provides routing power that is more than enough in most cases. L3HW shines the most at 10G+ speeds, and that isn't the area of the product.

gabacho4 · Fri Apr 08, 2022 7:18 pm

@raimondsp great response, even if unofficial. Offloading on the RB5009 doesn’t seem worth it at all based on your explanation. I’d much rather MikroTik focus on ROS 7 improvements and fine tuning things to make the RB5009 as powerful as it can be rather than spend time on something that appears to have no real benefit.

Fri Apr 08, 2022 8:07 pm

256 L3HW entries…is not enough to offload Inter-VLAN routing even between two /24 subnets.

I assume that claim is based on either 2×2⁸ or (2⁸)², both of which are > 256, but why would the switch chip need to be told each individual host mapping? I would have thought the necessary number of entries to be proportional to the total number of defined VLAN tags on each port: port 1 needs VLAN 10 and 20, port 2 needs VLAN 30, etc, so for a 9-port switch like the RB5009, I don't see why a practical inter-VLAN routing configuration would need more than a few dozen switch table entries.

Why the combinatorial explosion?

mkx · Fri Apr 08, 2022 9:56 pm

If we talk about switching VLAN-tagged traffic: switch chip needs to know which MAC is beyond which physical port and stores data in MAC Address Table. The only difference between "plain" switch and VLAN-enabled one is that VLAN-enabled can use one MAC Address Table per VLAN (some do it by default, others have it configurable).

When it comes to routing: router needs to know MAC address of next hop. If next hop is another router, then whole subnet (can be as large as /0) will have same MAC address of next hop, which means single entry in offloaded table. If next hop is device in connected network, each of destination IP addresses will have different MAC address.
And general CPU needs to fill the MAC/dst_network table before cerrain route can be offliaded because ASIC doesn't do ARP whohas procedures.

The bright side: if you're routing between 3 (or 5) /24 networks, only 3x2^8 (or 5x2^8) entries are needed ... so linear increase of number of routing entries, not exponential. Because router (in the narrowest possible meaning) only ever cares about next hop and doesn't care what was previous hop.

Fri Apr 08, 2022 10:07 pm

switch chip needs to know which MAC is beyond which physical port and stores data in MAC Address Table.

Even a cheap $20 NetGear switch has an 8k entry MAC address table, minimum. The Marvell product brief on this chip doesn't give that detail, but surely this 256-entry table raimondsp revealed is separate.

ASIC doesn't do ARP whohas procedures.

Yes, as mentioned above in the "gateway=" comment.

mkx · Fri Apr 08, 2022 10:13 pm

switch chip needs to know which MAC is beyond which physical port and stores data in MAC Address Table.

Even a cheap $20 NetGear switch has an 8k MAC address table. The Marvell product brief on this chip doesn't give that detail, but surely this 256-entry table raimondsp revealed is separate.

I guess that L2 MAC table is separate, it's still needed and used even with L3HW offload. The L3 stuff is all about determining next hop's MAC address (L2 switches don't do it, they use dst-MAC unchanged from frame received), but device still needs to do the L2 stuff afterwards (which is to deliver frame through correct egress port).

StubArea51 · Sat Apr 09, 2022 2:43 pm

highly doubt its real-life usage. It is not enough to offload Inter-VLAN routing even between two /24 subnets.

Please don't get me wrong - RB5009 is a great product for the price, and its quad-core CPU provides routing power that is more than enough in most cases. L3HW shines the most at 10G+ speeds, and that isn't the area of the product.

Thanks for clarifying, that makes a lot of sense now that we know the limitations of the ASIC. I still some value in developing it once other higher priority features in ROSv7 are addressed. There are a number of ISPs that want to use this in the last mile as a router that sits in transit to other routers which means it isn't directly facing L2 hosts. This is primarily due to the compact size and DC power options. With 10G PTP RF becoming more common, this could be a good use case.

Sat Apr 09, 2022 4:41 pm

switch chip needs to know which MAC is beyond which physical port and stores data in MAC Address Table.

Even a cheap $20 NetGear switch has an 8k entry MAC address table, minimum. The Marvell product brief on this chip doesn't give that detail, but surely this 256-entry table raimondsp revealed is separate.

ASIC doesn't do ARP whohas procedures.

Yes, as mentioned above in the "gateway=" comment.

in L3 HW offload thread in this forum was clarified that when using L3 HW acceleration for each host in a subnet you spend a L3 entry, because that doing L3 HW with only 256 entry available is not enough

Mon Apr 11, 2022 11:48 am

L2 MAC table is a separate one. RB5009 can store up to 16k MAC entries.

L3 HW Offload support on RB5009 [SOLVED]

L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009 [SOLVED]

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009

Re: L3 HW Offload support on RB5009