There is a larger thread in the general discussion where multiple people have reported the same issue but I didn't see a detailed post in the bug section outlining the issue so I'll try and make this as clean as possible to get the quickest resolution (either it is a bug, or its the intended functionality). Any thoughts are greatly appreciated, also here is the link to the larger thread where some other attempts have been made and also similar examples of it not working for other people.

Main Thread (Start of my post): viewtopic.php?p=985420#p984701
Other Example: viewtopic.php?p=985420#p929511
Other Example: viewtopic.php?p=985420#p963157

Version: 7.8rc2 (but tested on 7.7 stable and 7.8rc1 and it doesn't work there either)
Model: CCR2216-1G-12XS-2XQ
Notes:
1) No firewall rules blocking anything, just the accept rule for fasttrack.
2) In a more complex setup I am able to get non-VLAN traffic from another port on the CCR (like sfp28-2) to any VLAN on the bridge to be offloaded. But the key is that inter-VLAN traffic from a VLAN on the bridge to another VLAN on the bridge is not offloaded.
Steps to Reproduce:
1) Setup a downstream switch with multiple VLANs that will be trunked from a single port on the switch to a single port on the CCR. For example have a switch with 8 GbE ports and 2 SFP+ ports. Access ports for individual VLANs on ports 2-5 of the switch correspond to VLANs 2-5 and have them all trunked to a single SFP+ uplink that you connect to one of the CCR's sfp28 ports.
2) Setup a bridge on the CCR.
3) Add a bridge port with the desired sfp28 interface.
3) Create bridge VLAN entry with the VLAN's 3-5 tagged and the sfp28 port as well as bridge tagged.
4) Attempt to pass traffic (iperf3 or any traffic SMB/FTP etc.) from a workstation on VLAN 3 to another workstation on VLAN 4.
--- Expected functionality, is that within the appropriate connection limits etc, the traffic should be offloaded as per the documentation on L3HW offloading. Actual functionality is that the traffic is not offloaded.

Simple Config:

try removing this EDIT
and maybe try adding this

Unfortunately, nothing changed. Traffic is still not offloaded.

Try rebooting

Try disabling ando enabling Global L3 hw offload on switch

Maybe reboot again

Unfortunately, still no dice.

Try rebooting

Try disabling ando enabling Global L3 hw offload on switch

Maybe reboot again

One thing that would help to disambiguate:

l3 hw offload - stateless offload of IPv4/IPv6 routes into hardware
l3 fw offload - stateful offload of IPv4 connections and NAT (IPv6 fastpath/fasttrack yet to be implemented)

Brief list of what we discovered with fw offload in our lab and in prod for an ISP CG-NAT gw. More details in this thread: viewtopic.php?t=183142#p985541

Limitations of fw-offload (as of 7.6) -

- No LACP
- No VLANs
- No hairpinning on a single port
- must have dedicated in and out physical interfaces

Config requirements (once you've removed limitations above:

Must have l3-hw-offloading=yes on the switch chip *but* disabled on all ports
Must create mangle rules to use fasttrack on connections that you want to offload to hardware

MikroTik may be able to comment if the gap on these limitations has closed between 7.6 and 7.8rc2.

Updated title to reflect the situation of the issue clearly. Specifically, the FW compatible L3HW offload is not working with inter-vlan traffic. Simply doing L3HW offloading with no firewall expectations which is basically L2 switching + VLAN (L3) making it L3HW connected routes offloaded switching is not what I'm referring to in my post. This is soley about firewall compatible L3HW offloading (stateful firewall + NAT).

One thing that would help to disambiguate:

l3 hw offload - stateless offload of IPv4/IPv6 routes into hardware
l3 fw offload - stateful offload of IPv4 connections and NAT (IPv6 fastpath/fasttrack yet to be implemented)

Brief list of what we discovered with fw offload in our lab and in prod for an ISP CG-NAT gw. More details in this thread: viewtopic.php?t=183142#p985541

Limitations of fw-offload (as of 7.6) -

- No LACP
- No VLANs
- No hairpinning on a single port
- must have dedicated in and out physical interfaces

Config requirements (once you've removed limitations above:

Must have l3-hw-offloading=yes on the switch chip *but* disabled on all ports
Must create mangle rules to use fasttrack on connections that you want to offload to hardware

MikroTik may be able to comment if the gap on these limitations has closed between 7.6 and 7.8rc2.

This implies that if i use L3 FW offload I can't use L3 HW offload too, FW and HW simultaneously ?

This implies that if i use L3 FW offload I can't use L3 HW offload too, FW and HW simultaneously ?

I would say that's accurate because in fw offload you have to disable hw-offload on the port level (but enabled at the switch level) so the only way traffic is getting hw-offloaded is if it hits a mangle rule that enables fasttrack. Presumably this is so that mangle can enable/disable offload per flow instead of per port.

It could be that development hasn't reached a point where they can be used together and not a permanent limitation.

@IPANetEngineer

thank you for your answer

You can use them simultaneously, just you need to set the switch (port level) hw-offloading to enabled. But in doing so for that particular interface port you will not be able to use FW L3HW offload and HW offload at the same time. So in short, per documentation, at the port level you can only use one or the other but at the device level (CCR) you can have some ports that do L3HW FW offload and some that do L3HW connected routes offload.

That's prob a better way to say it.

More accurately - you can't use them together on the same set of ports.

Any thoughts on this, is it possible for a MikroTik engineer to replicate to at least confirm/deny if this is a bug or not?

this is a users forum

you must open a ticket with support:


https://help.mikrotik.com/servicedesk/servicedesk

I sent an email (as I don't have access to this servicedesk). Unfortunately nothing as of yet. Any possibility for any Mikrotik people to try and setup two VLANs on a bridge with each respective VLAN assigned to a bridge port that is associated with a different underlying physical interface and verify that the L3HW offloading works for -> inter-VLAN <- traffic? Additionally, I can also confirm that just having 2 VLANs on a bridge that is associated with the same underlying physical interface also doesn't work for L3HW offloading. This is all tested on 7.8 stable.

Also if any forum members know of a version of RouterOS where L3HW offloading was or is working for inter-vlan traffic please let me know as I would like to downgrade and test.

just today i deployed a CRS 317 doing inter VLAN routing with L3 HW offload 2 gbps of traffic with 1% of CPU usage, working ok with Ros 7.6

in my profile you can contact me to help you to solve the issue

My crs317 with 7.8 works fine with inter and intra vlan traffic on the same or different interfaces.

Do either of you mind posting a sanitized network config from your CRS that show how you are doing your L3HW offloading I might be able to glean something from it. I suspect maybe there is an issue between how the CRS is doing it versus the CCR.

Jesus, after a ridiculous amount of testing I figured out the problem. If you have IP-Firewall set as active and then set either of the two options like "Use IP Firewall For VLAN" or "Use IP Firewall For PPPoE" set, then it doesn't work. And I was told by a Mikrotik member to disable IP-Firewall and I did and there was no change. However, this is due to a bug, you need to first disable "Use IP Firewall For VLAN" and "Use IP Firewall For PPPoE" while the parent option "Use IP Firewall" is enabled and then disable the parent option "Use IP Firewall". Otherwise even though IP Firewall must be enabled for either of the lower two options to be selectable, disabling the parent option doesn't actually disable the other options so somewhere in the code the router still thinks the "Use IP Firewall For VLAN" and "Use IP Firewall For PPPoE" are still enabled even when the parent option "Use IP Firewall" is disabled.

Long story short, if you want L3HW FW Compatible Offloading, disable "Use IP Firewall" and then if you don't see "Bridge Fast Path Active" checked (make sure Allow Fast Path is checked as well), then try the below steps and when it becomes checked then you know you've got it working.

1) Enable Use IP Firewall (if it isn't already).
2) Disable Use IP Firewall For VLAN
2) Disable Use IP Firewall For PPPoE
3) Disable Use IP Firewall
4) Restart router.

@Mikrotik Mods, this thread can be closed since the problem is solved.

Jesus, after a ridiculous amount of testing I figured out the problem. If you have IP-Firewall set as active and then set either of the two options like "Use IP Firewall For VLAN" or "Use IP Firewall For PPPoE" set, then it doesn't work. And I was told by a Mikrotik member to disable IP-Firewall and I did and there was no change. However, this is due to a bug, you need to first disable "Use IP Firewall For VLAN" and "Use IP Firewall For PPPoE" while the parent option "Use IP Firewall" is enabled and then disable the parent option "Use IP Firewall". Otherwise even though IP Firewall must be enabled for either of the lower two options to be selectable, disabling the parent option doesn't actually disable the other options so somewhere in the code the router still thinks the "Use IP Firewall For VLAN" and "Use IP Firewall For PPPoE" are still enabled even when the parent option "Use IP Firewall" is disabled.

Long story short, if you want L3HW FW Compatible Offloading, disable "Use IP Firewall" and then if you don't see "Bridge Fast Path Active" checked (make sure Allow Fast Path is checked as well), then try the below steps and when it becomes checked then you know you've got it working.

1) Enable Use IP Firewall (if it isn't already).
2) Disable Use IP Firewall For VLAN
2) Disable Use IP Firewall For PPPoE
3) Disable Use IP Firewall
4) Restart router.

@Mikrotik Mods, this thread can be closed since the problem is solved.

I'm glad that L3HW FW-Compatible Offloading finally works on your side. We have identified an issue where disabling "Use IP Firewall" does not entirely disable the firewall if VLAN or PPPoE options are still enabled. It will be fixed in future releases. Meanwhile, disable all three options to make FastPath work.

Thank you for the feedback!