Problems with mangle-rules on RouterOS 7.12

OriiOn · Thu Nov 16, 2023 1:34 am

I am using two WAN connections on my router on interfaces eth11-WAN-A1 (backup WAN) and eth12-WAN-WE (primary WAN). I used the following 3 mangle rules to make sure that if someone accessed the services via eth11-WAN-A1, the outgoing packets would travel out through eth11-WAN-A1 as well:

/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=eth11-WAN-A1 new-connection-mark=MARK-WAN-A1 passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=MARK-WAN-A1 new-routing-mark=WAN-A1 passthrough=no
/ip firewall mangle add action=mark-routing chain=output connection-mark=MARK-WAN-A1 new-routing-mark=WAN-A1 passthrough=yes

On 7.12 if all 3 rules are active eth11-WAN-A1 would become unpingable and no more traffic going through. When disabling the second of these three rules, eth11-WAN-A1 is pingable, and works well again.

Why is the second rule causing problems on 7.12? This solution with mangle-rules used to work fine on RouterOS 6.
Is there a better solution to achieve my goal defined at the top without mangle rules?

anav · Thu Nov 16, 2023 3:12 am

Not sure why you have three rules it should be the first rule and one more............. combo of the other two.

/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=eth11-WAN-A1 new-connection-mark=MARK-WAN-A1 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=MARK-WAN-A1 new-routing-mark=WAN-A1 passthrough=no

OriiOn · Thu Nov 16, 2023 1:06 pm

Due to the mangle-rule troubles I have not upgraded my router yet to 7.12, but am still on 6.47.1

When doing what you suggest (disable rule #2, and remove the pass-through on rule #3), it appears as if the traffic is NOT routed out through the eth11-WAN-A1 ISP (which has a lower bandwidth), but instead through the primary ISP (with a higher bandwidth)

I test this by downloading a large file through the browser and look at the download speed. When disabling rule #2, the dl-speed is high, indicating it is going through the primary ISP.

Also, when I look at bytes/packets of rule #2 and #3, there appears to be 100x more packets handled by #2. So rule #2 appears to be important to get the job done. Unfortunatly rule #2 is the one causing the troubles on RouterOS 7.12

Any more suggestions what I can try?

anav · Thu Nov 16, 2023 2:57 pm

The only difference between 6 and 7, is that you need to add a table, and the extra route reflects the table not the routing mark.
Otherwise those two rules help ensure traffic that comes into WAN2 goes out WAN2.

Not sure why you think it wont..........
You could also try adding (if a static wanip)
/routing rule add src-address=wan2-IP action=lookup-only-in-table table=WAN-A1

OriiOn · Thu Nov 16, 2023 10:36 pm

I have set up a very simple configuration on a RB750Gr2, and tested with all releases of ROS7.

The following configuration works fine up until 7.2.1. I can ping WAN-2 IP successfully from a remote Windows machine.
When upgrading from 7.2.1 to 7.2.2 WAN-2 IP is no longer pingable. I receive "TTL expired in transit" when pinging from a Windows machine.

/interface ethernet set [ find default-name=ether4 ] name=eth11-WAN-2
/interface ethernet set [ find default-name=ether5 ] name=eth12-WAN-1

/routing table add disabled=no fib name=WAN-2
/ip address add address=88.xxx.xxx.106/30 comment="WAN-2" interface=eth11-WAN-2 network=88.xxx.xxx.104
/ip address add address=10.42.1.1/24 comment=LAN interface=bridge-local network=10.42.1.0

/ip route add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=88.xxx.xxx.105 routing-table=main scope=30 target-scope=10
/ip route add comment=WAN-2 disabled=no distance=2 dst-address=0.0.0.0/0 gateway=88.xxx.xxx.105 routing-table=WAN-2 scope=30 target-scope=10
/ip route add comment=WAN-1 disabled=no distance=1 dst-address=0.0.0.0/0 gateway=213.xxx.xxx.129 routing-table=main scope=30 target-scope=10

/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=eth11-WAN-2 new-connection-mark=MARK-WAN-2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=MARK-WAN-2 new-routing-mark=WAN-2 passthrough=no

When disabling the last mangle rule, I can obviously ping WAN-2 fine again.

anav · Thu Nov 16, 2023 11:51 pm

/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=eth11-WAN-2 new-connection-mark=MARK-WAN-2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=MARK-WAN-2 new-routing-mark=WAN-2 passthrough=no

OriiOn · Fri Nov 17, 2023 3:08 am

Thanks anav, I know you pointed me to chain=output right at the start of this thread. I tried that of course, however (at least on ROS6) it does NOT route the traffic out on WAN-2, as one would expect. Instead only chain=prerouting does that.

However for chain=prerouting to work properly, an additional routing rule is needed, to make sure only traffic outbound of the router is affected, and LAN traffic still uses the main routing table regardless. This rule makes sure of that:

/routing rule add action=lookup-only-in-table dst-address=10.42.0.0/16 table=main

So in total, these are the rules I am using on ROS6 to successfully make sure incoming traffic on WAN-2 will also exit on WAN-2

/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=eth11-WAN-2 new-connection-mark=MARK-WAN-2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=MARK-WAN-2 new-routing-mark=WAN-2 passthrough=no

/routing rule add action=lookup-only-in-table dst-address=10.42.0.0/16 table=main

Unfortunately this set of rules stopped working with ROS 7.2.2 -> I am getting a "TTL expired in transit" when pinging WAN-2 from a remote Windows machine.

LdB · Fri Nov 17, 2023 6:39 pm

I do your trick of putting the 3 rules because I got sick of what is obviously a bug in OS7 that either the pre-routing or the output mark routing doesn't work. I generally need it for out of band GSM access and I want traffic from the GSM to go back out the GSM and got that problem a lot.

If neither mark routing works ... reboot the router and generally one starts working

OriiOn · Sat Nov 18, 2023 3:39 am

Reboot does not work for me. In fact the issue is 100% reproducible (eth11-WAN-2 becomes unpingable). The configuration works fine up until 7.2.1 and stops working after 7.2.2. This can be reproduced with a very simple configuration. Tried and tested on RB1100, CCR1036 and RB750r2.

LdB · Sat Nov 18, 2023 9:35 am

I am on different hardware mainly CCR2004, CVC2116 which are ARM64 based and it's just annoying not fatal as per your hardware

OriiOn · Sat Nov 18, 2023 11:32 am

I am pretty certain this is not a hardware issue, but a software issue. What ROS version are you on?

Guscht · Sat Nov 18, 2023 11:56 am

I would read it this way:

/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=eth11-WAN-A1 new-connection-mark=MARK-WAN-A1 passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=MARK-WAN-A1 new-routing-mark=WAN-A1 passthrough=no
/ip firewall mangle add action=mark-routing chain=output connection-mark=MARK-WAN-A1 new-routing-mark=WAN-A1 passthrough=yes

A packet enters the router through eth11 and gets a conn-mark.
This packet gets passed to the next rule (1st rule = passthrough = yes).

The second rule matches, because the conn-mark from first rule is matched.
This packet is now placed into the routing table "WAN-A1". I assume in this table is route with a default desination (0.0.0.0/0) and so will always match (wont fall back to the main-table).

In other words: A packet enters from WAN to the router and goes straight back to WAN...
Is this the intended behaviour? If not, try to set in the first rule passthrough=no.

OriiOn · Sat Nov 18, 2023 1:40 pm

Thanks for your reply Guscht!

Here's the correct and complete setup that works - up until ROS 7.2.1

/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=eth11-WAN-2 new-connection-mark=MARK-WAN-2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=MARK-WAN-2 new-routing-mark=WAN-2 passthrough=no

/routing rule add action=lookup-only-in-table dst-address=<LAN> table=main

This setup stopped working for me with ROS 7.2.2 and higher.

Shon3 · Sun Jun 16, 2024 11:01 pm

EDIT: SOLVED.

The sudden connection speed after upgrade to v7.12 led me to a bit more googling. It seems, that Fasttrack is not compatible with Mangle. After applying Fasttrack firewall rule only to connections with "no connection mark", all started to work again

And it also means that Fasttrack did not work in v7.6 on Mikrotik hEX (model RB750Gr3)

===
@OriiOn

Have you resolved the problem?

I am facing exactly the same issue with model RB750Gr3 - Routing Marks worked till v7.6. and stopped working on v7.12.

(BTW, I can see a significant connection speed update - from max 200 Mbps on v7.6 up to 700 Mbps on v7.12.)

wg0 - Wireguard tunnel. The objective is that any connection comming from wg0 to Mikrotik must also return via wg0.
New connections outgoing from LAN must go via main routing table.

IP firewall mangle rules:

;;; Connection-mark for incoming wg0
      chain=prerouting action=mark-connection new-connection-mark=rtr1-conn passthrough=yes connection-mark=!rtr1-conn in-interface=wg0"
;;; Routing mark rtr1 for returning communication
      chain=prerouting action=mark-routing new-routing-mark=rtr1 passthrough=yes connection-mark=rtr1-conn in-interface=!wg0"

IP/Route:

dst-address=0.0.0.0/0 routing-table=rtr1 pref-src="" gateway=wg0 immediate-gw=wg0 distance=1 scope=30 target-scope=10 suppress-hw-offload=no

Routing/table:

;;; table for wg0 replies to incoming connections
       name="rtr1" fib

I have worked-around the bug by creating a routing policy rule to route any communication from my LAN server 10.11.13.21 via wg0 temporarily.

src-address=10.11.13.21/32 action=lookup table=rtr1

When I add filter for routing mark `rtr1`, this routing rule stops working:

src-address=10.11.13.21/32 routing-mark=rtr1 action=lookup table=rtr1

So either the routing mark is not applied or it is lost.

anav · Sun Jun 16, 2024 11:43 pm

Shon post complete config and will look.

/export file=anynameyouwish ( minus router serial #, any public WANIP information, keys etc.)

Shon3 · Mon Jun 17, 2024 7:37 pm

Shon post complete config and will look.

The issue is solved now - see my EDIT in comment above.
The solution came from this thread: viewtopic.php?t=200074

Problems with mangle-rules on RouterOS 7.12

Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12

Re: Problems with mangle-rules on RouterOS 7.12