Policy based routing

Shon3 · Thu Dec 28, 2023 7:31 pm

Hi,

In RouterOS 7.6, I would like to setup a route, which would catch all reply packets RELATED to connection coming from a given interface wg1 and route such packets via wg1 rather than the default route.

In other words, if an incoming connection appears on iface wg1, I need the response to be routed via wg1 again. The response will be addressed to any public IP (0.0.0.0/0). Otherwise, the mikrotik's default route is set to ISP's CGNAT on inteface wan0.

On linux, this can be achieved with this:

iptables -t mangle -I PREROUTING 1 -i wg1 -m conntrack --ctstate NEW -j CONNMARK --set-mark 2
iptables -t mangle -I PREROUTING 2 -m connmark --mark 2 -j CONNMARK --restore-mark  
ip rule add fwmark 2 lookup 2
ip route add default dev wg1 table 2

I have already tried to mark packets with routing-mark, which seems to work (count is increasing), and other steps as follows (gw 10.0.37.1 can be pinged, it is the other peer in wireguard tunnel on wg1).

/ip firewall mangle add chain=prerouting in-interface=wg1 action=mark-connection new-connection-mark=rtr1 passthrough=yes
/ip firewall mangle add chain=prerouting connection-mark=rtr1 action=mark-routing new-routing-mark=rtr1 passthrough=yes
/routing/table add name="rtr1" fib
/routing/rule/add routing-mark=rtr1 table=rtr1 action=lookup
/ip/route/add dst-address=0.0.0.0/0 gateway=10.0.37.1 routing-table=rtr1

Needless to say, the routing does not work. I would hope that routing table rtr1 would be used for any packets marked with routing mark rtr1, but something does not work.

Any idea?

Sob · Thu Dec 28, 2023 7:49 pm

Almost there. You need to limit the route marking rule, because this one applies also to incoming packets. As a result, they will be sent back, because route marks have maximum priority in ROS. Either that, or you'd need routes to local destinations in rtr1 table. The former is probably easier/simpler.

Same thing: viewtopic.php?t=202827

Edit: And you don't need the routing rule, it's implicit.

anav · Thu Dec 28, 2023 9:28 pm

Not understanding the additional complexity.

a. need route back into tunnel
created automatically on MT router by the use of the ip address of the wireguard on the router creates a DAC route.
Thus any incoming traffic from a source with wireguard IP, already has a route back........

b. Need a firewall rule allowing traffic exiting the wireguard tunnel to reach LAN devices.

c. ONLY NEED TO add rules, for any remote subnets coming in to access LAN devices.

So thus far not sure of the need for any routing rules or mangling let alone AND mangling.

Shon3 · Thu Dec 28, 2023 9:44 pm

Thanks for the hints. You were right, that this rule:

/ip firewall mangle add chain=prerouting connection-mark=rtr1 action=mark-routing new-routing-mark=rtr1 passthrough=yes

actually catched the incoming packets.

However, my setup is a bit more complicated - the remote client (78.136.141.22) connecting via WireGuard tunnel (10.0.37.3@wg1) actually is another WireGuard tunnel... So, the target computer for this connection is Mikrotik itself.

To find out more, I have added a new fw rule:

/ip firewall mangle add chain=output connection-mark=rtr1 action=mark-routing new-routing-mark=rtr1 passthrough=yes

and temporarilly added a manual route with dst-address=(IP of the remote client) to the `main` table:

/ip/route/add dst-address=78.136.141.22 gateway=wg1 routing-table=main

Then, everything works, the second tunnel opens and I can access services provided by the network of the remote client at 78.136.141.22.
The packet sniffer shows:

# Iface  Direction  Src addr    Src port  Dst Addr    Dst Port 
1 wg1           rx  78.136.141.22  37600  10.0.37.3      51872
2 wg1           tx  10.0.37.3      51872  78.136.141.22  37600

The counter in firewall rules for "mark routing" increases only for OUTPUT chain and does not change for PREROUTING chain. This seems correct.

Then, when I change the route from:

/ip/route/add dst-address=78.136.141.22 gateway=wg1 routing-table=main

to

/ip/route/add dst-address=78.136.141.22 gateway=wg1 routing-table=rtr1

then the connection breaks, the fw counter for "OUTPUT" chain does not increasy anymore and the packet sniffer shows that the replies (row 2 above) are being sent out via default route from main table...
I deduce that I have to find out a proper firewall rule to apply routing-mark on the reply packets (only):
- OUTPUT chain of mangle table has actually been marking packets, which were already routed via gateway wg1.
- PREROUTING chain of mangle table has only been marking incoming packets from wg1.

Sob · Thu Dec 28, 2023 10:19 pm

If the target is router itself, then output chain (instead of prerouting) is the right one for handling response packets, that's correct.

It should just work without any extra routes. That's if you need to deal only with incoming connections. First rule in prerouting marks incoming connections (you can add connection-mark=no-mark or connection-state=new to it, there's no need to re-mark it over and over with each packet). Then the one in output sets routing mark, which makes it use rtr1 routing table. And the only route there is the one via 10.0.37.1, which covers all destinations. So everything should be fine.

But it's possible that I'm missing something in that part about another WG tunnel, that's not entirely clear. If 78.xxx is another WG client connecting to server on router, but going through first tunnel, it may be possible that it's not only incoming connection. Because WG works both ways and if the connection breaks (there could be timeout when it's not used; keepalive should prevent that) and server has packet for client, it will try to contact it. So that would be outgoing connection. And it would use default routing table.

anav · Thu Dec 28, 2023 10:27 pm

A network diagram would allow us to better see through the fog of your explanation and the missing pieces which should have been provided up front.

Shon3 · Thu Dec 28, 2023 11:15 pm

Another try to find out if route marking works for me as it should.

I set route from main to rtr1 table:

/ip/route/add dst-address=78.136.141.22 gateway=wg1 routing-table=rtr1

Tunnel stops working, packets to 78.136.141.22 are sent via default route instead via gw1.

This time, this new modified fw rule

/ip firewall mangle add chain=output dst-address=78.136.141.22 action=mark-routing new-routing-mark=rtr1 passthrough=yes

continues to match the outgoing packets (counter increases) and presumably marks them with rtr1 routing mark.

But this is ignored by the route based on table rtr1, although after "mangle - output" there should be another routing adjustment as shown in this nice diagram: viewtopic.php?p=768157#p768157

So, it seems I need to look at the route rules.

Although as per earlier info in this thread the rule for rtr1 routing mark applied by fw should not be necessary, let's try it:

/routing/rule/add routing-mark=rtr1 action=lookup-only-in-table table=rtr1

And nothing happens, the packets are still routed via default route.

Let's change the rule to match dst-address instead:

/routing/rule/add dst-address=78.136.141.22  action=lookup-only-in-table table=rtr1

A voila - connection starts working again, the packets are routed via wg1.

P.S. Drawing a network diagram with a tunnel inside a tunnel might be a bit demanding for skills and imagination, but I will surely try to do so, probably on Saturday.

anav · Thu Dec 28, 2023 11:42 pm

Sorry I dont see tunnel within a tunnel at all............ All the more reason for network diagram LOL

Okay Can you confirm the source address coming on wireguard is not limited to.
a. single wireguard source addresses ( like from individual WG users )
b. private subnets from another device connected via WG.

THus, that leaves very little possibilities for this instance. One possibility that makes sense is talking about source addresses that could be described as 0.0.0.0/0
Thus you need 0.0.0.0/0 as the allowed IPs on the settings on the client MT device on its peer connection to a server WG device.
Example port forwarding from WG server device into WG tunnel toward servers on MT device.

In its simplest form.......
/routing table add fib name=useWG
/ip route
add dst-address=0.0.0.0/0 gateway=wireguard1 routing-table=useWG

/ip nat mangle
add chain=prerouting action=mark-connections connection-mark=no-mark
new-connection-mark=incoming-WG passthrough=yes in-interface=wireguard1
add chain=prerouting action=mark-routing connection-mark=incoming-WG \
new-routing-mark=useWG passthrough=no

PS I would upgrade to 7.13 to use wireguard, or at least 7.12

Sob · Fri Dec 29, 2023 2:47 am

Few things:

- If in rtr1 table you still have route to 0.0.0.0/0, as you did in first post and it was correct, then adding any other route there with same gateway is pointless, because the first one already covers any possible destination.

- Same for the rule to look up destination for packets with rtr1 mark in rtr1 table, that happens automatically.

- The last rule applies to packets without any routing mark (right now I'm not sure if only to them, but that's not important right now). It it makes a difference, then it seems that some packets to 78.xx are not part of incoming connections, but belong to connections initiated by router. You can check it:

/ip firewall mangle
add chain=output dst-address=78.136.141.22 connection-mark=rtr1 action=log log-prefix=marked
add chain=output dst-address=78.136.141.22 connection-mark=!rtr1 action=log log-prefix=unmarked

Shon3 · Sat Dec 30, 2023 2:01 pm

While drawing the diagrams, I have eventually decided on simplifying the network setup to minimise the complications.

Thanks a lot, @Sob, for feedback re. the routing marks - I will re-use the info in the new setup and if I fail, I may come back again

All the best to 2024!

Policy based routing

Policy based routing

Re: Policy based routing

Re: Policy based routing

Re: Policy based routing

Re: Policy based routing

Re: Policy based routing

Re: Policy based routing

Re: Policy based routing

Re: Policy based routing

Re: Policy based routing