Routing/Mangle issues with multiple WG tunnels

pawlisko · Mon Jun 20, 2022 5:35 pm

Hi,
I was about to make some changes to my setup (using 2 providers with failover) but I saw that my setup was not working.
I have 4 tunnels (4 different countries), I do routing 2 ways:
a) I sent traffic to 3 tunnels using IP rages (based on country IP allocation) - it created about 23k address entires
b) I use local address lists to force entire traffic from certain computers to specific tunnels (I use enable/disable addresses as it makes it easier to route certain IP at times)

It was working but it stopped - I assume maybe issue with rOS?. Code below. Please assume that wireguard is set up properly.

/routing table add comment="Table for WireGuard - Poland" disabled=no fib name=wg-pl
/routing table add comment="Table for WireGuard - Germany" disabled=no fib name=wg-de
/routing table add comment="Table for WireGuard - UK" disabled=no fib name=wg-uk
/routing table add comment="Table for WireGuard - France" disabled=no fib name=wg-fr
/ip route add dst-address=0.0.0.0/0 gateway=KeepSolidVPN-UK routing-table=wg-uk
/ip route add dst-address=0.0.0.0/0 gateway=KeepSolidVPN-France routing-table=wg-fr
/ip route add dst-address=0.0.0.0/0 gateway=KeepSolidVPN-Germany routing-table=wg-de
/ip route add dst-address=0.0.0.0/0 gateway=KeepSolidVPN-Poland routing-table=wg-pl
/ip firewall nat add action=masquerade chain=srcnat out-interface=KeepSolidVPN-Poland
/ip firewall nat add action=masquerade chain=srcnat out-interface=KeepSolidVPN-Germany
/ip firewall nat add action=masquerade chain=srcnat out-interface=KeepSolidVPN-UK
/ip firewall nat add action=masquerade chain=srcnat out-interface=KeepSolidVPN-France
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=VPN-IP-PL passthrough=yes src-address-list=local-pl
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=VPN-IP-UK passthrough=yes src-address-list=local-uk
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=VPN-IP-FR passthrough=yes src-address-list=local-fr
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=VPN-IP-DE passthrough=yes src-address-list=local-de
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=IP-Poland new-connection-mark=VPN-IP-PL passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=IP-Germany new-connection-mark=VPN-IP-DE passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=IP-UK new-connection-mark=VPN-IP-UK passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=VPN-IP-DE in-interface=LAN new-routing-mark=wg-de passthrough=no
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=VPN-IP-UK in-interface=LAN new-routing-mark=wg-uk passthrough=no
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=VPN-IP-FR in-interface=LAN new-routing-mark=wg-fr passthrough=no
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=VPN-IP-PL in-interface=LAN new-routing-mark=wg-pl passthrough=no

When the computer is in local lists with this scenario, it can't even log in to MT router

I tried route list:

/routing rule add action=lookup disabled=no src-address=Internal-IP/32 table=wg-de

But it is not reliable all the time

Any advice?

sindy · Mon Jun 20, 2022 6:45 pm

I would assume it is related to this recent change in RouterOS behaviour.

Try to add dst-address-type=!local to all your action=mark-routing rules, it should fix the issue of inability to access the router itself.

pawlisko · Tue Jun 21, 2022 6:00 am

I would assume it is related to this recent change in RouterOS behaviour.

Try to add dst-address-type=!local to all your action=mark-routing rules, it should fix the issue of inability to access the router itself.

I think you may be right.
So as you know I want to change setup for WG for failover etc, but I need to make sure that current works. Below is the current code

/routing table add comment="Table for WireGuard - Poland - ProtonVPN" disabled=no fib name=wg-pl-proton
/routing table add comment="Table for WireGuard - Poland - KeepSolidVPN" disabled=no fib name=wg-pl-ks
/routing table add comment="Table for WireGuard - Germany - ProtonVPN" disabled=no fib name=wg-de-proton
/routing table add comment="Table for WireGuard - Germany - KeepSolidVPN" disabled=no fib name=wg-de-ks
/routing table add comment="Table for WireGuard - UK - ProtonVPN" disabled=no fib name=wg-uk-proton
/routing table add comment="Table for WireGuard - UK - KeepSolidVPN" disabled=no fib name=wg-uk-ks
/routing table add comment="Table for WireGuard - P2P - ProtonVPN" disabled=no fib name=wg-p2p-proton
/routing table add comment="Table for WireGuard - P2P - KeepSolidVPN" disabled=no fib name=wg-p2p-ks
/ip firewall nat add action=masquerade chain=srcnat out-interface=ProtonVPN-Poland
/ip firewall nat add action=masquerade chain=srcnat out-interface=ProtonVPN-Germany
/ip firewall nat add action=masquerade chain=srcnat out-interface=ProtonVPN-UK
/ip firewall nat add action=masquerade chain=srcnat out-interface=ProtonVPN-P2P
/ip firewall nat add action=masquerade chain=srcnat out-interface=KeepSolidVPN-Poland
/ip firewall nat add action=masquerade chain=srcnat out-interface=KeepSolidVPN-Germany
/ip firewall nat add action=masquerade chain=srcnat out-interface=KeepSolidVPN-UK
/ip firewall nat add action=masquerade chain=srcnat out-interface=KeepSolidVPN-P2P
/ip route add dst-address=0.0.0.0/0 gateway=ProtonVPN-Poland routing-table=wg-pl-proton
/ip route add dst-address=0.0.0.0/0 gateway=ProtonVPN-Germany routing-table=wg-de-proton
/ip route add dst-address=0.0.0.0/0 gateway=ProtonVPN-UK routing-table=wg-uk-proton
/ip route add dst-address=0.0.0.0/0 gateway=ProtonVPN-P2P routing-table=wg-p2p-proton
/ip route add dst-address=0.0.0.0/0 gateway=KeepSolidVPN-Poland routing-table=wg-pl-ks
/ip route add dst-address=0.0.0.0/0 gateway=KeepSolidVPN-Germany routing-table=wg-de-ks
/ip route add dst-address=0.0.0.0/0 gateway=KeepSolidVPN-UK routing-table=wg-uk-ks
/ip route add dst-address=0.0.0.0/0 gateway=KeepSolidVPN-P2P routing-table=wg-p2p-ks
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=VPN-IP-PL passthrough=yes src-address-list=local-pl
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=VPN-IP-DE passthrough=yes src-address-list=local-de
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=VPN-IP-UK passthrough=yes src-address-list=local-uk
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=VPN-IP-P2P passthrough=yes src-address-list=local-p2p
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=IP-Poland new-connection-mark=VPN-IP-PL passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=IP-Germany new-connection-mark=VPN-IP-DE passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=IP-UK new-connection-mark=VPN-IP-UK passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting comment="WG - Poland - ProtonVPN" connection-mark=VPN-IP-PL dst-address-type=!local in-interface=LAN new-routing-mark=wg-pl-proton passthrough=no
/ip firewall mangle add action=mark-routing chain=prerouting comment="WG - Germany - ProtonVPN" connection-mark=VPN-IP-DE dst-address-type=!local in-interface=LAN new-routing-mark=wg-de-proton passthrough=no
/ip firewall mangle add action=mark-routing chain=prerouting comment="WG - UK - ProtonVPN" connection-mark=VPN-IP-UK dst-address-type=!local in-interface=LAN new-routing-mark=wg-uk-proton passthrough=no
/ip firewall mangle add action=mark-routing chain=prerouting comment="WG - P2P - ProtonVPN" connection-mark=VPN-IP-P2P dst-address-type=!local in-interface=LAN new-routing-mark=wg-p2p-proton passthrough=no
/ip firewall mangle add action=mark-routing chain=prerouting comment="WG - Poland - KeepSolidVPN" connection-mark=VPN-IP-PL dst-address-type=!local in-interface=LAN new-routing-mark=wg-pl-ks passthrough=no
/ip firewall mangle add action=mark-routing chain=prerouting comment="WG - Germany - KeepSolidVPN" connection-mark=VPN-IP-DE dst-address-type=!local in-interface=LAN new-routing-mark=wg-de-ks passthrough=no
/ip firewall mangle add action=mark-routing chain=prerouting comment="WG - UK - KeepSolidVPN" connection-mark=VPN-IP-UK dst-address-type=!local in-interface=LAN new-routing-mark=wg-uk-ks passthrough=no
/ip firewall mangle add action=mark-routing chain=prerouting comment="WG - P2P - KeepSolidVPN" connection-mark=VPN-IP-P2P dst-address-type=!local in-interface=LAN new-routing-mark=wg-p2p-ks passthrough=no

As you can see I added your suggestion - dst-address-type=!local

What I have now is weird behavior.

/ip firewall address-list add address=local-ip list=local-p2p

No internet access.

/ip firewall address-list add address=local-ip list=local-uk
/ip firewall address-list add address=local-ip list=local-pl
/ip firewall address-list add address=local-ip list=local-de

Works as designed

Previously I was not able to use lists -uk, -pl. -de. Now I can, but still can't use -p2p

As you will see the code - the difference is there is no destination list IP-P2P. I have IP-Poland, IP-Germany, IP-UK.

Any thoughts?

sindy · Tue Jun 21, 2022 8:15 am

dst-address-type=!local matches on any destination address except the own ones of the router. So adding this match condition to the action=mark-routing rules prevents packets from LAN hosts towards the router itself from being sent to some WG tunnel. If this helped, it means to me that the LAN hosts use the router itself as their DNS server, so until that modification, their DNS queries never got responded so they were reporting "no internet access".

How does adding local-ip to the local-p2p list affect internet access is unclear to me, especially as you haven't clarified what the local-ip stands for. A /32 IP of a LAN host you are testing from? A whole LAN subnet? Do you indeed use the router as a DNS server for the LAN hosts, or do you use some other DNS server (like PiHole) in LAN?

pawlisko · Tue Jun 21, 2022 2:21 pm

dst-address-type=!local matches on any destination address except the own ones of the router. So adding this match condition to the action=mark-routing rules prevents packets from LAN hosts towards the router itself from being sent to some WG tunnel. If this helped, it means to me that the LAN hosts use the router itself as their DNS server, so until that modification, their DNS queries never got responded so they were reporting "no internet access".

How does adding local-ip to the local-p2p list affect internet access is unclear to me, especially as you haven't clarified what the local-ip stands for. A /32 IP of a LAN host you are testing from? A whole LAN subnet? Do you indeed use the router as a DNS server for the LAN hosts, or do you use some other DNS server (like PiHole) in LAN?

So the first one makes sense as my MT is my DNS server - I am using DoH.

Second - local-ip stands for A /32IP of a LAN host I am testing. For quick testing I am using AppleTV with Speedtest app. Whenever I am using WAN (no tunnels), UK tunnel, PL tunnel, DE tunnel - it works, but it is now working with P2P tunnel

sindy · Tue Jun 21, 2022 3:45 pm

Whenever I am using WAN (no tunnels), UK tunnel, PL tunnel, DE tunnel - it works, but it is now working with P2P tunnel

I suppose you wanted to write not rather than now above.

Given that it behaves oddly, I'd first try to swap the routing marks in the rules to find out whether the issue is related to use of the ProtonVPN-P2P tunnel itself or to the connection marking and routing marking.

Next, I'd run /ip firewall connection print detail interval=1s where connection-mark=VPN-IP-P2P to see whether the connections get created - maybe ProtonVPN-P2P is just missing among permitted out-interfaces in /ip firewall filter rules?

In another words - the part of configuration you have provided seems allright to me, but there may be something elsewhere that prevents traffic via ProtonVPN-P2P from getting through.

/tool sniffer quick interface=ProtonVPN-P2P is the next step if the above test shows that connections get created but repl-packets counters stay at 0 for all of them.

pawlisko · Tue Jun 21, 2022 8:22 pm

Given that it behaves oddly, I'd first try to swap the routing marks in the rules to find out whether the issue is related to use of the ProtonVPN-P2P tunnel itself or to the connection marking and routing marking.

So P2P connection was showing as working but just in case I created another connection with new info and it worked.

So it is solved.

Thank you

Routing/Mangle issues with multiple WG tunnels [SOLVED]

Routing/Mangle issues with multiple WG tunnels

Re: Routing/Mangle issues with multiple WG tunnels [SOLVED]

Re: Routing/Mangle issues with multiple WG tunnels

Re: Routing/Mangle issues with multiple WG tunnels

Re: Routing/Mangle issues with multiple WG tunnels

Re: Routing/Mangle issues with multiple WG tunnels

Re: Routing/Mangle issues with multiple WG tunnels