RouterOS blatantly ignores pref-src. Can this really be a bug?

divB · Thu Nov 28, 2024 2:31 am

This is a followup question from viewtopic.php?t=212807 but posting new topic because original question is resolved.

Per request, I am starting with a network diagram:

Screenshot 2024-11-27 at 16.27.59.png

RouterOS runs a wireguard "server" for a road warrior setup. Sadly it's not possible to "bind" wireguard to specific interfaces, so they "listen" on any IP address. The road warrior uses 192.0.2.210:52810 as an endpoint. Now every reasonable person would expect that the response packet from such a request would have source address 192.0.2.210. Sadly RouterOS is always up for surprises and that's not the case. I assume due to an obscure implementation of wireguard, the source address is left unset and hence the operating system needs to decide what should be the source address for the locally generated packet. For me this is 192.0.2.177. Now you can imagine already that this breaks everything: The road warrier client requested a connection to 192.0.2.210 but gets a response from 192.0.2.177. It thinks this is MIM and discards the packet.

Now there is "pref-src" is exactly for this: It should determine the source address of a locally generated packet based on the route that is chosen. In my case the route comes from BGP and I could add a router filter to add pref-src. But for simplicity (and because it didn't work), I explicitly add a static route with low distance and pref-src set to 192.0.2.210:

/ip/route/add dst-address=0.0.0.0/0 routing-table=default_myas pref-src=192.0.2.210 gateway=192.0.2.176 distance=1
/ip/route/print detail where routing-table=default_myas                      
Flags: D - dynamic; X - disabled, I - inactive, A - active; c - connect, s - static, r - rip, b - bgp, o - ospf, i - is-is, d - dhcp, v - vpn, m - modem, y - bgp-mpls-vpn; H - hw-offloaded; + - ecmp 
   D b   dst-address=0.0.0.0/0 routing-table=default_myas gateway=172.20.215.129 immediate-gw=192.0.2.185%wg-bg1-ftth distance=200 scope=40 target-scope=30 suppress-hw-offload=no 

   D b   dst-address=0.0.0.0/0 routing-table=default_myas gateway=172.20.215.130 immediate-gw=192.0.2.176%wg-bg2-ftth distance=200 scope=40 target-scope=30 suppress-hw-offload=no 

 1  As   dst-address=0.0.0.0/0 routing-table=default_myas pref-src=192.0.2.210 gateway=192.0.2.176 immediate-gw=192.0.2.176%wg-bg2-ftth distance=1 scope=30 target-scope=10 suppress-hw-offload=no

However, RouterOS blatantly ignores this to its fullest extent and it drives me NUTS. NUTS. NUTS! :

/tool/sniffer/export 
# 2024-11-27 16:24:00 by RouterOS 7.15.3
# software id = 
#
/tool sniffer
set filter-ip-protocol=udp filter-port=51820
/tool/sniffer/packet/print detail
18 time=59.738 num=19 direction=rx interface=wg-bg1-ftth src-address=*.*.5.88:63419 dst-address=192.0.2.210:51820 protocol=ip ip-protocol=udp size=176 cpu=2 ip-packet-size=176 ip-header-size=20 dscp=0 
   identification=64534 fragment-offset=0 ttl=49 

19 time=59.738 num=20 direction=tx interface=wg-bg2-ftth src-address=192.0.2.177:51820 dst-address=*.*.5.88:63419 protocol=ip ip-protocol=udp size=120 cpu=1 ip-packet-size=120 ip-header-size=20 dscp=34 
   identification=32146 fragment-offset=0 ttl=64

You can see here, the request to 192.0.2.210:51820 enters via wg-bg1-ftth and the response (through perfectly valid interface wg-bg2-ftth) is sent with source 192.0.2.177:51820, despite explicitly asking RouterOS to use 192.0.2.210 via the pre-src in the default route.

While I don't think it's necessary, I re-post my entire config below.

Can anyone confirm that I am not totally crazy and this is a stupid bug in RouterOS?

If not, what the heck is wrong with this config?

# 2024-11-25 10:54:47 by RouterOS 7.15.3
# software id = 
#
/interface bridge
add name=br-main vlan-filtering=yes
add arp=disabled fast-forward=no name=dum0 protocol-mode=none
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no
/interface wireguard
add listen-port=13231 mtu=1420 name=wg-bg1-ftth
add listen-port=13232 mtu=1420 name=wg-bg2-ftth
add listen-port=51820 mtu=1420 name=wg-mobile
/interface vlan
add comment=ADM interface=br-main name=vlan1 vlan-id=1
add comment=FTTH interface=br-main name=vlan2 vlan-id=2
add comment=LAN interface=br-main name=vlan3 vlan-id=3
add comment=SRV interface=br-main name=vlan4 vlan-id=4
add comment=ADU interface=br-main name=vlan10 vlan-id=10
add comment=DEV interface=br-main name=vlan11 vlan-id=11
add comment=DEVP interface=br-main name=vlan12 vlan-id=12
add comment=DMZ44 interface=br-main name=vlan44 vlan-id=44
/interface vrrp
add interface=vlan1 name=vrrp1
add interface=vlan3 name=vrrp3 sync-connection-tracking=yes vrid=3
add interface=vlan4 name=vrrp4 vrid=4
add interface=vlan11 name=vrrp11 vrid=11
add interface=vlan12 name=vrrp12 vrid=12
add interface=vlan44 name=vrrp44 vrid=44
/interface list
add name=LAN
add comment="Admin Interfaces" name=ADM
add name=DMZ
add name=SRV
add name=DEV
add name=DEVP
add name=WAN
add include=ADM,DEV,DEVP,DMZ,LAN,SRV name=trusted
add name=peers
add name=WANMYAS
add name=ISP
add name=MACHINES
/routing ospf instance
add disabled=no name=ospf-instance-1 redistribute=static router-id=192.0.2.210
/routing ospf area
add disabled=no instance=ospf-instance-1 name=ospf-area-1
/routing table
add disabled=no fib name=default_myas
add disabled=no fib name=default_isp
add comment="Dummy table, acts only as a routing mark" disabled=no fib name=default_isp_only
/interface bridge port
add bridge=br-main interface=ether1
add bridge=br-main interface=veth1 pvid=3
/ip firewall connection tracking
set enabled=yes
/ip neighbor discovery-settings
set discover-interface-list=none lldp-med-net-policy-vlan=1
/interface bridge vlan
add bridge=br-main tagged=br-main,ether1 vlan-ids=2
add bridge=br-main tagged=br-main,ether1 vlan-ids=3
add bridge=br-main tagged=br-main,ether1 vlan-ids=4
add bridge=br-main tagged=br-main,ether1 vlan-ids=10
add bridge=br-main tagged=br-main,ether1 vlan-ids=11
add bridge=br-main tagged=br-main,ether1 vlan-ids=12
add bridge=br-main tagged=br-main,ether1 vlan-ids=44
add bridge=br-main tagged=br-main untagged=ether1 vlan-ids=1
add bridge=br-main tagged=br-main,ether1 vlan-ids=33
/interface detect-internet
set internet-interface-list=static lan-interface-list=static wan-interface-list=static
/interface list member
add interface=vlan3 list=LAN
add interface=vlan1 list=ADM
add interface=vlan2 list=WAN
add interface=vlan44 list=DMZ
add interface=wg-bg1-ftth list=WAN
add interface=wg-bg2-ftth list=WAN
add interface=vrrp3 list=LAN
add interface=vlan4 list=SRV
add interface=wg-bg1-ftth list=peers
add interface=wg-bg2-ftth list=peers
add interface=vlan11 list=DEV
add interface=vlan12 list=DEVP
add interface=wg-bg1-ftth list=WANMYAS
add interface=wg-bg2-ftth list=WANMYAS
add interface=vlan2 list=ISP
add interface=vlan33 list=ISP
add interface=vlan11 list=MACHINES
add interface=vlan12 list=MACHINES
add interface=vrrp44 list=DMZ
add interface=vrrp1 list=ADM
add interface=vrrp12 list=DEVP
add interface=vrrp4 list=SRV
add interface=vrrp11 list=DEV
add interface=vrrp11 list=MACHINES
add interface=vrrp12 list=MACHINES
/interface wireguard peers
add allowed-address=0.0.0.0/0,::/0 endpoint-address=*.*.*.* endpoint-port=51821 interface=wg-bg2-ftth name=bgate2-ftth persistent-keepalive=5s public-key=“********************************************”
add allowed-address=0.0.0.0/0,::/0 endpoint-address=*.*.*.* endpoint-port=51821 interface=wg-bg1-ftth name=bgate1-ftth persistent-keepalive=5s public-key=“********************************************”
add allowed-address=10.2.33.10/32 client-address=10.2.33.10/32 interface=wg-mobile name=miPhone public-key=“********************************************”
/ip address
add address=10.2.1.2/24 interface=vlan1 network=10.2.1.0
add address=10.2.4.2/24 interface=vlan4 network=10.2.4.0
add address=10.2.79.2/24 interface=vlan3 network=10.2.79.0
add address=192.0.2.177/31 interface=wg-bg2-ftth network=192.0.2.176
add address=192.0.2.186/29 interface=wg-bg1-ftth network=192.0.2.184
add address=192.0.2.210/28 interface=vlan44 network=192.0.2.208
add address=172.20.215.132 interface=dum0 network=172.20.215.132
add address=10.2.80.2/24 interface=vlan10 network=10.2.80.0
add address=192.168.222.2/24 interface=vlan11 network=192.168.222.0
add address=192.168.223.2/24 interface=vlan12 network=192.168.223.0
add address=10.2.79.254 interface=vrrp3 network=10.2.79.254
add address=10.2.33.1/24 interface=wg-mobile network=10.2.33.0
add address=192.168.223.254 interface=vrrp12 network=192.168.223.254
add address=192.168.222.254 interface=vrrp11 network=192.168.222.254
add address=10.2.1.254 interface=vrrp1 network=10.2.1.254
add address=10.2.4.254 interface=vrrp4 network=10.2.4.254
add address=192.168.5.102/24 interface=vlan33 network=192.168.5.0
add address=192.0.2.222 interface=vrrp44 network=192.0.2.222
/ip dhcp-client
add add-default-route=no interface=vlan2 script=":if (\$bound=1) do={\
    \n    /ip route set [find where comment=\"default_isp\"] gateway=\$\"gateway-address\" disabled=no\
    \n    } else={\
    \n    /ip route set [find where comment=\"default_isp\"] disabled=yes\
    \n    }\r\
    \n\r\
    \n" use-peer-ntp=no
/ip firewall address-list
add address=10.2.0.0/16 list=own_hosts
add address=192.0.2.0/24 list=own_hosts
add address=*.*.*.* list=border_gates
add address=*.*.*.* list=border_gates
add address=*.*.*.* list=own_hosts
add address=*.*.*.* list=own_hosts
add address=192.168.222.0/24 list=own_hosts
add address=192.168.223.0/24 list=own_hosts
add address=172.20.215.128/27 list=own_hosts
add address=0.0.0.0/8 comment="defconf: RFC6890" list=no_forward_ipv4
add address=169.254.0.0/16 comment="defconf: RFC6890" list=no_forward_ipv4
add address=224.0.0.0/4 comment="defconf: multicast" list=no_forward_ipv4
add address=255.255.255.255 comment="defconf: RFC6890" list=no_forward_ipv4
add address=127.0.0.0/8 comment="defconf: RFC6890" list=bad_ipv4
add address=192.0.0.0/24 comment="defconf: RFC6890" list=bad_ipv4
add address=192.0.2.0/24 comment="defconf: RFC6890 documentation" list=bad_ipv4
add address=198.51.100.0/24 comment="defconf: RFC6890 documentation" list=bad_ipv4
add address=203.0.113.0/24 comment="defconf: RFC6890 documentation" list=bad_ipv4
add address=240.0.0.0/4 comment="defconf: RFC6890 reserved" list=bad_ipv4
add address=0.0.0.0/8 comment="defconf: RFC6890" list=not_global_ipv4
add address=10.0.0.0/8 comment="defconf: RFC6890" list=not_global_ipv4
add address=100.64.0.0/10 comment="defconf: RFC6890" list=not_global_ipv4
add address=169.254.0.0/16 comment="defconf: RFC6890" list=not_global_ipv4
add address=172.16.0.0/12 comment="defconf: RFC6890" list=not_global_ipv4
add address=192.0.0.0/29 comment="defconf: RFC6890" list=not_global_ipv4
add address=192.168.0.0/16 comment="defconf: RFC6890" list=not_global_ipv4
add address=198.18.0.0/15 comment="defconf: RFC6890 benchmark" list=not_global_ipv4
add address=255.255.255.255 comment="defconf: RFC6890" list=not_global_ipv4
add address=224.0.0.0/4 comment="defconf: multicast" list=bad_src_ipv4
add address=255.255.255.255 comment="defconf: RFC6890" list=bad_src_ipv4
add address=0.0.0.0/8 comment="defconf: RFC6890" list=bad_dst_ipv4
add address=224.0.0.0/4 comment="defconf: RFC6890" list=bad_dst_ipv4
add address=10.2.0.0/16 list=private_lans
add address=192.168.0.0/16 list=private_lans
add address=172.20.215.128/27 list=border_gates
add address=10.2.4.10 list=hairpin_dst
add address=10.2.4.20 list=hairpin_dst
add address=10.2.0.0/16 list=hairpin_src
add address=192.168.222.0/24 list=hairpin_src
add address=192.168.223.0/24 list=hairpin_src
/ip firewall filter
add action=accept chain=input comment="defconf: accept ICMP after RAW" protocol=icmp
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=accept chain=input comment="SSH access from everywhere" dst-port=1983 protocol=tcp
add action=accept chain=input comment=Wireguard dst-port=51820 protocol=udp
add action=accept chain=input dst-port=8291 protocol=tcp src-address-list=private_lans
add action=accept chain=input dst-port=3784,4784,3785 in-interface-list=peers protocol=udp src-address-list=own_hosts
add action=accept chain=input in-interface-list=peers protocol=ospf
add action=accept chain=input dst-port=179 in-interface-list=peers protocol=tcp src-address-list=own_hosts
add action=accept chain=input dst-port=53 protocol=udp src-address-list=own_hosts
add action=accept chain=input dst-port=53 protocol=tcp src-address-list=own_hosts
add action=drop chain=input comment="defconf: drop all not coming from trusted" in-interface-list=!trusted
add action=accept chain=forward comment="PortFW: For some reason, Fasttrack doesn't work for these." connection-mark=conn_portfw connection-state=established,related
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=jump chain=forward comment="Traffic rules for new connections" connection-state=new jump-target=traffic_rules
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid log=yes log-prefix=INVALID
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
add action=drop chain=forward comment="defconf: drop bad forward IPs" src-address-list=no_forward_ipv4
add action=drop chain=forward comment="defconf: drop bad forward IPs" dst-address-list=no_forward_ipv4
add action=accept chain=forward comment="Check if counters >0. If not, DROP here" log=yes log-prefix="[ACPT]"
add action=accept chain=traffic_rules comment="Allow all ICMP echo" icmp-options=8:0-255 protocol=icmp
add action=accept chain=traffic_rules comment="Allow from ISPs only when DNAT" connection-nat-state=dstnat in-interface-list=ISP
add action=accept chain=traffic_rules comment="Allow internet for public 44" in-interface-list=DMZ out-interface-list=WANMYAS
add action=log chain=traffic_rules comment="Log drops (not wan)" in-interface-list=!WANMYAS log=yes log-prefix="[BLOCKED]" out-interface-list=!WANMYAS src-address=!192.168.222.150
add action=drop chain=traffic_rules
/ip firewall mangle
add action=mark-routing chain=output comment="wg-tunnels: bgate1 over FTTH" dst-address=*.*.*.* dst-port=51821 new-routing-mark=default_isp_only passthrough=no protocol=udp
add action=mark-routing chain=output comment="wg-tunnels: bgate2 over FTTH" dst-address=*.*.*.* dst-port=51821 new-routing-mark=default_isp_only passthrough=no protocol=udp
/ip firewall nat
add action=masquerade chain=srcnat comment="wg-tunnels: FTTH" routing-mark=default_isp_only
add action=masquerade chain=srcnat comment="Hairpin all PortFWs" dst-address-list=hairpin_dst src-address-list=hairpin_src
add action=masquerade chain=srcnat comment="defconf: masquerade WAN interface" out-interface=vlan2
/ip firewall raw
add action=accept chain=prerouting comment="defconf: enable for transparent firewall"
add action=accept chain=prerouting comment="defconf: accept DHCP discover" dst-address=255.255.255.255 dst-port=67 in-interface-list=LAN protocol=udp src-address=0.0.0.0 src-port=68
add action=drop chain=prerouting comment="defconf: drop bogon IP's" src-address-list=bad_ipv4
add action=drop chain=prerouting comment="defconf: drop bogon IP's" dst-address-list=bad_ipv4
add action=drop chain=prerouting comment="defconf: drop bogon IP's" src-address-list=bad_src_ipv4
add action=drop chain=prerouting comment="defconf: drop bogon IP's" dst-address-list=bad_dst_ipv4
add action=drop chain=prerouting comment="defconf: drop non global from WAN" in-interface-list=WAN src-address-list=not_global_ipv4
add action=drop chain=prerouting comment="defconf: drop forward to local lan from WAN" dst-address-list=private_lans in-interface-list=WAN
add action=drop chain=prerouting comment="defconf: drop local if not from default IP range" in-interface-list=LAN src-address=!10.2.79.0/24
add action=drop chain=prerouting in-interface-list=DEV src-address=!192.168.222.0/24
add action=drop chain=prerouting in-interface-list=DEVP src-address=!192.168.223.0/24
add action=drop chain=prerouting comment="defconf: drop bad UDP" port=0 protocol=udp
add action=jump chain=prerouting comment="defconf: jump to ICMP chain" jump-target=icmp4 protocol=icmp
add action=jump chain=prerouting comment="defconf: jump to TCP chain" jump-target=bad_tcp protocol=tcp
add action=accept chain=prerouting comment="defconf: accept everything else from LAN" in-interface-list=trusted
add action=accept chain=prerouting comment="defconf: accept everything else from WAN" in-interface-list=WAN
add action=drop chain=prerouting comment="defconf: drop the rest"
add action=drop chain=bad_tcp comment="defconf: TCP flag filter" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=bad_tcp comment=defconf protocol=tcp tcp-flags=fin,syn
add action=drop chain=bad_tcp comment=defconf protocol=tcp tcp-flags=fin,rst
add action=drop chain=bad_tcp comment=defconf protocol=tcp tcp-flags=fin,!ack
add action=drop chain=bad_tcp comment=defconf protocol=tcp tcp-flags=fin,urg
add action=drop chain=bad_tcp comment=defconf protocol=tcp tcp-flags=syn,rst
add action=drop chain=bad_tcp comment=defconf protocol=tcp tcp-flags=rst,urg
add action=drop chain=bad_tcp comment="defconf: TCP port 0 drop" port=0 protocol=tcp
add action=accept chain=icmp4 comment="defconf: echo reply" icmp-options=0:0 limit=5,10:packet protocol=icmp
add action=accept chain=icmp4 comment="defconf: net unreachable" icmp-options=3:0 protocol=icmp
add action=accept chain=icmp4 comment="defconf: host unreachable" icmp-options=3:1 protocol=icmp
add action=accept chain=icmp4 comment="defconf: protocol unreachable" icmp-options=3:2 protocol=icmp
add action=accept chain=icmp4 comment="defconf: port unreachable" icmp-options=3:3 protocol=icmp
add action=accept chain=icmp4 comment="defconf: fragmentation needed" icmp-options=3:4 protocol=icmp
add action=accept chain=icmp4 comment="defconf: echo" icmp-options=8:0 limit=5,10:packet protocol=icmp
add action=accept chain=icmp4 comment="defconf: time exceeded " icmp-options=11:0-255 protocol=icmp
add action=drop chain=icmp4 comment="defconf: drop other icmp" protocol=icmp
/ip route
add check-gateway=ping comment=default_isp disabled=no distance=1 dst-address=0.0.0.0/0 gateway=*.*.*.* routing-table=default_isp scope=30 suppress-hw-offload=no target-scope=10
/routing bgp connection
add as=64513 disabled=no local.address=172.20.215.132 .role=ibgp name=BorderGate1 remote.address=172.20.215.129/32 .as=64513 routing-table=default_myas
add as=64513 disabled=no local.address=172.20.215.132 .role=ibgp name=BorderGate2 remote.address=172.20.215.130/32 .as=64513 routing-table=default_myas
/routing ospf area range
add area=ospf-area-1 disabled=no prefix=172.20.215.128/27
add area=ospf-area-1 disabled=no prefix=192.0.2.0/24
/routing ospf interface-template
add area=ospf-area-1 auth=md5 auth-id=1 auth-key=**************** cost=100 dead-interval=10s disabled=no hello-interval=5s interfaces=wg-bg1-ftth,wg-bg2-ftth
add area=ospf-area-1 disabled=no interfaces=dum0 passive
add area=ospf-area-1 disabled=no interfaces=vlan44 passive
/routing rule
add action=lookup-only-in-table comment="Connections must never go through other links" disabled=no routing-mark=default_isp_only table=default_isp
add action=lookup comment="main table" disabled=no table=main
add action=lookup comment="default 44net" disabled=no src-address=192.0.2.0/24 table=default_myas
add action=lookup comment="default sonic" disabled=no table=default_isp
/tool sniffer
set filter-ip-address=192.0.2.210/32 filter-ip-protocol=udp filter-port=51820

Amm0 · Thu Nov 28, 2024 3:31 am

It's roughly the same as this issue: viewtopic.php?t=205278&hilit=wireguard

Wireguard, for some unknown reason, is not treated the same as "locally generated traffic". So pref-src= is I'm guessing a similar victim.

CGGXANNX · Thu Nov 28, 2024 3:50 am

It's known by MikroTik and they have recently updated the docs with a workaround

https://help.mikrotik.com/docs/pages/di ... ersions=37

Info
When you encounter issues with reply traffic having the wrong source address, using NAT to translate packet source addresses to your loopback interface is a common workaround. This approach helps ensure that the source address is consistent and correct when packets are routed back through the network.

divB · Thu Nov 28, 2024 4:05 am

Wow crazy. Thanks. I looked everywhere but did not find this. At least I'm not alone.
Ok, I really want to avoid NAT but I'm OK to use it as a workaround.
However, I am still not able to get it running. I am using the packet sniffer tool and just filter for UDP and the IP of my road warrior setup and often I just can't see any traffic going out ("tx").

From the Mikrotik link, it's really not clear to me what "translate packet source address to your loopback interface" really means.

I tried all possible variants using src-nat I could think of (i.e., source natting the wrong source IP, 192.0.2.177, to the right one, 192.0.2.210; both with a single snat rule as well as using mangle+nat).
I also tried dst-nat but no luck either (i.e., making a port forwarding from port 192.0.2:210:41820 to 192.0.2.210:51820)

Would you mind showing an example of the right NAT rule(s)?

CGGXANNX · Thu Nov 28, 2024 6:03 am

I haven't tested with your particular config, but on my router, when I want to control the source address as well as the outgoing interface for WireGuard reply-packets, I use dstnat rules:

* First create an address on the lo interface, let's say 10.20.30.40/32
* Add dstnat rule for destination 192.0.2.210/32, UDP dst-port 51820, with action=dst-nat to-addresses=10.20.30.40
* (Optional) in case you also want to control the outgoing interface, add routing rules for src-address 10.20.30.40

For your case, you'll probably only need the first two steps. Because incoming packets with destination to 192.0.2.210 get dstnat-ed to 10.20.30.40, their corresponding response packets will get un-NAT-ed accordingly and have their source address changed to 192.0.2.210, which is what you want. If it still doesn't work (like I said, I haven't tested with configurations similar to yours) you might need:

* An additional srcnat rule for src-address 10.20.30.40, with action=src-nat to-addresses=192.0.2.210.

And this srcnat rule must be placed ABOVE your masquerade rules int NAT table.

wiseroute · Thu Nov 28, 2024 6:38 am

@cgg,

First create an address on the lo interface, let's say 10.20.30.40/32

in case you didn't notice - that 210 is the bridge/loopback address which the wg listen to. the exact same thing as you have proposed.

the @op scenario and problem was full routing on 2 wan interfaces and the wg bridge (he expected no nat needed as he runs full routing). hence the rpf checks problem came in. he wonders why the wg source address is not 210 but rewritten to 177 - mitm/spoofing.

@normis,
please add some notes in the wiki - that this kind of scenario (wireguard listening on local bridge interface with multiple wan interfaces) currently won't work in full routing except with nat. so that other mt users understand the situation.

CGGXANNX · Thu Nov 28, 2024 7:27 am

in case you didn't notice - that 210 is the bridge/loopback address which the wg listen to. the exact same thing as you have proposed.

No, it's not the same. The most important thing is the dstnat rule. It doesn't matter which of the router's addresses the external user used as endpoint address for the WG connection. What needed is the additional DSTNAT operation (from one of the router's IP addresses to another one, that's why an additional IP address needs to be added to the router, and lo was chosen because that interface is always there).

lurker888 · Thu Nov 28, 2024 7:57 am

Actually this is not a bug, but simply an effect of how wireguard and linux routing works. By the way as far as I can tell, Mikrotik uses the stock linux implementation - and yes, the same thing happens on straight linux and yes, this is a frequently asked question on linux forums as well.

To break it down:
* wg sends out the packet and does not specify a source address (this is in line with the philosophy around wg - so I wouldn't hold my breath until this is changed)
* the routing decision is made and as the packet has no source address, pref addr is used - however your packet at this point has no routing mark attached, and quite correctly, the main table is consulted
* later, even if you apply a routing mark to the packet, in the routing adjustment phase of the packet flow, pref src is disregarded (because that's just how it works - at this point the packet has a source address and regardless of how it was acquired, it's assumed to be correct)

This behavior leads to a number of sometimes surprising effects. And yes, regardless of Mikrotik or non-Mikrotik, dnat is the universally accepted solution.

For other protocols this does not manifest so blatantly, because there is a clear request-response nature to them (e.g. ping: echo request - echo response; dns: again, request - response; and of course for all things TCP) and in these cases the source ip is set by the originator of the packet. Wg on the other hand does not differentiate meaningfully between client and server, and also supports roaming (changing ip addresses and ports) of each peer, and in line with this philosophy maintains that fw/routing should take care of the situation you are in. Thus every major contributor (as far as I can tell) maintains that dnat is not some workaround but an appropriate solution; of course everyone is entitled to their opinion. *shrug*

EDIT:
By the way, adding proper VRF support to wg would solve this. Although this was proposed, it never made it into the kernel (at least the last time I was aware of such an effort...)

rplant · Thu Nov 28, 2024 9:26 am

Hi,

My observations:
Actually wireguard in mikrotik does attempt to use the IP address that the incoming packet was sent to.

Looking at the packet on the output chain is too late. It has already gone through the routing process,
and had it's ip address changed, probably also natted.

If you use routing rules, you can change the routing table it uses based on the correct receiving IP address.

However this case seems a little unusual in that you have packets coming in interface A and replies leaving via interface B.
Which if you had no firewall rules at all on your router might work ok.

The outbound packets won't match the connection of the incoming packets, because they are on different interfaces.
So not being part of an existing connection, they will become a new connection started from inside the router and get Natted.

(In another thread MrZ mentioned that in wireguard the reply connection is a different connection to the incoming connection)

You could perhaps disable outbound NAT for packets with source of .210 ip address?

wiseroute · Thu Nov 28, 2024 9:39 am

@cgg,

It doesn't matter which of the router's addresses the external user used as endpoint address for the WG connection.

well, if @op would listen to put wg to listen on interface address 177 (which is persistent in terms of path) - then he won't have this headache resolving pref-src or nat or spoofing issues. please read his other thread as well so you get the picture. i did tell him to use nat - but he insisted on full routing scenario.

divB · Fri Nov 29, 2024 1:38 am

I haven't tested with your particular config, but on my router, when I want to control the source address as well as the outgoing interface for WireGuard reply-packets, I use dstnat rules:

* First create an address on the lo interface, let's say 10.20.30.40/32
* Add dstnat rule for destination 192.0.2.210/32, UDP dst-port 51820, with action=dst-nat to-addresses=10.20.30.40
* (Optional) in case you also want to control the outgoing interface, add routing rules for src-address 10.20.30.40

For your case, you'll probably only need the first two steps. Because incoming packets with destination to 192.0.2.210 get dstnat-ed to 10.20.30.40, their corresponding response packets will get un-NAT-ed accordingly and have their source address changed to 192.0.2.210, which is what you want. If it still doesn't work (like I said, I haven't tested with configurations similar to yours) you might need:

I really think I am not that dumb but this wireguard is one of the biggest headaches I have come across to date. It exceeds all the OSPF and BFD trouble.

I did what you said and I agree this is how it should work. Let me start what I did:

/interface bridge
add arp=disabled fast-forward=no name=dum1 protocol-mode=none
/ip address
add address=172.20.215.1 interface=dum1 network=172.20.215.1
/ip firewall filter
add action=accept chain=input comment=test dst-port=41820,51820 log=yes log-prefix="[***WG-IN]" protocol=udp
add action=accept chain=output log=yes log-prefix="[***WG-OUT]" port=51820,41820 protocol=udp
/ip firewall nat
add action=dst-nat chain=dstnat dst-address=192.0.2.210 dst-port=41820 log=yes log-prefix="[***WG-DNAT]" protocol=udp to-addresses=172.20.215.1 to-ports=51820

Created a dummy interface "dum1" with address 172.20.215.1
Created a DNAT rule to port forward from 192.0.2.210:41820 to 172.20.215.1:51820
Client now connects to 192.0.2.210:41820 which would then be "port forwarded" to internal 172.20.215.1:51820
I agree with you that "their corresponding response packets will get un-NAT-ed accordingly and have their source address changed to 192.0.2.210, which is what you want"
What RouterOS does just doesn't make any sense whatsoever.

Here is the log output:

15:13:06 wireguard,debug wg-mobile: [miPhone] ****: Receiving handshake initiation from peer (200.95.5.88:38846) extra:0 (einval) 
15:13:06 wireguard,debug wg-mobile: [miPhone] ****: Sending handshake response to peer (200.95.5.88:38846) 
15:13:06 firewall,info [***WG-DNAT] dstnat: in:wg-bg1-ftth out:(unknown 0), connection-state:new proto UDP, 200.95.5.88:38846->192.0.2.210:41820, len 176 
15:13:06 firewall,info [***WG-IN] input: in:wg-bg1-ftth out:(unknown 0), connection-state:new,dnat proto UDP, 200.95.5.88:38846->172.20.215.1:51820, NAT 200.95.5.88:38846->(192.0.2.210:41820->172.20.215.1:51820), len 176 
15:13:06 firewall,info [***WG-OUT] output: in:(unknown 0) out:vlan2, connection-state:new proto UDP, 125.170.121.236:51820->200.95.5.88:38846, len 120

WTF? Why is the source address of the return packet now 125.170.121.236? (This is the address of vlan2!). Because of DNAT, the return packet should be 192.0.2.210, that's the whole point of doing it and now again it picks some other random address.

Before we go into adding src-nat rules (in case they should be required) I would really like to understand what the heck is going on here. It defies any logic.

Do you have any further advice what's wrong here and why my DNAT rule is not cutting it?

EDIT: I also added a connection mark to the incoming connection via mangle ... but the output packet is still as "connection-state:new" and does not include my connection mark either! It seems connection tracking fails to be able to track this wireguard connection. If this is really the case, how on earth are you able to use DNAT for this?

EDIT2: I have never been so lost as with this ~!#$*@ wireguard. I've spent again hours trying any conceivable combination of DNAT+SNAT but there just doesn't seem to be any way:

SNAT does not work because:

Rewriting to any other (arbitrary) SADDR/SPORT works but only 192.0.2.210:51820 does not work!
192.0.2.210:51820 does NOT work because wireguard already listens on it! —> Rewritten packet is discarded and never transmitted (does not appear in packet sniffer)

DNAT does not work:

Response packet is not identified as part of the existing connection (see above) It’s always connection-state:new
Without assigning dum1 to default_myas table, the source address has the source IP of the main table (see above)
When assigning dum1 to default_myas, the source address is, again, 192.0.2.177

DNAT+SNAT does not work:

Idea: Use DNAT to move to a different port (51820-->41820)
DNAT 192.0.2.210:41820 —> 172.20.215.1:51820
SNAT 192.0.2.177:51820 —> 192.0.2.210:41820
And we run into the same issue as why SNAT does not work: 192.0.2.210:41820 is already taken by the DNAT

I am really going in circles and it seems everything about wireguard is buggy as crazy

Can't be that this is literally impossible?

EDIT3: Absolutely insane. The very reason that DNAT doesn't work is believe my original question which is because pref-src is ignored. Of course, if the source address is again 192.0.2.177, it won't be matched as a DNAT connection which would expect 172.20.215.1. But 172.20.215.1 is never chosen because of the default route. I can confirm this by forcing a dummy routing table that only contains "0.0.0.0/0 interface=dum1". Then the source address is properly taken as 172.20.215.1 and I can see that DNAT works. But then the packet is routed into dum1 and never arrives. It's really circles here! I don't think that DNAT is a solution of my question. The core is really as to WHY RouterOS ignores my request for pref-src

divB · Fri Nov 29, 2024 1:45 am

well, if @op would listen to put wg to listen on interface address 177 (which is persistent in terms of path) - then he won't have this headache resolving pref-src or nat or spoofing issues. please read his other thread as well so you get the picture. i did tell him to use nat - but he insisted on full routing scenario.

I answered to this already in the other posting. I would love if I could do that. But wg can't listen on a specific interface or address! It listens on all. If I miss something obvious, please do let me know.

Yes, I would prefer (and insist) on full routing scenario but I am starting to accept that WG implementation is buggy. I am fine to use NAT (S and/or D) as a bugfix but even after many days of trying many rules I am still not able to make it work, even with NAT. See previous answer just above. If you got any further advice on the NAT solution, I really appreciate it.

sirbryan · Fri Nov 29, 2024 2:23 am

I'm considering the idea of moving all routing away from the WG endpoint, and instead having the work split across two (or more) devices: WG tunnel terminators at each end, and have them hand off EOIP/VXLAN/IPIP/whatever to the next router(s) up/down the line. This way the WG router doesn't get confused with multiple routing tables, NAT, etc. I can still manage it with a secondary port plugged into the on-net routers, even move all management to a different VRF.

divB · Fri Nov 29, 2024 2:33 am

Actually this is not a bug, but simply an effect of how wireguard and linux routing works. By the way as far as I can tell, Mikrotik uses the stock linux implementation - and yes, the same thing happens on straight linux and yes, this is a frequently asked question on linux forums as well.

To break it down:
* wg sends out the packet and does not specify a source address (this is in line with the philosophy around wg - so I wouldn't hold my breath until this is changed)

I completely disagree with the statement it's not a bug. Saying it's a big is just an excuse for not fixing bad code. Yes, wireguard uses UDP and yes, UDP is stateless.

However, when the road warrier first sends a request to 192.0.2.210:51820, it creates a "virtual state" inside wireguard. It is the user space applications' duty to then fill out the source address (see https://blog.cloudflare.com/everything- ... ard-socket). I can't think of any meaningful reason why this shouldn't happen. There is no reason on earth, the response packet should ever come from a different source address than it was connected to. There is no reason to leave this decision to the operating system.

* the routing decision is made and as the packet has no source address, pref addr is used - however your packet at this point has no routing mark attached, and quite correctly, the main table is consulted

Hmm... Are you sure that's true? If the main table would have been consulted, the response packet wouldn't even find its way out, because the default route for the main table is vlan2 (see my posted config) and it be MASQ'ed and go out there.

I am fairly certain that the packet does use the right table ("default_myas") and right route because of this in my posted config:

/routing rule
add action=lookup-only-in-table comment="Connections must never go through other links" disabled=no routing-mark=default_isp_only table=default_isp
add action=lookup comment="main table" disabled=no table=main
add action=lookup disabled=no src-address=192.0.2.0/24 table=default_myas

Yet still, pref-src of that default route in table default_myas is ignored.

Again, if that route would not be used, not even the packet with source .177 would make its way out.

If you disagree, can you elaborate?

* later, even if you apply a routing mark to the packet, in the routing adjustment phase of the packet flow, pref src is disregarded (because that's just how it works - at this point the packet has a source address and regardless of how it was acquired, it's assumed to be correct)

This behavior leads to a number of sometimes surprising effects. And yes, regardless of Mikrotik or non-Mikrotik, dnat is the universally accepted solution.

But I'm not even applying any route marks. I just use "ip rule" that selects table "default_myas" whenever source address is 192.0.2.0/24. See my config.

For other protocols this does not manifest so blatantly, because there is a clear request-response nature to them (e.g. ping: echo request - echo response; dns: again, request - response; and of course for all things TCP) and in these cases the source ip is set by the originator of the packet. Wg on the other hand does not differentiate meaningfully between client and server, and also supports roaming (changing ip addresses and ports) of each peer, and in line with this philosophy maintains that fw/routing should take care of the situation you are in. Thus every major contributor (as far as I can tell) maintains that dnat is not some workaround but an appropriate solution; of course everyone is entitled to their opinion. *shrug*

Understand now what your argument is. Yet I still don't see what it would break if the response packet of a new (=unseen) connection comes from the right IP address.

Anyway, I am now OK to use DNAT but as I wrote, still no working setup for me in sight

EDIT:
By the way, adding proper VRF support to wg would solve this. Although this was proposed, it never made it into the kernel (at least the last time I was aware of such an effort...)

100%

The lack of VRF isn't only a pain for this example but for my complete setup: I establish wireguard tunnels for uplinks over different ISPs with dynamic IP. Right now I have to use dirty hacks with route marks (see my posted config). But the correct solution would be to put the respective wireguard connections into the VRFs of the respective ISPs ... and the resulting wg interfaces into my VRF "myas".

It's a major lacking feature

I have written Mikrotik and they answered it's on their list and they will implement this some time, but of course, who knows how long it would take.

lurker888 · Fri Nov 29, 2024 4:19 am

Actually this is not a bug, but simply an effect of how wireguard and linux routing works. By the way as far as I can tell, Mikrotik uses the stock linux implementation - and yes, the same thing happens on straight linux and yes, this is a frequently asked question on linux forums as well.

To break it down:
* wg sends out the packet and does not specify a source address (this is in line with the philosophy around wg - so I wouldn't hold my breath until this is changed)
I completely disagree with the statement it's not a bug. Saying it's a big is just an excuse for not fixing bad code. Yes, wireguard uses UDP and yes, UDP is stateless.

However, when the road warrier first sends a request to 192.0.2.210:51820, it creates a "virtual state" inside wireguard. It is the user space applications' duty to then fill out the source address (see https://blog.cloudflare.com/everything- ... ard-socket). I can't think of any meaningful reason why this shouldn't happen. There is no reason on earth, the response packet should ever come from a different source address than it was connected to. There is no reason to leave this decision to the operating system.

I agree with the Cloudflare sentiment, and as someone who writes network services professionally, neglecting this irritates me to no end. (Btw. you can feel the writer's frustration dripping between the lines

)

The problem is that wg *itself* is connection-less, i.e. there is no "new" connection, just another handshake.

Let's assume that wg did it the way you suggest. I am using my phone and in communication with a wg "server". I'm using mobile data. I walk into my favorite coffee shop. My phone connects to their wifi. My phone sends out wg packets with the src address appropriate for my previously used LTE connection and not with the address acquired over wifi. Connection fails.

Clearly the packet *should* go out on the default route and pick up its src address from the pref address there.

This is what I tried to explain. The two decisions:
* don't differentiate between peers as "server" and "client"
* support roaming IPs
don't allow for setting the src address.

It's not lazy coding, but a result of a philosophy regarding the protocol. One which obviously quite a few people don't agree with... but not exactly a bug.

* the routing decision is made and as the packet has no source address, pref addr is used - however your packet at this point has no routing mark attached, and quite correctly, the main table is consulted
Hmm... Are you sure that's true? If the main table would have been consulted, the response packet wouldn't even find its way out, because the default route for the main table is vlan2 (see my posted config) and it be MASQ'ed and go out there.

I am fairly certain that the packet does use the right table ("default_myas") and right route because of this in my posted config:
Code: Select all
/routing rule
add action=lookup-only-in-table comment="Connections must never go through other links" disabled=no routing-mark=default_isp_only table=default_isp
add action=lookup comment="main table" disabled=no table=main
add action=lookup disabled=no src-address=192.0.2.0/24 table=default_myas
Yet still, pref-src of that default route in table default_myas is ignored.

Again, if that route would not be used, not even the packet with source .177 would make its way out.

If you disagree, can you elaborate?

I do disagree. Refer to the packet flow compiled by the nice Mikrotik guys at: https://help.mikrotik.com/docs/spaces/R ... n+RouterOS

* you packet ingresses the diagram titled "routing" at K
* routing decision is made - please bear in mind that you packet has *no* src address, and so your third rule *does not* mach your packet - main is selected
* src addr is set based on pref src in main
* packet goes to output
* (if snat happens, it happens here - but now this is of no interest for us)
* at the end of output "routing adjustment" occurs - here the src address is already set and your third rule now matches: the packet is rerouted according to the *myas routing table; however in the routing adjustment phase, pref addr - as I wrote - is not consulted

Hence your packet goes out on the correct interface with the src address not set as you expected.

Please feel free to say so if you disagree on how/why this is happening...

* later, even if you apply a routing mark to the packet, in the routing adjustment phase of the packet flow, pref src is disregarded (because that's just how it works - at this point the packet has a source address and regardless of how it was acquired, it's assumed to be correct)

This behavior leads to a number of sometimes surprising effects. And yes, regardless of Mikrotik or non-Mikrotik, dnat is the universally accepted solution.
But I'm not even applying any route marks. I just use "ip rule" that selects table "default_myas" whenever source address is 192.0.2.0/24. See my config.

Explained above. In the "routing decision" phase your rule cannot and does not match the packet, only in the "routing adjustment" does it pick up the route it eventually takes.

For other protocols this does not manifest so blatantly, because there is a clear request-response nature to them (e.g. ping: echo request - echo response; dns: again, request - response; and of course for all things TCP) and in these cases the source ip is set by the originator of the packet. Wg on the other hand does not differentiate meaningfully between client and server, and also supports roaming (changing ip addresses and ports) of each peer, and in line with this philosophy maintains that fw/routing should take care of the situation you are in. Thus every major contributor (as far as I can tell) maintains that dnat is not some workaround but an appropriate solution; of course everyone is entitled to their opinion. *shrug*
Understand now what your argument is. Yet I still don't see what it would break if the response packet of a new (=unseen) connection comes from the right IP address.

Anyway, I am now OK to use DNAT but as I wrote, still no working setup for me in sight

EDIT:
By the way, adding proper VRF support to wg would solve this. Although this was proposed, it never made it into the kernel (at least the last time I was aware of such an effort...)
100%

The lack of VRF isn't only a pain for this example but for my complete setup: I establish wireguard tunnels for uplinks over different ISPs with dynamic IP. Right now I have to use dirty hacks with route marks (see my posted config). But the correct solution would be to put the respective wireguard connections into the VRFs of the respective ISPs ... and the resulting wg interfaces into my VRF "myas".

It's a major lacking feature

I have written Mikrotik and they answered it's on their list and they will implement this some time, but of course, who knows how long it would take.

Let me be a bit clearer on this: someone actually wrote the code and it was not included. I know not what the future holds

but if I were in a position to suggest what Mikrotik should do (which I am clearly not in,) I would probably wait until it's settled in the kernel.

I think VRF support would be more elegant, however - and more practically - if the underlying behavior is clearly understood, dnat is in 99% of cases totally fine.

Thank you for your unusually thoughtful reply. Hope this clears things up at least somewhat.

EDIT:
Messed up snat/dnat naming. Corrected.

divB · Fri Nov 29, 2024 4:55 am

Thank you, you cleared up some important points for me!

[...]
It's not lazy coding, but a result of a philosophy regarding the protocol. One which obviously quite a few people don't agree with... but not exactly a bug.

Ok, I get now what you are saying.
But please hear me out until the end of my response where I believe I can show that it might be impossible to use wireguard (in my described setup). This is really inconceivable to me...

I do disagree. Refer to the packet flow compiled by the nice Mikrotik guys at: https://help.mikrotik.com/docs/spaces/R ... n+RouterOS

* you packet ingresses the diagram titled "routing" at K
* routing decision is made - please bear in mind that you packet has *no* src address, and so your third rule *does not* mach your packet - main is selected
* src addr is set based on pref src in main
* packet goes to output
* (if snat happens, it happens here - but now this is of no interest for us)
* at the end of output "routing adjustment" occurs - here the src address is already set and your third rule now matches: the packet is rerouted according to the *myas routing table; however in the routing adjustment phase, pref addr - as I wrote - is not consulted

Hence your packet goes out on the correct interface with the src address not set as you expected.

Please feel free to say so if you disagree on how/why this is happening...

Thank you for walking me through this. I think it makes sense now.

Now I think we can proclaim that pref-src by definition is irrelevant in any table that's references from a rule that uses source address.
In other words, it doesn't work at all for policy based routing.
That's a HUGE bummer.

I think it would be more than important to add this as a remark in the documentation for pref-src.

I think VRF support would be more elegant, however - and more practically - if the underlying behavior is clearly understood, dnat is in 99% of cases totally fine.

Now here I am again claiming that DNAT or SNAT does actually not seem to be a solution at all. To avoid repeating things, would you mind checking my EDIT1-EDIT3 in this answer? viewtopic.php?p=1111805#p1111805

TLDR is: the main crux is that pref-src would be required to make DNAT work (but then I wouldn't need DNAT). And the issue with SNAT is that address/port is already in use.

I am afraid I have hit a big road block ... is there any other hope?

lurker888 · Fri Nov 29, 2024 5:40 am

Thank you, you cleared up some important points for me!

[...]
It's not lazy coding, but a result of a philosophy regarding the protocol. One which obviously quite a few people don't agree with... but not exactly a bug.
Ok, I get now what you are saying.
But please hear me out until the end of my response where I believe I can show that it might be impossible to use wireguard (in my described setup). This is really inconceivable to me...

I do disagree. Refer to the packet flow compiled by the nice Mikrotik guys at: https://help.mikrotik.com/docs/spaces/R ... n+RouterOS

* you packet ingresses the diagram titled "routing" at K
* routing decision is made - please bear in mind that you packet has *no* src address, and so your third rule *does not* mach your packet - main is selected
* src addr is set based on pref src in main
* packet goes to output
* (if snat happens, it happens here - but now this is of no interest for us)
* at the end of output "routing adjustment" occurs - here the src address is already set and your third rule now matches: the packet is rerouted according to the *myas routing table; however in the routing adjustment phase, pref addr - as I wrote - is not consulted

Hence your packet goes out on the correct interface with the src address not set as you expected.

Please feel free to say so if you disagree on how/why this is happening...
Thank you for walking me through this. I think it makes sense now.

Now I think we can proclaim that pref-src by definition is irrelevant in any table that's references from a rule that uses source address.

Well... assigning a source address based on source address does seem to be somewhat circular...

In other words, it doesn't work at all for policy based routing.
That's a HUGE bummer.

It does work for PBR, in fact VRFs are internally based on machinery that was originally put in place for PBR. It's just that the packet has to originate in the appropriate VRF (namespace.)

I think it would be more than important to add this as a remark in the documentation for pref-src.

Maybe... I'm sure that if you propose suitable language, Mikrotik will be happy to add it.

I think VRF support would be more elegant, however - and more practically - if the underlying behavior is clearly understood, dnat is in 99% of cases totally fine.
Now here I am again claiming that DNAT or SNAT does actually not seem to be a solution at all. To avoid repeating things, would you mind checking my EDIT1-EDIT3 in this answer? viewtopic.php?p=1111805#p1111805

TLDR is: the main crux is that pref-src would be required to make DNAT work (but then I wouldn't need DNAT). And the issue with SNAT is that address/port is already in use.

I am afraid I have hit a big road block ... is there any other hope?

That's just because you're not trying hard enough

Example:
(192.168.80.1 would be the normal address of the server, 192.168.80.8 is the client, 192.168.84.1 is the address we want the server to use.)

First we identify the traffic and assign it to the correct routing table. (WG on linux supports the fw-mark attribute, but Mikrotik doesn't seem to...)

/ip/firewall/mangle
add chain=output protocol=udp src-port=13231 action=mark-routing new-routing-mark=wg passthrough=yes

The actual address translation is triggered in dst-nat:

/ip/firewall/nat
add chain=dst-nat dst-address=192.168.84.1 protocol=udp dst-port=13231 action=dst-nat to-addresses=192.168.80.1

This results in outgoing packets using the correct routing table, and also the address is set. Logged as.

connection-state:established,dnat proto UDP, 192.168.80.1:13231->192.168.80.8:13231, NAT (192.168.80.1:13231->192.168.84.1:13231)->192.168.80.8:13231

Tested it, and it works flawlessly.

EDIT:
Syntax.
EDIT 2:
Initial solution was wrong.
EDIT 3:
Don't use this, it doesn't handle a specific case correctly. See next post.

lurker888 · Fri Nov 29, 2024 4:28 pm

I've played around a bit more and this is the nicest version:

/ip/firewall/mangle
add chain=output action=mark-packet new-packet-mark=wg passthrough=yes protocol=udp src-port=13231
add chain=output action=mark-routing new-routing-mark=wg passthrough=yes packet-mark=wg

/ip/firewall/nat
add chain=dstnat action=dst-nat to-addresses=192.168.80.1 protocol=udp dst-address=192.168.84.1 dst-port=13231
add chain=srcnat action=src-nat to-addresses=192.168.84.1 packet-mark=wg

wiseroute · Fri Nov 29, 2024 5:05 pm

@lurker,

ok. let us go back to the first problematic ip from the @op - which is 210. and not 177.

how about if we use this output nat just to force the output ip using 210 - because the @op said he doesn't have any problem reaching that ip 210 from the internet (full routing)??

iptables - t nat -a output - s 210 -o wan1 - j snat -to 210 ??

can't that simple line work?

210 is local vlan bridge ip. the same effect as loopback bridge - which is difficult to put some marking.

Amm0 · Fri Nov 29, 2024 5:18 pm

It is the user space applications' duty to then fill out the source address (see https://blog.cloudflare.com/everything- ... ard-socket).

That's a good article, which likely explains roughly what's going on. But the issue is UDP "app" is WG and lives in the kernel. Still 100% agree this is a "bug" — WG's logic shouldn't supersede the rest of RouterOS's routing logic. But again mucking with WG's code/logic may have risks & different side-effects... So I can see why this isn't a quick fix.

Yet still, pref-src of that default route in table default_myas is ignored.

I think @divB hit the nail on the head that pref-src= should actually be respected, since that's were WG should be getting the address from IMO...

Larsa · Fri Nov 29, 2024 5:32 pm

@divB, you're absolutely right to point out that this is a flawed implementation of WireGuard and it drove me nuts too before the root cause was identified. The good news is that it’s actually pretty easy for MikroTik to fix if they decide to. WireGuard works perfectly on Linux with the standard tools. For now, there are two pretty good workarounds; @Sindy's NAT solution or routing rules I mentioned in the link from @Amm0.

@lurker888, unfortunately mangle doesn’t work with WireGuard’s initial handshake process.

lurker888 · Fri Nov 29, 2024 5:34 pm

Hi,

I will assume that you probably meant to write:
iptables - t nat -a output -s 177 -o wan1 - j snat -to 210

In essence these correspond to my mangle/src-nat rules below:

/ip/firewall/mangle
add chain=output action=mark-packet new-packet-mark=wg passthrough=yes protocol=udp src-port=13231

/ip/firewall/nat
add chain=srcnat action=src-nat to-addresses=192.168.84.1 packet-mark=wg

So here we are in agreement. The only difference is that you id the packets based on their (according to op's wishes incorrect) ip, and I do it by being output (generated by router) and proto/port. I think mine is better

, but the approach is the same.

And this is all fine when the .210 (or .177) device initiates the connection. The appropriate conntrack entry is created with the appropriate translation.

The problem comes along when the other side sends the first packet. In this case the src-nat chain is not run. In fact the only chance src-nat would get to run is for the *response* packet, but src-nat is only run for connection state new packets - so actually no address translation is done.

For this case a dst-nat is needed, to enable the same translation for when the other side initiates the connection (from a conntrack perspective):

/ip/firewall/nat
add chain=dstnat action=dst-nat to-addresses=192.168.80.1 protocol=udp dst-address=192.168.84.1 dst-port=13231

My last rule simply makes sure that the lookup is done in the correct table: (This is there because src-nat is actually performed *after* routing adjustment, and as such it would not be able to influence table selection - for example exactly when src address is used as a selector in PBR.)

/ip/firewall/mangle
add chain=output action=mark-routing new-routing-mark=wg passthrough=yes packet-mark=wg

I agree with you that not all rules are necessary in all specific cases, I merely wanted to put the issue fully to rest. Some cases:
* no alternative routing table -> no mark routing is necessary
* connection is always initiated by the remote side -> src-nat is not necessary (and also mark-packet may be omitted)
* connection is always initiated by us -> no dst-nat is necessary

Through testing I have found, that if the connection somehow breaks externally, and there is constant traffic flowing through the tunnel, the second case (again, from a conntrack point of view) cannot be avoided.

That the remote side initiates the connection (third point) cannot really be avoided.

And if you want to make sure that the correct table is used, I would add the routing mark.

So in summary I think:
* to be correct whoever sends the first packet, both dst- and src-nat rules should be present
* if multiple routing tables are used, we should direct the router to use the correct one
* I prefer to identify the packets in mangle/output, but this is just a preference on my part (src address selection is kind of messy)

lurker888 · Fri Nov 29, 2024 5:38 pm

@divB, you're absolutely right to point out that this is a flawed implementation of WireGuard and it drove me nuts too before the root cause was identified. The good news is that it’s actually pretty easy for MikroTik to fix if they decide to. WireGuard works perfectly on Linux with the standard tools. For now, there are two pretty good workarounds; @Sindy's NAT solution or routing rules I mentioned in the link from @Amm0.

@lurker888, unfortunately mangle doesn’t work with WireGuard’s initial handshake process.

I just tested the rules as I have quoted on an rb5009 running 7.17rc1, and the mangle absolutely works for the initial handshake.

lurker888 · Fri Nov 29, 2024 5:40 pm

It is the user space applications' duty to then fill out the source address (see https://blog.cloudflare.com/everything- ... ard-socket).
That's a good article, which likely explains roughly what's going on. But the issue is UDP "app" is WG and lives in the kernel. Still 100% agree this is a "bug" — WG's logic shouldn't supersede the rest of RouterOS's routing logic. But again mucking with WG's code/logic may have risks & different side-effects... So I can see why this isn't a quick fix.

Yet still, pref-src of that default route in table default_myas is ignored.
I think @divB hit the nail on the head that pref-src= should actually be respected, since that's were WG should be getting the address from IMO...

In @divB's case it's impossible for pref-src to work, because his rule selects the table based on the source address. If there is no src address yet, there is nothing to match. If there already is a source address, there's nowhere to put the pref-src.

Larsa · Fri Nov 29, 2024 6:14 pm

I just tested the rules as I have quoted on an rb5009 running 7.17rc1, and the mangle absolutely works for the initial handshake.

Alright, good to know it works with 7.17rc1. Not sure when this changed, but it didn’t work before. If mangle works, that’s a third option along with NAT and routing rules.

lurker888 · Fri Nov 29, 2024 6:19 pm

I just tested the rules as I have quoted on an rb5009 running 7.17rc1, and the mangle absolutely works for the initial handshake.

Alright, good to know it works with 7.17rc1. Not sure when this changed, but it didn’t work before. If mangle works, that’s a third option along with NAT and routing rules.

Could you please point me to the mangle that doesn't work? I'd like to read up on it a bit... (I bet it's probably some load balancing/fail-over scenario)

Larsa · Fri Nov 29, 2024 6:33 pm

Same thread that Amm0 posted previously: viewtopic.php?t=205278

lurker888 · Fri Nov 29, 2024 7:04 pm

Same thread that Amm0 posted previously: viewtopic.php?t=205278

Thanks! I saw the thread, I just didn't want to get into it. This whole thread is about why it will not work in the way that you proposed.

And I have just set up a simulated multi-wan scenario (with a gre tunnel between my test devices serving as the second wan,) and with my proposed solution (adding dst-nat) it works correctly.

Why yours doesn't is actually *not* because the packet goes out on the wrong interface, but because it has a bad source address. And this is exactly because of the thing discussed here in detail.

The misunderstanding comes from the fact that it is quite hard to determine which interface a packet egresses in these situations, because the mangle/output rules apply the routing mark after the initial routing decision and before the routing adjustment step. (Therefore only during the routing adjustment phase can routing marks be used to select the routing table.) However firewall logging takes place *before* the routing adjustment, so in the logs you will always see the packet egressing the interface selected by the initial routing decision. (I have actually captured the packets external to the device (in the switch

) to verify this. Probably sniffing the packets will also yield a good answer because it uses raw sockets - though I have not verified this.)

Larsa · Fri Nov 29, 2024 7:27 pm

Perhaps you didn't read the whole thread and might have missed the most crucial parts:

1) During WG's initial handshake, there's no "connection state," so mangle rules can't apply
2) The initial handshake response always egresses through the default gateway unless you trick ROS into using another interface.

Maybe this changed in 7.17rc1, I dunno, but how it's handled is implementation dependent.

Handshake overview: https://www.wireguard.com/protocol/
Complete protocol stack: https://www.wireguard.com/papers/wireguard.pdf

lurker888 · Fri Nov 29, 2024 8:20 pm

I admit that I have not read everything in full detail, however I see conntrack functioning normally during initial handshake.

Initial incoming handshake as captured by dst-nat:
(rule: chain=dstnat action=dst-nat to-addresses=192.168.80.1 protocol=udp dst-address=192.168.200.1 dst-port=13231 log=yes log-prefix="DN")
DN dstnat: in:gre-tunnel1 out:(unknown 0), connection-state:new proto UDP, 192.168.200.8:13231->192.168.200.1:13231, len 176

Handshake response captured by mangle:
(rule: chain=output action=mark-packet new-packet-mark=wg passthrough=yes protocol=udp src-port=13231 log=yes log-prefix="MA")
MA output: in:(unknown 0) out:gre-tunnel1, connection-state:established,dnat proto UDP, 192.168.80.1:13231->192.168.200.8:13231, NAT (192.168.80.1:13231->192.168.200.1:13231)->192.168.200.8:13231, len 120

When routing is set (same packet as before, next mangle rule):
(rule: chain=output action=mark-routing new-routing-mark=wg passthrough=yes packet-mark=wg log=yes log-prefix="RT")
RT output: in:(unknown 0) out:gre-tunnel1, packet-mark:wg connection-state:established,dnat proto UDP, 192.168.80.1:13231->192.168.200.8:13231, NAT (192.168.80.1:13231->192.168.200.1:13231)->192.168.200.8:13231, len 120

And in the next rule I do a check mathcing on the routing mark, and it is correctly set:
(rule: chain=output action=passthrough routing-mark=wg log=yes log-prefix="RC")
RC output: in:(unknown 0) out:gre-tunnel1, packet-mark:wg connection-state:established,dnat proto UDP, 192.168.80.1:13231->192.168.200.8:13231, NAT (192.168.80.1:13231->192.168.200.1:13231)->192.168.200.8:13231, len 120

And the appropriate routing table is used, and the appropriate interface is egressed.

There is clearly a valid connection state associated with these packets and they are caught in both dst-nat (incoming) and mangle (outgoing).

Maybe I'm a bit dense, tired or don't exactly understand the point, could you elaborate what you mean by "no connection state", "can't apply mangle", "can't set routing table". They seem to work as intended.

Larsa · Fri Nov 29, 2024 8:33 pm

When I started the thread earlier, the initial handshake had to finish before the connection state became "established"[*] which prevented mangling from working. I see you’re using NAT, which might be affecting things similarly to routing rules. Have you tried running a packet trace (assuming on 7.17rc1) without NAT to see the difference?

[*] mangling only works on marked (established) connections.

lurker888 · Fri Nov 29, 2024 9:53 pm

When I started the thread earlier, the initial handshake had to finish before the connection state became "established"[*] which prevented mangling from working. I see you’re using NAT, which might be affecting things similarly to routing rules. Have you tried running a packet trace (assuming on 7.17rc1) without NAT to see the difference?

[*] mangling only works on marked (established) connections.

Yep. dst-nat annotates the conntrack entry, and without that annotation, the conntrack machinery cannot identify the handshake response as belonging to the same connection. (That's why it's there

)

Without the dst-nat rule, the handshake response is not identified as part of the same connection, the connection is in state new, and the connection-marks are lost. Additionally, for it to work correctly, the dst-nat to-addresses has to point to the pref-src of the route in the main table where it would egress. (This is only important for conntrack, the src address is later rewritten because of the dst-nat rule.) This (while absolutely doable) is combersome.

What I would ask from Santa - were he otherwise not occupied because of the season - would be:
* wg should by default, for handshake responses, use the incoming packets dst addr as src addr in the sento() call (without modifying the internal state associated with the wg peers)
* wg interfaces should have an optional local-address attribute (where all underlay packets, including handshake responses, would carry this src addr; this should generally be done by a bind() call)
* wg interfaces should have a VRF

What you write about mangling only being possible on established connections, well, that's not exactly true.

Let's mark the connection based on which interface it ingressed:

/ip/firewall/mangle
add chain=prerouting action=mark-connection new-connection-mark=thru-gre passthrough=yes connection-mark=no-mark in-interface=gre-tunnel1 log=yes log-prefix="CM"

In this case an incoming initial incoming handshake looks like this:

CM prerouting: in:gre-tunnel1 out:(unknown 0), connection-state:new proto UDP, 192.168.200.8:13231->192.168.200.1:13231, len 176
DN dstnat: in:gre-tunnel1 out:(unknown 0), connection-mark:thru-gre connection-state:new proto UDP, 192.168.200.8:13231->192.168.200.1:13231, len 176
MA output: in:(unknown 0) out:gre-tunnel1, connection-mark:thru-gre connection-state:established,dnat proto UDP, 192.168.80.1:13231->192.168.200.8:13231, NAT (192.168.80.1:13231->192.168.200.1:13231)->192.168.200.8:13231, len 120
RT output: in:(unknown 0) out:gre-tunnel1, packet-mark:wg connection-mark:thru-gre connection-state:established,dnat proto UDP, 192.168.80.1:13231->192.168.200.8:13231, NAT (192.168.80.1:13231->192.168.200.1:13231)->192.168.200.8:13231, len 120
RC output: in:(unknown 0) out:gre-tunnel1, packet-mark:wg connection-mark:thru-gre connection-state:established,dnat proto UDP, 192.168.80.1:13231->192.168.200.8:13231, NAT (192.168.80.1:13231->192.168.200.1:13231)->192.168.200.8:13231, len 120

As you can see, prerouting rules were correctly applied to a connecection-state=new (not established) connection, and they are preserved throughout the handshake.

divB · Sat Nov 30, 2024 3:56 am

I've played around a bit more and this is the nicest version:

/ip/firewall/mangle
add chain=output action=mark-packet new-packet-mark=wg passthrough=yes protocol=udp src-port=13231
add chain=output action=mark-routing new-routing-mark=wg passthrough=yes packet-mark=wg

/ip/firewall/nat
add chain=dstnat action=dst-nat to-addresses=192.168.80.1 protocol=udp dst-address=192.168.84.1 dst-port=13231
add chain=srcnat action=src-nat to-addresses=192.168.84.1 packet-mark=wg

Even though I still think that wireguard is totally broken and can't work in my setup, I'm not yet giving up hope.
Just to confirm, 192.168.80.1 would be 192.0.2.177 and 192.168.84.1 would be 192.0.2.210 in my case?
(I tried reversed as well)

Sadly still not

I added "--log" and "--log-prefix" to keep track of when the rules trigger.

Actually, only the DNAT ever triggers:

11-29 17:34:27 firewall,info [***WG-DNAT] dstnat: in:wg-bg1-ftth out:(unknown 0), connection-state:new proto UDP, 200.95.5.88:9081->192.0.2.210:51820, len 176

And I also don't get any "Receiving handshake initiation from peer" messages from wireguard in the logs. So the packet never reaches wireguard.

This is super weird because as soon as I change the DNAT rule to use 172.20.215.1 (the loopback interface), at least I see wireguard receiving it and sending a reply. I don't understand yet why the heck this is happening though.

Regardless, if I understand your solution correctly, you do NOT DNAT to a loopback address but the "victim" address .177 directly. That way, the source address is already set properly?

Here I want to re-emphasize the reason that I use .210 in the first place and not .177: .177 is just the IP of the P2P uplink. I just use a public one because I have one but it's common practice to use RFC1819 addresses for that. Now there are multiple such uplinks, I actually have .177/31, .186/29, .249/31, .253/31. The fact that .177 is used is purely "random" because that's just how iBGP selects the default route out. If some connection drops, it will be a different one. That's the reason I can't make the client connect to .177 and that also means we can't rely on these addresses to be even available.

divB · Sat Nov 30, 2024 4:13 am

Alright, good to know it works with 7.17rc1. Not sure when this changed, but it didn’t work before. If mangle works, that’s a third option along with NAT and routing rules.

Would you mind summarizing again explicitly what the first 2 options are ("1. NAT" and "2. routing rules")?

Re 1: I believe I described why the NAT one can't work here ... but I'm still hoping to be wrong.

Re 2: I assume you mean this thread. But I can't find routing rules posted by you. Do you mean rplant's answer? If so, wouldn't this just be the (D)NAT solution?

lurker888 · Sat Nov 30, 2024 4:30 am

I've played around a bit more and this is the nicest version:
Code: Select all
/ip/firewall/mangle
add chain=output action=mark-packet new-packet-mark=wg passthrough=yes protocol=udp src-port=13231
add chain=output action=mark-routing new-routing-mark=wg passthrough=yes packet-mark=wg

/ip/firewall/nat
add chain=dstnat action=dst-nat to-addresses=192.168.80.1 protocol=udp dst-address=192.168.84.1 dst-port=13231
add chain=srcnat action=src-nat to-addresses=192.168.84.1 packet-mark=wg
Even though I still think that wireguard is totally broken and can't work in my setup, I'm not yet giving up hope.
Just to confirm, 192.168.80.1 would be 192.0.2.177 and 192.168.84.1 would be 192.0.2.210 in my case?
(I tried reversed as well)

Sadly still not
I added "--log" and "--log-prefix" to keep track of when the rules trigger.

Actually, only the DNAT ever triggers:
Code: Select all
11-29 17:34:27 firewall,info [***WG-DNAT] dstnat: in:wg-bg1-ftth out:(unknown 0), connection-state:new proto UDP, 200.95.5.88:9081->192.0.2.210:51820, len 176
And I also don't get any "Receiving handshake initiation from peer" messages from wireguard in the logs. So the packet never reaches wireguard.

This is super weird because as soon as I change the DNAT rule to use 172.20.215.1 (the loopback interface), at least I see wireguard receiving it and sending a reply. I don't understand yet why the heck this is happening though.

Regardless, if I understand your solution correctly, you do NOT DNAT to a loopback address but the "victim" address .177 directly. That way, the source address is already set properly?

Here I want to re-emphasize the reason that I use .210 in the first place and not .177: .177 is just the IP of the P2P uplink. I just use a public one because I have one but it's common practice to use RFC1819 addresses for that. Now there are multiple such uplinks, I actually have .177/31, .186/29, .249/31, .253/31. The fact that .177 is used is purely "random" because that's just how iBGP selects the default route out. If some connection drops, it will be a different one. That's the reason I can't make the client connect to .177 and that also means we can't rely on these addresses to be even available.

Your initial version of IPs is correct. And up to this point only the dnat rule should be triggered.

Without the exact rules, it's quite hard to say... Maybe you could at least send us the rules that should interact with the incoming handshake packet (preferably all of them) and confirm (via logging) that they are indeed triggered.

Playing the guessing game: are you sure you are not filtering the incoming packet? Can you give the rule in filter input chain that accepts it?

wiseroute · Sat Nov 30, 2024 7:39 am

@lurker,

i will assume that you probably meant to write:

iptables - t nat -a output -s 177 -o wan1 - j snat -to 210

no no.. it is literally -s 210 -o wan1 -j snat -to 210

ok. let us try to break down @divb first scenario:
- vlan bridge/loopback/wireguard listen ip 210
- wan1 ip 177 no nat. full routing.

the problem:
the internet can reach 210 directly, but the reply always rewritten to 177 (hence got rejected by initiator).

try by analyzing these 2 povs :
1. using dnat for incoming internet via wan1 to vlan wg Bridge 210

iptables -t nat -a input -i wan1 -d 210 -j dnat -to 210.

2. using snat for outgoing vlan bridge to internet via wan1 (this is the actual problem: 210 rewritten to 177)

iptables -t nat -a output -o wan1 -s 210 -j snat -to 210.

just to make that ip 210 persistent across interface traffic.

please carefully note those incoming and outgoing interfaces corresponding to their iptables chain. no marking no rules - just simple nat.

hopefully they'll work - except if there is any other thing on the routing which alters their functionality.

as for the other ip's - just mirror those config.

divB · Sun Dec 01, 2024 1:45 am

@lurker,

i will assume that you probably meant to write:

iptables - t nat -a output -s 177 -o wan1 - j snat -to 210
no no.. it is literally -s 210 -o wan1 -j snat -to 210

ok. let us try to break down @divb first scenario:
- vlan bridge/loopback/wireguard listen ip 210
- wan1 ip 177 no nat. full routing.

the problem:
the internet can reach 210 directly, but the reply always rewritten to 177 (hence got rejected by initiator).

try by analyzing these 2 povs :
1. using dnat for incoming internet via wan1 to vlan wg Bridge 210

iptables -t nat -a input -i wan1 -d 210 -j dnat -to 210.

2. using snat for outgoing vlan bridge to internet via wan1 (this is the actual problem: 210 rewritten to 177)

iptables -t nat -a output -o wan1 -s 210 -j snat -to 210.

just to make that ip 210 persistent across interface traffic.

please carefully note those incoming and outgoing interfaces corresponding to their iptables chain. no marking no rules - just simple nat.

hopefully they'll work - except if there is any other thing on the routing which alters their functionality.

as for the other ip's - just mirror those config.

The commands you listed don't really work because dstnat can only be in output but not input (and snat in input but not output).

I have used these commands (pls let me know if I misunderstood):

/ip firewall nat
add action=dst-nat chain=dstnat dst-address=192.0.2.210 dst-port=51820 log=yes log-prefix="[***WG-DNAT]" protocol=udp to-addresses=192.0.2.210
add action=src-nat chain=srcnat dst-port=51820 log=yes log-prefix="[***WG-SNAT]" protocol=udp src-address=192.0.2.210 to-addresses=192.0.2.210

I do not see how both of them work at the same time. For the second rule, if address would already be 192.0.2.210 we wouldn't have an issue in the first place. Indeed, it's not even matched.

And the first rule (DNAT) does not work for the reason I mentioned already: Since wireguard answers with a random source address, it is never matched with the connection tracking entry of the DNAT connection. This can be seen by the fact that the output packet is "connection-state:new".

If I missed anything, please let me know

This is log output:

15:35:34 wireguard,debug wg-mobile: [miPhone] ***: Receiving handshake initiation from peer (200.95.7.232:33018) extra:0 (einval) 
15:35:34 wireguard,debug wg-mobile: [miPhone] ***: Sending handshake response to peer (200.95.7.232:33018) 
15:35:34 firewall,info [***WG-DNAT] dstnat: in:wg-bg1-ftth out:(unknown 0), connection-state:new proto UDP, 200.95.7.232:33018->192.0.2.210:51820, len 176 
15:35:34 firewall,info [***WG-IN] input: in:wg-bg1-ftth out:(unknown 0), connection-state:new proto UDP, 200.95.7.232:33018->192.0.2.210:51820, len 176 
15:35:34 firewall,info [***WG-OUT] output: in:(unknown 0) out:vlan2, packet-mark:wg connection-state:new proto UDP, 134.180.130.235:51820->200.95.7.232:33018, len 120

This is packet sniffer output:

58 time=187.238 num=59 direction=rx interface=wg-bg1-ftth src-address=200.95.7.232:33016 dst-address=192.0.2.210:51820 protocol=ip ip-protocol=udp size=176 cpu=1 ip-packet-size=176 ip-header-size=20 dscp=0 
   identification=52484 fragment-offset=0 ttl=49 

59 time=187.238 num=60 direction=tx interface=wg-bg2-ftth src-address=134.180.130.235:51820 dst-address=200.95.7.232:33016 protocol=ip ip-protocol=udp size=120 cpu=2 ip-packet-size=120 ip-header-size=20 dscp=34 
   identification=20323 fragment-offset=0 ttl=64

divB · Sun Dec 01, 2024 2:46 am

It seems finally I could get it working with the combined ideas of @lurker888, @wiseroute, @Larsa and myself.

Note that neither DNAT or SNAT by itself work for the reasons I mentioned a couple of times. It is important to indeed use a separate dummy device for DNAT. It is also necessary to create a dummy routine table and PBR rules based on source address

Anyway, let me just post my solution:

/interface bridge
add arp=disabled fast-forward=no name=dum1 protocol-mode=none

/ip address
add address=172.20.215.1 interface=dum1 network=172.20.215.1

/routing table
add comment="Dummy table containing a default route for everything from 172.20.215.1" fib name=bugfix_wg

/routing rule
add action=lookup-only-in-table src-address=172.20.215.1 table=bugfix_wg

/ip firewall mangle
add action=mark-routing chain=output log=yes log-prefix="[***WG-MANGLE-RT]" new-routing-mark=default_myas passthrough=yes protocol=udp src-address=172.20.215.1 src-port=51820

/ip firewall nat
add action=dst-nat chain=dstnat dst-address=192.0.2.210 dst-port=51820 log=yes log-prefix="[***WG-DNAT]" protocol=udp to-addresses=172.20.215.1 to-ports=51820

This is how I think it works:

The DNAT rule as prev proposed translates 192.0.2.210 --> 172.20.215.1
For the translation of the return packet to work, it is instrumental that wireguard will use 172.20.215.1 as source address
This is accomplished by using dummy table bugfix_wg and looking it up when the source address comes from the dum1 interface (src-address=172.20.215.1). (Note: Would it be correct what @lurker suggested that for the initial "routing decision" the src-address rule can't be selected because source address is empty, this wouldn't work. But my table is clearly selected)
Now the packet is in interface dum1 with source address 172.20.215.1. This is important because only now the source address can be reverted by the connection tracking of the DNAT rule!
However, now the packet would never get out because it's stuck in dum1 (via dummy routing table). Hence the mangle rule ensures that the routing decision is overruled and the "default_myas" table is selected

I did not expect this could ever work and this is the worst hack I have ever done. In 3 months from now I will have no clue how and why this configuration works. Only because of an (in my opinion), terrible wireguard design decision and implementation. Would just one of these be implemented, one wouldn't have to revert to such criminal hacks:

Do not just bind to any interface/address of the system; at least support "bind-address"
Do not just leave the source port unset (especially when there was no connection context before and a client just connected to the right endpoint address)
Do implement VRF for wireguard

divB · Mon Dec 02, 2024 12:37 am

One day later and this stopped working. The crux really seems to be the initial source address that is assigned on the first routing decision.

I'm sure I have the same config but different source address is chosen. Instead of 172.20.215.1, it's again the ISP IP or something else.

@lurker888: Do you know where this routing decision documented? In particular, how, based on /routing/rule and /ip/route is the address on the first step routing decision decided?
From my tests it still seems to me that pref-src or even the selected route is totally arbitrary for a local packet with unset source address.

For example, I tried to SNAT the external connection to the local address 172.20.215.1. That way, the wireguard response will always go to 172.20.215.1 and I can add a route to 172.20.215.1:

/ip/route/add dst-address=172.20.215.1/32 gateway=dum1 pref-src=172.20.215.1

Shouldn't this route be selected all the time now and pref-src be honored?

divB · Mon Dec 02, 2024 2:51 am

I think the issue with pre-src on 172.20.215.1/32 is that there is another direct connected route with higher priority.

Alright, another crazy option: SNAT to an invalid address and then use a static route to force pref-src on that prefix.
Disadvantage: All packages appear to come from the own address, visibility on road warrior IP address is lost

/ip route
add comment=bugfix-wg dst-address=172.20.215.254 gateway=vlan44 pref-src=192.0.2.210 routing-table=main
/ip firewall filter
add action=accept chain=input dst-address=192.0.2.210 dst-port=51820 protocol=udp
/ip firewall nat
add action=src-nat chain=input dst-address=192.0.2.210 dst-port=51820 protocol=udp to-addresses=172.20.215.254

Note that 172.20.215.254 is not assigned anywhere locally on the router.

Mimiko · Tue Apr 01, 2025 2:46 pm

Some one managed to solve this problem?

My environment have one wireguard instance bound to 13231. There is 2 different ISP each having 3 external IPs. This is used for load balancing internet access and access from internet to local dnated services. Only the problem with wireguard source IP and port can not set up.

I've tried the examples here with adjustment to my routing markings and ips. Nothing does work. Or there is no connection established, or the established connection does not pass crypted traffic. The only working setup is for client specify first (of mikrotik logic) external ip to connect and then it will work.

What I observed is, that if client starts a connection to any other ip (than the first one from mikrotik's point of view), then (accordingly to wireguard's philosophy) a new connection is created as response and that connection has a src IP set to first IP of the mikrotik's point of view. This is shown in connections tracking table. So there is no any tracking of connections as wireguard does not respond to the incoming packet, but rather creates a new one with destination of client's ip and port.

If I try to mark-routing the outbound packets accordingly to examples here, then indeed packet is send via specified provider, but the src address of packet is not correct for that provider. And so the packet is filtered at providers firewall as incorrect src addresses.
If I try to src-nat the output so the src address will be the one needed for that provider (and the one client addressed at first packet), then mikrotik routers change the src port to some random one and on client I see the packet coming from desired IP, but the port is not the servers wireguard port.

For reference, I am on 7.16.2 firmware.

Larsa · Tue Apr 01, 2025 2:56 pm

Have you tried policy routing using routing rules, with separate routing tables for each WireGuard instance?

Mimiko · Tue Apr 01, 2025 3:09 pm

Have you tried policy routing using routing rules, with separate routing tables for each WireGuard instance?

I've tried to use routing rules, but that does not help at all. Creating multiple wireguard instances is out of concept for multiwan access. Wireguard instances can not be bound to specific interface or ip.

The routing rules can capture routing marks (as src address and dst address are somehow dynamic) of packets but after output packed was created by wireguard and a routing decision and a src ip was assigned. So setting the routing mark at mangle/output does not change the src ip. And if a routing rule is applied, the packed is routed to desired outgoing interface, but the pref-source is not used to rewrite the src ip of the packet.

Larsa · Tue Apr 01, 2025 3:44 pm

It works. With routing rules, you force outbound traffic to use a specific egress interface, which automatically sets the source address to that WAN’s IP. But you can also use the Sindy NAT trick: "/ip firewall nat chain=dstnat dst-address-type=local in-interface=WAN2 protocol=udp dst-port=wg-port action=dst-nat to-addresses=ip.of.wan"

Mimiko · Tue Apr 01, 2025 4:06 pm

It works. With routing rules, you force outbound traffic to use a specific egress interface, which automatically sets the source address to that WAN’s IP. But you can also use the Sindy NAT trick: "/ip firewall nat chain=dstnat dst-address-type=local in-interface=WAN2 protocol=udp dst-port=wg-port action=dst-nat to-addresses=ip.of.wan"

Can you please supply an example of configuration with routing rules for this situation:
ISP1:
IP1 1.2.3.2/24
IP2 1.2.3.3/24
ISP2:
IP1 2.1.3.4/24
IP2 2.1.3.5/24

wireguard port: 13231

Clients should connect to any of the IP and receive packets back from same IP and same port (13231).

I dont see how NAT trick can handle multiple wan ip addresses on same WAN2 interfaces.

Larsa · Tue Apr 01, 2025 4:13 pm

Okay, but first try to explain what you're trying to accomplish without using too many technical terms.

Mimiko · Tue Apr 01, 2025 4:47 pm

Okay, but first try to explain what you're trying to accomplish without using too many technical terms.

That what is used the wireguard for:

Clients (some remote routers) should connect to any IP given by any ISP at specified port 13231. The mikrotik as a server only acts as responder for clients. Mikrotik should never initiate inital connection to clients. Mikrotik will only respond when client make first connection.

Clients does not have any incoming port open (they are behind some provider) and each client make connect from random IP and random port. The providers will have connection tracking (I hope) that response from wireguard will reach the clients. For this to happen wireguard should respond with same IP and port on which client had first contacted.

Practically this is a multi-wan multiple public IP setup.

lurker888 · Tue Apr 01, 2025 4:56 pm

I think I understand what @Mimiko want: simply to have many WAN addresses, and for wg to always answer on that address.

WG intentionally as a bit of a strange behavior (different from the usual stuff like ping, DNS over UDP, OpenVPN over UDP, etc.)

One way of doing what is asked for here is the following. It's not pretty, but it actually works and does so without abusing any functionality.

Create a bridge just for internal use. We will use this to mess with the wg traffic in order to get conntrack to be able to identify connections properly:

/bridge
add name=wg-br
/ip/address
add address=192.168.222.1/30 interface=wg-br

And then we NAT:

/ip/firewall/nat
add chain=dstnat dst-address=1.2.3.2 protocol=udp dst-port=13231 action=dst-nat to-address=192.168.222.1
add chain=dstnat dst-address=1.2.3.3 protocol=udp dst-port=13231 action=dst-nat to-address=192.168.222.1
(...)

/ip/firewall/nat
add chain=input dst-address=192.168.222.1 action=src-nat to-addresses=192.168.222.2

There you have it. Again, not pretty but it works.

EDIT: Because this appeases the gods of conntrack, this also makes connection marks sticky, so it can be used with multiple routing tables, VRFs, etc.

EDIT2. I finished the previous edit with "In fact the whole wg-br stuff can be hidden in a VRF is you want." This has been implemented in normal Linux for quite some time; it's done using the fwmark property of the wg interface (which marks *underlay* packets when being emitted) and then using routing rules based on this fwmark. Unsupported in the MT world, unfortunately.

Larsa · Tue Apr 01, 2025 6:35 pm

@lurker888: Great summary! Just wondering, why go with a bridge instead of assigning the addresses straight to the wg interface in this case?

@Mimiko: You’ve probably already thought about it, but just a heads-up that each client still needs a unique public key in a separate peer entry, otherwise WireGuard’s internal crypto routing won’t work properly.

lurker888 · Tue Apr 01, 2025 7:26 pm

@Larsa: Of course this also works with the two addresses on any interface (one assigned to the router and one not assigned, but routed to the interface in the "main" table - these are the criteria for conntrack to work). For me the wg interface is associated with the overlay traffic and in my applications is often included in a vrf, while we are tracking/natting underlay traffic in this instance. In other words for me it just "feels better."

It would also be valid to ask why input src-nat is omitted from the official packet flow diagrams by Mikrotik, while being fully and correctly supported.

Larsa · Tue Apr 01, 2025 7:55 pm

It would also be valid to ask why input src-nat is omitted from the official packet flow diagrams by Mikrotik, while being fully and correctly supported.

Yeah, that question has come up a few times before. Personally, I think those packet flow diagrams have a lot of room for improvement!

Mimiko · Wed Apr 02, 2025 8:53 am

EDIT: Because this appeases the gods of conntrack, this also makes connection marks sticky, so it can be used with multiple routing tables, VRFs, etc.

EDIT2. I finished the previous edit with "In fact the whole wg-br stuff can be hidden in a VRF is you want." This has been implemented in normal Linux for quite some time; it's done using the fwmark property of the wg interface (which marks *underlay* packets when being emitted) and then using routing rules based on this fwmark. Unsupported in the MT world, unfortunately.

Thank you. This does work. I already have marked and sticky connections implemented per each incoming WAN IP.
Also @Larsa, of course each client is configured with its own public key and all allowed addresses. It was just this thing about different WAN IP.

@lurker888 why define a rule in /ip/firewall/nat for each dst-address? I've omitted the dst-address and left only dst-port.
Using src-nat hides the ip address of the client. Is this possible to fix? The real ip address of the client give a good troubleshooting info.
If I remove the src-nat rule, the problem comes back.

Mimiko · Wed Apr 02, 2025 9:12 am

I've tried the solution in this post viewtopic.php?p=1136345#p1136345 last posts.

While using that method do send packet to correct gateway, the initial src ip of the packet generated by wireguard does not change even if I put pref-source on the route entry in fib.

lurker888 · Wed Apr 02, 2025 11:37 am

Thank you. This does work. I already have marked and sticky connections implemented per each incoming WAN IP.
Also @Larsa, of course each client is configured with its own public key and all allowed addresses. It was just this thing about different WAN IP.

@lurker888 why define a rule in /ip/firewall/nat for each dst-address? I've omitted the dst-address and left only dst-port.
Using src-nat hides the ip address of the client. Is this possible to fix? The real ip address of the client give a good troubleshooting info.
If I remove the src-nat rule, the problem comes back.

The src-nat is necessary for conntrack to be able to recognize the connections correctly (and to be able to apply the reverse translation for the dst-nat, and add the connection marks). This does mean that the source IP of the connections is masked. I know of no way around this. Of course you can log the packets in firewall or print them from the conntrack entries etc., however these are just workarounds.

Of course only specifying the dst-port for the dst-nat rules will also nat the intended connections. This however also has the effect of natting every packet that is udp/13231, and so no one who uses this port will be able to correctly communicate with anyone if their traffic is routed by your device. This is bad. (Maybe creating an address list would satisfy you?)

As to why the usual way of using output mangle results in the correct src address not being set, even though the correct routing table is selected: I wrote a *really* long post on this some time ago that takes you through the packet flow and the decisions the router makes step-by-step. You are only a search away. That thread also explains why wireguard is special in this regard.

Also, if you actually plan on using this dst-nat and input src-nat solution, take care to use a firewall udp-timeout larger than your wg keepalive, and of course a longer udp-stream-timeout. I use 35s, 10s and 6m respectively.

CGGXANNX · Wed Apr 02, 2025 11:48 am

Of course only specifying the dst-port for the dst-nat rules will also nat the intended connections. This however also has the effect of natting every packet that is udp/13231, and so no one who uses this port will be able to correctly communicate with anyone if their traffic is routed by your device. This is bad. (Maybe creating an address list would satisfy you?)

I think here using dst-address-type=local (instead of listing the dst-address entries) might work?

Mimiko · Wed Apr 02, 2025 12:50 pm

As to why the usual way of using output mangle results in the correct src address not being set, even though the correct routing table is selected: I wrote a *really* long post on this some time ago that takes you through the packet flow and the decisions the router makes step-by-step. You are only a search away. That thread also explains why wireguard is special in this regard.

Tried the search and is hard to find that post.

@CGGXANNX I use interface lists for this.

anav · Wed Apr 02, 2025 6:33 pm

HI Larsa and Lurker, have been attempting to follow these entangled threads but not making much headway other than Lurker seems to have come up with a way regardless of scenario to basically ensure that in a multi-wan scenario, RoS can be manipulated to ensure wireguard connections work properly. Notice I avoid the "S" word because in reality it is some bad ass work-around.

One of the threads pointed out that the usual mangle method didnt guarantee the proper WAN for output which was 'hacked' by sindy so that the return from a secondary WAN would indeed go out that WAN instead of the primary. This required the usual mangling of input and output chains for traffic into and out of a specific WAN, and then adding in Sindy's magic.
/ip firewall nat
chain=dstnat dst-address-type=local in-interface=WAN2 protocol=udp dst-port=wg-port action=dst-nat to-addresses=ip.of.wan.1

Which, in effect ensures that any traffic associated with the wireguard and WAN2 is dstnatted to WAN1 and thus on the way out of the router, the traffic is undestinatted so leaves the router with WAN2 IP.

++++++++++++++++++++++++++++++++++++++

The above seems to work with traffic arriving at the router as opposed to traffic originating from the traffic.
Can you state conceptually what you are doing that is different from the above... and what your are adding to the mix.......... ( in terms that a non-IT person can grasp ).

+++++++++++++++++++++++++++++++++++

As an aside MT recently introduced responder checkbox and it seems tied to a peer which Ithink incorrectly identifies its an issue with the remote peer whereas I think it tries to fix a problem at local router and thus should be part of the wiireguard interface menu not peer.
Does anyone know what this selection actually does, and does it have any relationship to the above .......

lurker888 · Wed Apr 02, 2025 7:41 pm

@Mimiko
My post that explains why wireguards has this idiosyncrasy in its design (right here on this thread):
viewtopic.php?p=1111816#p1111816

And the one that explains the source address assignment in detail: (You should probably scroll though the entire thread)
viewtopic.php?p=1122715#p1122715

@anav
I don't know if the question in the +++ / +++ was addressed to me, but: The workaround presented both by sindy and me rely on the same way of using dst-nat to achieve connection tracking. Mine is of course better (obviously

) in that sindy's fails if:
* the wan1 interface goes down (the address we dst-nat to is lost)
* the default route's pref src changes (for example because of wan1 going down)

My not-exactly-"s" works around these by: (1) dst-natting to a local address that can't go down and (2) using the input src-nat ensuring that wg underlay output packets are addressed to the (in some ways virtual) 192.168.222.2 address, the route to which (and its pref-src) cannot change.

Of course sindy's method preserves the source address, which is nicer. On the other hand in VRF-heavy scenarios input src-nat does wonders for overloaded source IPs (in fact conntrack cannot be correctly maintained without it).

anav · Wed Apr 02, 2025 8:57 pm

Thanks much lurker, that is most helpful for me and will take the time to digest traffic flows as you have manipulated them!!

Any thoughts on what the responder checkbox is trying to do??

lurker888 · Wed Apr 02, 2025 9:49 pm

Any thoughts on what the responder checkbox is trying to do??

Sorry, I meant to answer that one as well, just forgot by the time I got there.

Responder is a very useful feature. Wireguard in its default form (that generally should not be reconfigured by the user) behaves in the following way:
* the purpose of handshakes is to create an ephemeral symmetric key for data encryption (and MAC generation)
* an ephemeral key is discarded (invalidated, overwritten, shredded) 180s after its creation
* if a packet is to be sent to the other side (including keepalives) and the peer entry has a valid ephemeral key, then the data packet i sent using this key
* if when sending a packet there is an ephemeral key, but it is more than 150s old (180s - 30s) a new handshake is initiated - until the new handshake completes, but the existing key has not been discarded, the existingt key is used for communication
* if when sending a packet, there is no ephemeral key (that is valid) a handshake is initiated - traffic will resume when (after a successful handshake) a new ephemeral key is established

WG treats the two sides of the tunnel the same (both are peers) but it can't be overlooked that in many instances it is used as part of a client-server architecture. In these instances it is useful to preclude the server from initiating repeated handshakes. Marking the client peer on the server side as responder=yes does this - that is it simply instructs the side where it is set to never initiate a handshake. (Note that wg maintains a "current endpoint" ip/port separate from the configured "endpoint" address/peer, that is: if the client is no longer available, the server will send repeated and endless handshake requests to the last address of the client as long as it sees packets that should be routed to that client.)

There is a bit of a misconception that keepalive should *only* be enabled on the client side and disabled on the server. This is partially true in that this is a correct way of configuring things. However it is not mandatory to do so, and it is also correct to set keepalives on the responder=yes side. This simply has the effect of the server sending keepalives as well, which will be silently dropped if no handshake from the client side is initiated for some time (because with responder=yes, the server side will never initiate a handshake.)

Mimiko · Thu Apr 03, 2025 10:39 am

One of the threads pointed out that the usual mangle method didnt guarantee the proper WAN for output which was 'hacked' by sindy so that the return from a secondary WAN would indeed go out that WAN instead of the primary. This required the usual mangling of input and output chains for traffic into and out of a specific WAN, and then adding in Sindy's magic.
/ip firewall nat
chain=dstnat dst-address-type=local in-interface=WAN2 protocol=udp dst-port=wg-port action=dst-nat to-addresses=ip.of.wan.1

I'm wondering how dnating to onother ip of wan1 will assure that originated from wireguard packets will have src ip of second wan? Or i'm missing something.

@Mimiko
My post that explains why wireguards has this idiosyncrasy in its design (right here on this thread):
viewtopic.php?p=1111816#p1111816

And the one that explains the source address assignment in detail: (You should probably scroll though the entire thread)
viewtopic.php?p=1122715#p1122715

Thank you for the links.

Mimiko · Thu Apr 03, 2025 11:22 am

One of the threads pointed out that the usual mangle method didnt guarantee the proper WAN for output which was 'hacked' by sindy so that the return from a secondary WAN would indeed go out that WAN instead of the primary. This required the usual mangling of input and output chains for traffic into and out of a specific WAN, and then adding in Sindy's magic.
/ip firewall nat
chain=dstnat dst-address-type=local in-interface=WAN2 protocol=udp dst-port=wg-port action=dst-nat to-addresses=ip.of.wan.1

Which, in effect ensures that any traffic associated with the wireguard and WAN2 is dstnatted to WAN1 and thus on the way out of the router, the traffic is undestinatted so leaves the router with WAN2 IP.

Do you have a full example of the specific config to try?

anav · Thu Apr 03, 2025 1:21 pm

If you want help with your setup post a new thread and will need your traffic requirements and current config.

Mimiko · Thu Apr 03, 2025 3:57 pm

This is my test setup.

/interface bridge add arp=disabled frame-types=admit-only-vlan-tagged name=dum1 protocol-mode=none vlan-filtering=yes
/routing table add disabled=no fib name=bugfix_wg
/ip address add address=172.20.215.1/31 interface=dum1 network=172.20.215.0

/ip firewall filter add action=add-src-to-address-list address-list=WG-VIA-36 address-list-timeout=1m chain=input dst-address=X.X.X.36 dst-port=13231 protocol=udp
/ip firewall filter add action=add-src-to-address-list address-list=WG-VIA-37 address-list-timeout=1m chain=input dst-address=X.X.X.37 dst-port=13231 protocol=udp
/ip firewall filter add action=add-src-to-address-list address-list=WG-VIA-50 address-list-timeout=1m chain=input dst-address=Y.Y.Y.50 dst-port=13231 protocol=udp
/ip firewall filter add action=add-src-to-address-list address-list=WG-VIA-55 address-list-timeout=1m chain=input dst-address=Y.Y.Y.55 dst-port=13231 protocol=udp
/ip firewall filter add action=accept chain=input dst-port=13231 protocol=udp src-port=13231

/ip firewall mangle add action=add-src-to-address-list address-list=WG-VIA-36 address-list-timeout=1m chain=input dst-address=X.X.X.36 dst-port=13231 protocol=udp
/ip firewall mangle add action=add-src-to-address-list address-list=WG-VIA-37 address-list-timeout=1m chain=input dst-address=X.X.X.37 dst-port=13231 protocol=udp
/ip firewall mangle add action=add-src-to-address-list address-list=WG-VIA-50 address-list-timeout=1m chain=input dst-address=Y.Y.Y.50 dst-port=13231 protocol=udp
/ip firewall mangle add action=add-src-to-address-list address-list=WG-VIA-55 address-list-timeout=1m chain=input dst-address=Y.Y.Y.55 dst-port=13231 protocol=udp

/ip firewall mangle add action=mark-connection chain=output dst-address-list=WG-VIA-55 new-connection-mark=In-Via-55 passthrough=yes protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-connection chain=output dst-address-list=WG-VIA-36 new-connection-mark=In-Via-36 passthrough=yes protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-connection chain=output dst-address-list=WG-VIA-50 new-connection-mark=In-Via-50 passthrough=yes protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-connection chain=output dst-address-list=WG-VIA-37 new-connection-mark=In-Via-37 passthrough=yes protocol=udp src-address=172.20.215.1 src-port=13231

/ip firewall mangle add action=mark-routing chain=output dst-address-list=WG-VIA-36 new-routing-mark=Via-36 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output dst-address-list=WG-VIA-37 new-routing-mark=Via-37 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output dst-address-list=WG-VIA-50 new-routing-mark=Via-50 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output dst-address-list=WG-VIA-55 new-routing-mark=Via-55 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-50 new-routing-mark=Via-50 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-55 new-routing-mark=Via-55 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-56 new-routing-mark=Via-56 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-Y new-routing-mark=Via-Y passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-36 new-routing-mark=Via-36 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-37 new-routing-mark=Via-37 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-X new-routing-mark=Via-X passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231

/ip firewall nat add action=add-src-to-address-list address-list=WG-VIA-36 address-list-timeout=1m chain=dstnat dst-address=X.X.X.36 dst-address-type=local dst-port=13231 protocol=udp
/ip firewall nat add action=add-src-to-address-list address-list=WG-VIA-37 address-list-timeout=1m chain=dstnat dst-address=X.X.X.37 dst-address-type=local dst-port=13231 protocol=udp
/ip firewall nat add action=add-src-to-address-list address-list=WG-VIA-50 address-list-timeout=1m chain=dstnat dst-address=Y.Y.Y.50 dst-address-type=local dst-port=13231 protocol=udp
/ip firewall nat add action=add-src-to-address-list address-list=WG-VIA-55 address-list-timeout=1m chain=dstnat dst-address=Y.Y.Y.55 dst-port=13231 protocol=udp
/ip firewall nat add action=dst-nat chain=dstnat dst-address-type=local dst-port=13231 protocol=udp to-addresses=172.20.215.1
/ip firewall nat add action=accept chain=srcnat protocol=udp src-port=13231

/ip route add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=dum1 pref-src=172.20.215.1 routing-table=bugfix_wg scope=30 suppress-hw-offload=no target-scope=10
/routing rule add action=lookup-only-in-table disabled=no src-address=172.20.215.1/32 table=bugfix_wg

I can not set IP of dum1 bridge to /32 prefix as this somehow breaks BGP.

X.X.X.36 is the IP of WAN1 which is set to packets src generated by wireguard.
Client connects to Y.Y.Y.55 and at first connections the packets come back from Y.Y.Y.55 IP.
But if a ping is made from server (mikrotik) or something initiates a packet from wireguard (ie keepalive), the src IP of the initated packet is changed back to X.X.X.36.

The only real solution working is from @lurker888 with srcnat. But in this case mikrotik does not see the real IP of connected client.

lurker888 · Thu Apr 03, 2025 7:48 pm

I'm wondering how dnating to onother ip of wan1 will assure that originated from wireguard packets will have src ip of second wan? Or i'm missing something.

Both dst-nat and src-nat (and masquerade) actions do *both* a translation that is specified in the rule, *and* a translation of the packets in the reverse direction. If you enable logging in your firewall you will see this as
NAT realsrcip->(wan2ip->wan1ip)

This means that incoming packets have their dst addr altered from wan2ip to wan1ip, but *also* the response packets get their src addr altered to wan2ip. What was not explained very clearly in this thread is that this works (only) in case the lookup in the main table would result in the response packet going through wan1, so when emitted has the addressing wan1ip->realsrcip, which conntrack identifies and trnasforms to (wan1ip->wan2ip)->realsrcip.

In my version the incoming transform is (realsrcip->192.168.222.2)->(wan2ip->192.168.222.1) and the reverse translation is (192.168.222.1->wan2ip)->(192.168.222.2->realsrcip)

The only real solution working is from @lurker888 with srcnat. But in this case mikrotik does not see the real IP of connected client.

Well, thank you. I didn't find having to live with the real IP being masked as particularly onerous. WG packets are authenticated, and in case you want to do filtering in the firewall, the matchers have access to the pre-translation source ip.

If you meet the criteria that I've explained in the previous post about "main" routes and addresses not being lost, sindy's version works absolutely fine with leaving the source address in place. Of course only if correctly implemented

Mimiko · Fri Apr 04, 2025 7:51 am

If you meet the criteria that I've explained in the previous post about "main" routes and addresses not being lost, sindy's version works absolutely fine with leaving the source address in place. Of course only if correctly implemented

Public addresses are static, but providers may disconnect of some power outage or any other problem and reconnect back. Mostly it is constant.
I do not find any of Sindy's post related to this problem. Could you pinpoint it?

lurker888 · Fri Apr 04, 2025 9:52 am

If you meet the criteria that I've explained in the previous post about "main" routes and addresses not being lost, sindy's version works absolutely fine with leaving the source address in place. Of course only if correctly implemented
Public addresses are static, but providers may disconnect of some power outage or any other problem and reconnect back. Mostly it is constant.
I do not find any of Sindy's post related to this problem. Could you pinpoint it?

Well there you have it, if the interface goes down, you lose the address.

Sindy's method was discussed in a similarly long thread. I don't think any ready-made solution/config was available beyond what was discussed here. Correctly applying these requires a better than usual understanding of several of the interrelated mechanisms at play.

I saw the address-list based approach you offer for discussion in the other thread on this forum. I don't really get it.

Larsa · Fri Apr 04, 2025 11:37 am

@Mimiko, your only options are either routing rules or NAT, unless you’re on one of the newer ROS versions where they supposedly fixed the issue, i.e. that WireGuard’s initial handshake is going out via the default gateway instead of the inbound interface. Might’ve been mentioned in this or the other big thread on the topic. We’re running v7.15 ourselves and use routing rules for multi-WAN WireGuard.

Mimiko · Fri Apr 04, 2025 2:59 pm

I saw the address-list based approach you offer for discussion in the other thread on this forum. I don't really get it.

The idea of the poster behind the address list is to mimic connection tracking for wireguard. Ie. First connection comes to IP2 then put the client address in the address list for IP2. When wireguard respond, catch that packet and route via desired outbound gateway using routing tables and put the pref-source in header. But this does not work. I tried to use srcnat to IP2 if packet originates to dst IP of client in the address list, and this changes the src IP to IP2 as desiret, but also changes the src port of the packet to some random. I tried to put also src port same port to force the src ip and port, but then no packet will leave the router. This is a bug or its how it works I don't know. Anyway, address lists idea is to mimic connection tracking for wireguard. Setting timeout when adding src ip of client to address lists to same timeout of UDP packet in connection tracking will behave same with the timeout occurs, so a keepalive will always keep the address in the list.

Larsa · Fri Apr 04, 2025 5:45 pm

...ROS versions where they supposedly fixed the issue, i.e. that WireGuard’s initial handshake is going out via the default gateway instead of the inbound interface. Might’ve been mentioned in this or the other big thread on the topic.

@Mimiko, I found the post.

Turns out it was @lurker888 who mentioned in this thread (post #23) that the issue was fixed in v7.17. So if that’s true, you should be able to sort out your issues the usual way with mangle and routing marks.

anav · Fri Apr 04, 2025 7:21 pm

Lurker, gone down many a rabbit hole, I cannot seem to work my way through the noise of your solution..........
Context: Two WANS, WAN1 primary, and WAN2 secondary and wishing to use WAN2 as the wireguard connection.

If given a faux bridge 192.168.66.0/32 address
and given a listening port of 55555, WAN1IP = 24.X and WAN2IP = 24.y
What is your proposal for
a. NAT Rules
B. Mangle rules
C. IP routes???

We cannot interfere with other traffic on WAN2, https, DNS queries etc that may need to work as well, especially if WAN1 goes down.

lurker888 · Sat Apr 05, 2025 3:37 am

I was planning on writing out an example, so just for your reading pleasure: the official "rabbit hole solution" to multi-wan wireguard hosting.

We have two WANS. For this example, the two WAN connections are actually EoIP tunnels to another router. The base configuration is:

Interfaces:

/interface eoip
add name=eoip-wan1 remote-address=[other_router] tunnel-id=10
add name=eoip-wan2 remote-address=[other_router] tunnel-id=20

Addresses:

/ip address
add address=192.168.111.10/24 interface=eoip-wan1
add address=192.168.112.10/24 interface=eoip-wan2

Routes:

/routing table
add fib name=wan2

/ip route
add distance=1 dst-address=0.0.0.0/0 gateway=192.168.111.1
add distance=20 dst-address=0.0.0.0/0 gateway=192.168.112.1
add disabled=no dst-address=0.0.0.0/0 gateway=192.168.112.1 routing-table=wan2

WAN masquerade:

add action=masquerade chain=srcnat out-interface=eoip-wan1
add action=masquerade chain=srcnat out-interface=eoip-wan2

And a wireguard "server" with a single peer: (The other side has a 10s persistent keepalive configured.)

/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1
/interface wireguard peers
add allowed-address=192.168.50.0/24 interface=wireguard1 name=peer1 public-key="22SMLvYh7cQzj3/sGFeFLn7l0xVePXM59DLms3qnNUY=" responder=yes

And now the interesting bit.

Create a bridge with a /30 address:

/interface bridge
add name=br-wg protocol-mode=none

/ip address
add address=172.16.10.1/30 interface=br-wg

Mark connections according to their origins, add routing for wan2 and nat them:

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address=192.168.111.10 dst-port=13231 in-interface=eoip-wan1 new-connection-mark=wg-wan1 protocol=udp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address=192.168.112.10 dst-port=13231 in-interface=eoip-wan2 new-connection-mark=wg-wan2 protocol=udp
add action=mark-routing chain=output connection-mark=wg-wan2 new-routing-mark=wan2

/ip firewall nat
add action=dst-nat chain=dstnat connection-mark=wg-wan1 to-addresses=172.16.10.1
add action=dst-nat chain=dstnat connection-mark=wg-wan2 to-addresses=172.16.10.1
add action=src-nat chain=input connection-mark=wg-wan1 to-addresses=172.16.10.2
add action=src-nat chain=input connection-mark=wg-wan2 to-addresses=172.16.10.2

A packet capture:

(The capture is done on the opposing router where the corresponding interfaces are eoip-test1 and eoip-test2)

  #  TIME      INTERFACE   SRC-ADDRESS           DST-ADDRESS           IP-PROTOCOL  SIZE  CPU
  0  3.367     eoip-test1  192.168.33.1:13231    192.168.111.10:13231  udp           190    0
  1  3.373     eoip-test1  192.168.111.10:13231  192.168.33.1:13231    udp           134    1
  2  3.374     eoip-test1  192.168.33.1:13231    192.168.111.10:13231  udp            74    1
  3  13.542    eoip-test1  192.168.33.1:13231    192.168.111.10:13231  udp            74    1
  4  23.784    eoip-test1  192.168.33.1:13231    192.168.111.10:13231  udp            74    1
  5  47.754    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           190    0
  6  47.761    eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           134    1
  7  47.761    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    2
  8  58.342    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    2

This shows a connection first over wan1, then over wan2. The handshake initiations are the 190 byte packets, size 134 are the handshake responses and the 74 byte ones are keepalives. All is well.

A ping test was then conducted over the wg tunnel. The pings are 3s apart, and during the test, the wan1 connection was severed for about 10s.

  9  63.476    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 10  63.476    eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 11  66.484    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 12  66.485    eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 13  69.479    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 14  69.48     eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 15  72.479    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 16  72.48     eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 17  75.486    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 18  75.487    eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 19  78.486    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 20  78.487    eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 21  81.495    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 22  81.496    eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 23  84.498    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 24  84.499    eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 25  87.502    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 26  87.503    eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 27  90.499    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 28  90.5      eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 29  93.506    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 30  93.507    eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 31  96.512    eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 32  96.513    eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 33  99.51     eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 34  99.511    eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 35  102.512   eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    2
 36  102.513   eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
 37  112.742   eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    2
 38  112.742   eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    2
 39  122.982   eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    2
 40  133.222   eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    2
 41  143.462   eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    2
 42  153.702   eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    2
 43  163.942   eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    2
 44  174.182   eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    2

Traffic flows without interruption. (This is the problem with dst-natting to the primary wan address, there the traffic is interrupted if the address is lost.)

If you would also want to initiate wg connections from the router side and do it over wan2, even if wan1 is available, this is the additional rule you'll need: (Traffic may be interrupted for at most keepalive time when wan1 goes down.)

/ip firewall mangle
add action=mark-connection chain=output connection-mark=no-mark new-connection-mark=wg-wan2 protocol=udp src-port=13231

anav · Sat Apr 05, 2025 5:07 am

So in your example you have to manipulate both wans, not just wan2??

lurker888 · Sat Apr 05, 2025 5:44 am

So in your example you have to manipulate both wans, not just wan2??

I don't exactly get what you mean by "manipulate".

In the example I treat the WANs in a symmetric manner. We don't have to. If we remove everything that is related to the wg-wan1 mark, so these:

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address=192.168.111.10 dst-port=13231 in-interface=eoip-wan1 new-connection-mark=wg-wan1 protocol=udp

/ip firewall nat
add action=dst-nat chain=dstnat connection-mark=wg-wan1 to-addresses=172.16.10.1
add action=src-nat chain=input connection-mark=wg-wan1 to-addresses=172.16.10.2

Then the packet capture looks like this: (same scenario repeated with the above rules disabled)

 #  TIME    INTERFACE   SRC-ADDRESS           DST-ADDRESS           IP-PROTOCOL  SIZE  CPU
 0  13.332  eoip-test1  192.168.33.1:13231    192.168.111.10:13231  udp           190    1
 1  13.339  eoip-test2  192.168.111.10:13231  192.168.33.1:13231    udp           134    1
 2  13.34   eoip-test1  192.168.33.1:13231    192.168.111.10:13231  udp            74    0
 3  23.92   eoip-test1  192.168.33.1:13231    192.168.111.10:13231  udp            74    0
 4  34.16   eoip-test1  192.168.33.1:13231    192.168.111.10:13231  udp            74    0
 5  51.177  eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           190    1
 6  51.183  eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           134    1
 7  51.184  eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    1
 8  61.69   eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    1
 9  71.92   eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    1
10  73.467  eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    1
11  73.467  eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
12  76.476  eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    1
13  76.476  eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
14  79.478  eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    1
15  79.479  eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
16  82.483  eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    1
17  82.484  eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
18  85.489  eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    1
19  85.49   eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
20  88.487  eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    1
21  88.487  eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
22  91.487  eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    1
23  91.488  eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
24  94.484  eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp           138    1
25  94.485  eoip-test2  192.168.112.10:13231  192.168.33.1:13231    udp           138    1
26  104.56  eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    1
27  114.8   eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    1
28  125.05  eoip-test2  192.168.33.1:13231    192.168.112.10:13231  udp            74    1

So... exactly the same.

I don't know who those three measly lines would hurt, but if someone likes it better like this, I'm all for it.

anav · Sat Apr 05, 2025 2:00 pm

It goes to root reason. As I stated, WAN1 being primary WAN2 secondary wanting to use WAN2 for wireguard.
We only need to mangle for WAN2 and the problem was the router was sending return traffic via WAN1........

Thus we dsnatted to fool router to send traffic back out WAN2.......
You pointed out the weakness of this approach when WAN1 Fails..... and by using a faux bridge vice WAN1 in dstnat.
Thus in my mind I was not expecting any work for wan1....

I prefer to keep it simple, like adding EOIP to the mix is a distraction like throwing smoke grenades into the explanation ;-P
Your level of knowledge is far more comfortable and thus probably assume we know or see what you know or see as easily, not the case at least for me but appreciate the explanations and efforts!!

PS the peer settings were either wrong or incomplete.
If the other end is a router with a subnet,192.168.50.0/24, then one is missing the /32 address of the other ends wireguard address
OR
If the other end is simply a device, then the allowed address should be 192.168.50.2/32 ( assuming router wg ip is 192.168.50.1)

anav · Sat Apr 05, 2025 4:54 pm

My problem is not properly understanding connection tracking, Nothing more you can do LOL.
At least I kind of grasp your use of faux bridge and how traffic gets there, its after, the response traffic and mangle and routing that eludes me completely.

lurker888 · Sat Apr 05, 2025 6:27 pm

It goes to root reason. As I stated, WAN1 being primary WAN2 secondary wanting to use WAN2 for wireguard.
We only need to mangle for WAN2 and the problem was the router was sending return traffic via WAN1........

In my reply I pointed out that you don't *have* to do anything to wan1 traffic for everything to work out.

I often do configurations for scenarios where all participating networks are in VRFs (and some or all of them via virtual links) so I tend to orient my solutions so that everything is the same for all of them; this way there are no "special cases" to have to keep in mind. Everyone's usual application and point of view will of course be different. (One of the nice things about Mikrotik stuff is that they are useful in an unexpected variety of situations.)

Thus we dsnatted to fool router to send traffic back out WAN2.......
You pointed out the weakness of this approach when WAN1 Fails..... and by using a faux bridge vice WAN1 in dstnat.
Thus in my mind I was not expecting any work for wan1....

I prefer to keep it simple, like adding EOIP to the mix is a distraction like throwing smoke grenades into the explanation ;-P
Your level of knowledge is far more comfortable and thus probably assume we know or see what you know or see as easily, not the case at least for me but appreciate the explanations and efforts!!

EoIP is a perfectly cromulent ethernet interface. It's far easier to test things when you control both sides, plus it's always repeatable.

PS the peer settings were either wrong or incomplete.
If the other end is a router with a subnet,192.168.50.0/24, then one is missing the /32 address of the other ends wireguard address
OR
If the other end is simply a device, then the allowed address should be 192.168.50.2/32 ( assuming router wg ip is 192.168.50.1)

Actually the peer settings are exactly as given. (Generated by "export".)

How to manage the traffic inside the tunnels is outside the scope of the configuration given, but here is how it's done in this particular example:

An address is assigned on the router discussed:

/ip address
add address=192.168.50.2/24 interface=wireguard1

On the other side:

/interface wireguard
add listen-port=13231 mtu=1420 name=wg-test
/interface wireguard peers
add allowed-address=192.168.50.0/24 endpoint-address=192.168.112.10 endpoint-port=13231 interface=wg-test name=peer4 persistent-keepalive=10s public-key="+XbZ8kxNGw2i6my6hOzSK3Ih2VMH03gjzckpr4+5cj0="

/ip address
add address=192.168.50.1/24 interface=wg-test

And the ping is done from the initiator side:

/tool/ping address=192.168.50.2 interval=3

My problem is not properly understanding connection tracking, Nothing more you can do LOL.
At least I kind of grasp your use of faux bridge and how traffic gets there, its after, the response traffic and mangle and routing that eludes me completely.

I'll try to explain. It may be useful, my apologies if you don't find it so. It still might be nice for someone else encountering the same problem to have it written down somewhere.

The first thing is that actually the normal way of "apply connection mark, add a routing table, mark routing on output" thing *would* work just fine with wireguard. The problem is that the source address of the outgoing packet is determined using the "main" routing table, and thus conntrack can't identify the connection (thus treating it as new, and the connection mark is therefore "not there"). If we disregard this and force the routing table, even though the table selection and the egress interface will be correct, the source address will still be incorrect.

Connection tracking (for general udp flows - which wireguard is) looks only at the src/dst addresses and ports. So the steps are the following for a connection initiated from 192.168.33.1 to 192.168.112.10, as in the packet captures.
* we use dst-nat to change the dst address to 172.16.10.1; this also means that conntrack will expect the reply packets to come from 172.16.0.1, and will change the src address on the reply packet during forwarding to the original one (for wan2 this is 192.168.112.10).
* we use input src-nat to change the src address to 172.16.10.2; this means that conntrack will expect the reply packets to be addressed to 172.16.10.2, and will change the dst address on the reply packets to the original (in our example 192.168.33.1)

Now onto wg.
* wg thus receives the incoming packets from 172.16.10.2 (as translated by the input src-nat) therefore it accepts this as "current endpoint address" for the peer and so the dst address of the outgoing packets will be this
* for determining the src address, a lookup in the "main" table is done, and a route to 172.16.10.0/30 is found (added dynamically by the address assignment), and for this the appropriate src address is 172.16.10.1 - thus this is selected

Back to conntrack:
* the packet - as seen by conntrack - is from 172.16.10.1 to 172.16.10.2
* this fits the expectation for a reply packet as discussed above and the connection is identified: state is set to established, and the connection mark is also correctly set
* the src address - as described above - is translated to the original (for wan2, 192.168.112.10)
* the dst address - as described above - is translated to the original (in our example, 192.168.33.1)

We have correct addressing, we have the connection mark, we're happy. Packet marking for routing, routing table and egress interface selection can now proceed in the usual way.

anav · Sat Apr 05, 2025 9:25 pm

My bad that is valid, but this is assuming the remote router is an MT router. ( client peer for handshake) ........ makes sense, so other peers connecting to the local router can easily re-enter the tunnel and reach the remote router via the local router, so to speak.
The local router needs
allowed IPs=192.168.50.1/32 You cant have both ends with allowed addresses at 192.168.50.0/24 LOL

+++++++++++++++++++++++++++++++++++++

Hmm interesting comment:
The first thing is that actually the normal way of "apply connection mark, add a routing table, mark routing on output" thing *would* work just fine with wireguard. The problem is that the source address of the outgoing packet is determined using the "main" routing table, and thus conntrack can't identify the connection (thus treating it as new, and the connection mark is therefore "not there"). If we disregard this and force the routing table, even though the table selection and the egress interface will be correct, the source address will still be incorrect.

Makes me wonder if we shouldn't simply
add dst-address=RemoteWANIP gateway=WAN2 table=main distance=2
add dst-address=0.0.0.0/0 gateway=WAN1 table=main distance=4
add dst-address=0.0.0.0/0 gateway=WAN2 table=main distance=6

Fix our problem LOL if we had flexibility in what entry arguments were for destination address.
++++++++++++++++++++++++

The first crux of my lack of understanding is exactly here, everything above I followed.....
* for determining the src address, a lookup in the "main" table is done, and a route to 172.16.10.0/30 is found (added dynamically by the address assignment), and for this the appropriate src address is 172.16.10.1 - thus this is selected

I simply dont understand this process, where does this rule come from? Is this some basic networking stuff I would know if I actually studied IT and networking??
Why is the table or routing even in the picture for this decision???............ since the destination packets were sent to 172.16.10.1 wouldnt that automatically mean the that address would be the source address of reply packets... Since this is not simply user/device to user/device, and is ROUTER traffic OWN traffic, I do not understand how it works.

I do understand that conntrac sees 172.16.10.1 sending traffic to 172.16.10.2
What I dont understand is why the connection marks on any of this traffic was removed...........
What I dont understand is how the router can magically put them back on without admin direction.........
Where was conntrack involved in the process of changing the addresses>> Conntrac is not involved after DSTNAT in prerouting its before, and of course before the next block of input srcnat.

(as per packet flow diagram)

Connection tracking is only seen again in output just before mangle output which I suppose is where we move that traffic out WAN2 with the connection marks and route mark.

I do understand the concepts of un-dst and un-src so that we have a traffic flow that now resembles 192.168.112.10 to 192.168.33.1.
I do understand that the original dst port (listening port on local device) and original src port ( listening port from remote device) were maintained throughout the process and need no shenanigans.

The other thing that is not clear to me is how this fools the router at all vis-a-vis avoiding WAN1. In SINDYs rule we fool the router in the last step, by getting the router to un-dsnat out wan2 vice WAN1.
We do no such thing in your example. We piddle around a faux bridge, which is mainly to avoid the scenario if WAN1 dies. but I dont see how we avoid WAN1 in main table, which SIndys rule addresses.

lurker888 · Sat Apr 05, 2025 10:22 pm

My bad that is valid, but this is assuming the remote router is an MT router. ( client peer for handshake) ........ makes sense, so other peers connecting to the local router can easily re-enter the tunnel and reach the remote router via the local router, so to speak.
The local router needs
allowed IPs=192.168.50.1/32 You cant have both ends with allowed addresses at 192.168.50.0/24 LOL

As described, the side whose configuration was presented has this peer entry, and this address (and the dynamic route it adds):

/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1
/interface wireguard peers
add allowed-address=192.168.50.0/24 interface=wireguard1 name=peer1 public-key="22SMLvYh7cQzj3/sGFeFLn7l0xVePXM59DLms3qnNUY=" responder=yes

/ip address
add address=192.168.50.2/24 interface=wireguard1

The other side has this:

/interface wireguard
add listen-port=13231 mtu=1420 name=wg-test
/interface wireguard peers
add allowed-address=192.168.50.0/24 endpoint-address=192.168.112.10 endpoint-port=13231 interface=wg-test name=peer4 persistent-keepalive=10s public-key="+XbZ8kxNGw2i6my6hOzSK3Ih2VMH03gjzckpr4+5cj0="

/ip address
add address=192.168.50.1/24 interface=wg-test

Nothing wrong with it and works fine. Both sides can ping each other, transit traffic, etc.

Using a /32 allowed address on one end and a wider subnet on the other is a common config. So is using /32 on both. So is using the same /30 on both.

+++++++++++++++++++++++++++++++++++++

Hmm interesting comment:
The first thing is that actually the normal way of "apply connection mark, add a routing table, mark routing on output" thing *would* work just fine with wireguard. The problem is that the source address of the outgoing packet is determined using the "main" routing table, and thus conntrack can't identify the connection (thus treating it as new, and the connection mark is therefore "not there"). If we disregard this and force the routing table, even though the table selection and the egress interface will be correct, the source address will still be incorrect.

Makes me wonder if we shouldn't simply
add dst-address=RemoteWANIP gateway=WAN2 table=main distance=2
add dst-address=0.0.0.0/0 gateway=WAN1 table=main distance=4
add dst-address=0.0.0.0/0 gateway=WAN2 table=main distance=6

Fix our problem LOL if we had flexibility in what entry arguments were for destination address.
++++++++++++++++++++++++

Absolutely correct. You don't even have to have a smaller distance for the /32 route - longer prefixes are always preferred, regardless of distance. If you know the peer's address is advance this is actually fine to do.

The first crux of my lack of understanding is exactly here, everything above I followed.....
* for determining the src address, a lookup in the "main" table is done, and a route to 172.16.10.0/30 is found (added dynamically by the address assignment), and for this the appropriate src address is 172.16.10.1 - thus this is selected

I simply dont understand this process, where does this rule come from? Is this some basic networking stuff I would know if I actually studied IT and networking??
Why is the table or routing even in the picture for this decision???............ since the destination packets were sent to 172.16.10.1 wouldnt that automatically mean the that address would be the source address of reply packets...

For normal services (such as DNS over udp, OpenVPN over udp, etc.) you are correct and the source address is where the request was sent to. Wireguard follows a different architecture, and leaves the source address to be assigned by the initial output routing stage. (It is actually done to allow easy peer roaming.)

My post with the full description:
viewtopic.php?p=1122715#p1122715
To quote:

The *other* - and in your current case more important - story concerns the source address of packets. Source addresses are normally assigned either (1) by the application layer (e.g. a DNS server receives a UDP query, it should clearly specify (to the kernel) as the source address of the reply the address the query was received on), (2) by other networking layers, e.g. TCP will originate packets from the address the SYN was received on, or from which the SYN was sent during the handshake. But if still a packet has no source address (often referenced as the "address of last resort") the packet acquires the "pref src address" of the route it takes. And here's the strangeness: this assignment happens during the initial "routing decision" and is not (can't be) corrected in the "routing adjustment" phase. Wireguard - by its design - doesn't assign source addresses to its outgoing packets, which leads to a situation where if the packet is to be redirected in the "mangle postrouting" chain, it will go out on the correct interface, but with the wrong source address. Happy world.

I do understand that conntrac sees 172.16.10.1 sending traffic to 172.16.10.2
What I dont understand is why the connection marks on any of this traffic was removed...........

Packets don't carry connection marks (it's not in the IP packet), they are assigned by conntrack. (There are some exceptions.) The packet output by wireguard has no marks whatsoever on it initially. The packet goes:

[emitted by wg] -> initial routing decision -> conntrack -> mangle output -> final routing decision (routing adjustment) -> and so on

The packet is annotated with the connection mark in the conntrack phase. Until then, there is no associated connection mark. (On normal linux, wg interfaces have a property fwmark, which allows all packets emitted by wg to be marked on creation - this is not currently available on Mikrotik.)

What I dont understand is how the router can magically put them back on without admin direction.........

Again, packets on interfaces (ethernet, virtual interfaces, all of them) come into the router "naked". Conntrack labels them based on the conntrack entry it associates with them. When a mangle assigns a connection mark, it is noted in the conntrack entry the packets nf metadata references.

Where was conntrack involved in the process of changing the addresses>> Conntrac is not involved after DSTNAT in prerouting its before, and of course before the next block of input srcnat.

(as per packet flow diagram)

Again, dst-nat modifies the conntrack entry. That's its primary purpose. The translation is done immediately to be in line with the (freshly modified) conntrack entry.

Connection tracking is only seen again in output just before mangle output which I suppose is where we move that traffic out WAN2 with the connection marks and route mark.

I do understand the concepts of un-dst and un-src so that we have a traffic flow that now resembles 192.168.112.10 to 192.168.33.1.
I do understand that the original dst port (listening port on local device) and original src port ( listening port from remote device) were maintained throughout the process and need no shenanigans.

The other thing that is not clear to me is how this fools the router at all vis-a-vis avoiding WAN1. In SINDYs rule we fool the router in the last step, by getting the router to un-dsnat out wan2 vice WAN1.
We do no such thing in your example. We piddle around a faux bridge, which is mainly to avoid the scenario if WAN1 dies. but I dont see how we avoid WAN1 in main table, which SIndys rule addresses.

Un-dstnat only changes the address. The interface is decided by routing. In mangle output a routing mark "wan2" is applied based on the connection mark "wg-wan2", and the output interface is determined in the "routing adjustment" phase based on the routing mark "wan2" to be the wan2 interface.

anav · Sun Apr 06, 2025 1:09 am

(I see what you are doing with wireguard just dont agree with it. There is no case where both sides of a connection need 50.0/24 that I can see.)

Regarding the contrack and wireguard and dual WAN etc......... I approached it from a different angle so it makes sense to me.
The initial problem before ANY fixes is that The outbound traffic from the router, or should I say source address the router was selecting for the return traffic was the WAN1 IP. So our mangling was working but really the problem was not the mangling, it was the fact that the wrong source IP address was on the return traffic from the wireguard processing itself and thus got rejected by the peer client.
I dont really understand how the router selects source of outbound traffic from the router itself but it would appear to be based on what interfaces and their ip routes are available at the main table...........

The sindy dstnat rule simply ensured that at the last minute although the router chose wan1IP for the return outbound source, it was undstnatted to wan2 IP and thus the path (route) like before was correct and this time the source address was what was expected by the peer client.

Your solution is similar, we still need the mangle to ensure the path is correct but you decided to use a nasty trick of faux address to avoid the WAN1 problem (if becomes unavailable and thus fails with the above dsntat rule ).........

So We have to ensure the source address of the traffic leaving the wireguard handshake is something else, not the chosen WAN1 by the router. We don't dstnat the traffic heading for WAN2 with WANIP1, we dstnat to an existing but unused will not go away/fail unused BRIDGE 172.16.10.1 ip address. We also ensure we sourcenat the traffic going to the wireguard process, from the WANIP of the remote router, with another address on the Bridge --> 172.168.10.2, forcing the wireguard process to use a reply dst address of 172,168.10.2

Previously the router, for wireguard source address outbound, would choose the best interface ( and thus source address) based on the optimal available route on the main table, and that was WAN1 but we have now an available dynamic accurate route which is preferable for 172.16.10.2 ( not sure if its because the interface and dst-address 172.16.10.2 are related or due to prefix differenence from WAN1 ) and the router chooses 172.16.10.1 for source and off it goes.... In summary, we now have traffic coming from the wireguard process with destination address of 172.16.10.2 and with source address of 172.16.10.1

Thus mission accomplished we have a source address coming from the wireguard process that is not WAN1, and the router sees it as 172.16.10.1 and un-dstnats that back to the( the original destination address, WANIP 2 of the local router ( which is now the new source address ). Now the remote router will see the correct source address on the return traffic it sees on the handshake. 172.16.10.1 becomes 192.168.112.10 ( in your examples). At the same time, the Router un-srcnats the destination traffic reply IP from wireguard process ( 172.16.10.2) to the original incoming source address of 192.168.33.1 ( the new destination address ).

Traffic leaves from WAN2 due to our mangle output routing-mark, and traffic leaves the router with source address of WAN2.

+++++++++++++++

Notes: Helpful --> For normal services (such as DNS over udp, OpenVPN over udp, etc.) you are correct and the source address is where the request was sent to. Wireguard follows a different architecture, and leaves the source address to be assigned by the initial output routing stage. (It is actually done to allow easy peer roaming.)

Notes: Helpful --> Something I keep forgetting too: Absolutely correct. You don't even have to have a smaller distance for the /32 route - longer prefixes are always preferred, regardless of distance. If you know the peer's

Notes: Helfpul, so the un-dst nat takes place in ROUTING ADJUSTMENT and the unsourcenat takes place in post-routing??>>
[emitted by wg] -> initial routing decision -> conntrack -> mangle output -> final routing decision (routing adjustment) -> and so on

anav · Sun Apr 06, 2025 1:16 am

THis is the most interesting part about your post.
The packet is annotated with the connection mark in the conntrack phase. Until then, there is no associated connection mark. (On normal linux, wg interfaces have a property fwmark, which allows all packets emitted by wg to be marked on creation - this is not currently available on Mikrotik.)

Should I report this as a bug, or a recommendation? Is it outside the realm of wireguard protocol and just a function that linux applies to all such entities by design and perhaps MT was lazy??

Note: Made suggestion to MT.

Mimiko · Mon Apr 07, 2025 8:39 am

@Mimiko, your only options are either routing rules or NAT, unless you’re on one of the newer ROS versions where they supposedly fixed the issue, i.e. that WireGuard’s initial handshake is going out via the default gateway instead of the inbound interface. Might’ve been mentioned in this or the other big thread on the topic. We’re running v7.15 ourselves and use routing rules for multi-WAN WireGuard.

I've tried routing rules and that does not work either. Its the wireguard goning default route or no route.
Had read the change log starting with 7.17 (I have 7.16.2) and ther is no mention of any fixing for wireguard. The problem is not with initial handshake where wireguard responds. But with subsequent packets were wireguard initiates the sending.

Larsa · Mon Apr 07, 2025 8:53 am

...The problem is not with initial handshake where wireguard responds. But with subsequent packets were wireguard initiates the sending.

Do you mean the subsequent packets from the clients after they’ve connected to the WireGuard server, or are you referring to when the WireGuard server sends traffic back towards the clients, such as handshake responses or keepalive packets?

Mimiko · Mon Apr 07, 2025 9:50 am

...The problem is not with initial handshake where wireguard responds. But with subsequent packets were wireguard initiates the sending.

Do you mean the subsequent packets from the clients after they’ve connected to the WireGuard server, or are you referring to when the WireGuard server sends traffic back towards the clients, such as handshake responses or keepalive packets?

The subsequent packets from the client will go to the handshaked server IP. But from wireguard's philosophy of being hidden, if from the server side a packed is send via wireguard, then here the src IP of wireguard which client expects will change.

Larsa · Mon Apr 07, 2025 10:13 am

WireGuard’s philosophy does not mean much unless there is a real issue to solve. If you are referring to a solid issue, like reply packets going out through the wrong WAN interface and the client rejecting them or something similar, could you please clarify what exact behavior you are seeing or referring to, as I asked in my previous post?

lurker888 · Mon Apr 07, 2025 10:41 am

So here's my response.

(I see what you are doing with wireguard just dont agree with it. There is no case where both sides of a connection need 50.0/24 that I can see.)

I just wanted to send a ping across. This is what I typed first. There's no hidden agenda.

So our mangling was working but really the problem was not the mangling, it was the fact that the wrong source IP address was on the return traffic from the wireguard processing itself and thus got rejected by the peer client.

You are absolutely right that the source address is the issue, just not why. Wireguard always accepts correctly authenticated packets from *any* address. Either of two things actually go wrong:
* If you emit a packet with a wrong source address (probably so wrong, it's not even the same subnet or AS,) well-configured routers (by that I mean that they have some form of RPF enabled) simply drop these
* If the packet manages to get to the other endpoint, it sets it's "current endpoint address" to this source address

Notes: Helfpul, so the un-dst nat takes place in ROUTING ADJUSTMENT and the unsourcenat takes place in post-routing??>>
[emitted by wg] -> initial routing decision -> conntrack -> mangle output -> final routing decision (routing adjustment) -> and so on

I'm not going to go into as much detail as I was initially inclined to do. Where the *decision* to apply src- or dst-nat to a connection is clear in the packet flow diagram. Where exactly the address translation happens is sneakily not shown in any of them. It might give a hint as to what is happening if we look at what we get if we enable logging for an src-natted packet:

(This has no relation to wireguard, just a general natted packet that came my way.)

firewall,info forward: in:vlan100-local-trusted out:wan, connection-state:established,snat src-mac 04:d9:f5:1e:33:71, proto UDP, 192.168.80.229:51413->81.183.229.86:1034, NAT (192.168.80.229:51413->89.133.122.28:51413)->81.183.229.86:1034, len 116

What we can surmise immediately from what we see is that *all* the information (the original source address, and the one it is replaced with) is readily available for netfilter to show. And it's available for the entire journey it takes through the while netfilter framework, from the moment the connection is identified in the stage of processing usually labelled "connection tracking" until the time it leaves netfilter. The actual src-nat and dst-nat actions just modify these values and set flags. And when we match on the dst- or src-address (or ports) in some chain of the firewall, we are actually matching different addresses under the same name. Sneaky isn't it.

Larsa · Mon Apr 07, 2025 11:06 am

Just to be very clear: the WireGuard protocol requires that the destination address of the initial handshake request MUST match the source address of the initial handshake response, otherwise the session will be dropped. If the handshake succeeds, then it is fine for the address to change, and WireGuard will update its internal endpoint accordingly.

lurker888 · Mon Apr 07, 2025 12:00 pm

Just to be very clear: the WireGuard protocol requires that the destination address of the initial handshake request MUST match the source address of the initial handshake response, otherwise the session will be dropped. If the handshake succeeds, then it is fine for the address to change, and WireGuard will update its internal endpoint accordingly.

In the Wireguard Whitepaper (https://www.wireguard.com/papers/wireguard.pdf) section "5.4.2 First Message: Initiator to Responder" clearly specifies that

sender := I_i (4 bytes) [...] is generated randomly (rho^4) when this message is sent, and is used to tie subsequent replies to the session begun by this message.

And in section "5.4.3 Second Message: Responder to Initiator" it is clear that this is sent back to identify the session:

receiver := I_i (4 bytes)

So the handshake peer is indeed identified by its "sender id" (a random number) and not by its IP address.

I was not satisfied with this, so I took a look at the source code of the kernel implementation. The handshake request is validated in receive.c:

static void wg_receive_handshake_packet(struct wg_device *wg,
					struct sk_buff *skb)
{
[...]
	switch (SKB_TYPE_LE32(skb)) {
[...]
	case cpu_to_le32(MESSAGE_HANDSHAKE_RESPONSE): {
		struct message_handshake_response *message =
			(struct message_handshake_response *)skb->data;

		if (packet_needs_cookie) {
			wg_packet_send_handshake_cookie(wg, skb,
							message->sender_index);
			return;
		}
		peer = wg_noise_handshake_consume_response(message, wg);
		if (unlikely(!peer)) {
			net_dbg_skb_ratelimited("%s: Invalid handshake response from %pISpfsc\n",
						wg->dev->name, skb);
			return;
		}
		wg_socket_set_peer_endpoint_from_skb(peer, skb);
		net_dbg_ratelimited("%s: Receiving handshake response from peer %llu (%pISpfsc)\n",
				    wg->dev->name, peer->internal_id,
				    &peer->endpoint.addr);
		if (wg_noise_handshake_begin_session(&peer->handshake,
						     &peer->keypairs)) {
			wg_timers_session_derived(peer);
			wg_timers_handshake_complete(peer);
			/* Calling this function will either send any existing
			 * packets in the queue and not send a keepalive, which
			 * is the best case, Or, if there's nothing in the
			 * queue, it will send a keepalive, in order to give
			 * immediate confirmation of the session.
			 */
			wg_packet_send_keepalive(peer);
		}
		break;
	}

packet_needs_cookie is only set if the handshake queue is too long (i.e. DoS)

peer is only determined by the message contents: peer = wg_noise_handshake_consume_response(message, wg)

And based on this, the endpoint address/port is updated and the session is started.

Nowhere is the source IP of the response validated, in fact the (current) endpoint address of the peer is updated based on it.

Mimiko · Mon Apr 07, 2025 3:04 pm

Just to be very clear: the WireGuard protocol requires that the destination address of the initial handshake request MUST match the source address of the initial handshake response, otherwise the session will be dropped. If the handshake succeeds, then it is fine for the address to change, and WireGuard will update its internal endpoint accordingly.

WireGuard’s philosophy does not mean much unless there is a real issue to solve. If you are referring to a solid issue, like reply packets going out through the wrong WAN interface and the client rejecting them or something similar, could you please clarify what exact behavior you are seeing or referring to, as I asked in my previous post?

This 2 a related.

My use case have 2 mikrotiks. One acts as server (virtually as it has responder=yes), and another acts as client (virtually, no responder option set). From wireguard perspective both a peers (not servers not clients). So any of them can initiate communication to known IP address of another part. First via config if set, then via learned src IP.
Now to my example. Both mikrotiks have 2 ISPs. The wireguard are listen on 13231 port and on both mikrotiks this port is opened for input in firewall.

I witnessed two scenarious when mangling the rules on "server".

1) The "client" start a first connection. The "client" send the packed randomly via one of the ISP to the IP address of "server" assigned to second ISP. The "server" receives the packet and by rules in firewall responds using second ISP and IP address. Back to "client". Ok. As for know this seems to work ok. If a ping is generated from "server" to "client", then the "client" will receive the packet from first ISP of the "server", and because "client" is connected directly to internet and allows incoming packets to the 13231 port, it will receive it, mach the peer public key and will update the public ip of the "server" to the one received now. Then "client" sends back response to "server"'s ping but choose to emerge via first its provider. "server" receive that packet as it have the port opened and based of "client"s public key also updates "client"s current public IP. Subsequently both miktoriks use the first ISP of each one to communicate.

2) The "client" start first connection. The "client" send the packed randomly via one of the ISP to the IP address of "server" assigned to second ISP. The "server" receives the packet and no rules in firewall a configured, so "server" from start responds via first ISP to "client". But because "client" have the port opened, it receives the response, recognize by the public key and updates the current endpoint IP. Then "client" sends a packed to "server" but choose to emerge via first its provider. And happens same as previous one.

Both mikrotiks will end up communicating via first ISP regardless of which initial ISPs or IPs were used. This is true if both of them have the input port opened.

Now think of "client" behind a firewall and is nated for accessing internet. (ie access via mobile data). "client" will not have any incoming port opened.

CGGXANNX · Tue Apr 08, 2025 1:36 am

And now the interesting bit.

Create a bridge with a /30 address:

i had some free time to do a couple of tests today, and this point can be further simplified: we can just add a /32 address to the lo interface. No needs for the extra dummy bridge and the two separate IP addresses for src-nat and dst-nat.

/ip address
add address=10.20.30.40 interface=lo network=10.20.30.40

Then 10.20.30.40 can be used instead of both your 172.16.10.1 and 172.16.10.2.

What i find interesting is that, until at least the middle of 2024, the srcnat rule was not necessary. I dug into my old chat logs with online acquaintances where we used to test and discuss this issue and this solution from @rplant used to be all what was needed: viewtopic.php?t=205278#p1061641. My acquaintances and I did the small modification where instead of the dummy bridge, lo was used, other than that the single dstnat rule used to be enough. That did produce the effect that the reply to the handshake had the IP address of the lo interface as source. And the routing rule could steer it to the right table. I had assumed that was still the case when I wrote the reply in #5 of this thread.

But something must have changed in RouterOS in the 2nd half of 2024, because when I redid those old configs from last year today, the additional srcnat rule is now definitely required, as explained by @lurker888 multiple times in this thread.

anav · Tue Apr 08, 2025 2:39 am

Hi CGX...
What the heck is lo LOL, an existing interface on the router that is there all the time??

CGGXANNX · Tue Apr 08, 2025 3:08 am

It's the loopback interface, that can be used since 7.14 viewtopic.php?t=202612

*) system - expose "lo" and "vrf" interfaces;

One of the use-cases is to replace the need for adding dummy bridge interfaces if you want extra internal IP addresses for the router.

anav · Tue Apr 08, 2025 4:01 am

How does it relate to the input chain rule then??
add chain=input action=accept dst-address=127.0.0.1

and you are saying
Then 10.20.30.40 can be used instead of both your 172.16.10.1 and 172.16.10.2.

Does this mean the following.

/ip firewall nat
add action=dst-nat chain=dstnat connection-mark=wg-wan2 to-addresses=10.20.30.40
add action=src-nat chain=input connection-mark=wg-wan2 to-addresses=10.20.30.40

instead of
add action=dst-nat chain=dstnat connection-mark=wg-wan2 to-addresses=172.16.10.1
add action=src-nat chain=input connection-mark=wg-wan2 to-addresses=172.16.10.2