NAT in output chain for Wireguard

ivugrinec · Thu Mar 17, 2022 9:52 pm

Can someone please try to explain why is that; for Wiregurad in RouterOS 7.1.3
as observed in postrouting log when using mark connection, mark packet and mark routing to route the traffic through PPPoE interface with static ip address there is NAT translation active and therefore the src port is wrong in output chain?

- but if main routing table is changed in a way that default route is changed in a way that it exits the PPoE interface - NAT is not applied.

Sample from the log:

I observe NAT-ing on postrouting log.

 19:59:49 firewall,info MC ==> output: in:(unknown 0) out:ether3-WAN1, proto UDP, 172.16.16.2:5555->149.7.81.125:5555, [b]NAT (172.16.16.2:5555->A.B.C.88:53386)[/b]->149.7.81.125:5555, len 120
 19:59:49 firewall,info MP ====>> output: in:(unknown 0) out:ether3-WAN1, proto UDP, 172.16.16.2:5555->149.7.81.125:5555, [b]NAT (172.16.16.2:5555->A.B.C.88:53386)[/b]->149.7.81.125:5555, len 120
 19:59:49 firewall,info MR ====>>> output: in:(unknown 0) out:ether3-WAN1, proto UDP, 172.16.16.2:5555->149.7.81.125:5555, [b]NAT (172.16.16.2:5555->A.B.C.88:53386)[/b]->149.7.81.125:5555, len 120
 19:59:49 firewall,info RESULT =====>>>>> postrouting: in:wg0 out:PPPoE, proto UDP, 172.16.16.2:5555->149.7.81.125:5555, [b]NAT (172.16.16.2:5555->A.B.C.88:53386)[/b]->149.7.81.125:5555, len 120

The error i think is creating issues is here the: A.B.C.88:53386
How to avoid this?
---------------------------

If i redirect the traffic out via PPPoE via default route in main table this works fine:
(The default route in "main" table is on PPPoE interface, No NAT, src port is 5555 (as expected by the wg client).

 20:03:24 firewall,info MC ==> output: in:(unknown 0) out:PPPoE, proto UDP, A.B.C.88:5555->149.7.81.125:5555, len 60
 20:03:24 firewall,info MP ====>> output: in:(unknown 0) out:PPPoE, proto UDP, A.B.C.88:5555->149.7.81.125:5555, len 60
 20:03:24 firewall,info MR ====>>> output: in:(unknown 0) out:PPPoE, proto UDP, A.B.C.88:5555->149.7.81.125:5555, len 60
 20:03:24 firewall,info =====>>>>> postrouting: in:wg0 out:PPPoE, proto UDP, A.B.C.88:5555->149.7.81.125:5555, len 60

Now we i have: A.B.C.88:5555->149.7.81.125:5555 and it works fine.
So how do i get that the output IP is A.B.C.88 and src port is 5555 without having to change the default route (defacto; stop seeing NAT in the logs)?

What could i do to try to use mangle facility (MC->MP->MR) to route the packets out through PPPoE but with PPPoE (static) IP address as source IP and src port 5555?
In other words is there something i could do to have the (lower, working) output but using anything else except changing the default route (or adding static routes)?

Sob · Fri Mar 18, 2022 12:39 am

Problem with changing routing using mangle rules is that it happens too late. If you look at this diagram:

https://wiki.mikrotik.com/wiki/Manual:P ... ng_Diagram

where K is starting point when router itself sends some packet, then there's routing decision where it also decides what source address should be used, and only then is output chain and mangle rules, where you can change routing. But it won't change source address, that was already decided. And you can see that, when default route points to PPPoE, source address is A.B.C.88. But when it points to the other interface, source is 172.16.16.2.

Srcnat tries to keep original port. But it's only possible when <new source address>:<port> <-> <destination address>:<port> is not already taken. If it is, source port has to be changed. In your case, it's probably that WG client connected from 149.7.81.125:5555 to A.B.C.88:5555 and connection tracking made a record for this connection. Then router tried to send packet from 172.16.16.2:5555 to 149.7.81.125:5555, which srcnat would normally change to A.B.C.88:5555, but it can't because there's already entry for A.B.C.88:5555 <-> 149.7.81.125:5555.

The solution should be to fix routing decision to do it right the first time:

/routing table
add disabled=no fib name=wan2
/ip route
add dst-address=0.0.0.0/0 gateway=PPPoE routing-table=wan2
/routing rule
add src-address=A.B.C.88/32 action=lookup-only-in-table table=wan2

I'm not completely sure how foolproof it is. It's reliable when the source is given address. But I don't know how exactly it happens in this case that it's chosen. Anyway, for me it works.

ivugrinec · Fri Mar 18, 2022 7:22 am

Beautiful. Spot on.
Not only it works, i now understand why i didn't work before and there is basic understanding of the routing process here.
You are more than awesome. Kudos!!

Sob · Fri Mar 18, 2022 3:31 pm

Too bad I don't completely understand why it works.

In case the source address is A.B.C.88, i.e. if the service would specifically set it as source, it's clear. But it can't be doing that, because if it did, it would be A.B.C.88 before too (not 172.16.16.2), only it would be routed wrong way. But if it uses unspecified address and real one is decided as part of routing decision, then how can routing rule having src-address=A.B.C.88 as condition influence it, when at that point it shouldn't have A.B.C.88 yet?

Kingdom for an explanation!

mkx · Fri Mar 18, 2022 4:25 pm

Too bad I don't completely understand why it works.

You shouldn't be admitting this on a such a public place ... I'm sure somebody will hold it against you for a while now

mkx · Fri Mar 18, 2022 4:36 pm

Kingdom for an explanation!

How about this: according to packet flow you posted, new packet will first hit routing decission and might select 172.16.16.2 as src-address.
Next box it hits is output chain of events and the last one is routing adjustment. I don't really know it, but by the sounds of it this might be the moment where the new route is selected?
Next comes postrouting which includes src-nat ... But yes, this doesn't explain why selecting alternative routing table works in the first place.

Sob · Fri Mar 18, 2022 5:31 pm

Output is too late, by then there's already source address and route. Route can be changed using mangle, and that's what the following routing adjustment is for. This is about the initial routing decision and address selection.

When sending packet, it's possible to select (bind to) specific address or let the system decide. And that depends on routing, e.g. if I have two LANs (.1 and .2), then when target is in .1, source address will be also from .1 (whatever router has), for .2 it will be router's address from .2. So if here there's default route using WAN1 and that interface has 172.16.16.2, it will be selected as source. And when default route used WAN2, it selected A.B.C.88 from there.

But now there's rule saying "if source address is A.B.C.88, then use WAN2", but where did A.B.C.88 come from? AFAIK (but I'm not sure about it) there isn't anything between process choosing specific address and not choosing it, just some kind of hint. If it was, then it could work, because WG knows what address received packet from client, so it could use it as source if it wanted to.

ivugrinec · Fri Mar 18, 2022 9:42 pm

Thank you sob, mkx. The thing is that this has been observed on other (not just WG) mangle's of output chain.
I cant' be sure, as i am not on a level you are - but i believe that this part is somewhat different than it was on RouterOS 6.
I've started to notice the issues with mangle and output chain after i upgraded my router to ROS7.

anav · Sun Mar 20, 2022 4:25 pm

@Mkx, Sob knows everything. If he is confused then likely the coders are too. He even suggested that MT should invoke automatic Routes for all allowed IP addresses (to mimic wg-quick etc.), but separately from any IP address entered under /ip addresses.
The funny thing will be how that works when the allowed IPs has IP addresses for wireguard interfaces to assist in troubleshooting and pinging.
They will have to add some extra code........ if Allowed IP address=wg sbunet address, THEN skip making automated route (as its already been made through the normal routing assigned to subnets existing on the router).

Sob · Sun Mar 20, 2022 6:16 pm

@anav: The idea with routes was inspired by @mozerd, poor guy now has constant nightmares that WG in RouterOS is incomplete, so this option would save him, and it's true that I saw other people miss adding routes before. But that's OT here. Regarding this thread, it's your chance. Go to https://kernel.org/, download the source (I think RouterOS v7 has some 5.6.x, but I'm not sure and it's likely to be the same in all), get reading and find out why it works like this.

mozerd · Sun Mar 20, 2022 6:28 pm

@anav: The idea with routes was inspired by @mozerd, poor guy now has constant nightmares that WG in RouterOS is incomplete, so this option would save him, and it's true that I saw other people miss adding routes before.

@Sob ....

Understanding modern Linux routing (and wg-quick)

anav · Sun Mar 20, 2022 7:41 pm

Nice article.
I particularly like this part...

"To summarize, the effect of this rule is to respect all manual routes that the administrator might have added to the main table. However, if the packet didn’t match any of the specific routes, then instead of applying the default route, we’re proceeding to the next rule.
So the effect of the rule is to route everything that reached it through the VPN, with one exception: the mysterious not fwmark 0xca6c......

So its not clear to me it actually uses the Allowed IP lists at all (and this is only for keys and peers selection etc) and wg quick or simply uses the traffic going through the wireguard interface to create routes on the fly.... ???????
.

mozerd · Sun Mar 20, 2022 8:07 pm

In the Tik world WG has 2 states for Routing Table … either off or by default on …. When the WG interface has a IP address Routes are added automatically … when the WG interface does not have IP address assigned Table is off and you must add routes manually.

[edit] forgot to mention that cryptokey routing is ALWAYS in effect when routes are added in manually …. MUCH easier when IP is assigned to the WG INTERFACE and that is why it’s important to use WGI …..I=interfaceIP …..never forget that WG is strictly P2P —- always regardless how hard anyone tries to bend to C/S … in the WG world C/S is an abomination …. WG is Peer to Peer regardless of the number of Peers. If you think of it in any other way you do not understand P2P networking ….

IMO why the difference is important:

Peer-to-Peer Network:
This model does not differentiate the clients and the servers, In this each and every node is itself client and server. In Peer-to-Peer Network, Each and every node can do both request and respond for the services. WG is a Peer to Peer VPN Protocol … that means that the VPN it creates services a P2P network with a lot of flexibility …. Not a C/S servicing protocol …

So Tik is a WG Peer that acts like a distribution Hub for other Peers ….
Very good example of Hub and Spoke with auto generated routes

msatter · Wed Apr 20, 2022 4:08 pm

I assume to separate packet streams, timestamps in the header can be used.

NAT in output chain for Wireguard [SOLVED]

NAT in output chain for Wireguard

Re: NAT in output chain for Wireguard [SOLVED]

Re: NAT in output chain for Wireguard

Re: NAT in output chain for Wireguard

Re: NAT in output chain for Wireguard

Re: NAT in output chain for Wireguard

Re: NAT in output chain for Wireguard

Re: NAT in output chain for Wireguard

Re: NAT in output chain for Wireguard

Re: NAT in output chain for Wireguard

Re: NAT in output chain for Wireguard

Re: NAT in output chain for Wireguard

Re: NAT in output chain for Wireguard

Re: NAT in output chain for Wireguard

Who is online