It seems that in Router OS 7.10 stable, in certain situations ICMP packets (maybe others but I didn't see any others) are being leaked and exiting through the default WAN route instead of going through the Wireguard tunnel or being dropped. To see the issue you need to have a sniffer monitoring the output chain of the router. So either put something upstream to the router or attach a rule that monitors the output chain of the router with the protocol icmp.

Scenario:
*Tested on CCR2216
1) Setup some WAN connection via dhcp client on router os.
2) Setup a wireguard tunnel to another location (can be a VPN service, other site, other router whatever)
3) Masquerade that tunnel interface so multiple LAN clients can connect through and use it.
4) Setup some LAN client to connect to some target on the other end and start a download or some longer running process that you can interrupt. (In my particular case I have some wifi clients connecting to the facebook app or downloading google play store updates). I used policy routing rules to route the entire LAN subnet (where my wifi clients come connect into) into the wireguard tunnel, and added a 0.0.0.0/0 default rule in a seperate routing table.
5) Force disconnect the LAN clients (I switch the phone to airplane mode to drop it from the wifi)

Result:
Since the client has now been forcefully disconnected, whatever in transit packets that require some response will still be flowing or trying to return, the router tries to respond with ICMP code 3 type 3 (port unreachable) or sometimes code 3 type 7 (destination host unknown) which makes sense. However, it does so by using the default route WAN connection instead of sending the response back through the wireguard tunnel to the intended target.

Outcome:
In some situations your privacy or anonymity can be at risk if these packets leak into the default route and then reach the target server. I don't think this is what is expected to occur. All packets should go through the wireguard tunnel if generated for or as a result of a client that was originally trafficing through the wireguard tunnel. Or simply drop these packets. In the mean time I have setup a block rule on the output chain of my router to prevent my router from sending ICMP. This obviously does not impact forwarding of ICMP from other clients on my network which is good.

Notes:
1) Tested with Fasttrack both disabled and enabled, this didn't make a difference.
2) Tested with L3HW off and on, didn't make a difference. (I didn't expect it to because the packets are going to CPU anyway for a wireguard tunnel).

Very interesting, could you test this in Mangle if the output counter increases?
I have here a interface list named WireGuard and you have most likely one interface for WireGuard and then you use that in in/out-interface.

I think that ICMP response by the router is not packet marked so this may not work.

Update: I could not reproduce your problem and maybe that has to do that I have all WireGuard traffic is connection and routing marked. The packet mark extra.

Update 2: if your client create their own tunnels than ICMP packet is not leaking when the tunnel is terminated, because the tunnel outside is.

I also suspected that the policy rule (shown below) probably isn't doing the same thing as connection marking through mangle but that seems odd. Wouldn't they work in the same way? I mean to steer all traffic RouterOS likely is attaching an internal connection-mark to packets coming from the specified interface and then it knows to route them through the particular target interface. Based on some additional testing, I agree, I think the packet that is generated from the router must be some other kind of process that doesn't get marked by connection-marking. In the output logs, the packet is marked as "related" when it is sent by the router, so clearly it knows it was generated in response to some other packet that originally came from inside the wireguard tunnel. I suspect at this point there is some code missing to route it back through the same route it came from.

What do you mean interface, what are you trying to send through wireguard a subnet ??

I have the same rule and and yours routes traffic coming in through WiFi. Output is local so you have to match on the connection-mark you set. In Mangle output you can route it or kill it by setting TTL to 0.

What do you mean interface, what are you trying to send through wireguard a subnet ??

I have a single wireguard address, so I'm masquerading that interface "WG1" with a src-nat rule. I have X clients on my LAN network that connect through wifi that I want to route all traffic through the WG1 interface. The policy rule simply looks up and routes all traffic according to the rules in that mullvad_general table which is just a default route 0.0.0.0/0 which has a nexthop or gateway of the WG1 interface. Everything here works as expected. The issue is when the router responds to traffic after a client has forcefully disconnected or there is an interruption and it sends ICMP back to the target it sends it through the default route in the main table. Instead of sending it back through the expected WG1 interface.

I have the same rule and and yours routes traffic coming in through WiFi. Output is local so you have to match on the connection-mark you set. In Mangle output you can route it or kill it by setting TTL to 0.

With regards to your approach are you saying I need to do something like below.

1) Keep policy routing rule.
2) Add mangle rule that marks incoming packets coming into the input chain of the router that originated from the WG tunnel with an action of mark-packet.
3) Add a mangle rule on the output chain that routes marked packets back into the WG tunnel?

Your suggestion makes sense and I will try it.

Regarding the current functionality of the router I guess it makes sense that this is happening because technically the router is only using the main table for it's own routes and when it recieves an action like "respond to ICMP" it is no longer subject to the policy routing rule that is routing from the wifi interface. For these particular situations I would prefer to either have an option for the router not to respond to ICMP for packets that originate from a wireguard connection, or it should automatically send the response back through the wireguard tunnel it is acting on behalf of (best approach).

So after adding some more logging, the issue cannot be fixed by mangle rules outside of just dropping ICMP that originates from the router. What happens is before the packet reaches the state of invalid (which is then dropped by appropriate rules) it remains in a related state and this triggers the router to respond via ICMP. To be clear, I orginally thought the router was responding to a ICMP packet but the actual situation is that flow of TCP packets that cannot be delivered generate a ICMP packet from the router back to the source letting them know the packet cannot be delivered etc. So the original issue still remains, the router will try to repond appropriately (this is the expected process) to a packet that has been natted and cannot be delivered with variations of ICMP code 3. As seen below it tries to respond to the source IP (52.50.114.126) and even seems to know that it needs to pass through the WG tunnel (10.65.230.150) but ends up sending it through the default WAN interface (WAN 4000).

hello blacksnow,

To be clear, I orginally thought the router was responding to a ICMP packet but the actual situation is that flow of TCP packets that cannot be delivered generate a ICMP packet from the router back to the source letting them know the packet cannot be delivered etc.

it (the icmp 3 generation) depends on remote server requirements - which it needs to know the status of remote device accessing its service/ports (ie. doing keep alive while file transfer nor the tcp flags status from remote clients etc) for the server to close its open connection. so when the server sense it doesn't get the ack or fin flags - the server raise question to the offending ip - to remote device with icmp message (which is unfortunately your router masquerade ip). the router doesn't have the logic to return it back to the source if the conntrack is disabled (since the router still have that phone macaddr on the table). so the router looks like *blatantly* forwarding unknown traffic everywhere.

the key : conntrack, and remote service requirements (keep alive, status flags etc)


hope this helps.

Mixing users routing within a subnet is never clean.
Suggest trying one SSID reserved (one vlan) for those wishing to go out mullvad.
Instead of identifying X out of total users Y on vlan ( ssid ) by firewall address list and mangling them to go out mullvad.

So the packet return is flagged related and so you can combine that with that it is generated by the router self on behalf of the disconnected client. Then you can kill that packet returning anywhere, by killing it. Routing in Mangle is not available on output so that avenue is closed.
Please test this line if it really stops those ICMP packets going out the WAN.

Optional narrowing off the is adding check on connection-marking of even if the packet marked encrypted. You can also narrow the icmp-options to 3:7

ps. filtering on related is not needed because it is always related when the router answers on behalf.


Normal output: https://help.mikrotik.com/docs/display/ ... rOS-Output

Encrypted output starting at K and loops one time. To me, your logging does not show that the packet is encrypted.



Update, not knowing if traffic is encrypted or not I made the rule only kill those that are not going out through the VPN. You can also narrow it by reversing is to only filter WAN. [Post-routing] traffic is encrypted, blue lines, so the rule underneath is obsolete.

@ msatter

ps. filtering on related is not needed because it is always related when the router answers on behalf.

agreed. related means conntrack should be enabled. otherwise we will see huge amount of alien Traffic.

but. why do we bother to stop the end result of a stale connection (outgoing to any other link than the one they were coming from) - if we can filter it using - m state -- state invalid, untracked at inbound traffic on the tunnel?

@wisroute thanks and then invalid will catch that. Unmarked is traffic that is not present in connection tracking.

invalid - a packet that does not have a determined state in connection tracking (usually - severe out-of-order packets, packets with wrong sequence/ack number, or in case of a resource over usage on the router), for this reason, an invalid packet will not participate in NAT (as only connection-state=new packets do), and will still contain original source IP address when routed. We strongly suggest dropping all connection-state=invalid packets in firewall filter forward and input chains

Reading this I think this is not the golden bullet because the incoming traffic can be valid and have a active connection tracking. Only when the router want to push the traffic to the client the client says, won't accept it because I don't have an active memory of that connection.

So the invalid + untracked limits but does not account for all traffic. This is my thought on that.

ps. the default configuration has already a drop on invalid packet in filter in the input chain.

@ msatter,

Reading this I think this is not the golden bullet because the incoming traffic can be valid and have a active connection tracking. Only when the router want to push the traffic to the client the client says, won't accept it because I don't have an active memory of that connection.

absolutely

but this,

So the invalid + untracked limits but does not account for all traffic. This is my thought on that.

so, afaik, which I always did to make the work easier... i always taught myself there are 2 modes of firewalling :
1. passive firewall, which is the most easiest for me to calculate the results, examples :

-m state --state !related, established -j drop

drop is the key in passive Firewall. no hassle. no back info. just drop. efficient.

2. active/aggressive firewall, i don't like it - its cpu and bandwidth intensive. example:

-m state --state ! related, established -j reject

that reject will send reply to remote server - which could be anything from host unreachable, tcp reset etc.

so, in this very topic - i think it is obvious that either the @OP have no firewall at all on the tunnel - or... the @OP set the firewall reject mode hence there are routers activities for stale connections.

just a thought. nice discussion btw

So I do have a firewall on all interfaces, and specifically these WG tunnels are treated as WAN connections in terms of my rules because they can reach the outside internet. I have invalid packets dropped at the very top of my rule list (rule 1 & 2) for both the input chain as well as the forward chain. This works because I see the counters increasing and I've logged/tested. But with regards to the WG tunnel, the connection is not marked as invalid right away because remember when you force disconnect you do not send a FIN packet so the NAT entry is not closed. Only when the server tries to continue sending TCP packets that don't get ACK in return then it says "hey are you there at all". This is when the connection gets closed by the server end and the result is the NAT entries are deleted, then the packets are marked as invalid and dropped as expected. So the issue is really in between the connection closing and the NAT entry being deleted, the router responds with ICMP.

With regards to the solutions presented, they are in one form or another just the same as adding a firewall filter rule that blocks ICMP from the router (so output chain) going out the WAN interface. You can do this a variety of ways but the result is the same the packets are dropped. My question and this thread revolves around should this be the behavior of the router, should there be an option to not respond to ICMP on behalf of a client that was traversing through a WG tunnel or should there be the capability to route the ICMP automatically back through the associated WG tunnel.

hello blacksnow,

So the issue is really in between the connection closing and the NAT entry being deleted, the router responds with ICMP.

yes. but this chicken and the eggs thing could be overlap one another. which is who will close the door first: the server or the router?

supposed that remote server https keep alive at 5 second before closing the connection maybe not enough for the router to detect the absence of your phone macaddr on the table.

My question and this thread revolves around should this be the behavior of the router, should there be an option to not respond to ICMP on behalf of a client that was traversing through a WG tunnel or should there be the capability to route the ICMP automatically back through the associated WG tunnel.

as i said on my previous post, your router doesn't have the logic to send back those *alien Traffic* unless you create that routing and connection mark for the wg tunnel.

mikrotik wiki has that very nice pcc how to example

if that pcc doesn't work for your icmp 3,

you can create 3 layers of firewall in drop mode - not reject mode. because action=reject will still process any stale connection.

(put your wg accept rules before these lines)

1. you already did masquerade on wq tunnel - so let's put one filter on this natted interface :

prerouting -m state --state !related, established -j drop.

2. extra layer on wg input

input -i wg -m state --state ! related, established -j drop

3. extra layer on wg forward

forward -i wg -o any -m state --state ! related, established -j drop

4. let us see how it works

good luck

So I tested adding a connection-mark similar to the load balancing setups and what you all have been suggesting and you are correct that when you have connection-marks setup, it will route accordingly assuming you place the appropriate rules to route specific connection-marks comming off of the router output chain into the appropriate tunnel. Somewhat annoyed I have to setup connection-marks in addition to policy rules when I want to steer all traffic coming from a particular interface including any related traffic generated on behalf of those clients. I still think this is something that should be handled automatically if one chooses to use policy rules rather than the mangle approach. Both should essentially do the same thing, attach a connection-mark and use that to steer the packets into the right place. I even verified that connection-marks remain on the connection even when the router is responding via ICMP on behalf of a client so it only makes sense that the policy rules should be able to route that traffic as well.

Summary:
Simply using policy routing rules will steer traffic for 99.9% of cases, however with regards to ICMP since the router will respond on behalf of a client (due to disconnection or otherwise) unless the packets have been marked specifically using mangle rules (connection marking) then the router will not know how to route the ICMP packets that it generates back through the appropriate tunnel. The solution is to either setup connection marks and appropriate routes, or to drop all ICMP traffic coming off the router output chain or for more granularity you can use connection marks and drop only ICMP traffic that has the WG mark coming off the output chain if you choose not to route it. This still seems like a bug/missing functionality to me and I'm hoping a Mikrotik engineer can take a look and weigh in.

Just had a situation where TCP and UDP packets were leaking. I'm going to switch my setup to use mangle rather than policy routing, I would recommend the same for anyone else unless you want leaked packets.

My addition to completely resolve the issue and maintain other desired functionality. Added at the very top of my filter list.

I had till now not any related traffic going out so testing is not possible here. Can you put this log line in and check if this is detecting the same packets. It is a Postrouting and allow any last minute routing by routing adjustment to be done. It looks only a traffic coming form your router itself (10.0.8.254 ?).
Traffic that is routed last minute correctly is not detected but traffic that wants to exit through the WAN is detected and can after changing the rule also be halted.

But your choice to halt all answers on behalf is also good.

This mangle rule also detects traffic, just for reference the router is 10.0.8.1, the 10.0.8.254 is another wifi client.

Thanks for confirming this and for the correction. The position of the rule is in filter fine and it is then just before SRC-NAT is applied. In NAT there is since ROS 7 also an input and output chain so you can detect there also if something is leaking from by router self.

Changed the title back to ICMP only as I did many tests and was unable to replicate TCP/UDP packets leaking so I suspect I probably saw those show up when I was messing around with configs. I do continue to see the related ICMP packets being dropped as per the original scenario I outlined above.

Changed the title back to ICMP only as I did many tests and was unable to replicate TCP/UDP packets leaking so I suspect I probably saw those show up when I was messing around with configs. I do continue to see the related ICMP packets being dropped as per the original scenario I outlined above.

Now you have connection marked traffic going through the WG did you still see related ICMP traffic trying to out through the WAN?

I did not see any and high in Mangle I have a reRoute rule for sub sequential traffic from the client and now also a separate one catching connection marked traffic generated by the router self. This will inform the server on the other side of the WG connection that host is not reachable anymore. I think that is best solution to handle this problem.

That will render the filter rule useless. then it will serve as backup in case I make a mistake in Mangle in the future.

With connection marking I did not see any pop-up in my related or just pure ICMP catch rule on the output chain of the router. Connection-marking definitely ensures that both established and related marked connections are routed through to the right place. However, connection marking is very tedious for my use case. It is much easier to use the policy rules to steer the entire incoming interface's traffic to use a particular routing table. So for me I continue to do it that way versus the connection marking route and therefore I still need the drop rule on the router output chain.

I agree the best approach and most complete is through connection marking, however it is too many rules to maintain especially when you have multiple routing tables and you want to be able to easily change things on the fly.

Hi there blacksnow, can you summarize in simple terms what is going on. How the leak occurs.
How is this remedied by an output chain rule?
Did you send a supout to MT for them to digest?

@ blacksnow,

It is much easier to use the policy rules to steer the entire incoming interface's traffic to use a particular routing table. So for me I continue to do it that way versus the connection marking route and therefore I still need the drop rule on the router output chain.

well, after all these approaches you have tested for your icmp leakage concern, the first thing I have had noticed was your wg masquerade setup - which actually already doing partially good easy job to streamline forwarding traffic from your phone to the internet via wg tunnel.

then that icmp 3 problem came in.

hmm, how about this setup - maybe you could test this first, so that you don't have to maintain too much marking?
you already have outbound masquerade on wg tunnel. now, how about you put another masquerade on wg inbound? dstnat as wg ip.

so double masquerade. outbound and inbound.

so that your router nor lan device sees wg internet as 1 entity - which is the wg ip.

have a lab. and let us know the results

Sure @anav so here is whats up.

Background:
Whenever there are issues with delivering packets, fragmentation etc, the router can reply on behalf of the client indicating to the source through ICMP that something isn't perfect. The router by default uses the main routing table to determine where/how to deliver packets, this includes any packets that the router will generate on behalf of any clients connected to the router unless otherwise specified by mangle rules (connection marks, routing marks etc).

Summary of Issue:
Using routing policy rules is not the exactly same as using connection marks in mangle. Policy rules are explicit to the src address, dst address, routing mark or interface upon which they are applied and must be matched accordingly at each routing decision, connection marks are universal or much more permanent in the sense that once the connection has been marked the mark lives with the connection no matter where it goes or who picks it up meaning all packets (established and related) once the first packet has been connection marked. Routing policy rules happen in the routing phase and are applied to each new packet individually each time it goes through the routing decision step (which makes sense they are part of the routing tool) but as a result the following scenario can occur.

1) There is a routing table named wg1_route which has only a default route of 0.0.0.0/0 that goes through interface WG1 (a successfully setup wireguard tunnel to some destination X).
2) Policy routing rule is setup to route all traffic originating from Home_Wifi (a vlan interface, but this can be any interface ether1 or whatever) using only entries in the routing table wg1_route (shown below).
3) All traffic is routed as expected when it originates from Home_Wifi per the policy routing rules, however when a related connection originates from the router (ICMP in reply to indicate a host is unreachable etc) it is not caught by the policy routing rule and defaults to the main routing table meaning it leaks through that default wan route (shown below).
Bug/Issue:
Whether you use the src address field, the dst address field, or interface the routing policy rules alone will not route outgoing related packets that originate from the router using the desired routing table. Even using mangle routing marks do not route properly if the packet is originating from the router. If you use connection marks it works as expected because the connection is marked first when it exits Home_Wifi and that mark lives with the connection and all associated packets until the connection dies.


Solutions:
1) Block all related connections coming off the router using the rule below, this blocks related ICMP packets that are coming from the router on behalf of LAN client. It will also break things like if you are hosting a ftp server on your routers WAN connection in my case this isn't an issue for me you can switch the protocol to ICMP to further nail this down without breaking other functionality. 2) Use connection marks in mangle to route packets.

Okay I got lost at 3!!

3) All traffic is routed as expected when it originates from Home_Wifi per the policy routing rules, however when a related connection originates from the router (ICMP in reply to indicate a host is unreachable etc) it is not caught by the policy routing rule and defaults to the main routing table meaning it leaks through that default wan route (shown below).

I am sitting pretty on vlanX, be it a wired or wireless subnet.
I have routing rule that says for such src-address all traffic goes out WG.
Above that I have other routing rules NORMALLY that basically state dst-address to any other local addresses use table main.
Therefore any local use is correctly done through table main be it originating traffic on subnet or return traffoc to local subnets.

I fail to see any other traffic thay may leak.
You are being much too vague for non-networking me, when you say related connections originates from router blah blah blah.................

Anav I have a similar setup. I have 1 routing rule at the very top that routes anything destined for the 10.0.0.0/8 range to only use the main table so all local connections work as expected. Everything you have is fine and essentially what I am doing as well +/- some random stuff. The important part is that when the router replys via ICMP to a source it will use the default routing table and it will not send the response through the WG tunnel. Easy steps to replicate based on your setup.

1) Add a log rule at position 1 in your firewall filters that operates on the output chain, protocol ICMP, and destination interface being the WAN connection (not the WG tunnel, your actual WAN connection that you get from the ISP) you can also leave the destination interface blank if you don't have a bunch of other noisy traffic that you will have to parse through.
2) Use a device on vlanX that is routed using policy routing rules through a WG tunnel to connect to speedtest.net and start a speedtest.
3) Forcefully disconnect that client during the download portion of the speedtest. Literally put the wifi device on airplane mode or disconnect the ethernet wire if your using a hard wired device.
4) Leave the device disconnected for the remaining portion of this.
5) Within 30 seconds to 1 minute your log rule will start logging the router sending out ICMP codes out your default WAN connection towards the source of whatever speedtest server you were using on behalf of the forcefully disconnected client.

The NAT will look like (Client Device LAN IP -> WG Masquerade IP) -> ISP WAN IP and the target exit interface will be your default ISP WAN interface, not the WG tunnel which means the packet is "leaking".

I full believe you but dont understand the scenario.
Why would the router be responding to anything from the subnet. Its traffic is either going out the net or to local traffic.
Its the exact details you are lacking.......... or an explanation at my low level.........

My question is how is icmp traffic in any way shape or form related to the subnet going out wireguard, practical example???

Give it a rest Anav.

Happy you understand what is triggering the leak but I dont............. Butt out ........

Just bumping this with more detail as I finally got around to creating a SUP-149838.

Description: When using Router OS V7 (7.15beta9) with the following configuration it is observed (via packet logging) that the router is not correctly routing related packets generated by the router on behalf of client devices (such as ICMP packets) through the proper routes via established connection/routing marks on the original traffic that prompted the related packet to be generated.

Context: For a simple setup and reproducibility, you have two routers, the first (router V) has an upstream WAN with a public IP of 2.2.2.2 configured, 1 LAN network range (10.0.5.1/24) and 1 wireguard instance with 1 peer (peer A). Peer A is a site-to-site tunnel (tunnel P) that connects to another router (router Q) that has it’s own upstream WAN connection with a public IP of 3.3.3.3 and a corresponding wireguard peer (peer B) to allow peer A from router V to connect.

Using connection marking we are able to route all of the traffic from the LAN network on router V (10.0.5.0/24) through the tunnel established between peer A on router V to peer B on router Q. All traffic destined for WAN produced directly from the client 10.0.5.15 appears as if it is coming with a source IP of 3.3.3.3, this works perfectly fine. No firewall rules are necessary at this time, but do include a log rule on the output chain for all related packets.

Issue: The problem occurs when a LAN client on router V’s network suddenly drops/loses connection or any other reason that requires sending an ICMP packet by the router on behalf of the client to some external target (lets say AWS endpoint 5.5.5.5). For example if a client drops suddenly without closing the connection Router V senses this issue because packets are continuing to come from tunnel P with a destination of 10.0.5.15 (our hypothetical LAN client) and since no response from the client is sent back the router decides to send an ICMP packet (type 3, code 1) to the sender which in this case is 5.5.5.5. However, instead of sending this ICMP packet through tunnel P to the target as the connection-marking would have designated, it instead sends it through it’s own upstream WAN default route through the public IP 2.2.2.2.

Reproducibility: Using the context section provided in this ticket, create a network that follows that layout. Once setup, have a LAN client on router V initiate a connection to youtube or some other streaming service. Start watching a video, and then either physically or otherwise sever the connection between the LAN client and the router abruptly (mid-playback). You will see that after about 30-60 seconds router V will send related ICMP packets (visible in the output chain) of (type 3, code 1) to various IP’s (aws, google and other cdns). The point is in the log the packets will be marked with the appropriate connection mark but they will not be routed through the tunnel as the connection mark would indicate. Instead they will be sent via router V’s default upstream WAN connection.

Notes: An alternative reason could be that the logging is incorrectly stating what is happening and the packets are actually routed properly, but this seems unlikely.

Clarifying, I incorrectly stated the following quoted below, after deeper investigation I realized that not even this works.

If you use connection marks it works as expected because the connection is marked first when it exits Home_Wifi and that mark lives with the connection and all associated packets until the connection dies.

This is testing with connection-marking.
This also occurs with other types of ICMP codes, which is frustrating because those ICMP codes are actually necessary. Like for example saying a smaller MTU is necessary via ICMP (type 3, code 4).

So in short, while I can block these packets, it is disadvantageous for me to do so because the ICMP codes actually serve a purpose.

Can you reproduce the issue with a site like this too: https://websocketking.com/? After loading the page make sure to connect to the demo/echo endpoint and wait for the log to say `Connected to wss://echo.websocket.org`. Asking because the youtube streaming platform might bring quite a bit of complexity to the table, but if you can repro it by terminating an open websocket connection too then that would be much easier to look into.

Also it would be very helpful if you could add a network diagram image and routeros configurations if you already have those