I have an IPsec tunnel between two MTs (RB750GL and hEX 750G r3, both with current version 6.39.2), set up exactly according https://wiki.mikrotik.com/wiki/Manual:I ... Sec_Tunnel.
The tunnel itself works correct, I can make connections from devices from both sides to each other.
The only problem is with traceroute between two sites - the router on the other side returns only asterisks.
Ping against router and all devices on the other side works.

This is my topology:





This is the result of Windows tracert command from 192.168.2.1 (trying to trace route from PC from one network to PC on the other network):



Similar result directly from router's trace route tool:


And this is what I think is related to the problem - ICMP packets from tracert addressed to WAN interface of the remote router.


I was searching for similar problem a couple of days, playing with firewall/NAT/mangle rules, altering TTL values on both routers, but didn't find any solution.
Does anyone have the same behaviour on the IPsec tunnel? Does anyone know the cause / solution of this?

Are you creating a Tunnel or are you using Transport for the IPSEC policy's? There is a difference in behaviour depending on which option you went with and how you configured the IPSEC policies. A little to much to explain here, but you can read up on the differences and it may shed some light for you.

I was searching for similar problem a couple of days, playing with firewall/NAT/mangle rules, altering TTL values on both routers, but didn't find any solution. Does anyone have the same behavior on the IPsec tunnel? Does anyone know the cause / solution of this?

^^ Ya, that was me. Thankfully you're in the club that wants traceroute to work so you and I are friends now. That means I will help you get 'er working!

So, it could be a few things. An export will likely show us the problem. Like the first reply indicates. If you're using a straight IPSec tunnel (no underlying tunneling technology like GRE, IPIP or EoIP) then the router likely has no way of sending an ICMP Time Exceeded message back which is what the traceroute utility is looking for. This is because well, the protocol is ICMP not TCP or UDP. If you used the ALL flag for protocol then it could be the src-address or dst-address needs to include to the router IP for ICMP.

This is 1 of the reasons I typically do IPSec in transport mode with GRE instead of a straight IPSec tunnel mode VPN. And could be verified by "/ip ipsec policy export" and posted here. (Dynamic routing - excluding BGP - is my other leading reason for GRE)

If the boxes above are checked or you are already using a tunnel underneath a transport mode IPSec connection then we need to look at the firewall rules. The traceroute tool works because it sends a packet to the destination with a TTL value set to 0 (maybe it starts at 1 I can't remember) then increments it by 1 until it reaches the destination. Each "hop" or router is supposed to send back an ICMP Time Exceeded message which includes it's IP address. This is how traceroute learns the routers address. The fact that you are seeing only 1 line of asterisks means that the TTL is being decremented appropriately but the tool isn't hearing the ICMP Time Exceeded message. It is possible you have an over zealous firewall rule dropping the wrong kinds of ICMP messages.

This could be verified by "/ip firewall export" and posted here.

The Torch traffic is fairly interesting. Setting log=yes on an IP firewall filter log will get us type and code. It is possible that message is unrelated. Something like protocol=icmp, action=accept and log=yes only in the rule and all the way at the top should suffice. It seems odd the message would be destined to the WAN IP but sourced from the an IP assigned to the LAN. I'd expect the TTL exceeded to be sourced and destined to both WAN IPs not one or the other and that's only if MikroTik doesn't read inside the IPSec message and matches the TTL to outside of IPSec tunnel traffic. It's possible there is a bug somewhere in their but I'd be a bit surprised. Either way a more detailed log entry may show us it isn't the Time Exceeded message we're expecting and it might just be a benign ping from something like a check-gateway.

I labbed it up quick, 192.168.1.2 is the "WAN" IP of the remote router and 10.1.2.2 is the "Internal IP" of my Traceroute client.

16:35:36 firewall,info output: in:(none) out:ether1, proto ICMP (type 11, code 0), 192.168.1.2->10.1.2.2, len 56
16:35:39 firewall,info output: in:(none) out:ether1, proto ICMP (type 11, code 0), 192.168.1.2->10.1.2.2, len 56

So the firewall generates an ICMP Time Exceeded message as you'd expect but it's sourced from the interface where the traffic arrived (WAN). This means it has to be sent back to the client and it egresses ether1. I'd have to really study the packet flow diagram to see if it is possible to capture this traffic and have it tunneled to the client where it could be decrypted and delivered to the client.

I did try a few things, creating a policy for traffic sourced from WAN to the remote LAN seems to activate as expected per the packet flow diagram but doesn't seem to get decrypted and sent on by the remote router. From there I lose it. I suppose with IPSec debugging I could look at the packet level to see if it in fact arrives and if it is discard or decrypted and try to follow it fro there ... but it's Friday so ya.

Maybe someone else has some additional thoughts?

If it's simple IPSec tunnel with policy only for 192.168.2.0/24 <-> 192.168.5.0/24, then it won't catch icmp time exceeded from 11.22.33.44, it will go out unencrypted towards public internet and because it's for 192.168.2.1 (private address), it will be dropped just a little later.

There would have to be additional policies for 192.168.2.0/24 <-> 11.22.33.44 and 88.77.66.55 <-> 192.168.5.0/24. Or go the same way as many others did before, use some "real tunnel" (GRE/IPIP/...) and only use IPSec in transport mode to secure it. It will behave like a "normal" interface, take any addresses, etc.

Are you creating a Tunnel or are you using Transport for the IPSEC policy's? There is a difference in behaviour depending on which option you went with and how you configured the IPSEC policies. A little to much to explain here, but you can read up on the differences and it may shed some light for you.

Tunnel.

If it's simple IPSec tunnel with policy only for 192.168.2.0/24 <-> 192.168.5.0/24, then it won't catch icmp time exceeded from 11.22.33.44, it will go out unencrypted towards public internet and because it's for 192.168.2.1 (private address), it will be dropped just a little later.

There would have to be additional policies for 192.168.2.0/24 <-> 11.22.33.44 and 88.77.66.55 <-> 192.168.5.0/24. Or go the same way as many others did before, use some "real tunnel" (GRE/IPIP/...) and only use IPSec in transport mode to secure it. It will behave like a "normal" interface, take any addresses, etc.

Thank you all for helping. Now I understand the cause of the problem. I thought that even the Time Exceeded reply would easily travel back through IPsec tunnel as any other packet.
So as suggested I set up a GRE tunnel with IPsec; that was successful, the traceroute between both LANs finally worked as expected.
BUT I had other problems then:
1) I couldn't trace / ping the remote WAN IP from LAN (probably because the ICMP requests were encrypted and sent into the tunnel instead of going out through WAN)
2) the performance of that tunnel was very low. Both sites have 5 Mbps upload; when using IPsec I get full speed. When used GRE/IPsec, the transfer speed dropped below 1 Mbps.
3) I tried experimenting with other policy settings as suggested by Sob, but I almost locked myself out of the remote router (which is on different continent), so I rather stopped experimenting for now.

So I rolled back to my former config without GRE, I think I can live without the internal trace, I thought there would be an easier solution to this.
Anyway thank you all very much for explaining the problem and for your help!

I managed to have traceroute and tracepath going between 2 different RFC1918 subnets in a site2site ipsec vpn behind public IPs. That without using the fancy solution of creating GRE protected interfaces (the poor man Routeros equivalent of the Virtual Tunnel Interface that you have on Cisco or Juniper):

(192.168.1.0/24)--Routeros on PublicIP1------INTERNET-------Routeros on PublicIP2---(192.168.2.0/24)

It's a tipical Site2Site VPN:

Doing tracepath and/or traceroute from 2 linux machine on either lan gives an icmp out on the WAN interface that obviously is dropped from the upstream internet gateway.


My solution was (as already suggested here) creating a new ipsec policy for the public IP that correctly gives out



Anyway, putting on the following rules on the two sites I could remove the ipsec policy for the public ip addresses and have the tracepath/traceroute going well:



the gateway=ether1 is something put here but not really needed. Its not the public interface (that really is a PPoE thing). The policy rule takes the traffic before the route rule, that is needed only to correctly "source" tha ICMP response packet originated from the router.

The final output is like this:


So, the ipsec-encapsulating-decapsulating routers correctly respond to the ICMP packets, from the right (and tunneled) IP, without involving the public IP, and this way there is no need for a new IPsec policy.