Hi,

I have a topology with 4 Mikrotik CHRs (version 6.43.13 - LongTerm):
- Edge: R1 and R2
- Core: R3 and R4

Edge routers have IPSEC toward Routers A and B with eBGP peering configured. Primary path from remote AS is through R1 and backup is through R2.
Edge and Core routers have IPIP tunnels with IPSEC and iBGP peering configured.

Image is on https://ibb.co/L0Grh4z


Routers A and B have routes to 1.1.1.1/32 and 1.1.2.1/32 and Core routers R3 and R4 have routes to 100.100.1.0/24 and 100.100.2.0/24.
Traffci goes fine if path is
- Router A => R1 => R3 or R4
- Router B => R2 => R3 or R4

I'm testing redundancy, and when links from R1 to R3&R4 are down, so only way for traffic is yellow arrrow, Router A => R1 => R2 and then R3 or R4 (doesn't matter), I have a problem.
When I sniff traffic on R2, I can see ingress traffic on IPIP tunnel R1-R2, but no egress traffic to R3 or R4. In log I can see:

[admin@R2 /ip route> /tool sniffer quick port=1812 freeze-frame-interval=10s
INTERFACE TIME NUM DIR SRC-MAC DST-MAC VLAN SRC-ADDRESS DST-ADDRESS PROTOCOL SIZE CPU FP
IPIP_R1-R2 5.656 1 <- 100.100.1.1:1812 (radius) 1.1.1.1:1812 (radius) ip:udp 581 1 no
IPIP_R1-R2 7.67 2 <- 100.100.2.2:1812 (radius) 1.1.2.1:1812 (radius) ip:udp 581 1 no

What is interesting; if I disable IPSEC policies on R2 for IPSEC to Router B, traffic goes as it should, so it seems to me that traffic is simply triggering wrong policy. On R2, policies to Router B are configured manually and for IPIP tunnels, Mikrotik configures dynamic policies of course.

Routers A and B are Junipers with route-based IPSEC configured. So I configured policies with level Require because on the other side, only one SA is up.
I also added those policies with action NONE:

/ip ipsec policy
add action=none dst-address=1.1.1.1/32 src-address=100.100.1.0/24
add action=none dst-address=1.1.1.1/32 src-address=100.100.2.0/24
add action=none dst-address=1.1.2.1/32 src-address=100.100.1.0/24
add action=none dst-address=1.1.2.1/32 src-address=100.100.2.0/24
add dst-address=100.100.2.0/24 proposal="PROPOSAL_1" sa-dst-address=4.4.4.4 sa-src-address=172.31.32.27 src-address=1.1.1.1/32 tunnel=yes
add dst-address=100.100.1.0/24 proposal="PROPOSAL_1" sa-dst-address=4.4.4.4 sa-src-address=172.31.32.27 src-address=1.1.1.1/32 tunnel=yes
add dst-address=100.100.2.0/24 proposal="PROPOSAL_1" sa-dst-address=4.4.4.4 sa-src-address=172.31.32.27 src-address=1.1.2.1/32 tunnel=yes
add dst-address=100.100.1.0/24 proposal="PROPOSAL_1" sa-dst-address=4.4.4.4 sa-src-address=172.31.32.27 src-address=1.1.2.1/32 tunnel=yes


Dynamic policy R2 to R1, R2 and R3:

16 DA ;;; IPIP_R1-R2
src-address=172.31.32.27/32 src-port=any dst-address=10.0.50.102/32 dst-port=any protocol=ipencap action=encrypt level=require ipsec-protocols=esp tunnel=no proposal=default ph2-count=2

17 DA ;;; IPIP_R2-R4
src-address=172.31.32.27/32 src-port=any dst-address=192.168.255.30/32 dst-port=any protocol=ipencap action=encrypt level=require ipsec-protocols=esp tunnel=no proposal=default ph2-count=2

19 DA ;;; IPIP_R2-R3
src-address=172.31.32.27/32 src-port=any dst-address=192.168.255.40/32 dst-port=any protocol=ipencap action=encrypt level=require ipsec-protocols=esp tunnel=no proposal=default ph2-count=2

For tshoot reasons, I configured mangle rules just to track packets.
In log I see:

10:50:01 firewall,info MANGLE_IN_NONE prerouting: in:IPIP_R1-R2 out:(unknown 0), proto UDP, 100.100.1.10:1812->1.1.2.1:1812, len 581
10:50:03 firewall,info MANGLE_IN_NONE prerouting: in:IPIP_R1-R2 out:(unknown 0), proto UDP, 100.100.2.20:1812->1.1.2.1:1812, len 581

When policies to Router A and B are disabled, then traffic goes fine and in sniff and log I see:

[admin@R2] /log> /tool sniffer quick port=1812 freeze-frame-interval=10s
INTERFACE TIME NUM DIR SRC-MAC DST-MAC VLAN SRC-ADDRESS DST-ADDRESS PROTOCOL SIZE CPU FP
IPIP_R1-R2 53.627 1 <- 100.100.2.20:1812 (radius) 1.1.1.1:1812 (radius) ip:udp 581 0 no
IPIP_R2-R3 53.627 2 -> 100.100.2.20:1812 (radius) 1.1.1.1:1812 (radius) ip:udp 581 0 no
IPIP_R2-R3 54.667 3 <- 1.1.1.1:1812 (radius) 100.100.2.20:1812 (radius) ip:udp 196 1 no
IPIP_R1-R2 54.667 4 -> 1.1.1.1:1812 (radius) 100.100.2.20:1812 (radius) ip:udp 196 1 no

11:02:20 firewall,info MANGLE_IN_NONE prerouting: in:IPIP_R1-R2 out:(unknown 0), proto UDP, 100.100.2.20:1812->1.1.1.1:1812, len 581
11:02:20 firewall,info FORWARD____LOG forward: in:IPIP_R1-R2 out:IPIP_R2-R3, proto UDP, 100.100.2.20:1812->1.1.1.1:1812, len 581
11:02:20 firewall,info MANGLE_TEST postrouting: in:(unknown 0) out:IPIP_R2-R3, proto UDP, 100.100.2.20:1812->1.1.1.1:1812, len 581
11:02:20 firewall,info MANGLE_TEST postrouting: in:(unknown 0) out:IPIP_R2-R3, proto UDP, 100.100.2.20:1812->1.1.1.1:1812, len 581
11:02:20 firewall,info MANGLE_POSTROUTING_OUT_NONE postrou: in:(unknown 0) out:IPIP_R2-R3, proto UDP, 100.100.2.20:1812->1.1.1.1:1812, len 581
11:02:21 firewall,info FORWARD____LOG forward: in:IPIP_R2-R3 out:IPIP_AWS_R1-R2, proto UDP, 1.1.1.1:1812->100.100.2.20:1812, len 196


Does anyone have a clue how to configure this to work even when IPSEC on R2 to Router B is enabled?

Tnx.

It doesn't seem like just hitting a wrong policy because the policy's traffic selector is normally matched at the very last moment before the packet would be sent out via the gateway determined by the regular routing, i.e. after all the firewall processing (including even src-nat in postrouting) has been done. So if the packet just about to be sent to R3 or R4 would have been mistakenly stolen by some policy, your action=log rules in mangle and filter would have normally logged the pass through filter's forward and mangle's postrouting chains, which they haven't, so it seems as if the filter doesn't even get them.

Is the action=log rule the very first one in filter's forward chain? I assume it is, as people already running setups of this complexity when asking their first ever question on the forum usually don't make stupid mistakes, but I better ask to exclude elementary issues. If it is, I suspect there is something hidden that drops packets which reverse-match a traffic selector of an active policy, so they should normally ingress via an SA associated to that policy, but they actually ingress some other way. If it is the case, it should be possible to replicate such behaviour using a much simpler setup (of 4 devices in total) and give the supout.rif from the affected router to support@mikrotik.com.

I suspect there is something hidden that drops packets which reverse-match a traffic selector of an active policy, so they should normally ingress via an SA associated to that policy, but they actually ingress some other way. If it is the case, it should be possible to replicate such behaviour using a much simpler setup (of 4 devices in total)

Wow. So I gave it a try on my own and it really behaves like that. The setup is simple, actually two devices are enough, you set up an action=encrypt IPsec policy for some source and destination subnets along the normal route, and at the sending end you place before it an exceptional policy with action=none, preventing packets with a particular destination address in the dst-address range of the action=encrypt policy from being tunnelled, so these packets arrive in plaintext to the same ingress interface of the MUT (Mikrotik Under Test) to which the tunnel transport packets arrive. And whereas the payload packets decrypted and decapsulated from the transport ones are seen by /tool sniffer and both the mangle and filter chains of /ip firewall, the plaintext packets are seen only by /tool sniffer and the mangle chain, but not by the filter chain, although the rules in chain=mangle have just action=log and these logging rules in chain=forward of /ip firewall filter are the topmost ones.

In particular the rules in question looked like this in my case: So it is definitely worth a support ticket.

As a workaround, you'll likely have to use GRE tunnels inside the IPsec ones between A and R1 and between B and R2.

@sindy: Your log-prefix and ipsec-policy don't match (in two rules). But you're right, this:

... something hidden that drops packets which reverse-match a traffic selector of an active policy ...

is there. I'm under impression that it should happen, but I can't find any reference to that. With (hopefully) same setup as yours (policy for 172.23.0.0/20 on local side, but individual addresses excluded by action=none on remote), I get this: Both packets to .2 and .4 don't match local policy (they are expected to come encypted, but they are not). They both increate in-template-mismatches in /ip ipsec statistics. Packet to .4 disappears between prerouting and forward. Packet to .2 gets to input, but it looks like it disappears after that, because there's no reply in output.

As a workaround, you'll likely have to use GRE tunnels inside the IPsec ones between A and R1 and between B and R2.

When you are free to configure it, you can use GRE or IPIP tunnels inside IPsec but do not use IPsec tunnels, rather use IPsec transport.
Then you do not incur the extra overhead of two nested tunnels.
When configuring GRE or IPIP tunnel with IPsec encryption key on RouterOS, it is done correctly.

I see IPIP/IPsec tunnels are used in the right part of the diagram, why not for those two links on the left?

Your log-prefix and ipsec-policy don't match (in two rules)

That's true, but luckily it had no impact on my analysis because after adding the rules I was actually using just /ip firewall mangle print stats to observe what's going on.

I didn't come to my mind to check /ip ipsec statistics print but checking it now has also shown that I was accumulating in-template-mismatches. And it didn't come to my mind because once the packets made it to mangle, the IPsec processing should have been already over for them.

I'm under impression that it should happen, but I can't find any reference to that.

And that's what makes me so crazy about it. I hate undocumented functionalities, which on top of being undocumented appear not to behave logically.

At times when the ipsec-policy matcher did not exist in firewall, there was a bit of nervosity on how to distinguish between packets from IP A to IP B which came in via IPsec tunnel and plaintext packets from IP A to IP B which came in via the same interface as the transport ones of the tunnel.

In that topic, gents from Mikrotik have never stated that the IPsec processing (which you don't expect to ever touch plaintext packets, do you?) addresses that situation by dropping those received plaintext packets that match an inverse of IPsec policy's traffic selector. And then the ipsec-policy matcher appeared, which for me was a sign that the concern was real and the ipsec-policy matcher was the solution.

But now it appears to me that there is actually a hidden rule at the beginning of /ip firewall filter (because filter has the magic power to drop packets) which drops packets which came in as plaintext and match the mirror of a traffic selector of one of the policies which are active and have action=encrypt. To check, I've added the action=none policy mirroring the one used at sending side also to the MUT (receiving side), et voilà - the packets inverse-matching it started being accepted.

The drawback of this "we anticipate what you want so we'll do it whether you actually want it or not, and we won't tell you" is that the traffic selectors have no relationship to interfaces, so the hidden rule in filter drops the plaintext packet regardless whether it came in through the same interface through which it should have come encrypted (which is probably what we want) or whether it came in through some other interface (which is what we definitely don't want and what was the OP's trigger to start this topic).

A loosely related topic is the ipsec-policy=out,... matching. There, I never understood how it might work given that the manual says that the IPsec processing is done after the postrouting chain of the firewall; as the headers of egress packets are obviously actually compared to policies' traffic selectors already at filter stage, escaping the nat processing could have also been done automatically (following the "we anticipate what you want..." principle mentioned above), but it is not (and for a good reason, as sometimes you need to use the nat processing to choose whether the packet will be sent out via IPsec or not). So that needs to be explained better too, because if the ipsec-policy=out,... matches the packet header before nat processing, it is kind of counter-intuitive, and if was matching the packet header after nat processing, it would require some dry run of the packet through the firewall to know the final outcome already at filter stage.

So @dario111, please either open a ticket at support@mikrotik.com and note it down here, or say expressly that you won't, so that we don't open several tickets for the same issue.

Well, in linked thread, I see this post from @mrz confirming that unencrypted packets do get discarded, even though his previous post seems to tell otherwise.

Although maybe we're mixing two things together. It seems that droppping unencrypted packets that were supposed to come encrypted is what IPSec should naturally do, so you should be safe even without ipsec-policy=in,ipsec, as long as you have correct policy (peer config must be active, but not necessarily connected). This works well with statically configured policies like site to site VPN. But not e.g. with IPSec for L2TP, because there will be IPSec policy only when client asks for it first. And that's what ipsec-policy matcher is for, to match packets that actually came encrypted (or will be encrypted when going out).

The ipsec-policy=out,... if fine except for NAT part. You can't use it to match packets after they were encrypted (because it's the very last thing that happens to them, after firewall), but it can match packets that will be encrypted. It would be almost the same, if it wasn't for NAT that can mess it up. But I'm willing to see it as natural property of NAT, it messes up things.

About how exactly IPSec drops work, I don't know. But there's something similar with rp_filter, it allows packets in prerouting, but they never reach input or forward. It's equally confusing when you don't expect it.

Well, in linked thread, I see this post from @mrz confirming that unencrypted packets do get discarded

There are two aspects of this:

• does the RFC really require this to happen even if the plaintext packets haven't via the same interface through which the SA transport packets come in (and if it does, is it a legitimate requirement given the scenarios like the OP one with two IPsec tunnels used for redundancy and a failover link between the devices on which those tunnels are terminated)?
• if the RFC is right, is it OK that the plaintext packets start getting in as soon as the SA goes down (which may happen due to a bug, there was some CVS (no idea whether it ever affected Mikrotik) which allowed to crash the IPsec stack and gain unencrypted path to the device). Because - the reason for dropping plaintext packets which should have arrived encrypted is security, whereas allowing them to get in when the tunnel breaks down (or before it gets established) is anything but secure.

And that's again the same story... this particular part of the behaviour is so counter-intuitive (which is not the same as incorrect!) that when writing the manual, you cannot rely on everyone to know the RFC and have to mention that.

And even if it should behave like that, the RFC also says the following:

This is an auditable event. The audit log entry for this event SHOULD include the current date/time, SPI if available, IPsec protocol if available, source and destination of the packet, and any other selector values of the packet that are available.

Needless to say that there is nothing in the log even with ipsec logging on. You cannot find even the explanation of the detailed meaning of in-template-mismatches in /ip ipsec statistics in the Mikrotik wiki...

You can't use it to match packets after they were encrypted (because it's the very last thing that happens to them, after firewall), but it can match packets that will be encrypted.

Yeah, right, but what's the purpose of this matcher in filter then? To decide whether to NAT them or not makes sense, but to filter packets based on them being eligible for encryption is almost useless; what you really need is to prevent packets from leaking out if they fail to get encrypted (because the tunnel is down, for example). Normally, if there is a policy, it should prevent the packets from taking the normal path regardless whether the SA serving that policy is active or not. If this contradicts some requirement of the RFC, your should be able to set an action=discard policy next to the action=encrypt one, but that doesn't work (the latter one stays inactive even if the former one becomes inactive if their traffic selectors are exactly the same). So on top of specifying the policy for the packets to be encrypted, you have also to specify a route for them which sends them to a blackhole interface, so you have to set the same thing at two places if you want to be sure.

The whole policy-based IPSec tunnels with "stealing" outgoing packets and encrypting them is a little confusing, at first at least. Interfaces don't play any role here, it doesn't really matter where packets go to, they use regular routing in unencrypted state, but then IPSec snatches them and can send them elsewhere.

Or another problem, I like to add unreachable routes for private addresses, so any leaks to unused subnets are automatically stopped by router even without firewall. But it of course breaks tunnel-mode IPSec to remote private subnet, because there's no route for it and I have to add a fake one, which is ugly solution. But recently I saw some Linux IPSec implementation and it was adding such routes itself.

Route-based IPSec tunnels with own interfaces (as supported by other vendors) would be easier to understand. But RouterOS doesn't support that and even with recent big IPSec changes there's no word about it. I'm not sure how would it play along with policy-based tunnels, this part could make things even more confusing. But otherwise it sounds as very nice thing to have.

..., but to filter packets based on them being eligible for encryption is almost useless; what you really need is to prevent packets from leaking out if they fail to get encrypted (because the tunnel is down, for example).

This seems to work. Even if tunnel is down (e.g. because remote site doesn't respond at all), filtering by policy works, even if router is only trying to contact with peer. And if it's not active at all (e.g. you disable local end), then there's no policy and you'll get match with ipsec-policy=out,none (or not with ipsec-policy=out,ipsec).

Route-based IPSec tunnels with own interfaces (as supported by other vendors) would be easier to understand. But RouterOS doesn't support that and even with recent big IPSec changes there's no word about it.

But IPIP tunnels over IPsec transport are close enough. While not compatible with the implementations by other vendors they (or GRE tunnels over IPsec transport) offer the same convenience and work very well on MikroTik RouterOS, as long as the external addresses are static.

@sindy; please open the ticket as if it's not a problem for you.

Thanks.

For this specific setup, the other side of IPSEC tunnel is not configured by me and I don't have access for that. It's route based VPN configured on Juniper and the only thing I can do on Mikrotik is policy-based VPN.

Between my Mikrotiks I use IPIP over IPSEC and it works fine.

Br,
Dario

In the course of discussion while you were on vacation it turned out that the behaviour observed is requested by RFC4301. So it becames a much more subtle topic whether that RFC mandates that to happen always or whether it can be interpreted more to the favor of your scenario, i.e. that the behaviour be linked to particular interface. And even if I'd manage to convince Mikrotik's R&D (via support, I have no direct link to them), it would be an implementation nightmare so it would take months at best to get it integrated into the stable branch.

So I've thought about a workaround consisting in setting up the same type of IPsec tunnel also between R1 and R2, but after overcoming some minor issues, I've found that you cannot fool the IPsec stack - not only it checks whether the packet comes in via "some" SA but only accepts it if it came via the particular SA associated to the first policy it reverse-matches. And even if that approach succeeded, it would just force you to extend the policy-based IPsec also to the links to R3, R4, otherwise the same issue would happen when R1 would drop packets received directly from R3 because they would reverse-match the IPsec policy used on the backup route via R2.

So there is essentially no other solution to your scenario than splitting R1, R2 into two pieces each, where the pieces closer to RA, RB will be just converters between the IPsec tunnels made the Juniper way and the IPIP over IPsec tunnels used in the rest of the network, and the R1-R2 link will be set up between the pieces closer to R3, R4.

A brief look into RFC4301 section 4.4.1 confirms that the requirement to drop traffic ingressing outside its proper SA is only relevant for the traffic crossing the "IPsec boundary". However, it also says that it is not mandatory to have a possibility to define individually whether a given interface serves as an IPsec boundary or not, so Mikrotik uses a single SPD. Which means it would require at least to maintain a list of interfaces on which IPsec handling should be applied (or not applied).

And we also get here to the common "we have to work more to get less money" paradox - if they implement this, the product will become more flexible (and thus competitive), but it will cause less CHR licences to be sold as the splitting of the functionality into two CHRs as suggested in the previous post will not be necessary.