We have a client with 6 sites using IPsec. Every now and again, possibly once a week, sometimes once a month, IPsec just stops working randomly at one of the sites.

In order to demonstrate the symptoms of the problem I have attached a diagram. On the diagram Installed SAs tab you will notice a source IP address x.x.186.50 trying to communicate with x.x.7.3 but 0 current bytes. x.x.186.50 is the client's remote Fortigate IPsec server, and x.x.7.73 is a MikroTik based IPsec endpoint. It appears data from the remote side to us is not always flowing.

Phase 1 is always established, but Phase 2 fails randomly.

We tried various things over time, such as rebooting, setting clocks, dabbling with configuration, rechecking and rechecking configuration but it appears the problem is entirely random. And sometimes random things fixes it. At one stage I had a theory that if the tunnel is initiated from their side it works, but fiddling with "Send Initial Contact" has not made any difference.

We've had many chats to the client about this but they have many more international IPsec VPNs and only our MikroTik configuration is failing.

I have also included a log file. It's difficult to say exactly where the negotiation is failing but it loops at the "received a valid R-U-THERE, ACK sent"

Log file:

Make sure the IPSec responder has both passive=yes and send-initial-contact=no set.

That should solve your problems.

EDIT:

I changed the title of this post from "phase 2 intermittently refuses to start" to a more generic title because it appears actually Phase 2 is starting. However, traffic is not flowing from the remote endpoint to the local router as described in my first post.

We received additional information from the client that their Fortigate log shows this

Received ESP packet with unknown SPI.

Fortigate has an article in their knowledgebase explaining that this could be because the SPIs don't agree. http://kb.fortinet.com/kb/microsites/mi ... alId=11654

The workaround appears to be enabling dead peer detection on both sides, but we cannot do this, only on our side. The certainly seems plausible becauseI can see these constant "sendto information notify" messages.

Our situation is greatly compounded that 5 other sites are working and that the client's firewall is under change control. The setup also always worked for many years, so they claim it cannot be a configuration error on their side.

Make sure the IPSec responder has both passive=yes and send-initial-contact=no set.

I looked up some Fortigate documentation and I cannot find a passive setting nor send initial contact.

Which device is the initiator and which device is the responder?

What ROS version are your MikroTiks running?

The workaround appears to be enabling dead peer detection on both sides, but we cannot do this, only on our side.

DPD setting is not negotiable, so you can safely turn it on on your side only (with whatever DPD interval you feel appropriate) without risking to break the either phase negotiation. And chances are it is enabled on the other side actually.

Which device is the initiator and which device is the responder?

If send initial contact is any indicator the MikroTik is the initiator. The client first told us their Fortigate can be both initiator and responder but when we questioned them again they said there is no such setting.

What ROS version are your MikroTiks running?

We were on v6.5 but tried downgrading to v5.26. We then upgraded to v6.7 so currently it's v6.7.

DPD setting is not negotiable, so you can safely turn it on on your side only (with whatever DPD interval you feel appropriate) without risking to break the either phase negotiation. And chances are it is enabled on the other side actually.

Thanks. We've experimented with DPD trying various values. It's on both the remote side (Fortigate) and the near side (MikroTik).

I have pasted their config for your perusal. I have also started a bounty on Server Fault because we are about to loose this client.
http://serverfault.com/questions/559820 ... nknown-spi

First thing to note, phase1 lifetime settings do not match. You have 1day while they have 8 hours (28800 second). Make them match. Also make sure everything else matches as well. Set proposal check setting to strict, it will ensure everything matches (otherwise tunnel will stop working).

First thing to note, phase1 lifetime settings do not match. You have 1day while they have 8 hours (28800 second). Make them match. Also make sure everything else matches as well. Set proposal check setting to strict, it will ensure everything matches (otherwise tunnel will stop working).

Thank you. We have tried setting it to 8 hours as well but at this point of the frustration the entry was the MikroTik default of 24 hours.

So here is the solution, which again, is completely random and does not make sense. I have attached another diagram inline to show you what happened.

We fixed it by:

1. Turning off PPPoE at client.
2. Installing completely new router (Router B) and tested at Border. It worked at Border.
3. Switching off new router B at border.

AND THEN, WITHOUT MAKING A SINGLE CHANGE, the client's end-point Router A started working. So just adding a duplicate router at the border and taking this router offline again made the original router work.

So add this fix to the list of things we've done:

1. Reboot. That worked once.
2. Create new tunnel with new IP. That worked once but only once. After changing IP back client endpoint came live again.
3. Change time servers.
4. Fiddle with every possible setting.
5. Wait. Once, after a day, it just came right. This time, even after days, nothing came right.

So I postulate that there is an incompatibility on either Fortigate or MikroTik side which only happens at very random situations. The only things we haven't been able to try is upgrade firmware on Fortigate. Maybe there is hidden corrupt configuration value or timing issue invisible to configurer.

I further speculate that the issue is caused by random timing issues causing SPI mismatch. That justs happens randomly. I'll add to this post when it happens again.