I have a RB1100AHx2 which is currently terminating 250 IPSec vpn's
The problem is if the router is rebooted it gets hit with all the vpn connection attempts at the one time, the cpu maxes out and all connections seem to fail on phase 1. Occasional I got one connection up but it dident last long.
What do I need to change to fix this so I can reboot the router and the connections will all re-establish?
Re: IPSec too many connections at once
Do you have 'Send Initial Contact' enabled on all the peers in the RB1100AHx2?
Re: IPSec too many connections at once
No the RB1100AHx2 is not set to send the initial contact, it is left to the remote routers to initiate the connection when required
Re: IPSec too many connections at once
What version of ROS, although not documented, it seems some work has been done on later versions, I am currently on 5.18.
Have you turned on ipsec logning (e.g. /system logging add topic=ipsec ...), I used to, but in 5.x it has become extreamly verbose, so I use topics=ipsec,!debug now.
What DH-group do you use, the large ones are extreamly resouce intensive.
Are there any clues in the log?
Have you turned on ipsec logning (e.g. /system logging add topic=ipsec ...), I used to, but in 5.x it has become extreamly verbose, so I use topics=ipsec,!debug now.
What DH-group do you use, the large ones are extreamly resouce intensive.
Are there any clues in the log?
Re: IPSec too many connections at once
Its running the latest 5.19
I turned on the logging only after the issue happens to work out why none of the vpn's were connecting and the messages were about the phase 1 failing.
DH group is 14 (modp2048)
The problem does appear to be the router getting bogged down trying to do all the key exchanges at the one time (the client routers are too good at re-establishing the connection immediately)
To get all the VPN's back up I had to block all IPSec connections, then allow the routers a few at a time in explicit rules, this obviously does not scale well or work in an automated way.
When enabling the routers in blocks they all came up almost instantly so if there was some way to throttle the connections and only allow a few per second that may solve the issue.
I have had a look at the advanced settings in the firewall rules like the limit option but I am unsure if this would work or how to properly configure it.
I turned on the logging only after the issue happens to work out why none of the vpn's were connecting and the messages were about the phase 1 failing.
DH group is 14 (modp2048)
The problem does appear to be the router getting bogged down trying to do all the key exchanges at the one time (the client routers are too good at re-establishing the connection immediately)
To get all the VPN's back up I had to block all IPSec connections, then allow the routers a few at a time in explicit rules, this obviously does not scale well or work in an automated way.
When enabling the routers in blocks they all came up almost instantly so if there was some way to throttle the connections and only allow a few per second that may solve the issue.
I have had a look at the advanced settings in the firewall rules like the limit option but I am unsure if this would work or how to properly configure it.
Re: IPSec too many connections at once
I would contact support. 250 tunnels on a RB1100AHx2 doesn't sound unreasonable.
I'm no queue expert, but I wonder if limiting trafic to UDP/500 would be a (short term) solution, but you need someone else to help you there.
I'm no queue expert, but I wonder if limiting trafic to UDP/500 would be a (short term) solution, but you need someone else to help you there.
Re: IPSec too many connections at once
Apart from that the box should handle this by it self, this may be a work around:
be warned though, I haven't tried this my self, but it may be worth a try, or something to build on.
Code: Select all
/ip firewall filter
add action=jump chain=input dst-port=500 jump-target="IPSec sluice" protocol=\
udp
add chain="IPSec sluice" dst-limit=1,5,src-address
add action=drop chain="IPSec sluice"Re: IPSec too many connections at once
Thanks for that, I think that may be a neater solution then what I currently have.
I had played about with the limit option but found it wasent suitable as it was just letting in random packets I need all IKE packets from a host at a time, I ended up using the limit option to add the addresses to a list with a short time out then allowing the list.
I then ran in to problems that after the tunnel is established there still had to be IKE traffic so I added yet another list based on established tunnels to allow IKE.
This does work as the VPN's can drop and dont need any manual intervention to get them back up, but I still dont think its the nicest solution, the CPU still gets maxed out for a bit until the tunnel list expires and it takes 4mins to get all the tunnels back up and running.
I have also discovered that if you add, remove, enable or disable a peer all the ipsec VPN's are dropped! is this normal behaviour?
I had played about with the limit option but found it wasent suitable as it was just letting in random packets I need all IKE packets from a host at a time, I ended up using the limit option to add the addresses to a list with a short time out then allowing the list.
I then ran in to problems that after the tunnel is established there still had to be IKE traffic so I added yet another list based on established tunnels to allow IKE.
This does work as the VPN's can drop and dont need any manual intervention to get them back up, but I still dont think its the nicest solution, the CPU still gets maxed out for a bit until the tunnel list expires and it takes 4mins to get all the tunnels back up and running.
I have also discovered that if you add, remove, enable or disable a peer all the ipsec VPN's are dropped! is this normal behaviour?
Re: IPSec too many connections at once
I still urge you to write to MikroTik support, don't expect them to pick up from the forum.
The problem with all tunnels being renegociated when adding/removing peers can't I remember seeing myself, and reports about it is old (2+ years). I even tried on a 5.18 box with 449 enabled peers and currently 375 active SAs, and it nothing unusual happened (this was on a x86 box though)
The problem with all tunnels being renegociated when adding/removing peers can't I remember seeing myself, and reports about it is old (2+ years). I even tried on a 5.18 box with 449 enabled peers and currently 375 active SAs, and it nothing unusual happened (this was on a x86 box though)