I'm looking for some pointers to increase the throughput on our L2TP/IPSec connections. As we are getting more and more 1Gbps links, I'm observing that our VPN site-to-site infrastructure cannot keep up. The observations are similar to the discussion in the threads IPSEC performance problem and EOIP TCP problem.

I chose L2TP/IPSec for the following reasons:

• The routers out in the field easily connect back to our hubs which have static IPs and there are no NAT involved
• By setting up IPSec in transport mode, I'm provided interfaces that can be used by OSPF to dynamically establish routes
• It allows me to run IPv6 on our network and connect sites over the same L2TP IPv4 based links

The drawback is that our MTU setting is 1290 to handle the IPv6 traffic, but it has worked well for us since the link speeds have been in the 100M/b range. A speedtest between a CCR1009-8G-1S-1S+ and a RB4011 on a 1G link measuring 16ms latency produce these bandwidth Test results (10s Average, IPv4, ROS 6.48.1):

• WAN to WAN using UDP: 970 Mbps
• WAN to WAN using TCP: 900 Mbps
• L2TP/IPsec (transport mode, HW accelleration) using UDP: 675 Mbps
• L2TP/IPsec (transport mode, HW accelleration) using TCP: 375 Mbps

The tunnel is set up with AES-256-CBC + SHA256 using IKE2 and both routers have the H flag displayed on the "Intsalled SA" tab, so it appears that our IPSec setting is correct in utilizing hardware accelleration.

My questions are:

• There is no published IPSec data for the CCR1009-8G-1S-1S+. The RB4011s performance table is not specific to our configuration, but for a single tunnel it appears that it should handle the link. Am I correct, or do we need better hardware to be able to saturate the 1G link?
• There is a large difference between UDP and TCP using L2TP/IPSec. Why is that? The link is relative quick (latency 16ms) and there are no detected errors. According to the referenced threads, posters suggest contacting Mikrotik support. Is this speed difference a limitation of V6 that will be addressed in V7?
• Is there any tuning that can be done besides making sure that HW accelleration is active?

Hopefully my test data can help someone with comparison throughput.

It seems that all CCR1009 have same CPU built in and I would assume that they all feature same HW encryption device with same performance. So you can have a look at performance tables of some other CCR1009. Performance table for CCR1009-7G-1C-1S+ indicates that realistic max IPsec throughput for single connection would be around 500Mbps. RB4011 has quite higher performance figures in this respect.

I don't know exactly the reason for TCP being so much slower than UDP. I guess it has to do with slightly increased round-trip latency (UDP is not affected by that) and improper (too slow) TCP window scaling. If there were some errors (retransmissions) then performance would be even lower. Possibly TCP window size does not increase nearly enough to facilitate full link bandwidth: with 1Gbps link speed and 16ms RTT, TCP window size should have been 2MB while most OSes use 64k initial window size. It should increase with longer test durations though.

vikinggeek
Its known issue with Mikrotik RouterBoards: viewtopic.php?t=146665#p769858
You should contact technical support or replace hardware router to CHR.

@vikinggeek

It seems we are using quite the same network layout. It would be nice to share more experiences. We have not been able to use L2TP (IPSec or not) stable for longer than about 2 weeks uptime. We are using CCR1036 all over. Contact me if you feel alike.

@skraw: I've not had any stability problem with our L2TP connections (except 6.48, but that's another story, 6.48.1 solved it). The only challenge for me is the single stream IPSec performance. According to the CCRs' data sheets, it could be limited to 500Mbps. I'm still waiting for clarification from MT Support. I'm in Pacific Time Zone at we could chat off board to brainstorm some ideas. I've seen suggestion for various type of aggregation, but I'm not sure this will improve single stream. My use case is file transfer/NextCloud sync which may be tricked into open multiple sockets i.e. streams and thus aggregation spread performance over multiple CPU threads. According to the CCRs' data sheets most of them support more than 1Gbps aggregated bandwidth. Not sure how this would work with the OSPF routing unless we get one virtual interface exposed to it.