We have several core routers housed in data centres in key cities. "Branch" offices (typically co-working spaces, some homes, mobile workers) .

My plan was to have all those remote locations connect via a VPN to a core router, and tunnel their traffic over that connection.

My challenge now is throughput. According to this incredibly detailed testing by Rick Frey http://rickfreyconsulting.com/mikrotik-vpns/ almost every tunneling possibility in the Mikrotik ecosystem has around 90% throughput loss.

I'm confused as to why there is such a performance hit, and what my options might be for at least improving transfer speeds.

On the link I am testing with at the moment (a 10/1 LTE link) I can only achieve about 1,000 kpbs down.

I've tried OpenVPN and SSTP and both have the same ceiling for downlink speed. IPSec and EoIP are out of the question because of the dynamic IP / NAT possibilities.

For SSTP I've ended up with max-mtu: 1460 and max-mru: 1420 with little to no improvement.

CPU / resources aren't an issue.

Where is all the throughput going? What are some suggestions as to better ways to link sites? Are there any other tweaking suggestions - or is the 90% loss just something I have to live with. Do other routers perform so badly?

Open VPN and SSTP are TCP based protocols.This limits your performance.

IPSEC (with L2TP) performs better, especially if you reduce MTU to avoid the packet reordering bug.

I think that report is not really valid. It claims a 90% loss but it starts with a very fast connection and uses a very slow router.
So, when using encryption the VPN performance is limited by the encryption load on the CPU (this router does not have HW Accel for encryption)
and the VPN has a fixed maximum speed. Then your loss is proportional to how fast the initial link was.

When your link is slower and your router is faster, the loss will be much less. On a GRE/IPsec link I observe no more loss than
can be explained by the header overhead added to each packet. This depends on your average packet size, when you send
only small packets (e.g. VoIP) the header overhead will be relatively more than when you mainly send full-size packets.

Modern MikroTik models like the RB750G r3 that have hardware accelleration for encryption are not limited the way the CRS is.
They can run a VPN tunnel at the full speed the writer of that report has. He should re-test with that router.

In your case, with LTE, it is much less of a problem because of the lower actual rate.
Another issue is with the CCR routers. They have accelleration of the encryption, but they have 9 or more cores that do it in
parallel. The result is that packets are randomly re-ordered as a result of the encryption process. This leads to problems
when the receiving system does not do a good job at resequencing the packets in its TCP implementation.
As a result, you will have slow TCP performance on Windows, while there is little influence when you use Linux.

As so many users use Windows, the only practical way for MikroTik to solve this is to implement some way of re-ordering
or at least to make sure the packets belonging to a single TCP stream are not re-ordered (e.g. by binding a stream to a
core instead of using them dynamically for all data entering the router). But this has been in the works for years.
When you are not using a CCR you are not affected by this problem.

Are you sure you can reproduce that a direct session between the public IP addresses of your LTE and central site is running
at full speed but a VPN is running at 10% of that? Or could it also be that the LTE in fact never reaches its advertised
speed when transferring data from your central site?

Are you sure you can reproduce that a direct session between the public IP addresses of your LTE and central site is running
at full speed but a VPN is running at 10% of that? Or could it also be that the LTE in fact never reaches its advertised
speed when transferring data from your central site?

I've run up a temporary box in the same data centre and tested from there. I know it's not _exactly_ the same server - but it's close enough - and given the bandwidth is so low at the remote endpoint, I think the test is valid.

I've also tried an ADSL link, and the speed ceiling remains the same at about 1000 kbps. The ADSL link achieves about 3 Mbps, so is a lot slower than the LTE link.

We are using a CCR (CCR1009-8G-1S-1S+) but we don't have any windows computers - they are all mac, centos or debian (with a couple of QNAP servers).

I'd much prefer to use IPSec with L2TP but the LTE connection introduces NAT (and is obviously not a fixed address either) - thus I only have SSTP or OpenVPN as options afaik.

L2TP/IPsec works OK over NAT but sometimes you need to relax the IPsec rules a bit e.g. when the
connection is in fact double-NAT. This happens when your LTE devices does NAT and your provider
puts you behind NAT as well.
You can do this by removing the IPsec settings in the L2TP server settings and manually setting
an IPsec Peer for the L2TP like this:

/ip ipsec peer
add enc-algorithm=aes-128 generate-policy=port-override local-address=\
11.22.33.44 passive=yes secret=yoursecretkey

Then it will work OK as long as there is only a single client behind the same public IP address.

Please do some test where the packet order is not important (e.g. using UDP) and see if the reordering
that the CCR does is a problem for you. In that case file a problem report at MikroTik, they need to
be aware that this problem is really an issue for their customers (they already are, but they don't seem
to translate that into solutions)

There really should not be any limitations at those bandwidths, people are routinely running VPNs
that achieve >50 Mbps throughput on a suitable connection.

Open VPN and SSTP are TCP based protocols.This limits your performance.

IPSEC (with L2TP) performs better, especially if you reduce MTU to avoid the packet reordering bug.

Reorder bug? what sort of MTU would you suggest? I am planing to start connecting all my data centre sites using IPSEC, GRE and OSPF.

Thanks in advance.

Also is this bug present on RB750Gr3? which is whats being used at most sites right now apart from DC's which are using a mixture of servers running softether or openswan.

You can find the information here:
http://wiki.mikrotik.com/wiki/Manual:IP ... imizations
and
http://forum.mikrotik.com/viewtopic.php?f=1&t=112545

You can find the information here:
http://wiki.mikrotik.com/wiki/Manual:IP ... imizations
and
http://forum.mikrotik.com/viewtopic.php?f=1&t=112545

Thanks very much for the information.