Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

nathan1 · Sat Sep 24, 2016 2:39 am

Fixed. See below and recent posts.
Update: 3/12/2017: 6.39rc51 is shown to have fixed the re-ordering issue. Testing is still being done to confirm performance. Please test if you can. This has been confirmed by fixed by two users. Bandwidth @ 1ms is around 700Mbit (1Gbit link) single stream TCP on MTU 1400.
Update: 2/14/2017: Still not fixed. Latest update from Alex Hart: http://forum.mikrotik.com/viewtopic.php ... 02#p583602
Update: 1/18/2017: Still not fixed. alexjhart and I have both checked our tickets with Mikrotik and received the following response:

We will continue to work on this problem when ike2 main features will be finished.

Update: 10/18/2016: Mikrotik has just released another hardware accelerated platform (MIPS based) that does NOT have a re-ordering problem: http://forum.mikrotik.com/viewtopic.php ... 59#p563380

This post *ONLY* impacts the Tile based platforms. The CCR platform is still suitable if your traffic is low enough (~<250Mbit), if you disable hardware acceleration: http://forum.mikrotik.com/viewtopic.php ... 01#p558911

-----

Since Mikrotik has been rather silent about an actual ETA about fixing this, I will create a thread to update as versions are released to notify everyone if it is fixed or not.

The IPSec hardware acceleration is useless in general, packets are randomly reordered creating disastrous effects on TCP and UDP streams. Until this is fixed, you should only use non-hardware accelerated implementations (CTR).

References:
http://forum.mikrotik.com/viewtopic.php?t=106960
http://forum.mikrotik.com/viewtopic.php ... 48#p538801
http://forum.mikrotik.com/viewtopic.php?t=84465
http://forum.mikrotik.com/viewtopic.php?t=98526
http://forum.mikrotik.com/viewtopic.php?t=106857

6.37 (NOT fixed):

# date; ping -c 10 -l 10 10.50.85.140
Fri Sep 23 19:34:11 EDT 2016
PING 10.50.85.140 (10.50.85.140) 56(84) bytes of data.
64 bytes from 10.50.85.140: icmp_seq=3 ttl=62 time=0.742 ms
64 bytes from 10.50.85.140: icmp_seq=4 ttl=62 time=0.755 ms
64 bytes from 10.50.85.140: icmp_seq=6 ttl=62 time=0.749 ms
64 bytes from 10.50.85.140: icmp_seq=8 ttl=62 time=0.742 ms
64 bytes from 10.50.85.140: icmp_seq=9 ttl=62 time=0.750 ms
64 bytes from 10.50.85.140: icmp_seq=1 ttl=62 time=0.801 ms
64 bytes from 10.50.85.140: icmp_seq=7 ttl=62 time=0.763 ms
64 bytes from 10.50.85.140: icmp_seq=10 ttl=62 time=0.753 ms
64 bytes from 10.50.85.140: icmp_seq=2 ttl=62 time=0.788 ms
64 bytes from 10.50.85.140: icmp_seq=5 ttl=62 time=0.777 ms

--- 10.50.85.140 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.742/0.762/0.801/0.019 ms, pipe 10

alexjhart · Mon Sep 26, 2016 5:58 pm

I asked for another update on 8/30/16 via email to my ticket on this issue and was basically told to stop asking for updates and just watch the changelog.

nathan1 · Mon Sep 26, 2016 9:28 pm

I asked for another update on 8/30/16 via email to my ticket on this issue and was basically told to stop asking for updates and just watch the changelog.

Yep, this is basically what I got the last time I tried to get information. Rather disappointing.

Ascendo · Wed Sep 28, 2016 11:19 am

I wonder if this is related to the issue we are having with the RB1100AHx2: after some time (1-2days) encryption drops from 500+mbps to around 20mbps. Our solution is to disable/enable the IPSec encrypted EoIP tunnel with the scheduler.

We gave up with support on this as our workaround only results in 1-2 dropped pings every 12 hours and it's only replication traffic being affected.

mikruser · Fri Sep 30, 2016 3:15 pm

In my opinion it will never be fixed.
This is a problem in hardware (tilera).

pe1chl · Fri Sep 30, 2016 3:31 pm

Fixing it requires software handling to separate the traffic into streams, like with fair queuing.
A hash is computed from each packet header (at least source and destination IP, additionally maybe port numbers or other traffic keys).
Encryption is queued to the hardware in such a way that either a single stream is queued to a single processor, or the requests
are assigned sequence numbers and the resulting output is queued and resequenced before being sent on the network.

It is likely to cause a drop in performance, especially in benchmark tests (where the "test traffic" may be identified as a single stream
and thus processed serially rather than being distributed over the processors). So probably there will be whiners who observe a low
performance in their tests, while the performance is OK in normal use.

It will not be a simple fix, I guess... both technically and politically.

alexjhart · Fri Sep 30, 2016 6:08 pm

In my opinion it will never be fixed.
This is a problem in hardware (tilera).

Mikrotik has already agreed to fix it and have discussed possible solutions with me. I don't think this is an if, but when. Unfortunately, they won't commit to a timeline, so I'm not sure when it will be.

nathan1 · Mon Oct 03, 2016 11:55 pm

Even if RouterOS somehow forced all of the packets for a single IPSec session (not per inner flow) to hit a single core so they remain ordered then the performance would still be better than the software encryption workaround, at least in many use cases. I'd even be happy to designate a single core for all of my IPSec sessions.

Can anyone from Mikrotik chime in on why hardware IPSec acceleration is even advertised? I can't think of a single real world network that can run it without strange problems. It seems like it should be stricken from the feature set entirely right now.

Are any end users actually using it and getting expected behaviors?

pe1chl · Tue Oct 04, 2016 11:10 am

I can't think of a single real world network that can run it without strange problems.

I think it is mostly a Windows problem. I run IPsec sessions with that behaviour between Linux systems and between Windows and Linux
with the data mainly flowing from Windows to Linux, and they do not appear to be affected too much. Apparently the TCP implementation
in Linux is much better than in Windows.

Are any end users actually using it and getting expected behaviors?

There is something else: I use L2TP/IPsec. When I use it between CCR and RB951G I get the nonsequenced behaviour
when running your test, but when I have a connection btween CCR and RB2011 it behaves nicely. I have configured compression
at the L2TP (PPP) level on all boxes, but it appears it is only successfully negotiated between RB2011 and CCR, and not
between RB951G and CCR, with the same configuration. It is not clear to me why that is. A difference is that my RB951G boxes
are behind NAT (so IPsec with NAT-T) and my RB2011 is not. I don't think that should affect the PPP layer, though.
Anyway, when compression is active at the PPP layer it resequences the packets much like I described before as a requirement
at the IPsec layer when multicore hardware encryption is done.

mikruser · Tue Oct 04, 2016 6:41 pm

In my opinion it will never be fixed.
This is a problem in hardware (tilera).
Mikrotik has already agreed to fix it and have discussed possible solutions with me. I don't think this is an if, but when. Unfortunately, they won't commit to a timeline, so I'm not sure when it will be.

Look at the date of the message http://forum.mikrotik.com/viewtopic.php?t=84465
Its "Apr 28, 2014". Mikrotik team can not fix it within a few years. I no longer believe in their promises.

mikruser · Tue Oct 04, 2016 6:48 pm

Even if RouterOS somehow forced all of the packets for a single IPSec session (not per inner flow) to hit a single core so they remain ordered then the performance would still be better than the software encryption workaround, at least in many use cases. I'd even be happy to designate a single core for all of my IPSec sessions.

Can anyone from Mikrotik chime in on why hardware IPSec acceleration is even advertised? I can't think of a single real world network that can run it without strange problems. It seems like it should be stricken from the feature set entirely right now.

Are any end users actually using it and getting expected behaviors?

I monitored CPU usage on CCR1009 when copying a file via L2TP/IPsec tunnel (100Mbit WAN):

ccr_ipsec.png

very strange - why CCR uses 3 cores with very low usage???
copying speed very slow ~30...35 Mbit/s

nathan1 · Tue Oct 04, 2016 6:58 pm

Even if RouterOS somehow forced all of the packets for a single IPSec session (not per inner flow) to hit a single core so they remain ordered then the performance would still be better than the software encryption workaround, at least in many use cases. I'd even be happy to designate a single core for all of my IPSec sessions.

Can anyone from Mikrotik chime in on why hardware IPSec acceleration is even advertised? I can't think of a single real world network that can run it without strange problems. It seems like it should be stricken from the feature set entirely right now.

Are any end users actually using it and getting expected behaviors?
I monitored CPU usage on CCR1009 when copying a file via L2TP/IPsec tunnel:
ccr_ipsec.png
very strange - why CCR uses 3 cores with very low usage???

Is your L2TP/IPSec tunnel is using hardware accelerated crypto? If so, then it makes sense. The Mikrotik is load balancing on a per packet basis, which effectively will distribute the load randomly across some set of cores. You would see a behavior that looks like this - this is exactly what creates the reordering.

mikruser · Tue Oct 04, 2016 7:19 pm

>>Is your L2TP/IPSec tunnel is using hardware accelerated crypto?
Yes

>>The Mikrotik is load balancing on a per packet basis, which effectively will distribute the load randomly across some set of cores.
Ok, how can i change this to per-connection basis? One core should be enough to handle 100Mbit/s

nathan1 · Tue Oct 04, 2016 7:33 pm

>>Is your L2TP/IPSec tunnel is using hardware accelerated crypto?
Yes

>>The Mikrotik is load balancing on a per packet basis, which effectively will distribute the load randomly across some set of cores.
Ok, how can i change this to per-connection basis? One core should be enough to handle 100Mbit/s

You can't. This is entirely the problem we are talking about here. Switch your settings to non-hardware accelerated crypto (like aes-128-ctr), a CCR1009 can handle over 200Mbit with software encryption.

alexjhart · Tue Oct 04, 2016 9:00 pm

I have given Mikrotik evidence that router-to-router-ipsec-with-hardware-encryption traffic is received out-of-order, therefore affecting the encapsulated traffic, and they have confirmed this is a problem. It does affect throughput. How much depends on latency, protocol, etc. SMB with Windows is one that is greatly impacted but the traffic being out of order. They know what the problem is and how they are going to fix. Mikrotik staff have confirmed with me multiple times in person and via email. Unfortunately, they just haven't been able to commit to a timeline on getting the new driver pushed out. The new driver will fix the quality issue by having packets arrive at the far router in order. I believe they will do this by forcing the ipsec connection to use a single core, rather than sending packets to different cores and have them processed at different times (out of order). My work around, which people like nathan1 also use, is to force software encryption. While that means less overall throughput with the tunnel, the user's application is actually able to push a lot more traffic because of better quality tunneling (for example 150Mbps vs 5-30Mbps using the same test and only changing hardware vs software ipsec encryption driver on routers). I would rather have users get a better overall experience than just get a good reading on speedtest.net. Who cares about speedtest.net if they have slow data transfers on applications they actually use.

pe1chl · Tue Oct 04, 2016 9:12 pm

SMB with Windows is one that is greatly impacted but the traffic being out of order.

I wonder why you reported the problem to MikroTik and expect them to fix it, instead of to Microsoft who are the actual owner of the problem?
After all, IP specifies an unsequenced unreliable datagram delivery, and routers are free to re-order packets.
For example, you could have 10 connections between two locations and routers could distribute the traffic over those 10 connections
without obligation to resequence them. It is the task of the end system to do that, and apparently Microsoft is failing at that job.

See https://en.wikipedia.org/wiki/Internet_Protocol under the Reliability header, heck even Microsoft writes this on
Technet: https://technet.microsoft.com/nl-nl/lib ... s.10).aspx under Network Interface layer.

nathan1 · Tue Oct 04, 2016 10:19 pm

SMB with Windows is one that is greatly impacted but the traffic being out of order.
I wonder why you reported the problem to MikroTik and expect them to fix it, instead of to Microsoft who are the actual owner of the problem?
After all, IP specifies an unsequenced unreliable datagram delivery, and routers are free to re-order packets.
For example, you could have 10 connections between two locations and routers could distribute the traffic over those 10 connections
without obligation to resequence them. It is the task of the end system to do that, and apparently Microsoft is failing at that job.

See https://en.wikipedia.org/wiki/Internet_Protocol under the Reliability header, heck even Microsoft writes this on
Technet: https://technet.microsoft.com/nl-nl/lib ... s.10).aspx under Network Interface layer.

Wait a minute here, packets are not guaranteed to be in order, but in general, they are and just about every method of bonding/load sharing does hashing in order to maintain this. Doing per packet load balancing is an absolute disaster for UDP flows. While the general TCP settings for Linux as defaulted and tuned by major distributions is favorable to some amount of re-ordering, it is not impervious to be impacted. You will see more re-transmissions on a single TCP stream over a single EoIP/IPSec session between two physically connected CCR1009s that are using hardware encryption vs. one that isn't. This is without a doubt a Mikrotik problem. I agree that it feels like they will never fix it but I'm hoping we are wrong about that.

Per packet load balancing is often a requested feature on platforms - for very specific applications. Mikrotik is the first vendor that I have ever seen to ship it as "default" and with no way to even disable it. Normally you would expect an L2 or an L3+L4 hash and something like per packet would be very explicitly configured.

http://www.cisco.com/c/en/us/td/docs/io ... /pplb.html
https://www.juniper.net/documentation/e ... rview.html

roadracer96 · Wed Oct 05, 2016 2:59 am

The problem with misordered packets is that the receiving side has to wait for everything to come in so it can reorder them and feed them to the application. This results in the connection slowing down and scaling the window size smaller in order to guarantee delivery. You could tweak the settings in Linux to make it fast but that isn't real world.

Between waiting for bgp and IPv6 problems to be resolved and IPSec promises to be made good on (after being told repeatedly that "it works fine"), we've pulled most of the mikrotiks out of our environment. Sure. It cost us more money but the hardware we put in worked out of box without having to beg.

alexjhart · Wed Oct 05, 2016 3:37 am

The problem with misordered packets is that the receiving side has to wait for everything to come in so it can reorder them and feed them to the application. This results in the connection slowing down and scaling the window size smaller in order to guarantee delivery. You could tweak the settings in Linux to make it fast but that isn't real world.

Between waiting for bgp and IPv6 problems to be resolved and IPSec promises to be made good on (after being told repeatedly that "it works fine"), we've pulled most of the mikrotiks out of our environment. Sure. It cost us more money but the hardware we put in worked out of box without having to beg.

What hardware are you replacing with for IPSec?

nathan1 · Wed Oct 05, 2016 7:29 pm

The problem with misordered packets is that the receiving side has to wait for everything to come in so it can reorder them and feed them to the application. This results in the connection slowing down and scaling the window size smaller in order to guarantee delivery. You could tweak the settings in Linux to make it fast but that isn't real world.

Between waiting for bgp and IPv6 problems to be resolved and IPSec promises to be made good on (after being told repeatedly that "it works fine"), we've pulled most of the mikrotiks out of our environment. Sure. It cost us more money but the hardware we put in worked out of box without having to beg.
What hardware are you replacing with for IPSec?

I'm curious as well. I've considered Vyos on a SuperMicro SC515 or similar to replace my CCRs but so far I haven't pulled the trigger.

roadracer96 · Sat Oct 08, 2016 2:20 am

Brocade MLXs. Having 400gbit of IPSec throughout and 2.4million hardware route scale is just a happy side effect. The reality is. We collapsed a lot of devices into 1 per building (9 total MLXs). The internet facing units have the newer 20 port 10gig IPSec enabled line cards.

nathan1 · Sat Oct 08, 2016 3:32 am

Brocade MLXs. Having 400gbit of IPSec throughout and 2.4million hardware route scale is just a happy side effect. The reality is. We collapsed a lot of devices into 1 per building (9 total MLXs). The internet facing units have the newer 20 port 10gig IPSec enabled line cards.

Very nice looking units but you definitely need a bit more physical space. I'm hoping to find a 1U solution to possibly the replace my CCRs at some point.

roadracer96 · Sun Oct 09, 2016 5:10 pm

Maybe that look at all he brocade 7450s. They have an available IPSec module. Now. Keep in mind this isn't really useful for road warrior. It's only designed for router to router links.

nathan1 · Sun Oct 09, 2016 5:39 pm

I only use the CCRs for site to site IPSec so the 7450 looks like a solid alternative. Thanks for the pointer.

nathan1 · Mon Oct 17, 2016 6:01 pm

So the hex r3 was just released and supposedly supports hardware encryption on a MIPS chip. Has anyone found out if this platform maintains order? It would be amusing if they do which would mean these $60 units outperform our $500+ units.

http://www.mikrotik.com/download/share/hexr3.pdf
http://forum.mikrotik.com/viewtopic.php?t=113068

alexjhart · Mon Oct 17, 2016 6:06 pm

So the hex r3 was just released and supposedly supports hardware encryption on a MIPS chip. Has anyone found out if this platform maintains order? It would be amusing if they do which would mean these $60 units outperform our $500+ units.

http://www.mikrotik.com/download/share/hexr3.pdf
http://forum.mikrotik.com/viewtopic.php?t=113068

The 3011 hardware features encryption in hardware too, but their driver doesn't support that device yet. I'd be surprised if the hex r3 already had software support for the hardware encryption.

mrz · Mon Oct 17, 2016 6:16 pm

Yes, it does. It was already announced that hex 3 has HW support.

nathan1 · Mon Oct 17, 2016 6:17 pm

Yes, it does. It was already announced that hex 3 has HW support.

mrz, can you speak to the problem we are all fighting here? Does the hex3 maintain packet order per-flow?

alexjhart · Mon Oct 17, 2016 6:19 pm

Yes, it does. It was already announced that hex 3 has HW support.

Hey, Mikrotik finally joined the thread.

Does routeros have support for the hardware support the hex 3 offers, or is it like the 3011 that doesn't yet?

nathan1 · Mon Oct 17, 2016 6:21 pm

Yes, it does. It was already announced that hex 3 has HW support.
Hey, Mikrotik finally joined the thread.

Does routeros have support for the hardware support the hex 3 offers, or is it like the 3011 that doesn't yet?

Hey Alex, did you take a look at http://www.mikrotik.com/download/share/hexr3.pdf ? It seems to suggest (and I think mrz just confirmed) end to end support for HW accel.

alexjhart · Mon Oct 17, 2016 6:33 pm

Yes, it does. It was already announced that hex 3 has HW support.
mrz, can you speak to the problem we are all fighting here? Does the hex3 maintain packet order per-flow?

I would love to know this too.

mrz · Mon Oct 17, 2016 6:44 pm

Hex v3 is not affected by reordering problem you see on CCRs.

Fraction · Mon Oct 17, 2016 6:56 pm

Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Not 100% sure what nathan1 meant by adding ping output to first post, but this ping is going aes-128-cbc eoip tunnel (hex3 - hap lite):
64 bytes from 192.168.50.1: seq=0 ttl=63 time=16.875 ms
64 bytes from 192.168.50.1: seq=1 ttl=63 time=15.062 ms
64 bytes from 192.168.50.1: seq=2 ttl=63 time=15.640 ms
64 bytes from 192.168.50.1: seq=3 ttl=63 time=15.546 ms
64 bytes from 192.168.50.1: seq=4 ttl=63 time=15.652 ms
64 bytes from 192.168.50.1: seq=5 ttl=63 time=15.066 ms
64 bytes from 192.168.50.1: seq=6 ttl=63 time=14.897 ms
64 bytes from 192.168.50.1: seq=7 ttl=63 time=15.763 ms
64 bytes from 192.168.50.1: seq=8 ttl=63 time=15.866 ms
64 bytes from 192.168.50.1: seq=9 ttl=63 time=15.708 ms
64 bytes from 192.168.50.1: seq=10 ttl=63 time=15.107 ms
64 bytes from 192.168.50.1: seq=11 ttl=63 time=15.716 ms
64 bytes from 192.168.50.1: seq=12 ttl=63 time=14.844 ms
64 bytes from 192.168.50.1: seq=13 ttl=63 time=15.183 ms
64 bytes from 192.168.50.1: seq=14 ttl=63 time=15.769 ms
64 bytes from 192.168.50.1: seq=15 ttl=63 time=16.155 ms
64 bytes from 192.168.50.1: seq=16 ttl=63 time=15.043 ms
64 bytes from 192.168.50.1: seq=17 ttl=63 time=15.379 ms
64 bytes from 192.168.50.1: seq=18 ttl=63 time=15.769 ms
64 bytes from 192.168.50.1: seq=19 ttl=63 time=15.172 ms
64 bytes from 192.168.50.1: seq=20 ttl=63 time=15.759 ms

It just hap lite and 10Mbps ADSL on the other side, so can't say anything about performance or anything, but so far it seems to be working ok.

alexjhart · Mon Oct 17, 2016 7:00 pm

Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)
Not 100% sure what nathan1 meant by adding ping output to first post, but this ping is going aes-128-cbc eoip tunnel (hex3 - hap lite):
64 bytes from 192.168.50.1: seq=0 ttl=63 time=16.875 ms
64 bytes from 192.168.50.1: seq=1 ttl=63 time=15.062 ms
64 bytes from 192.168.50.1: seq=2 ttl=63 time=15.640 ms
64 bytes from 192.168.50.1: seq=3 ttl=63 time=15.546 ms
64 bytes from 192.168.50.1: seq=4 ttl=63 time=15.652 ms
64 bytes from 192.168.50.1: seq=5 ttl=63 time=15.066 ms
64 bytes from 192.168.50.1: seq=6 ttl=63 time=14.897 ms
64 bytes from 192.168.50.1: seq=7 ttl=63 time=15.763 ms
64 bytes from 192.168.50.1: seq=8 ttl=63 time=15.866 ms
64 bytes from 192.168.50.1: seq=9 ttl=63 time=15.708 ms
64 bytes from 192.168.50.1: seq=10 ttl=63 time=15.107 ms
64 bytes from 192.168.50.1: seq=11 ttl=63 time=15.716 ms
64 bytes from 192.168.50.1: seq=12 ttl=63 time=14.844 ms
64 bytes from 192.168.50.1: seq=13 ttl=63 time=15.183 ms
64 bytes from 192.168.50.1: seq=14 ttl=63 time=15.769 ms
64 bytes from 192.168.50.1: seq=15 ttl=63 time=16.155 ms
64 bytes from 192.168.50.1: seq=16 ttl=63 time=15.043 ms
64 bytes from 192.168.50.1: seq=17 ttl=63 time=15.379 ms
64 bytes from 192.168.50.1: seq=18 ttl=63 time=15.769 ms
64 bytes from 192.168.50.1: seq=19 ttl=63 time=15.172 ms
64 bytes from 192.168.50.1: seq=20 ttl=63 time=15.759 ms

It just hap lite and 10Mbps ADSL on the other side, so can't say anything about performance or anything, but so far it seems to be working ok.

hap lite isn't the new hex 3 (mmips). Your test is showing software performance, not hardware encryption, which this thread talks about.

Fraction · Mon Oct 17, 2016 7:11 pm

Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)
Not 100% sure what nathan1 meant by adding ping output to first post, but this ping is going aes-128-cbc eoip tunnel (hex3 - hap lite):
64 bytes from 192.168.50.1: seq=0 ttl=63 time=16.875 ms
64 bytes from 192.168.50.1: seq=1 ttl=63 time=15.062 ms
64 bytes from 192.168.50.1: seq=2 ttl=63 time=15.640 ms
64 bytes from 192.168.50.1: seq=3 ttl=63 time=15.546 ms
64 bytes from 192.168.50.1: seq=4 ttl=63 time=15.652 ms
64 bytes from 192.168.50.1: seq=5 ttl=63 time=15.066 ms
64 bytes from 192.168.50.1: seq=6 ttl=63 time=14.897 ms
64 bytes from 192.168.50.1: seq=7 ttl=63 time=15.763 ms
64 bytes from 192.168.50.1: seq=8 ttl=63 time=15.866 ms
64 bytes from 192.168.50.1: seq=9 ttl=63 time=15.708 ms
64 bytes from 192.168.50.1: seq=10 ttl=63 time=15.107 ms
64 bytes from 192.168.50.1: seq=11 ttl=63 time=15.716 ms
64 bytes from 192.168.50.1: seq=12 ttl=63 time=14.844 ms
64 bytes from 192.168.50.1: seq=13 ttl=63 time=15.183 ms
64 bytes from 192.168.50.1: seq=14 ttl=63 time=15.769 ms
64 bytes from 192.168.50.1: seq=15 ttl=63 time=16.155 ms
64 bytes from 192.168.50.1: seq=16 ttl=63 time=15.043 ms
64 bytes from 192.168.50.1: seq=17 ttl=63 time=15.379 ms
64 bytes from 192.168.50.1: seq=18 ttl=63 time=15.769 ms
64 bytes from 192.168.50.1: seq=19 ttl=63 time=15.172 ms
64 bytes from 192.168.50.1: seq=20 ttl=63 time=15.759 ms

It just hap lite and 10Mbps ADSL on the other side, so can't say anything about performance or anything, but so far it seems to be working ok.

hap lite isn't the new hex 3 (mmips). Your test is showing software performance, not hardware encryption, which this thread talks about.

There is hap lite in the other side of the tunnel, but I have new hex3 on my side and because the encryption method is aes-128-cbc, I presume it is using hw encryption..

nathan1 · Mon Oct 17, 2016 7:39 pm

Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)
Not 100% sure what nathan1 meant by adding ping output to first post, but this ping is going aes-128-cbc eoip tunnel (hex3 - hap lite):
64 bytes from 192.168.50.1: seq=0 ttl=63 time=16.875 ms
64 bytes from 192.168.50.1: seq=1 ttl=63 time=15.062 ms
..

It is important to do the ping with -l 10 to preload a burst. The default timing of ping will not allow you to see a re-order unless your termination devices are heavily loaded. Note the sequence numbers in my first post vs. yours - yours looks good, mine are re-ordered.

Fraction · Mon Oct 17, 2016 10:25 pm

Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)
Not 100% sure what nathan1 meant by adding ping output to first post, but this ping is going aes-128-cbc eoip tunnel (hex3 - hap lite):
64 bytes from 192.168.50.1: seq=0 ttl=63 time=16.875 ms
64 bytes from 192.168.50.1: seq=1 ttl=63 time=15.062 ms
..
It is important to do the ping with -l 10 to preload a burst. The default timing of ping will not allow you to see a re-order unless your termination devices are heavily loaded. Note the sequence numbers in my first post vs. yours - yours looks good, mine are re-ordered.

Yes, I noticed the difference, thats why I posted it.

mrz said that everything should be ok with this, and it looked better than yours.

But ok, I "forgot" to use preload so here is the new one, different destination address, but still using the same tunnel:

sudo ping -c 10 -l 10 192.168.110.1
PING 192.168.110.1 (192.168.110.1) 56(84) bytes of data.
64 bytes from 192.168.110.1: icmp_seq=1 ttl=62 time=15.9 ms
64 bytes from 192.168.110.1: icmp_seq=2 ttl=62 time=17.9 ms
64 bytes from 192.168.110.1: icmp_seq=3 ttl=62 time=19.7 ms
64 bytes from 192.168.110.1: icmp_seq=4 ttl=62 time=22.0 ms
64 bytes from 192.168.110.1: icmp_seq=5 ttl=62 time=23.9 ms
64 bytes from 192.168.110.1: icmp_seq=6 ttl=62 time=26.4 ms
64 bytes from 192.168.110.1: icmp_seq=7 ttl=62 time=28.5 ms
64 bytes from 192.168.110.1: icmp_seq=8 ttl=62 time=30.5 ms
64 bytes from 192.168.110.1: icmp_seq=9 ttl=62 time=32.6 ms
64 bytes from 192.168.110.1: icmp_seq=10 ttl=62 time=34.7 ms

--- 192.168.110.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 15.929/25.261/34.718/6.060 ms, pipe 10

nathan1 · Tue Oct 18, 2016 4:46 am

Yes, I noticed the difference, thats why I posted it. mrz said that everything should be ok with this, and it looked better than yours.
But ok, I "forgot" to use preload so here is the new one, different destination address, but still using the same tunnel:

sudo ping -c 10 -l 10 192.168.110.1
PING 192.168.110.1 (192.168.110.1) 56(84) bytes of data.
64 bytes from 192.168.110.1: icmp_seq=1 ttl=62 time=15.9 ms
64 bytes from 192.168.110.1: icmp_seq=2 ttl=62 time=17.9 ms
64 bytes from 192.168.110.1: icmp_seq=3 ttl=62 time=19.7 ms
64 bytes from 192.168.110.1: icmp_seq=4 ttl=62 time=22.0 ms
64 bytes from 192.168.110.1: icmp_seq=5 ttl=62 time=23.9 ms
64 bytes from 192.168.110.1: icmp_seq=6 ttl=62 time=26.4 ms
64 bytes from 192.168.110.1: icmp_seq=7 ttl=62 time=28.5 ms
64 bytes from 192.168.110.1: icmp_seq=8 ttl=62 time=30.5 ms
64 bytes from 192.168.110.1: icmp_seq=9 ttl=62 time=32.6 ms
64 bytes from 192.168.110.1: icmp_seq=10 ttl=62 time=34.7 ms

--- 192.168.110.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 15.929/25.261/34.718/6.060 ms, pipe 10

Yep, looks good. Now I just need to replace my 24 CCR1009-8G-1S-1S+ with these, a bargain at $1440 for all of them. The form factor isn't quite right for me and I need SFP+ so that isn't quite a reality. I'm glad to see this piece of hardware is being advertised with accurate and useful hardware acceleration though.

Mikrotik, can we be next? Disable all but one of my Tile cores for IPSec, to force serialization. I will be better off than I am right now, I'd take this as a real stop-gap fix. Really

Jeroen1000 · Tue Oct 18, 2016 12:19 pm

Does anyone know whether this occurs with regular TCP/UDP streams too (so without HW encryption)? Secondly, is SSTP working ok or is that HW accelerated too?
Bit of a shocker this thread:-)

nathan1 · Tue Oct 18, 2016 2:43 pm

Does anyone know whether this occurs with regular TCP/UDP streams too (so without HW encryption)? Secondly, is SSTP working ok or is that HW accelerated too?
Bit of a shocker this thread:-)

Yes, it impacts all traffic. Note that you can use IPSec ciphers/hashes that will not use hardware encryption - allowing the platform to still work but slower (MUCH slower). This only affects Tile as far as we know (CCR*).

alexjhart · Tue Oct 18, 2016 6:06 pm

Does anyone know whether this occurs with regular TCP/UDP streams too (so without HW encryption)? Secondly, is SSTP working ok or is that HW accelerated too?
Bit of a shocker this thread:-)
Yes, it impacts all traffic. Note that you can use IPSec ciphers/hashes that will not use hardware encryption - allowing the platform to still work but slower (MUCH slower). This only affects Tile as far as we know (CCR*).

To clarify, this affects all types of traffic that is being encrypted by hardware offloaded encryption on tile chip. So if you are using ccr without encryption, that traffic isn't affected, (for example, http tcp without encryption is fine). That's why Nathan1 and I are forcing aes-ctr, which isn't supported by hardware acceleration and therefore software driven and not affected.

As for sstp, I don't believe it uses hardware acceleration, but could be wrong. If it doesn't, it won't be affected. I found one mum presentation that said it wasn't. I recall that tilera supports aes-cbc for hardware encryption. If you pass traffic and watch the cpu, you'll know pretty quick of it is using hardware acceleration (cpu utilization isn't affected nearly as much).

Hope that helps

Jeroen1000 · Wed Oct 19, 2016 11:09 am

Thanks Alex and Nathan. Since I don't need more than 100 megabit CTR is ok. It's considered safe so there is no security trade off at least. Thanks for maintaining this thread pushing for a fix!

sallen · Thu Oct 20, 2016 11:17 pm

Can someone tell me if this reasoning is correct? Even though the new hex r3 can support HW accelerated encryption without these reordering issues, it seems that there is no way to use one of those successfully on a site-to-site VPN with a CCR one one endpoint. As I understand it, the only way to force software encryption on the CCR is to use CTR mode, and the only way for hex r3 to use HW encryption is to use CBC mode. So therefore you are stuck, and there is no way to get decent speeds (for Windows SMB TCP traffic) between the two in a site-to-site IPSEC configuration.

I understand that Mikrotik says the ordering problem is being fixed (but when? ROS v7?). But can we temporarily get an option in the v6.x version to disable HW acceleration on the CCR platform, so that we can do software CBC on the CCR and hardware CBC on the hex 3r?

nathan1 · Thu Oct 20, 2016 11:28 pm

Can someone tell me if this reasoning is correct? Even though the new hex r3 can support HW accelerated encryption without these reordering issues, it seems that there is no way to use one of those successfully on a site-to-site VPN with a CCR one one endpoint. As I understand it, the only way to force software encryption on the CCR is to use CTR mode, and the only way for hex r3 to use HW encryption is to use CBC mode. So therefore you are stuck, and there is no way to get decent speeds (for Windows SMB TCP traffic) between the two in a site-to-site IPSEC configuration.
...

Wow, if that is true, you have yourself in an even tighter jam than Alex and I are in. Maybe Fraction can chime in, I believe he was running this exact setup that you describe but he was peaking at 100Mbit.

Edit...thinking about this some more. You are definitely right if that is true, thats a big problem. Fraction, have you tried pushing ~100Mbit with that hex setup setup? According to to the hex r3 PDF (http://www.mikrotik.com/download/share/hexr3.pdf) , it seems like you are only going to manage around 40Mbit. Do you have some alternate config that seems to be working well?

alexjhart · Mon Nov 07, 2016 8:23 pm

http://forum.mikrotik.com/viewtopic.php ... 26#p566410

nathan1 · Mon Nov 07, 2016 8:39 pm

http://forum.mikrotik.com/viewtopic.php ... 26#p566410

Have you picked up on something that may offer us relief or just another reference point?
Do you feel like they are trolling us? Their newsletter specifically states:

http://download2.mikrotik.com/news/news_73.pdf
To get the best performance two simple rules must be taken into account:
• avoid packet fragmentation;
• there must be no packet reordering.

And then one of their flagship platforms (CCR) that they have advertised as capable of high performance IPSec does exactly that (tons of reordering).

alexjhart · Mon Nov 07, 2016 10:33 pm

http://forum.mikrotik.com/viewtopic.php ... 26#p566410
Have you picked up on something that may offer us relief or just another reference point?

Just another reference point. I felt the same way when I saw the notes in the newsletter. I am somewhat comforted that they understand the important of the CCR IPsec driver issue.

Pengu1n · Tue Nov 22, 2016 5:40 pm

Got 1 hEX r3 for backup link. Connected it to RB2011 with IPSEC (aes128cbc-sha1-dh2). Many packet losses. Firmware 6.37.1. Today upgraded to 6.37.2. Same result.
Changing proposal to aes128ctr solves problem but IPSEC performance decreases: CPU load about 25-30% with 25mbps.
No such problem with RB1100AHx2 v6.31. hEX is going to table to wait for new firmware (and new bugs).
Thank you Mikrotik.

mrz · Fri Nov 25, 2016 1:40 pm

try to switch from sha1 to md5.

pe1chl · Fri Nov 25, 2016 2:49 pm

I understand that Mikrotik says the ordering problem is being fixed (but when? ROS v7?). But can we temporarily get an option in the v6.x version to disable HW acceleration on the CCR platform, so that we can do software CBC on the CCR and hardware CBC on the hex 3r?

I vote for that as well! Preferably settable at the policy level (so you can do hardware accel for some and don't do it for others).
I now have several VPN (L2TP/Ipsec) clients that are Linux-based printers that have no issue at all with reordering, and one
GRE//IPsec VPN to a hEX r3 on a LAN with Windows workstations where the reordering is a problem. I would like to disable
hardware accel on that statically defined policy but keep it on the other ones. However, disable hardware accel for all AES-CBC
on the CCR would be a first step.

mikruser · Thu Dec 01, 2016 12:03 pm

I understand that Mikrotik says the ordering problem is being fixed (but when? ROS v7?). But can we temporarily get an option in the v6.x version to disable HW acceleration on the CCR platform, so that we can do software CBC on the CCR and hardware CBC on the hex 3r?
I vote for that as well!

it has already been proposed: http://forum.mikrotik.com/viewtopic.php?f=1&t=113911

mikruser · Thu Dec 01, 2016 12:06 pm

Why reordering issue occurs with hardware multicore, but not occurs with software multicore?

th0massin0 · Thu Dec 01, 2016 3:59 pm

It may be sticked to architecture (of CPU).

alexjhart · Thu Dec 01, 2016 9:56 pm

My latest update from support on this (from yesterday) is:
"We are working on the ipsec problem right now."

I'm not sure what that means for timeline, but it does show they are giving attention to this issue that I brought up with them about 1 year ago now.

nathan1 · Fri Dec 02, 2016 6:09 am

My latest update from support on this (from yesterday) is:
"We are working on the ipsec problem right now."

I'm not sure what that means for timeline, but it does show they are giving attention to this issue that I brought up with them about 1 year ago now.

Have you been poking them via email or did they reach out to you as an update? Crossing my fingers...

alexjhart · Wed Dec 14, 2016 6:35 pm

Just poking via ticket email.

pchott · Tue Jan 03, 2017 11:05 am

Just poking via ticket email.

Any news on this poking??

alexjhart · Tue Jan 03, 2017 10:59 pm

Just poking via ticket email.
Any news on this poking??

I just replied to my ticket again for another update. I haven't heard from them in over a month. Maybe 6.39 will be the lucky release?

alexjhart · Wed Jan 04, 2017 8:37 pm

Just poking via ticket email.
Any news on this poking??
I just replied to my ticket again for another update. I haven't heard from them in over a month. Maybe 6.39 will be the lucky release?

Support said:

We are still working on ike2, right after this we will continue to work on reordering problem.

If there last update was true, it seems like they must have taken time off to work on ike2 instead for a bit.

nathan1 · Thu Jan 19, 2017 3:13 am

My ticket on this as of 1/18/2017:

We will continue to work on this problem when ike2 main features will be finished.

The wait goes on.

alexjhart · Thu Jan 19, 2017 3:35 am

My ticket on this as of 1/18/2017:
We will continue to work on this problem when ike2 main features will be finished.
The wait goes on.

Thanks for joining in the struggle to get this fixed. Don't forget to update your starting post in the thread.

alexjhart · Tue Feb 14, 2017 7:02 pm

My last update says they're still working on ike2, but they will try to fix this in "one of the next versions". Unfortunately, the fix will be for lower core routers (CCR1009 and CCR1016) first. "Because solution for these routers are almost ready. For others, not yet." Hopefully support for the more expensive 1036 and 1072 will follow shortly after. We'll see. I'm not sure what the technical explanation is for not being able to do all at once.

Ascendo · Fri Feb 17, 2017 8:32 am

We've been struggling to get GCM (i.e. software only encryption) working between our CCR1009s and Cisco ASAs. As such I'm in a real pickle having recommended Mikrotik.

We'll take any kind of workaround right now (e.g. run CCR1009 as single core router) until there is a proper fix.

For now, RB1100AHx2 is still the best router Mikrotik makes. Going down the Tilera path has been a mistake.

nathan1 · Fri Feb 17, 2017 5:20 pm

We've been struggling to get GCM (i.e. software only encryption) working between our CCR1009s and Cisco ASAs. As such I'm in a real pickle having recommended Mikrotik.

We'll take any kind of workaround right now (e.g. run CCR1009 as single core router) until there is a proper fix.

For now, RB1100AHx2 is still the best router Mikrotik makes. Going down the Tilera path has been a mistake.

I've tried to suggest the single core workaround without success. Try contacting support, a little more demand certainly can't hurt. What kind of throughput are you able to get out of IPSec on a RB1100AHx2?

Ascendo · Fri Feb 17, 2017 6:34 pm

We've been struggling to get GCM (i.e. software only encryption) working between our CCR1009s and Cisco ASAs. As such I'm in a real pickle having recommended Mikrotik.

We'll take any kind of workaround right now (e.g. run CCR1009 as single core router) until there is a proper fix.

For now, RB1100AHx2 is still the best router Mikrotik makes. Going down the Tilera path has been a mistake.
I've tried to suggest the single core workaround without success. Try contacting support, a little more demand certainly can't hurt. What kind of throughput are you able to get out of IPSec on a RB1100AHx2?

We get around 750mbit/sec with EoIP + IPSEC. Would get a bit more if we had used the correct ports according to the block diagram. We did contact support - same story, no timeframe for a fix but 1009/1016 will be sorted first.

nathan1 · Fri Feb 17, 2017 6:55 pm

We've been struggling to get GCM (i.e. software only encryption) working between our CCR1009s and Cisco ASAs. As such I'm in a real pickle having recommended Mikrotik.

We'll take any kind of workaround right now (e.g. run CCR1009 as single core router) until there is a proper fix.

For now, RB1100AHx2 is still the best router Mikrotik makes. Going down the Tilera path has been a mistake.
I've tried to suggest the single core workaround without success. Try contacting support, a little more demand certainly can't hurt. What kind of throughput are you able to get out of IPSec on a RB1100AHx2?
We get around 750mbit/sec with EoIP + IPSEC. Would get a bit more if we had used the correct ports according to the block diagram. We did contact support - same story, no timeframe for a fix but 1009/1016 will be sorted first.

750Mbit/sec EoIP + IPSec is pretty nice. I'm stuck with 14 of the 1009s doing 250Mbit at best. If I had known about the re-ordering issue before I deployed these 1009s, I'd be on the AHx2.
I think we are now going on over a year without resolution now and the CCR platform continues to be advertised as a high performance IPSec platform.

pe1chl · Fri Feb 17, 2017 7:12 pm

750Mbit/sec EoIP + IPSec is pretty nice. I'm stuck with 14 of the 1009s doing 250Mbit at best. If I had known about the re-ordering issue before I deployed these 1009s, I'd be on the AHx2.
I think we are now going on over a year without resolution now and the CCR platform continues to be advertised as a high performance IPSec platform.

Do you have issues in day-to-day operation or only when running benchmarks on Windows machines?
I ask because we use IPsec on CCR1009 without problem. But the traffic is from/to many different machines, not a single-connection-TCP-benchmark.
But even that works fine when between Linux systems.

nathan1 · Fri Feb 17, 2017 7:21 pm

750Mbit/sec EoIP + IPSec is pretty nice. I'm stuck with 14 of the 1009s doing 250Mbit at best. If I had known about the re-ordering issue before I deployed these 1009s, I'd be on the AHx2.
I think we are now going on over a year without resolution now and the CCR platform continues to be advertised as a high performance IPSec platform.
Do you have issues in day-to-day operation or only when running benchmarks on Windows machines?
I ask because we use IPsec on CCR1009 without problem. But the traffic is from/to many different machines, not a single-connection-TCP-benchmark.
But even that works fine when between Linux systems.

We only have Linux machines but we have a lot of non-TCP flows, which the issue wreaks havoc on. Linux does cope well with the TCP re-ordering but it creates a very unstable flow, bandwidth and window sizes fluctuate significantly. At this point I have just entirely disabled the hardware acceleration and the network is very stable but throughput is lower than the physical capability. The biggest show stopper was with OSPF running over the EoIP sessions, it caused constant flapping and made for complete instability of the routing core and in turn the network.

alexjhart · Fri Feb 17, 2017 7:33 pm

We've been struggling to get GCM (i.e. software only encryption) working between our CCR1009s and Cisco ASAs. As such I'm in a real pickle having recommended Mikrotik.

We'll take any kind of workaround right now (e.g. run CCR1009 as single core router) until there is a proper fix.

For now, RB1100AHx2 is still the best router Mikrotik makes. Going down the Tilera path has been a mistake.
I've tried to suggest the single core workaround without success. Try contacting support, a little more demand certainly can't hurt. What kind of throughput are you able to get out of IPSec on a RB1100AHx2?
We get around 750mbit/sec with EoIP + IPSEC. Would get a bit more if we had used the correct ports according to the block diagram. We did contact support - same story, no timeframe for a fix but 1009/1016 will be sorted first.
750Mbit/sec EoIP + IPSec is pretty nice. I'm stuck with 14 of the 1009s doing 250Mbit at best. If I had known about the re-ordering issue before I deployed these 1009s, I'd be on the AHx2.
I think we are now going on over a year without resolution now and the CCR platform continues to be advertised as a high performance IPSec platform.

That's absolutely the case here too. We went with the tilera chip because of the promised hardware encryption throughput, but don't get it because of this driver problem. Unfortunately, we spent even more on 1036 and 1072 units. Insult to injury, they'll be the last to be fixed. I started reporting the issue to mikrotik well over 1 year ago. In the case of 10Gbps interfaces, we don't even have the option to replace with the AHx2. So I'm stuck waiting for them to get their act together or going with a different vendor.

StubArea51 · Wed Feb 22, 2017 10:47 pm

Glad to see this issue is getting attention still. Been waiting to see a fix on it ever since Alex brought it up at the 2016 US MUM.

Ascendo · Thu Feb 23, 2017 8:34 pm

Glad to see this issue is getting attention still. Been waiting to see a fix on it ever since Alex brought it up at the 2016 US MUM.

An actual update from Mikrotik staff would be great...

mrz · Fri Mar 10, 2017 3:20 pm

What's new in 6.39rc51 (2017-Mar-10 12:50):

!) tile - fixed IPsec hardware acceleration out-of-order packet problem, significantly improved performance;

nathan1 · Fri Mar 10, 2017 3:35 pm

What's new in 6.39rc51 (2017-Mar-10 12:50):

!) tile - fixed IPsec hardware acceleration out-of-order packet problem, significantly improved performance;

Thanks for the update! Testing soon...

nathan1 · Fri Mar 10, 2017 4:57 pm

What's new in 6.39rc51 (2017-Mar-10 12:50):

!) tile - fixed IPsec hardware acceleration out-of-order packet problem, significantly improved performance;

mrz, is EoIP in this version obeying the default proposals? I've attempted switching to hardware accelerated proposals and then recreating the EoIP but it doesn't seem quite right. Any suggestions on what configuration I should be using to test this fixed hardware acceleration?

 0  * name="default" auth-algorithms=sha256 enc-algorithms=aes-256-cbc lifetime=30m pfs-group=modp1024
 
  0  D  ;;; XXX
       address=169.254.17.11/32 local-address=169.254.17.12 auth-method=pre-shared-key 
       secret="XXXXX" generate-policy=no policy-template-group=default exchange-mode=main 
       send-initial-contact=yes nat-traversal=yes proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128,3des 
       dh-group=modp1024 lifetime=1d dpd-interval=2m dpd-maximum-failures=5

alexjhart · Fri Mar 10, 2017 5:47 pm

What's new in 6.39rc51 (2017-Mar-10 12:50):

!) tile - fixed IPsec hardware acceleration out-of-order packet problem, significantly improved performance;

This is fantastic news. Does this work with all models, or as suggested by support earlier will it just be a subset at this time? Any details on what the specific behavior is now vs before?

mrz · Fri Mar 10, 2017 7:12 pm

What's new in 6.39rc51 (2017-Mar-10 12:50):

!) tile - fixed IPsec hardware acceleration out-of-order packet problem, significantly improved performance;
mrz, is EoIP in this version obeying the default proposals? I've attempted switching to hardware accelerated proposals and then recreating the EoIP but it doesn't seem quite right. Any suggestions on what configuration I should be using to test this fixed hardware acceleration?
Code: Select all
 0  * name="default" auth-algorithms=sha256 enc-algorithms=aes-256-cbc lifetime=30m pfs-group=modp1024
 
  0  D  ;;; XXX
       address=169.254.17.11/32 local-address=169.254.17.12 auth-method=pre-shared-key 
       secret="XXXXX" generate-policy=no policy-template-group=default exchange-mode=main 
       send-initial-contact=yes nat-traversal=yes proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128,3des 
       dh-group=modp1024 lifetime=1d dpd-interval=2m dpd-maximum-failures=5 

Peer proposal is phase1 and has nothing to do with hardware acceleration. Phase2 proposal can be selected in "/ip ipsec proposal" menu, this is hardware acceleration related.

mrz · Fri Mar 10, 2017 7:12 pm

What's new in 6.39rc51 (2017-Mar-10 12:50):

!) tile - fixed IPsec hardware acceleration out-of-order packet problem, significantly improved performance;
This is fantastic news. Does this work with all models, or as suggested by support earlier will it just be a subset at this time? Any details on what the specific behavior is now vs before?

Fix is for all CCR models.

nathan1 · Fri Mar 10, 2017 7:36 pm

Peer proposal is phase1 and has nothing to do with hardware acceleration. Phase2 proposal can be selected in "/ip ipsec proposal" menu, this is hardware acceleration related.

Phase 2 was not completing for me with sha256/aes-256-cbc, just seemed to stall. What proposal do you recommend testing with?

nathan1 · Sat Mar 11, 2017 12:41 am

So I believe that I have the rc working with sha1/aes-128-cbc but performance is not what I'd expect. Re-ordering does appear to be fixed but performance isn't anywhere near line rate (1Gbit in this case).

mrz, can you elaborate on what we should be seeing for performance on a single IPSec session with a UDP flow (or many TCP connections)? It feels as if the performance is nearly the same as a software aes ctr.

Has anyone else been able to test?

Ascendo · Sat Mar 11, 2017 9:55 am

Nathan1, what sort of speeds are you getting per flow, and how high is the latency between the devices? At least there is some progress...

nathan1 · Sat Mar 11, 2017 12:44 pm

Nathan1, what sort of speeds are you getting per flow, and how high is the latency between the devices? At least there is some progress...

Latency is .7ms RTT and I was seeing around 300Mbit, very similar to the ctr. Blasting UDP peaks around 350Mbit, in clear I can push 1Gbit. I was only able to briefly test so hopefully someone can confirm or deny what I am seeing so I know which direction to go.

Ascendo · Sat Mar 11, 2017 2:03 pm

Nathan1, what sort of speeds are you getting per flow, and how high is the latency between the devices? At least there is some progress...
Latency is .7ms RTT and I was seeing around 300Mbit, very similar to the ctr. Blasting UDP peaks around 350Mbit, in clear I can push 1Gbit. I was only able to briefly test so hopefully someone can confirm or deny what I am seeing so I know which direction to go.

We currently get ~2Mbit file copies over a 85ms link, so I'll take 300 any day! You mentioned EoIP - are you seeing more than one core being maxed out during the UDP test?

nathan1 · Sat Mar 11, 2017 3:24 pm

Nathan1, what sort of speeds are you getting per flow, and how high is the latency between the devices? At least there is some progress...
Latency is .7ms RTT and I was seeing around 300Mbit, very similar to the ctr. Blasting UDP peaks around 350Mbit, in clear I can push 1Gbit. I was only able to briefly test so hopefully someone can confirm or deny what I am seeing so I know which direction to go.
We currently get ~2Mbit file copies over a 85ms link, so I'll take 300 any day! You mentioned EoIP - are you seeing more than one core being maxed out during the UDP test?

You can do 300 with the software encryption without the rc fix. What settings are you using? I use EoIP and a single core does max out with software but not hardware.

Ascendo · Sat Mar 11, 2017 3:37 pm

Nathan1, what sort of speeds are you getting per flow, and how high is the latency between the devices? At least there is some progress...
Latency is .7ms RTT and I was seeing around 300Mbit, very similar to the ctr. Blasting UDP peaks around 350Mbit, in clear I can push 1Gbit. I was only able to briefly test so hopefully someone can confirm or deny what I am seeing so I know which direction to go.
We currently get ~2Mbit file copies over a 85ms link, so I'll take 300 any day! You mentioned EoIP - are you seeing more than one core being maxed out during the UDP test?
You can do 300 with the software encryption without the rc fix. What settings are you using? I use EoIP and a single core does max out with software but not hardware.

We were unable to get GCM working with the particular Cisco ASA version we have on the remote end, so we've been stuck using hardware encryption (aes-128-cbc). Most of our use case is high latency low bandwidth so not sure we can add much to the discussion right now.

We do run some RB1100AHx2 with a 1Gb link between our primary and DR site (EoIP) which we were hoping to replace with CCRs at some point. I'll try replacing them with some CCR1009s next week (with the RC code) to see if they work any better.

nathan1 · Sat Mar 11, 2017 3:48 pm

We were unable to get GCM working with the particular Cisco ASA version we have on the remote end, so we've been stuck using hardware encryption (aes-128-cbc). Most of our use case is high latency low bandwidth so not sure we can add much to the discussion right now.

We do run some RB1100AHx2 with a 1Gb link between our primary and DR site (EoIP) which we were hoping to replace with CCRs at some point. I'll try replacing them with some CCR1009s next week (with the RC code) to see if they work any better.

I see. I think in your situation, since it is a compatibility issue, you will see huge improvements from the rc.

nathan1 · Sun Mar 12, 2017 3:37 pm

So I did some additional testing, I can actually push around 800Mbit with a UDP flow and CPU usage is pretty good. However, when I run a TCP benchmark, CPU usage is very high (networking/cpu under profile) which I believe is constraining the TCP throughput now. I think I may have an issue with fragmentation/DF/MSS that I am looking into.

Edit: I have confirmed that there is some issue with fragmentation in my setup during the hardware IPSec test. It is not clear what is going on but if I force the TCP MSS on iperf, I can confirm heavily increased throughput nearing the UDP test. It seems the increased CPU usage is due to the CCR being forced to fragment/reconstruct the TCP packets.

TLDR: This fix looks like what we have all been waiting for.

Ascendo · Sun Mar 12, 2017 4:50 pm

So I did some additional testing, I can actually push around 800Mbit with a UDP flow and CPU usage is pretty good. However, when I run a TCP benchmark, CPU usage is very high (networking/cpu under profile) which I believe is constraining the TCP throughput now. I think I may have an issue with fragmentation/DF/MSS that I am looking into.

Edit: I have confirmed that there is some issue with fragmentation in my setup during the hardware IPSec test. It is not clear what is going on but if I force the TCP MSS on iperf, I can confirm heavily increased throughput nearing the UDP test. It seems the increased CPU usage is due to the CCR being forced to fragment/reconstruct the TCP packets.

TLDR: This fix looks like what we have all been waiting for.

That's interesting. Any chance of doing a single-threaded windows copy (explorer/robobocopy) test? Very interested to see if it beats our ~450mbps result on the RB1100AHx2.

nathan1 · Sun Mar 12, 2017 5:13 pm

So I did some additional testing, I can actually push around 800Mbit with a UDP flow and CPU usage is pretty good. However, when I run a TCP benchmark, CPU usage is very high (networking/cpu under profile) which I believe is constraining the TCP throughput now. I think I may have an issue with fragmentation/DF/MSS that I am looking into.

Edit: I have confirmed that there is some issue with fragmentation in my setup during the hardware IPSec test. It is not clear what is going on but if I force the TCP MSS on iperf, I can confirm heavily increased throughput nearing the UDP test. It seems the increased CPU usage is due to the CCR being forced to fragment/reconstruct the TCP packets.

TLDR: This fix looks like what we have all been waiting for.
That's interesting. Any chance of doing a single-threaded windows copy (explorer/robobocopy) test? Very interested to see if it beats our ~450mbps result on the RB1100AHx2.

I would but I can't do it easily - I don't have any windows machines in my infrastructure.

alexjhart · Sun Mar 12, 2017 5:59 pm

So I did some additional testing, I can actually push around 800Mbit with a UDP flow and CPU usage is pretty good. However, when I run a TCP benchmark, CPU usage is very high (networking/cpu under profile) which I believe is constraining the TCP throughput now. I think I may have an issue with fragmentation/DF/MSS that I am looking into.

Edit: I have confirmed that there is some issue with fragmentation in my setup during the hardware IPSec test. It is not clear what is going on but if I force the TCP MSS on iperf, I can confirm heavily increased throughput nearing the UDP test. It seems the increased CPU usage is due to the CCR being forced to fragment/reconstruct the TCP packets.

TLDR: This fix looks like what we have all been waiting for.
That's interesting. Any chance of doing a single-threaded windows copy (explorer/robobocopy) test? Very interested to see if it beats our ~450mbps result on the RB1100AHx2.
I would but I can't do it easily - I don't have any windows machines in my infrastructure.

When I get a chance, I'll test and report back.

Ascendo · Sun Mar 12, 2017 6:14 pm

So I did some additional testing, I can actually push around 800Mbit with a UDP flow and CPU usage is pretty good. However, when I run a TCP benchmark, CPU usage is very high (networking/cpu under profile) which I believe is constraining the TCP throughput now. I think I may have an issue with fragmentation/DF/MSS that I am looking into.

Edit: I have confirmed that there is some issue with fragmentation in my setup during the hardware IPSec test. It is not clear what is going on but if I force the TCP MSS on iperf, I can confirm heavily increased throughput nearing the UDP test. It seems the increased CPU usage is due to the CCR being forced to fragment/reconstruct the TCP packets.

TLDR: This fix looks like what we have all been waiting for.
That's interesting. Any chance of doing a single-threaded windows copy (explorer/robobocopy) test? Very interested to see if it beats our ~450mbps result on the RB1100AHx2.
I would but I can't do it easily - I don't have any windows machines in my infrastructure.

Fair enough

I'll pick up some CCR1009s during the week and try replacing the existing RB1100AHx2s and see how it goes.

Now the only question is when the fix will be added to the bugfix/current builds. Hopefully the RC is stable enough for production.

Ascendo · Mon Mar 13, 2017 7:48 am

We have deployed this to a couple of small, low bandwidth remote sites (CCR1009) and are happy to report that the performance is now exactly what it should be. Will hopefully do some testing this week with more throughput but so far so good.

@MT - thanks for getting this sorted. Hopefully it'll be put into current/bugfix version soon.

theprojectgroup · Mon Mar 13, 2017 10:00 pm

Looking good here on CCR1016-12G, tested before and after the update

Site2Site IPIP Tunnel Spain (fibre 300mbits ISP: consumer) <-----------> Germany (fibre 100mbits ISP: m-net corp) with latency:60ms

SMB2 traffic:

speed.png

nathan1 · Wed Mar 15, 2017 1:07 am

Also confirming that this is looking very good. I have deployed it across my global network, backed up by a set of CCRs using the old software setup, in the event that the rc fails. Seeing peak performance around 800Mbit single TCP stream on my fastest connection (1Gbit @ .7ms). Currently using sha1/aes-128-cbc. I've had a few strange issues with 256 not wanting to go to phase 2 every time, not sure why. I did drop my MTU to compensate for the extra overhead vs. ctr. I am currently running at 1400 on the L2 VLANs.

mrz · Wed Mar 15, 2017 9:03 am

Itcould be great if anyone with 72core router could verify if it work,too.

alexjhart · Wed Mar 15, 2017 9:32 am

Itcould be great if anyone with 72core router could verify if it work,too.

I should be able to do that soon and will get back to you.

anuser · Sun Mar 19, 2017 9:51 pm

We have deployed this to a couple of small, low bandwidth remote sites (CCR1009) and are happy to report that the performance is now exactly what it should be. Will hopefully do some testing this week with more throughput but so far so good..

@Ascendo: What throughput do you get with IPSec on CCR1009?

@All: Does anyone use CCR1009 with EoIP and IPSec enabled with latest software? What throughput do you get in this situation?

Ascendo · Mon Mar 20, 2017 7:00 am

We have deployed this to a couple of small, low bandwidth remote sites (CCR1009) and are happy to report that the performance is now exactly what it should be. Will hopefully do some testing this week with more throughput but so far so good..
@Ascendo: What throughput do you get with IPSec on CCR1009?

@All: Does anyone use CCR1009 with EoIP and IPSec enabled with latest software? What throughput do you get in this situation?

The site we've deployed to only has 10mbps and it now maxes out the link when doing file copies to the remote DC (currently 69ms latency). Previously we'd be lucky to get 2mbps.

We have not tried replacing the pair of RB1100AHx2s which are running EoIP+IPSEC yet (1ms 1Gbps link).

mrz · Mon Mar 20, 2017 10:29 am

@anuser: if it is any use to you we got 2.4Gbps on CCR1009 with multiple streams (UDP 1400byte packets)

pchott · Mon Mar 20, 2017 10:42 am

@mrz: With UDP there is usual not such great error as at TCP.

I think the relevant speed test ist TCP. On my test i can get over 2.5Gb (before anythis changes) over UDP but under 300Mb over TCP. Since now is all in production I was not able to test this update-

mrz · Mon Mar 20, 2017 10:44 am

~2Gbps TCP 1360MSS

pchott · Mon Mar 20, 2017 10:48 am

~2Gbps TCP 1360MSS

Great to hear that. Thank you for info. I am looking forward that i can test this on CCR1036 that I use.

Mon Mar 20, 2017 10:51 am

~2Gbps TCP 1360MSS

Impressive!

Thank you for the baseline mrz

mikruser · Mon Mar 20, 2017 10:52 am

When we can expect this fix in "bugfix" branch?

anuser · Mon Mar 20, 2017 12:06 pm

~2Gbps TCP 1360MSS

Great! Is this EoIP including IPSec? Or IPSec only? Which parameter did you use?

mrz · Mon Mar 20, 2017 12:30 pm

It is pure ipsec.
EoIP over ipsec should be a bit slower because of overhead and ability to send smaller packets over the tunnel.

And note that these numbers cannot be achieved on single tunnel.

Siona · Mon Mar 20, 2017 1:21 pm

What's your single tunnel performance?

Wysłane z mojego ONEPLUS A3003 przy użyciu Tapatalka

mrz · Mon Mar 20, 2017 1:25 pm

1tun 650/700Mbps total 1.3Gbps (UDP), did not test with TCP.

nathan1 · Wed Mar 22, 2017 2:05 am

FYI - I am running single tunnel EoIP and can achieve single flow 700Mbit TCP - MTU 1400.

/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha1 disabled=no enc-algorithms=aes-128-cbc lifetime=30m name=default pfs-group=modp1024

normalcy · Wed Mar 22, 2017 4:05 am

Has anyone tried GRE over IPSec transport? Or SMB traffic on windows clients? That's been what we have had issues with.

Looking forward to this fix filtering down - I'll be sorely tempted to use current rather than waiting for bugfix....

nathan1 · Wed Mar 22, 2017 12:36 pm

Has anyone tried GRE over IPSec transport? Or SMB traffic on windows clients? That's been what we have had issues with.

Looking forward to this fix filtering down - I'll be sorely tempted to use current rather than waiting for bugfix....

theprojectgroup tested SMB: viewtopic.php?f=1&t=112545&start=50#p588615
The re-ordering is definitely fixed so as long as your MTU is good, you should see the performance that you expected to see back when we all put CCRs in place.

alexjhart · Mon Mar 27, 2017 8:11 pm

Itcould be great if anyone with 72core router could verify if it work,too.

Has anyone tried GRE over IPSec transport? Or SMB traffic on windows clients? That's been what we have had issues with.

Looking forward to this fix filtering down - I'll be sorely tempted to use current rather than waiting for bugfix....

I'm using GRE with IPSEC. Also SMB traffic with Windows clients. I was able to get the RC loaded on a 1072 and a few 1036 CCRs this weekend. Initial testing shows that I can do at least the amount of throughput on hardware as software encryption was doing (CBC vs CTR), so that is a big step in the right direction. I have yet to do a capture and check how the packet order is looking. I'll try to do that soon. I'm not seeing anywhere near the throughput in an encrypted tunnel as am I in the same tunnel with encryption turned off. I don't know if that is directly related to the CCR1072 (since Mikrotik specifically asked about verifying it).

So I'm not ecstatic yet about the results, but I see that you guys are getting much more so maybe I have some more tweaking to do.

nathan1 · Fri Mar 31, 2017 10:16 pm

Itcould be great if anyone with 72core router could verify if it work,too.

Has anyone tried GRE over IPSec transport? Or SMB traffic on windows clients? That's been what we have had issues with.

Looking forward to this fix filtering down - I'll be sorely tempted to use current rather than waiting for bugfix....
I'm using GRE with IPSEC. Also SMB traffic with Windows clients. I was able to get the RC loaded on a 1072 and a few 1036 CCRs this weekend. Initial testing shows that I can do at least the amount of throughput on hardware as software encryption was doing (CBC vs CTR), so that is a big step in the right direction. I have yet to do a capture and check how the packet order is looking. I'll try to do that soon. I'm not seeing anywhere near the throughput in an encrypted tunnel as am I in the same tunnel with encryption turned off. I don't know if that is directly related to the CCR1072 (since Mikrotik specifically asked about verifying it).

So I'm not ecstatic yet about the results, but I see that you guys are getting much more so maybe I have some more tweaking to do.

These are the results that I got before I sorted out my MTU. The fragmentation will fry it. Have you confirmed that you aren't getting fragmentation over the tunnel?

normalcy · Sat Apr 01, 2017 12:48 pm

Thanks for the feedback nathan1 and alexjhart (and for you persistence chasing this!!).

My testing CCR1016 has been borrowed by a colleague but I look forward to trying this out on it when I get it back. We have multiple CCR1036 acting as VPN concentrators (never got the chance to swap them out after discovering this issue) that will benefit from this.

sbeauchamp · Mon Apr 03, 2017 10:46 pm

Should I expect this to be fixed on CCR1009-8G-1S-1S+? I haven't had a chance to test yet, but I ask because i notice this model isn't listed for sale anymore.

pe1chl · Mon Apr 03, 2017 11:41 pm

Should I expect this to be fixed on CCR1009-8G-1S-1S+? I haven't had a chance to test yet, but I ask because i notice this model isn't listed for sale anymore.

I think there is no reason to fear... the new model is mostly the same except it does not have the switch chip and it has more directly connected links to the processor instead.
The software running on those models is exactly the same. That is one advantage of MikroTik over many other manufacturers: common software over the entire product range, and thus less reason to terminate software support on out of sale models.

alexjhart · Tue Apr 04, 2017 2:05 am

These are the results that I got before I sorted out my MTU. The fragmentation will fry it. Have you confirmed that you aren't getting fragmentation over the tunnel?

So far as I can tell, this is not the case. However, if you made changes and got the performance expected, perhaps I am missing something. Any advice?

nathan1 · Tue Apr 04, 2017 2:27 am

These are the results that I got before I sorted out my MTU. The fragmentation will fry it. Have you confirmed that you aren't getting fragmentation over the tunnel?
So far as I can tell, this is not the case. However, if you made changes and got the performance expected, perhaps I am missing something. Any advice?

Run a ping with DF at the max size you THINK you can move across the tunnel and then run the packet sniffer on the encapsulating interface and the source interface, see if if you see one big packet followed by a smaller one. I also used ping with do-not-fragment=yes to ping mikrotik-mikrotik to determine the actual MTU. In my case, with EoIP and the change of encryption, my MTU was slightly decreased vs. what I had on CTR software.

Edit: Another way that I proved I was having MTU problems was to force the MTU down between two hosts that talk over the tunnel. With Linux, I was able to do this with iproute2 but I'm not sure how you do this with Windows? Perhaps use a UDP benchmark where you can set the datagram length and start low around 1200 just to get a baseline.

sbeauchamp · Tue Apr 04, 2017 2:48 pm

Heres what im seeing with the CSR1009-8g-1s-1s+

IPIP tunnel with IPSEc, holds ~200mbps
turning on QoS takes a pretty big hit down to ~70mbps
turning off connection tracking seems to give a 30-60mbps bump depending on the situation
no QoS and no connection tracking reaches about 300mbps.
upload seems odd, about 35mbps is where it tops out regardless of what features are turned on. (this may be on my end, not sure at the moment)

I'm using simple queue to prioritize certain types of traffic such as voice. Is there a better way or any more optimal settings to increase my performance from what I see above? Im using mangle rules to identify and mark the packets then use this queue config below:

0 name="LOU-HUB" target=lsv,lsv parent=none packet-marks="" priority=8/8 queue=default/default limit-at=0/0 max-limit=0/0
burst-limit=0/0 burst-threshold=0/0 burst-time=1s/0s bucket-size=0.1/0.1 total-queue=default

1 name="BGP-LOU1" parent=LOU-HUB packet-marks=BGP priority=1/1 queue=default/default limit-at=10k/10k max-limit=500k/500k
burst-limit=500k/500k burst-threshold=500k/500k burst-time=1s/1s bucket-size=0.1/0.1

2 name="BFD-LOU1" parent=LOU-HUB packet-marks=BFD priority=1/1 queue=default/default limit-at=10k/10k max-limit=10k/10k
burst-limit=10k/10k burst-threshold=10k/10k burst-time=1s/1s bucket-size=0.1/0.1

3 name="VOIPUDSP-LOU1" parent=LOU-HUB packet-marks=VOIP priority=2/2 queue=default-small/default-small limit-at=384k/384k
max-limit=512k/512k burst-limit=512k/512k burst-threshold=512k/512k burst-time=1s/1s bucket-size=0.1/0.1

4 name="SIGNALING-LOU1" parent=LOU-HUB packet-marks=SIP,SKINNY priority=3/3 queue=default/default limit-at=50k/50k
max-limit=500k/500k burst-limit=500k/500k burst-threshold=500k/500k burst-time=1s/1s bucket-size=0.1/0.1

5 name="OTHER-LOU1" parent=LOU-HUB packet-marks=OTHER priority=8/8 queue=default/default limit-at=50k/50k max-limit=500M/500M
burst-limit=500M/500M burst-threshold=500M/500M burst-time=1s/1s bucket-size=0.1/0.1 total-queue=default

the target in number 0 is my IPIP interface. For some reason it adds it in there twice. If i try to just have it once, the rule becomes invalid. It seems to work like this though.

mrz · Tue Apr 04, 2017 5:59 pm

It is like in any other configuration, adding simple queues or software queues on interface will significantly reduce performance. Stream is classified before encryption and all actions with packets from that stream are done on the same core.

sbeauchamp · Thu Apr 06, 2017 2:59 pm

It is like in any other configuration, adding simple queues or software queues on interface will significantly reduce performance. Stream is classified before encryption and all actions with packets from that stream are done on the same core.

That makes sense. This issue does indeed seem fixed to me. Im unfamiliar with how the releases go here, will this fix get pushed down to the bugfix line 6.37.x at some point?

I should expect the bigger CCR models with higher core counts to achieve higher total bandwidth (than the 1009) then with IPSEC+Queues as it does spread the streams among the cores correct?

royalpublishing · Mon Apr 10, 2017 5:55 pm

It is like in any other configuration, adding simple queues or software queues on interface will significantly reduce performance. Stream is classified before encryption and all actions with packets from that stream are done on the same core.
That makes sense. This issue does indeed seem fixed to me. Im unfamiliar with how the releases go here, will this fix get pushed down to the bugfix line 6.37.x at some point?

I should expect the bigger CCR models with higher core counts to achieve higher total bandwidth (than the 1009) then with IPSEC+Queues as it does spread the streams among the cores correct?

I wouldn't get your hopes up too much about the higher core CCR's, I see the exact same thing you are seeing throughput wise. I am on 6.39rc58 on all of my devices (CCR1036 core and CCR1016 remote sites), but unfortunately even after trying all of the different hardware encryption combinations (SHA1/SHA256 and AES-128/192/256 CBC) I still cannot obtain an upload speed greater than around 42Mbps, but more realistically it sits about the mid 30's. I've been dealing with this problem since June of 2013 and with these updates I finally thought to myself "oh thank god they've finally got the problem fixed and I'll actually be able to use the full potential of my fiber connections", but that is not the case. Even with the re-ordering problem "fixed", I have seen zero difference whatsoever in upload throughput. In fact, I still get higher upload throughput using the software encryption. Even if I completely disable my queue trees, I see no changes in throughput. I do understand that having some queues, firewall/mangle rules, and connection tracking enabled will degrade performance, but I'm not sure that is even the cause of my issues here. I can still sit here and watch the state sequence errors increase while I try to upload a large file. It shouldn't be a fragmentation issue either as I've got mangle rules set up on both ends for MSS of 1360. I desperately would like to find a fix for this.

Jocksor · Mon Apr 10, 2017 7:12 pm

In my case 6.39rc58 improved bandwidth from ~70Mbps to ~250Mbps using EoIP + IPsec + Firewall w/o queues. Download\upload are the same.

It is like in any other configuration, adding simple queues or software queues on interface will significantly reduce performance. Stream is classified before encryption and all actions with packets from that stream are done on the same core.
That makes sense. This issue does indeed seem fixed to me. Im unfamiliar with how the releases go here, will this fix get pushed down to the bugfix line 6.37.x at some point?

I should expect the bigger CCR models with higher core counts to achieve higher total bandwidth (than the 1009) then with IPSEC+Queues as it does spread the streams among the cores correct?
I wouldn't get your hopes up too much about the higher core CCR's, I see the exact same thing you are seeing throughput wise. I am on 6.39rc58 on all of my devices (CCR1036 core and CCR1016 remote sites), but unfortunately even after trying all of the different hardware encryption combinations (SHA1/SHA256 and AES-128/192/256 CBC) I still cannot obtain an upload speed greater than around 42Mbps, but more realistically it sits about the mid 30's. I've been dealing with this problem since June of 2013 and with these updates I finally thought to myself "oh thank god they've finally got the problem fixed and I'll actually be able to use the full potential of my fiber connections", but that is not the case. Even with the re-ordering problem "fixed", I have seen zero difference whatsoever in upload throughput. In fact, I still get higher upload throughput using the software encryption. Even if I completely disable my queue trees, I see no changes in throughput. I do understand that having some queues, firewall/mangle rules, and connection tracking enabled will degrade performance, but I'm not sure that is even the cause of my issues here. I can still sit here and watch the state sequence errors increase while I try to upload a large file. It shouldn't be a fragmentation issue either as I've got mangle rules set up on both ends for MSS. I desperately would like to find a fix for this.

royalpublishing · Mon Apr 10, 2017 7:37 pm

@Jocksor, unfortunately I'm not using EoIP. I'm just using the basic IPSec in tunnel mode. On the IPSec wiki page, there are some optimizations for the RB1100AHx2 to set the irq's, would this help on the CCR?

sbeauchamp · Mon Apr 10, 2017 8:48 pm

@Jocksor, unfortunately I'm not using EoIP. I'm just using the basic IPSec in tunnel mode. On the IPSec wiki page, there are some optimizations for the RB1100AHx2 to set the irq's, would this help on the CCR?

I tried some of those 1100 optimizations on the CCR, but in my case it didn't really make a difference.

on the 1100 and on CHR x86 systems turning RPS off and assigning 1 CPU core per interface, and setting everything else to a core unused by interfaces seemed to greatly improve performance in my case. RPS actually seems to work correctly on the CCR, at least under crypto and with the latest RC software.

By the way on your CCRs, what are you seeing for your download speeds now?

royalpublishing · Mon Apr 10, 2017 9:56 pm

@Jocksor, unfortunately I'm not using EoIP. I'm just using the basic IPSec in tunnel mode. On the IPSec wiki page, there are some optimizations for the RB1100AHx2 to set the irq's, would this help on the CCR?
I tried some of those 1100 optimizations on the CCR, but in my case it didn't really make a difference.

on the 1100 and on CHR x86 systems turning RPS off and assigning 1 CPU core per interface, and setting everything else to a core unused by interfaces seemed to greatly improve performance in my case. RPS actually seems to work correctly on the CCR, at least under crypto and with the latest RC software.

By the way on your CCRs, what are you seeing for your download speeds now?

Unfortunately, I can really only do a speed test in one direction at the moment until all of my other fiber circuits get turned up. Currently my main site has a 100/200Mbps fiber connection and one of the remote sites has a 40/100Mbps fiber connection, so I'm fairly limited in the amount of testing I can do. Interestingly enough, I can run a tcp bandwidth test from the main site to the branch and can get like 75Mbps, but an SMB upload from a Windows 8 machine only gets the 35Mbps like I was saying. I'm going to do some more testing with various types of uploads like from a linux box, ftp, robocopy, etc to see if maybe that's where I'm hitting the wall.

nathan1 · Wed Apr 12, 2017 4:58 am

@Jocksor, unfortunately I'm not using EoIP. I'm just using the basic IPSec in tunnel mode. On the IPSec wiki page, there are some optimizations for the RB1100AHx2 to set the irq's, would this help on the CCR?
I tried some of those 1100 optimizations on the CCR, but in my case it didn't really make a difference.

on the 1100 and on CHR x86 systems turning RPS off and assigning 1 CPU core per interface, and setting everything else to a core unused by interfaces seemed to greatly improve performance in my case. RPS actually seems to work correctly on the CCR, at least under crypto and with the latest RC software.

By the way on your CCRs, what are you seeing for your download speeds now?
Unfortunately, I can really only do a speed test in one direction at the moment until all of my other fiber circuits get turned up. Currently my main site has a 100/200Mbps fiber connection and one of the remote sites has a 40/100Mbps fiber connection, so I'm fairly limited in the amount of testing I can do. Interestingly enough, I can run a tcp bandwidth test from the main site to the branch and can get like 75Mbps, but an SMB upload from a Windows 8 machine only gets the 35Mbps like I was saying. I'm going to do some more testing with various types of uploads like from a linux box, ftp, robocopy, etc to see if maybe that's where I'm hitting the wall.

What is the latency between these sites? Can your saturate the connection using the same testing method, without the IPSec? The numbers you mention (100/200Mbps) I had previously achieved no problem with the software encryption. I can easily exceed them with the ordering fix w/hardware encryption.

mikruser · Tue Jun 27, 2017 5:33 pm

I repeat my question:
When we can expect this fix in "bugfix" branch?

mrz · Tue Jun 27, 2017 5:43 pm

Probably when v6.39 will become a bugfix.

mikruser · Mon Oct 02, 2017 5:38 pm

But when 6.39 become a bugfix???

very strange situation:

6.41.x = RC
6.40.x = Current
6.39.x = WTF??? Where??? When???
6.38.x = Bugfix

Mon Oct 02, 2017 7:38 pm

6.39.x = WTF??? Where??? When???

When it is stable enough, I believe.

normalcy · Wed Oct 18, 2017 3:55 am

I've just seen 6.39.3 come into the bugfix train.

Can anyone confirm it includes the fix? Unless I'm blind I didn't see it mentioned in the system -> packages brief release notes.

Out of the office for another week before I can try it on a cloud core.

Wed Oct 18, 2017 11:07 am

Unless I'm blind I didn't see it mentioned in the system -> packages brief release notes.

It was mentioned in the changelog for the original 6.39 release, so should be fixed in the latest 6.39 release as well.

miro · Fri Oct 20, 2017 11:54 am

Hi guys.
Do you have to change MSS to 1360, to achieve max performance or you still have reordering issue, if you don't change MSS... ?

Fri Oct 20, 2017 1:18 pm

The original problem was not related to MSS and/or packet fragmentation. The usual stream of non-fragmented TCP packets was also affected.

nathan1 · Sun Nov 05, 2017 2:38 am

Hi guys.
Do you have to change MSS to 1360, to achieve max performance or you still have reordering issue, if you don't change MSS... ?

Yes, I have my MTU dropped to 1400 on my EoIP interfaces. Last I checked, if packets need to be fragmented, it will fallback to software. This might look like a re-ordering issue in terms of performance but it is a different problem.

clamp-tcp-mss=yes mtu=1400 dont-fragment=no
using sha1/aes-128 cbc/modp1024

Who is online