Community discussions

MikroTik App
 
Fabian
just joined
Topic Author
Posts: 8
Joined: Wed Mar 04, 2015 3:23 pm

Cloud Core IPSEC performance

Wed Mar 04, 2015 4:02 pm

This week i've received my very first mikrotik router. The CCR1009-8G-1S-1S+PC, which i'd like to use to replace a cisco ASA5505.

Configuring the CCR was quite easy, including different vlans, routing, firewall chains and ipsec, but the ipsec throughput is a bit disappointing.

My cable connection should be able to push 200 Mbps (down) / 40 Mbps (up) and routed/natted the CCR does this at ease. After reading http://wiki.mikrotik.com/wiki/Manual:IP ... encryption i expected the CCR to forward/encrypt ipsec traffic at the same speed as plain traffic.

With AES256 + SHA1 i'm hardly able to get 10 Mbps and with 3des a single transfer slowly creeps up to 30 Mbps. A cisco ASA5540 serves as VPN concentrator which has 1Gbps of bandwith. The old ASA5505 was able to encrypt/ipsec at 100Mbps and the CCR is configured as drop-in replacement for the ASA5505 so the ipsec configuration was to match the old configuration.

Phase1 AES256 + SHA1 + DH Group 5
Phase2 AES256 + SHA1 (no pfs)

Looking at the ipsec statistics *something* seems wrong. But so far i haven't been able to figure out what. Do i have to lower my expectations or should i be able to push 200Mbps / 40 Mbps ipsec traffic with the CCR ?
[admin@MikroTik] /ip ipsec statistics> print
                  in-errors: 0
           in-buffer-errors: 0
           in-header-errors: 0
               in-no-states: 0
   in-state-protocol-errors: 9
       in-state-mode-errors: 0
   in-state-sequence-errors: 43105
           in-state-expired: 0
        in-state-mismatches: 0
           in-state-invalid: 0
     in-template-mismatches: 16591
             in-no-policies: 0
          in-policy-blocked: 0
           in-policy-errors: 0
                 out-errors: 0
          out-bundle-errors: 0
    out-bundle-check-errors: 0
              out-no-states: 866
  out-state-protocol-errors: 297
      out-state-mode-errors: 0
  out-state-sequence-errors: 0
          out-state-expired: 297
         out-policy-blocked: 0
            out-policy-dead: 0
          out-policy-errors: 0
[admin@MikroTik] /ip ipsec> peer print
Flags: X - disabled, D - dynamic
 0    address=[Colo ASA5540] local-address=0.0.0.0 passive=no port=500 auth-method=pre-shared-key secret=":)" generate-policy=no policy-template-group=default exchange-mode=main send-initial-contact=yes
      nat-traversal=no proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-256 dh-group=modp1536 lifetime=1d lifebytes=0 dpd-interval=2m dpd-maximum-failures=1

[admin@MikroTik] /ip ipsec> proposal print
Flags: X - disabled, * - default
 0  * name="default" auth-algorithms=md5,sha1,sha256,null enc-algorithms=3des,aes-128-cbc,aes-256-cbc,camellia-256,aes-256-ctr lifetime=30m pfs-group=modp1024

 1    name="asa5540" auth-algorithms=sha1 enc-algorithms=aes-256-cbc lifetime=30m pfs-group=none

[admin@MikroTik] /ip ipsec> policy print
Flags: T - template, X - disabled, D - dynamic, I - inactive, * - default
 0 T * group=default src-address=::/0 dst-address=::/0 protocol=all proposal=default template=yes

 1     src-address=[public subnet] src-port=any dst-address=0.0.0.0/0 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=[CCR Wan address] sa-dst-address=[Colo ASA5540] proposal=asa5540
       priority=0
 
Fabian
just joined
Topic Author
Posts: 8
Joined: Wed Mar 04, 2015 3:23 pm

Re: Cloud Core IPSEC performance

Fri Mar 13, 2015 4:58 pm

To limit any issues caused by incompatibility between cisco and MT, i've bought another MT.

Site1: CCR1009-8G-1S-1S
Site2: CCR1009-8G-1S-1S+PC

IPsec forwarding performance hasn't improved much. Average speeds of 5MB/sec with spikes up to 9MB/sec. Without VPN i'm able to generate a steady flow at 23MB/sec

At the receiving end i've noticed a fast increasing amount of in-state-sequence-errors.
in-state-sequence-errors: 58923
CPU load during a filetransfer
> /system resource monitor
          cpu-used: 11%
  cpu-used-per-cpu: 0%,0%,100%,1%,0%,0%,0%,0%,0%
       free-memory: 1844416KiB
Is this supposed to be 'AES Hardware Encyption' ?
 
sallen
just joined
Posts: 11
Joined: Tue Feb 25, 2014 12:57 am

Re: Cloud Core IPSEC performance

Fri Mar 13, 2015 7:59 pm

I have this same problem with the RB850Gx2. It maxes out around 20 Mbps throughput on a 60 Mbps line when accessing a NATed machine behind the router. When I do a throughput test against the router itself, I get the full 60 Mbps. There is something seriously wrong with the IPSEC tunnel. The dumb thing is that both the CPUs are only at 2-5% usage, and it is only passing about 600 pps.

I actually just ordered a CCR1009-8G-1S-1S+PC to attempt to fix it. But now I see that you are having the same problem?!

Mikrotik, this needs to be fixed! It's been like this since at least 6.18.
 
Fabian
just joined
Topic Author
Posts: 8
Joined: Wed Mar 04, 2015 3:23 pm

Re: Cloud Core IPSEC performance

Sat Mar 14, 2015 1:45 pm

To rule out any other networking issues, i've configured a ipip tunnel.

Test results from Lan to Wan (remote site):
down [  5]  0.0-60.1 sec  1.35 GBytes   194 Mbits/sec
 up   [  3]  0.0-60.0 sec   285 MBytes  39.8 Mbits/sec
Which come really close the specs of the broadband internet connection (200/40).

Back to cisco asa for me :(
 
alchemyx
just joined
Posts: 24
Joined: Mon Apr 16, 2012 9:43 pm

Re: Cloud Core IPSEC performance

Mon Mar 23, 2015 4:10 pm

Here are my specs that work fine with about 200 Mbps of traffic:
[alchemyx@xxx] /ip ipsec> peer print
Flags: X - disabled, D - dynamic 
 0    address=0.0.0.0/0 local-address=0.0.0.0 passive=no port=500 auth-method=rsa-signature certificate=serwer-obecny remote-certificate=none generate-policy=port-override policy-template-group=default exchange-mode=main-l2tp 
      send-initial-contact=yes nat-traversal=yes hash-algorithm=sha1 enc-algorithm=3des,aes-256 dh-group=modp1024 lifetime=1d dpd-interval=2m dpd-maximum-failures=5 
[alchemyx@xxx] /ip ipsec> proposal print 
Flags: X - disabled, * - default 
 0  * name="default" auth-algorithms=sha1 enc-algorithms=aes-256-cbc lifetime=30m pfs-group=none 
[alchemyx@xxx] /ip ipsec> policy print 
Flags: T - template, X - disabled, D - dynamic, I - inactive, * - default 
 0 T * group=default src-address=::/0 dst-address=::/0 protocol=all proposal=default template=yes 
[alchemyx@xxx] /ip ipsec> statistics print 
                  in-errors: 0
           in-buffer-errors: 0
           in-header-errors: 0
               in-no-states: 0
   in-state-protocol-errors: 0
       in-state-mode-errors: 0
   in-state-sequence-errors: 0
           in-state-expired: 0
        in-state-mismatches: 0
           in-state-invalid: 0
     in-template-mismatches: 0
             in-no-policies: 0
          in-policy-blocked: 0
           in-policy-errors: 0
                 out-errors: 0
          out-bundle-errors: 0
    out-bundle-check-errors: 0
              out-no-states: 0
  out-state-protocol-errors: 0
      out-state-mode-errors: 0
  out-state-sequence-errors: 0
          out-state-expired: 0
         out-policy-blocked: 0
            out-policy-dead: 0
          out-policy-errors: 0
 
_saik0
Member Candidate
Member Candidate
Posts: 129
Joined: Sun Aug 26, 2007 11:18 pm

Re: Cloud Core IPSEC performance

Tue Apr 07, 2015 12:38 am

Hmm, strongly considering buying a ccr1009 to replace my rb2011 for a 100/100 link.
My setup relies on l2tp over ipsec, so i've been reading a lot about ipsec throughput on ccr1009.
Turns out many people have issues and there are LOTS of threads regarding ccr and ipsec.

Beside you two guys, what throughput are people actually getting in the end?
Should I really just get an old RB1100AH?
 
troffasky
Member
Member
Posts: 436
Joined: Wed Mar 26, 2014 4:37 pm

Re: Cloud Core IPSEC performance

Tue Apr 07, 2015 1:03 am

Here are my specs that work fine with about 200 Mbps of traffic:
<snip config>
[alchemyx@xxx] /ip ipsec> statistics print 
                  in-errors: 0
           in-buffer-errors: 0
           in-header-errors: 0
               in-no-states: 0
   in-state-protocol-errors: 0
       in-state-mode-errors: 0
   in-state-sequence-errors: 0
           in-state-expired: 0
        in-state-mismatches: 0
           in-state-invalid: 0
     in-template-mismatches: 0
             in-no-policies: 0
          in-policy-blocked: 0
           in-policy-errors: 0
                 out-errors: 0
          out-bundle-errors: 0
    out-bundle-check-errors: 0
              out-no-states: 0
  out-state-protocol-errors: 0
      out-state-mode-errors: 0
  out-state-sequence-errors: 0
          out-state-expired: 0
         out-policy-blocked: 0
            out-policy-dead: 0
          out-policy-errors: 0
I am curious, how long had the SA been up at the point you printed these stats? Even on an apparently perfectly working IPsec tunnel, I see lots of non-zero values, mostly on the *-state-* parameters. Having checked a few routers, not one of them has all zeroes. Have you done a packet capture to confirm you're actually encrypting the traffic? :-D
 
royalpublishing
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Sep 23, 2013 5:47 pm

Re: Cloud Core IPSEC performance

Tue Jul 14, 2015 11:27 pm

Another upset/disappointed customer here. Unfortunately I am seeing the same performance problems on CCR1036 and CCR1016 as everybody else. Using IPSec, I can't get more than 25-30 Mbps throughput per tunnel. The speed is consistently around 25 Mbps with occasional jumps up to 27 or 28 Mbps but never above 30 Mbps and the CPU usage never goes above 1% on any of my routers. I have a pretty minimal access list, am using queuing, am doing some packet tagging, and connection tracking is on. As with everyone else here complaining, I desperately need this fixed as soon as humanly possible and NO downgrading to the RB1100AHx2 is not an acceptable answer!
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7195
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Cloud Core IPSEC performance

Wed Jul 15, 2015 10:02 am

Two reasons for this:
* fragmented packets
* out of order packets

You can see in ipsec stats a lot of state-sequence-errors, which means that packets arrive out of order and get dropped by ipsec driver.

If it is UDP traffic make sure that you send packets with size that do fit without fragmentation. If it is TCP reduce TCP MSS until packet is not fragmented.
 
XTX
newbie
Posts: 27
Joined: Sat Jun 04, 2011 4:34 pm

Re: Cloud Core IPSEC performance

Wed Jul 15, 2015 5:59 pm

Two reasons for this:
* fragmented packets
* out of order packets

You can see in ipsec stats a lot of state-sequence-errors, which means that packets arrive out of order and get dropped by ipsec driver.

If it is UDP traffic make sure that you send packets with size that do fit without fragmentation. If it is TCP reduce TCP MSS until packet is not fragmented.
So if I have a road warrior setup l2tp/ipsec and I'm seeing a lot of state-sequence-errors on the MT, then I have to reduce the TCP MSS on the client machines that connect to the MT ?
 
royalpublishing
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Sep 23, 2013 5:47 pm

Re: Cloud Core IPSEC performance

Wed Jul 15, 2015 7:21 pm

Two reasons for this:
* fragmented packets
* out of order packets

You can see in ipsec stats a lot of state-sequence-errors, which means that packets arrive out of order and get dropped by ipsec driver.

If it is UDP traffic make sure that you send packets with size that do fit without fragmentation. If it is TCP reduce TCP MSS until packet is not fragmented.
Unfortunately, at one of my remote testing sites using IPSec and IPIP tunnels, I already have reduced the MSS to 1418 and have tried the new MSS clamping feature and see the same speed results as mentioned earlier. Any other suggestions?
 
royalpublishing
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Sep 23, 2013 5:47 pm

Re: Cloud Core IPSEC performance

Thu Jul 16, 2015 5:09 pm

Two reasons for this:
* fragmented packets
* out of order packets

You can see in ipsec stats a lot of state-sequence-errors, which means that packets arrive out of order and get dropped by ipsec driver.

If it is UDP traffic make sure that you send packets with size that do fit without fragmentation. If it is TCP reduce TCP MSS until packet is not fragmented.
Unfortunately, at one of my remote testing sites using IPSec and IPIP tunnels, I already have reduced the MSS to 1418 and have tried the new MSS clamping feature and see the same speed results as mentioned earlier. Any other suggestions?
After toying around a bit and scouring the dozens of posts on this forum that contains the words "ipsec poor performance" I found out that by changing the auth and encryption algorithms from sha1 and 3des to sha512 and aes cbc 256 I was effectively able to up my maximum bandwidth to around 42.5Mbps which is an improvement over the 30Mbps I was seeing before. Now if Mikrotik could only work on getting the 3.4 Gbps of IPSec VPN throughput that they advertised on other forum posts that would be fantastic.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7195
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Cloud Core IPSEC performance

Fri Jul 17, 2015 12:29 pm

Hardware encryption is enabled ONLY when SHA1 and AES-CBC algorithms are used.

sha256 3des aes-gcm etc will work in software.
 
royalpublishing
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Sep 23, 2013 5:47 pm

Re: Cloud Core IPSEC performance

Fri Jul 17, 2015 6:42 pm

Hardware encryption is enabled ONLY when SHA1 and AES-CBC algorithms are used.

sha256 3des aes-gcm etc will work in software.
There's something extremely disturbing about the fact that when I enable "hardware encryption" using the specified algorithms above I get worse speeds than before. In my testing, when I tried SHA1 with AES-CBC 128,192, or 256, I only get a maximum 15 Mbps of throughput in one direction. Sorry if it seems I may have hijacked this thread.
 
mikruser
Long time Member
Long time Member
Posts: 578
Joined: Wed Jan 16, 2013 6:28 pm

Re: Cloud Core IPSEC performance

Tue Jun 07, 2016 4:52 pm

Two reasons for this:
* fragmented packets
* out of order packets
May be you will stop lying, and agree that two reason is:
* CCR hardware is heap of shit
* ROS code for CCR is heap of shit
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 2185
Joined: Mon Jan 14, 2008 1:53 pm
Location: Over the Rainbow
Contact:

Re: Cloud Core IPSEC performance

Wed Jun 08, 2016 1:35 am

Hardware encryption is enabled ONLY when SHA1 and AES-CBC algorithms are used.

sha256 3des aes-gcm etc will work in software.
If only this was easy to see in the UI... something like showing which algorithms are hardware accelerated in System/Routerboard or System/Resources.

Even pfsense does this!