Community discussions

MikroTik App
  4. Other topics
  5. RouterBOARD hardware

RB1100AHx4 IPsec site-to-site performance

Post Reply
 
miguelgoncalves
just joined
Topic Author
Posts: 11
Joined: Tue Jul 19, 2011 12:43 am

RB1100AHx4 IPsec site-to-site performance

Tue Nov 23, 2021 2:32 pm

Good morning!

I am running two RB1100AHx4 (firmware 6.47.10) within the same ISP with 2 ms latency between them. Both routers are connected at 1 Gbps to the ISP.

When establishing a GRE tunnel with IPsec I can't get more than 500 Mbps on a single stream between two servers: one behind each RB1100.

It seems the decryption is limited to a single core (ingress traffic from remote site over the tunnel):
Code: Select all
[admin@edge1] > system resource cpu print interval=1
 # CPU                                             LOAD         IRQ        DISK
 0 cpu0                                              2%          2%          0%
 1 cpu1                                              7%          4%          0%
 2 cpu2                                              1%          1%          0%
 3 cpu3                                             99%         99%          0%

When traffic goes to the remote site it is distributed across the cores:
Code: Select all
[admin@edge1] > system resource cpu print interval=1
 # CPU                                             LOAD         IRQ        DISK
 0 cpu0                                              1%          1%          0%
 1 cpu1                                             53%         52%          0%
 2 cpu2                                              0%          0%          0%
 3 cpu3                                             17%         16%          0%

[admin@edge1] > system resource cpu print interval=1
 # CPU                                             LOAD         IRQ        DISK
 0 cpu0                                              4%          4%          0%
 1 cpu1                                             51%         50%          0%
 2 cpu2                                              1%          1%          0%
 3 cpu3                                             16%         10%          0%

[admin@edge1] > system resource cpu print interval=1
 # CPU                                             LOAD         IRQ        DISK
 0 cpu0                                              1%          1%          0%
 1 cpu1                                             48%         46%          0%
 2 cpu2                                              2%          2%          0%
 3 cpu3                                             16%         14%          0%

The policies and proposals are the same at both ends:
Code: Select all
/ip ipsec profile
set [ find default=yes ] dh-group=modp2048 enc-algorithm=aes-256 \
    hash-algorithm=sha256 lifetime=8h
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc \
    lifetime=1h pfs-group=modp2048

And the hardware accelaration is being used:
Code: Select all
[admin@edge1] > ip ipsec installed-sa print count-only where state=mature
24
[admin@edge1] > ip ipsec installed-sa print where !hw-aead
Flags: H - hw-aead, A - AH, E - ESP
[admin@edge1] >

Also there does not seems packets are being fragmented:
Code: Select all
[admin@edge1] > tool sniffer quick interface=ether1-xxx-aa ip-address=XXX.YYY.AA.BB
INTERFACE         TIME    NUM DIR SRC-MAC           DST-MAC           VLAN   SRC-ADDRESS          DST-ADDRESS       PROTOCOL   SIZE CPU FP
ether1-xxx-aa    0.997  36562 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.997  36563 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.997  36564 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.997  36565 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.997  36566 ->  22:22:22:22:22:22 11:11:11:11:11:11        TT.VV.123.456        XXX.YYY.AA.BB     ip:ipse...  122   1 no
ether1-xxx-aa    0.997  36567 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.997  36568 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.997  36569 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.997  36570 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.998  36571 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.998  36572 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.998  36573 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.998  36574 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.998  36575 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.998  36576 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no
ether1-xxx-aa    0.998  36577 <-  11:11:11:11:11:11 22:22:22:22:22:22        XXX.YYY.AA.BB        TT.VV.123.456     ip:ipse... 1514   3 no

I tried disabling the IPsec encryption and I can reach gigabit line speed. So it seems it's the decryption that is slowing things down.

Has anyone been able to reach gigabit VPN speeds? Is there a more performant router that can do this? Should I switch to CHR?

Thanks!
Top
 
mikruser
Long time Member
Long time Member
Posts: 578
Joined: Wed Jan 16, 2013 6:28 pm

Re: RB1100AHx4 IPsec site-to-site performance

Tue Nov 23, 2021 3:21 pm

Try aes-128 gcm
https://en.wikipedia.org/wiki/Galois/Counter_Mode
Top
 
miguelgoncalves
just joined
Topic Author
Posts: 11
Joined: Tue Jul 19, 2011 12:43 am

Re: RB1100AHx4 IPsec site-to-site performance

Wed Nov 24, 2021 1:09 pm

It does help a bit but very far from the 1 Gbps I need.

In a lab environment I set up 2 RB1100AHx4 and tried the bandwidth test between them with IPsec on and off. With IPsec on the performance drops to half.
Code: Select all
#
# Router R1
#
/interface gre
add name=gre-tunnel1 remote-address=10.0.0.2
/interface ethernet switch port
set 0 default-vlan-id=0
set 1 default-vlan-id=0
set 2 default-vlan-id=0
set 3 default-vlan-id=0
set 4 default-vlan-id=0
set 5 default-vlan-id=0
set 6 default-vlan-id=0
set 7 default-vlan-id=0
set 8 default-vlan-id=0
set 9 default-vlan-id=0
set 10 default-vlan-id=0
set 11 default-vlan-id=0
set 12 default-vlan-id=0
set 13 default-vlan-id=0
set 14 default-vlan-id=0
set 15 default-vlan-id=0
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip address
add address=10.0.0.1/30 interface=ether1 network=10.0.0.0
add address=192.168.1.1/30 interface=gre-tunnel1 network=192.168.1.0
/system identity
set name=R1
/tool bandwidth-server
set authenticate=no

Code: Select all
# 
# Router R2
#
/interface gre
add name=gre-tunnel1 remote-address=10.0.0.1
/interface ethernet switch port
set 0 default-vlan-id=0
set 1 default-vlan-id=0
set 2 default-vlan-id=0
set 3 default-vlan-id=0
set 4 default-vlan-id=0
set 5 default-vlan-id=0
set 6 default-vlan-id=0
set 7 default-vlan-id=0
set 8 default-vlan-id=0
set 9 default-vlan-id=0
set 10 default-vlan-id=0
set 11 default-vlan-id=0
set 12 default-vlan-id=0
set 13 default-vlan-id=0
set 14 default-vlan-id=0
set 15 default-vlan-id=0
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip address
add address=10.0.0.2/30 interface=ether1 network=10.0.0.0
add address=192.168.1.2/30 interface=gre-tunnel1 network=192.168.1.0
/system identity
set name=R2
/tool bandwidth-server
set authenticate=no

Results:
Code: Select all
[admin@R2] > tool bandwidth-test address=10.0.0.1 protocol=tcp
                status: running
              duration: 30s
            rx-current: 934.0Mbps
  rx-10-second-average: 941.6Mbps
      rx-total-average: 941.6Mbps
           random-data: no
             direction: receive
      connection-count: 20
        local-cpu-load: 55%

[admin@R2] > tool bandwidth-test address=192.168.1.1 protocol=tcp
                status: running
              duration: 30s
            rx-current: 927.7Mbps
  rx-10-second-average: 926.6Mbps
      rx-total-average: 925.9Mbps
           random-data: no
             direction: receive
      connection-count: 20
        local-cpu-load: 49%

So, not a big speed difference when going over the GRE tunnel (192.168.1.1) or over the directly connected ethernet ports (10.0.0.1).

To enable IPsec just issued the following on both routers:
Code: Select all
#
# Router R1
#
/interface gre
add allow-fast-path=no ipsec-secret=kunyev4mtcmtxhcoen5c7y456 name=gre-tunnel1 remote-address=10.0.0.2
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=null enc-algorithms=aes-128-gcm

#
# Router R2
#
/interface gre
add allow-fast-path=no ipsec-secret=kunyev4mtcmtxhcoen5c7y456 name=gre-tunnel1 remote-address=10.0.0.1
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=null enc-algorithms=aes-128-gcm

And the difference is dramatic (941.7 Mbps between the ethernet ports and 533.8 Mbps between the GRE/IPsec endpoints - it was 925.9 Mbps without encryption):
Code: Select all
[admin@R2] > tool bandwidth-test address=10.0.0.1 protocol=tcp             
                status: running
              duration: 30s
            rx-current: 943.0Mbps
  rx-10-second-average: 943.3Mbps
      rx-total-average: 941.7Mbps
           random-data: no
             direction: receive
      connection-count: 20
        local-cpu-load: 53%

[admin@R2] > tool bandwidth-test address=192.168.1.1 protocol=tcp                                                        
                status: running
              duration: 30s
            rx-current: 536.3Mbps
  rx-10-second-average: 530.5Mbps
      rx-total-average: 533.8Mbps
           random-data: no
             direction: receive
      connection-count: 20
        local-cpu-load: 39%

Can anyone from Mikrotik shed a light here?

What speeds can the hardware acceleration reach?

Thanks in advance!
Top
 
mikruser
Long time Member
Long time Member
Posts: 578
Joined: Wed Jan 16, 2013 6:28 pm

Re: RB1100AHx4 IPsec site-to-site performance

Wed Nov 24, 2021 1:35 pm

You can create a ticket to technical support (but I know in advance that they will answer you)
"Adding or enabling any additional RouterOS feature apart from IPsec policies can reduce the throughput significantly." (c) emils
You can read viewtopic.php?t=97880 and add your results.

i recommend you use CHR. I already use several CHR routers on 1Gbps WAN links with GRE+IPsec tunnels (aes-128 gcm). Even with 1vCPU routers can 950Mbps.
Top
 
atakacs
Member Candidate
Member Candidate
Posts: 121
Joined: Mon Mar 07, 2016 5:39 pm

Re: RB1100AHx4 IPsec site-to-site performance

Thu Nov 25, 2021 10:09 pm

i recommend you use CHR. I already use several CHR routers on 1Gbps WAN links with GRE+IPsec tunnels (aes-128 gcm). Even with 1vCPU routers can 950Mbps.
That's rather impressive.
Top
Post Reply
  4. Other topics
  5. RouterBOARD hardware