MikroTik

2) How could I obtain promised 450Mbit of IPSec throughput? Currently I have only about 130Mbit.

iperf -c <ip_of_plain_server> -B <ip_of_strongswan_server> -t 30 -l 1400
...
[  3]  0.0-30.0 sec   550 MBytes   154 Mbits/sec
iperf -c <ip_of_plain_server> -B <ip_of_strongswan_server> -t 30 -l 1400 -P 10
...
[SUM]  0.0-30.0 sec   518 MBytes   145 Mbits/sec

iperf -c <ip_of_strongswan_server> -B <ip_of_plain_server> -t 30 -l 1400
...
[  3]  0.0-30.0 sec   743 MBytes   208 Mbits/sec
iperf -c <ip_of_strongswan_server> -B <ip_of_plain_server> -t 30 -l 1400 -P 10
...
[SUM]  0.0-30.0 sec   601 MBytes   168 Mbits/sec

iperf -c <ip_of_plain_server> -B <ip_of_strongswan_server> -t 30 -l 1400
...
[  3]  0.0-30.0 sec   351 MBytes  98.0 Mbits/sec
iperf -c <ip_of_strongswan_server> -B <ip_of_plain_server> -t 30 -l 1400
...
[  3]  0.0-30.0 sec   276 MBytes  77.2 Mbits/sec
iperf -c <ip_of_plain_server> -B <ip_of_strongswan_server> -t 30 -l 1400 -P 10
...
[SUM]  0.0-30.0 sec   348 MBytes  97.1 Mbits/sec
iperf -c <ip_of_strongswan_server> -B <ip_of_plain_server> -t 30 -l 1400 -P 10
...
[SUM]  0.0-30.0 sec   201 MBytes  56.3 Mbits/sec

HW Crypto Engine 200 Mbps IPSec throughput

P.S. I'm wondering how could Mikrotik get 400+ Mbits on this chip, when vendor tell us:
http://www.mediatek.com/en/products/con ... /mt7621na/

HW Crypto Engine 200 Mbps IPSec throughput

¯\_(ツ)_/¯

P.P.S.
Hovever, I have one hypothesis.
CPU of RB750Gr3 was loaded by about 48%.
Seems to me, we have another free core which could also be loaded by IPSec, but I'm not sure what do I need - another tunnel or what?

Yes, that looks close to the limit. What packet size did you use?

BTW this is a RB750Gr3 topic not RB850.

Please use English not Russian. English is the official language for this forum.

Yes, that looks close to the limit. What packet size did you use?

root@iperf1:~# iperf3 -c 172.17.18.200 -V
iperf 3.0.7
Linux iperf1 4.4.19-1-pve #1 SMP Wed Sep 14 14:33:50 CEST 2016 x86_64 GNU/Linux
Time: Wed, 26 Oct 2016 08:41:13 GMT
Connecting to host 172.17.18.200, port 5201
      Cookie: iperf1.1477471273.352359.6942f333580
      TCP MSS: 1386 (default)
[  4] local 172.17.19.200 port 45350 connected to 172.17.18.200 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  53.5 MBytes   449 Mbits/sec   17    322 KBytes       
[  4]   1.00-2.00   sec  52.7 MBytes   442 Mbits/sec    0    426 KBytes       
[  4]   2.00-3.00   sec  51.9 MBytes   436 Mbits/sec    7    344 KBytes       
[  4]   3.00-4.00   sec  52.5 MBytes   440 Mbits/sec    5    338 KBytes       
[  4]   4.00-5.00   sec  53.4 MBytes   448 Mbits/sec    1    319 KBytes       
[  4]   5.00-6.00   sec  52.8 MBytes   443 Mbits/sec    5    208 KBytes       
[  4]   6.00-7.00   sec  51.9 MBytes   435 Mbits/sec    0    348 KBytes       
[  4]   7.00-8.00   sec  53.4 MBytes   448 Mbits/sec    2    326 KBytes       
[  4]   8.00-9.00   sec  52.2 MBytes   438 Mbits/sec    2    300 KBytes       
[  4]   9.00-10.00  sec  53.2 MBytes   446 Mbits/sec    0    409 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   527 MBytes   442 Mbits/sec   39             sender
[  4]   0.00-10.00  sec   525 MBytes   441 Mbits/sec                  receiver
CPU Utilization: local/sender 1.2% (0.1%u/1.1%s), remote/receiver 3.3% (0.3%u/3.0%s)

@dhoulbrooke

do you get 400+ throughput regardless of the direction of iperf traffic flow? I assume yes since your test using two RB750r3 - one will be in opposite direction of the other at any time

What CPU utilisation do you see on both RB750r3 when you hit 400+? thanks

CPU/IPsec performance looks great, missing switch features doesn't. Cancelled my order today because of this :'(

What CPU utilisation do you see on both RB750r3 when you hit 400+? thanks

Hopefully a version out with an SFP port.

Hopefully a version out with an SFP port.

+1

I feel sorry for Mikrotik, they release a great new product and straight away people are wanting more features !

BlackVS, test, please, LT2P/IPsec with AES-256. About your stranges with CPU, I think this is global firmware bug, I found it at all firmwares after 6.34.4: http://forum.mikrotik.com/viewtopic.php?t=110714. Support say, that all is ok, buy more powerful router. As we see, most very powerful routers has likely bugs, so number of cores and frequency give, because some developers cannot use it.

Redo the test without fragmentation and you will see different result.

Slow TCP might be due to CCR reordering packets.

Hopefully a version out with an SFP port.

+1

I feel sorry for Mikrotik, they release a great new product and straight away people are wanting more features !

Frankly I find to have 1 SFP port a must these days. I can always add a switch to an ethernet.

I am not impressed with the new RB750Gr3. Mainly, a lot of things working differently.
Cannot make static address in DHCP, when address is not in any pool, .
The worst thing, if i change DHCP range on Quick Set page and click on Apply, lost it's ip address and the device will be inaccessible. Then reset, load last working configuration and go on...
Waiting for a new, working release.

I am not impressed with the new RB750Gr3. Mainly, a lot of things working differently.
Cannot make static address in DHCP, when address is not in any pool, .
The worst thing, if i change DHCP range on Quick Set page and click on Apply, lost it's ip address and the device will be inaccessible. Then reset, load last working configuration and go on...
Waiting for a new, working release.

Could you try it again with a more current version of Winbox?

sfp port ? what for ? SFP+ in 10GBps -capable devices make sense for uplink interfaces, but for generic devices - not really any use for.
in such case - simpler to use media converter. unless if you need to use something exotic, like SFP/SFP+ -cased VDSL modem, EPON interface or LTE interface, but thats niche application and solvable w/o SFP usage.
CCR's had SFP and some of RB3011 and RB2011 if you desperately need it

Redo the test without fragmentation and you will see different result.

Sure.
But... hm... in such case we will get results not from real world due to I have complex network and real clients send usual ethernet MTU packets.
Of course fragmentation take place and in the case of non-fragmentation we will get better results.

I suspect that your issue is more likely related to 6.38.rc19. I have been trying to build a config on an old RB750GL, in preparation for a new RB750Gr3, and it was giving me fits taking some configuration elements (mostly IPv6 related). The router becomes inaccessible an eventually restarts with completely blank config.

Anyway, can you downgrade the RB750Gr3 to 6.37.1 and try?

I am not impressed with the new RB750Gr3. Mainly, a lot of things working differently.
Cannot make static address in DHCP, when address is not in any pool, .
The worst thing, if i change DHCP range on Quick Set page and click on Apply, lost it's ip address and the device will be inaccessible. Then reset, load last working configuration and go on...
Waiting for a new, working release.

Could you try it again with a more current version of Winbox?

The result is same with Webfig and the latest version of Winbox.

Never use quickset

My only disappointment is the 16MB flash. I'm a huge proponent of partitions. There have been times where I have had to revert back after upgrading, due to a bug. Nothing like being able to just boot the other partition! I would be willing to pay the extra $2 to have 32 or 64MB flash.

My only disappointment is the 16MB flash. I'm a huge proponent of partitions. There have been times where I have had to revert back after upgrading, due to a bug. Nothing like being able to just boot the other partition! I would be willing to pay the extra $2 to have 32 or 64MB flash.

Amen .. +1

I like A-B booting in principle (e.g on Chromebooks), but how would it work in this case, would revert to old partition be automated or manual - how would it be triggered? A software watchdog? A hardware watchdog? A switch on the outside? Would an admin need to confirm new version is alright by issuing an explicit command?

sfp port ? what for ? SFP+ in 10GBps -capable devices make sense for uplink interfaces, but for generic devices - not really any use for.
in such case - simpler to use media converter. unless if you need to use something exotic, like SFP/SFP+ -cased VDSL modem, EPON interface or LTE interface, but thats niche application and solvable w/o SFP usage.
CCR's had SFP and some of RB3011 and RB2011 if you desperately need it

SFP is just another interface type, we use it extensively for runs where conduits are tight, TV over fibre, distance etc. SFP doesn't just mean FAST speeds. It is merely another form of connectivity. I prefer not installing media converters as it is just another thing that requires power and space. I do like the Hex POE with SFP that came out recently, have not gotten my hands on one yet - but that would fit our installs nicely (except the pasive POE) - hence why a non POE RB750Gr3 with an SFP would be great.

technically its "interface for interface or interfaces" with full case/mount for devices in.
SPF+ had other advantages aside speed.
for lower than 10Gbps speed there is no need for SFP/SFP+ port, basically.
surely many "prefer not" many thing, and expect it "as granted" within devices for prices as they are without it. and then other consumers start become whine about their over-pricing or more frequently downperforming, because money saved on essentials from design/production budget, wasted for irrelevant for 90% uses(outside backbone or SMB border of med-size or huge networks), SFP/SFP+ interfaces.
personally i would like routers to be able to make me coffee and tell me fresh anecdotes and help me catch cool chicks and do it for free, of course, but thats not likely to happen. same with SFP within SOHO or most SMB devices.
but talking practically - you may notice that RB 2011/RB3011 and CCR(and some of PPC devices?)have SFP ports, so likely they may design later such model you requested, just within another product line.
likely it would be something like RB2011r2 or something like that. MIkrotik tend to hade several Different models: 1. one most basicmodelwithout wifi or SFP, with slightly downclocked(no need for relevant CPU, happly) and bit smaller RAM(bottleneck, so make much sense too). 2. one model with wifi. 3. one model with wifi and sfp. 4. one model with wifi, sfp 5. and one model with SFP only. but it would be labeled or cased as rb750Gr3 or alike that but separated products, i guess. also as i pointed 4x-core chip on similar core(cost as much as already used, one watt hungrier)also exist and open some options too.

for lower than 10Gbps speed there is no need for SFP/SFP+ port, basically.

I just did a quick test with 2x RB750Gr3 in tunnel mode and was able to quite happily get ~436Mbps:

Note this was running RouterOS 6.38rc15.

user@IngTegration:~$ iperf3  -c 172.16.1.120
Connecting to host 172.16.1.120, port 5201
[  4] local 192.168.89.224 port 47172 connected to 172.16.1.120 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  41.6 MBytes   349 Mbits/sec   10    382 KBytes       
[  4]   1.00-2.00   sec  40.8 MBytes   342 Mbits/sec    1    362 KBytes       
[  4]   2.00-3.00   sec  41.4 MBytes   347 Mbits/sec    1    348 KBytes       
[  4]   3.00-4.00   sec  40.5 MBytes   339 Mbits/sec    3    329 KBytes       
[  4]   4.00-5.00   sec  40.3 MBytes   338 Mbits/sec   10    335 KBytes       
[  4]   5.00-6.00   sec  41.3 MBytes   346 Mbits/sec    5    318 KBytes       
[  4]   6.00-7.00   sec  39.5 MBytes   332 Mbits/sec   19    311 KBytes       
[  4]   7.00-8.00   sec  40.1 MBytes   337 Mbits/sec    2    293 KBytes       
[  4]   8.00-9.00   sec  39.6 MBytes   332 Mbits/sec    1    277 KBytes       
[  4]   9.00-10.00  sec  40.1 MBytes   337 Mbits/sec    0    372 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   405 MBytes   340 Mbits/sec   52             sender
[  4]   0.00-10.00  sec   403 MBytes   338 Mbits/sec                  receiver

/ip ipsec proposal
set [ find default=yes ] enc-algorithms=3des

/ip ipsec peer
add address=172.16.2.1/32 enc-algorithm=3des nat-traversal=no secret=password

/ip ipsec policy
set 0 disabled=yes dst-address=0.0.0.0/0 src-address=0.0.0.0/0
add dst-address=172.16.2.1/32 sa-dst-address=172.16.2.1 sa-src-address=172.16.2.3 src-address=172.16.2.3/32 tunnel=yes

I wonder how you can reach those numbers. I'm testing with ROS 6.38rc31 and i get:

/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc

Having VTI support would mean that GRE is not required to run OSPF between the IPSEC endpoints, and in turn remove that overhead.

RB750Gr3 switch chip does not have full VLAN tagging/untagging support yet, it is planned to implement it in future. Currently, you should use RB750Gr3 switch chip only for basic switching.
RB750Gr3 is also powerful enough to handle software based VLANs: http://wiki.mikrotik.com/wiki/Manual:In ... p_examples

/interface bridge
add name=br-vlan5
/ip dhcp-client
add default-route-distance=0 dhcp-options=hostname,clientid disabled=no interface=ether1

/interface bridge
add name=br-vlan5
/ip dhcp-client
add dhcp-options=hostname,clientid disabled=no interface=ether1
/interface vlan
add interface=ether1name=ether1-vlan5 vlan-id=5
/interface bridge port
add bridge=br-vlan5 interface=ether1-vlan5

...after that i updated the Firmware too...

...after that i updated the Firmware too...

Why did you have to upgrade the firmware, was there a problem ?
[and now you bricked the device ?]

I've been suspicious about VLANs, since Ubiquiti Edgerouter-x doesn't support them either, Mediatek seems to have screwed the pooch on delivering well documented switch drivers to their customers.

I have tested with Netinstall but im not running with Windows, so i have to try a other Computer. But the Bootloop is so fast that the Link on the Cable not seem coming before the Board restart, i have to tried that with a Windows Computer and NetInstall again, i hope that this will work :-/

So just to confirm what i am reading, Hex v3 to Strongswan ipsec in transport will yeild somewhere around 100Mbits, and in tunnel can yield more? I want to put one of these in front of my 2011 for the aes128 HW encryption wanna make sure i can hit at least 75/75

So just to confirm what i am reading, Hex v3 to Strongswan ipsec in transport will yeild somewhere around 100Mbits, and in tunnel can yield more? I want to put one of these in front of my 2011 for the aes128 HW encryption wanna make sure i can hit at least 75/75

Finally received a RB750Gr3 yesterday to also replace a RB2011, these are my results for a gre over aes256-sha1-modp4096 ipsec tunnel to strongswan on a 180/30Mbit/s connection (cpu usage ~50%, probably limited to 1 core?):

Not bad at all. Wish they'd make a "cheap rackmount" version of this too!

Not bad at all. Wish they'd make a "cheap rackmount" version of this too!

Maybe someone will make a bracket that can hold 1-3 of these in a 1U panel?
Could be made to fit some other MikroTik models as well...

(cpu usage ~50%, probably limited to 1 core?)

So just to confirm what i am reading, Hex v3 to Strongswan ipsec in transport will yeild somewhere around 100Mbits, and in tunnel can yield more? I want to put one of these in front of my 2011 for the aes128 HW encryption wanna make sure i can hit at least 75/75

Finally received a RB750Gr3 yesterday to also replace a RB2011, these are my results for a gre over aes256-sha1-modp4096 ipsec tunnel to strongswan on a 180/30Mbit/s connection (cpu usage ~50%, probably limited to 1 core?):

Not bad at all. Wish they'd make a "cheap rackmount" version of this too!

So just to confirm what i am reading, Hex v3 to Strongswan ipsec in transport will yeild somewhere around 100Mbits, and in tunnel can yield more? I want to put one of these in front of my 2011 for the aes128 HW encryption wanna make sure i can hit at least 75/75

Finally received a RB750Gr3 yesterday to also replace a RB2011, these are my results for a gre over aes256-sha1-modp4096 ipsec tunnel to strongswan on a 180/30Mbit/s connection (cpu usage ~50%, probably limited to 1 core?):

Not bad at all. Wish they'd make a "cheap rackmount" version of this too!

Thanks for this response! just placed my order, ive decided to just place it in front of my 2011 and just do a dual NAT thing so the 2011 will do most of the routing to network, and all the Hex needs to do is pass route to it.

Well that seems slightly silly. Why the double NAT? Just have the Hex do all the routing and turn the 2011 into a switch and AP (if your's is wireless). The Hex has a better CPU and more memory.

Could be due to (absence of) /system routerboard settings boot-delay feature.

This time is needed to initialize interface etc.

Try adding a delay in your script, e.g.:

Code: Select all

# mar/02/2016 04:00:00 by RouterOS 6.35rc15
# software id = XXXX-XXXX
#
:delay 15s
/interface bridge
add name=bridge-lan

So just to confirm what i am reading, Hex v3 to Strongswan ipsec in transport will yeild somewhere around 100Mbits, and in tunnel can yield more? I want to put one of these in front of my 2011 for the aes128 HW encryption wanna make sure i can hit at least 75/75

Finally received a RB750Gr3 yesterday to also replace a RB2011, these are my results for a gre over aes256-sha1-modp4096 ipsec tunnel to strongswan on a 180/30Mbit/s connection (cpu usage ~50%, probably limited to 1 core?):

Can anybody help?

RB750Gr3 switch chip does not have full VLAN tagging/untagging support yet, it is planned to implement it in future. Currently, you should use RB750Gr3 switch chip only for basic switching.
RB750Gr3 is also powerful enough to handle software based VLANs: http://wiki.mikrotik.com/wiki/Manual:In ... p_examples

RB750Gr3 switch chip does not have full VLAN tagging/untagging support yet, it is planned to implement it in future. Currently, you should use RB750Gr3 switch chip only for basic switching.
RB750Gr3 is also powerful enough to handle software based VLANs: http://wiki.mikrotik.com/wiki/Manual:In ... p_examples

RB750Gr3 switch chip does not have full VLAN tagging/untagging support yet, it is planned to implement it in future. Currently, you should use RB750Gr3 switch chip only for basic switching.