RB750Gr3 - Report and questions

godlike · Fri Oct 21, 2016 1:08 am

I boght brand new RB750Gr3 aka new HEx.
Upgraded to 6.37.1 RouterOS(also tried 6.38r15)

1) Switch functionality does not work - there is no vlan support, could not modify Ethernet header and so on.
Example:

/interface ethernet switch port> set 4 vlan-header=add-if-missing 
failure: vlan header mode not supported

2) Switch is not shown in Winbox:

Hovever, switch could be obtained via CLI:

/interface ethernet switch> print
Flags: I - invalid 
 #   NAME         TYPE             MIRROR-SOURCE       MIRROR-TARGET       SWITCH-ALL-PORTS
 0   switch1      MediaTek-MT7621  none                none

Switch is also absent in WebFig
3) Profile does not work:
All the time I can only see:

4) Hardware encryption working not as promised:
a) one 100Mbit tunnel:

iperf -c 10.254.201.2 -l 1400 -t 30 -P 10
...
[SUM]  0.0-19.1 sec   202 MBytes  89.0 Mbits/sec

b) two simultaneous 100Mbit tunneles:

iperf -c 10.254.201.2 -l 1400 -t 30 -P 10
...
[SUM]  0.0-30.1 sec   226 MBytes  63.0 Mbits/sec
iperf -c 10.254.200.2 -l 1400 -t 30 -P 10 -p 5002
...
[SUM]  0.0-30.1 sec   245 MBytes  68.2 Mbits/sec

result repeated many times
CPU load during these tests:

My IPSec settings:

/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc
/ip ipsec peer
add address=x.x.x.x/32 enc-algorithm=aes-128 local-address=y.y.y.y nat-traversal=no secret=***
add address=x.x.x.x/32 enc-algorithm=aes-128 local-address=z.z.z.z nat-traversal=no secret=***
/ip ipsec policy
add dst-address=x.x.x.x/32 protocol=gre sa-dst-address=x.x.x.x sa-src-address=y.y.y.y src-address=y.y.y.y/32
add dst-address=x.x.x.x/32 protocol=gre sa-dst-address=x.x.x.x sa-src-address=z.z.z.z src-address=z.z.z.z/32
/ip ipsec proposal> print
Flags: X - disabled, * - default 
 0  * name="default" auth-algorithms=sha1 enc-algorithms=aes-128-cbc lifetime=30m pfs-group=modp1024

So, I have few questions to Mikrotik staff:
1) Will RB750Gr3 have full Switch functionality, or, at least, vlan support? As I digged internet, vlan(and even q-in-q) functionality is working in openwrt for this chipset.
2) How could I obtain promised 450Mbit of IPSec throughput? Currently I have only about 130Mbit.
3) Will OpenVPN be able to use hardware acceleration?
Thank you in advance for your answers!

I could answer any questions regarding this device if you guys have ones.
P.S. packet reordering from this post:
http://forum.mikrotik.com/viewtopic.php?t=112545
does not influence RB750GR3:

ping -c 10 -l 10 10.254.201.2
PING 10.254.201.2 (10.254.201.2) 56(84) bytes of data.
64 bytes from 10.254.201.2: icmp_seq=1 ttl=63 time=48.1 ms
64 bytes from 10.254.201.2: icmp_seq=2 ttl=63 time=48.2 ms
64 bytes from 10.254.201.2: icmp_seq=3 ttl=63 time=48.3 ms
64 bytes from 10.254.201.2: icmp_seq=4 ttl=63 time=48.3 ms
64 bytes from 10.254.201.2: icmp_seq=5 ttl=63 time=48.4 ms
64 bytes from 10.254.201.2: icmp_seq=6 ttl=63 time=48.5 ms
64 bytes from 10.254.201.2: icmp_seq=7 ttl=63 time=48.7 ms
64 bytes from 10.254.201.2: icmp_seq=8 ttl=63 time=48.8 ms
64 bytes from 10.254.201.2: icmp_seq=9 ttl=63 time=49.0 ms
64 bytes from 10.254.201.2: icmp_seq=10 ttl=63 time=49.1 ms

--- 10.254.201.2 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 48.129/48.586/49.133/0.318 ms, pipe 10

risk · Fri Oct 21, 2016 9:17 am

I've been suspicious about VLANs, since Ubiquiti Edgerouter-x doesn't support them either, Mediatek seems to have screwed the pooch on delivering well documented switch drivers to their customers.

Thank you for confirming.

Meanwhile I can confirm VLANs are working fine on that chipset with OpenWRT on 2 different devices the DIR-860L B1 and MQmaker WiTi development board, hopefully Mikrotik fixes their drivers.

mrz · Fri Oct 21, 2016 11:47 am

2) How could I obtain promised 450Mbit of IPSec throughput? Currently I have only about 130Mbit.

sha1/aes-128-cbc ipsec between two hex v3 devices. Traffic generator running with 1400byte packets.

godlike · Fri Oct 21, 2016 11:52 am

My IPSec tunnels are built between hexv3<->strongswan
Is there a way to improve throughput in my case?
What about other two questions?
Thanks!

P.S.
@mrz, could you please share IPSec config(without confidential information, of course) and RouterOS version from your tests?
Thank you in advance!

mrz · Fri Oct 21, 2016 12:56 pm

There is nothing specific in ipsec configuration, just basic tunnel mode with sha1/aes-128-cbc.
Are you sure that hardware on which you are running openswan is capable of encrypting/decrypting more than 300Mbps?

godlike · Fri Oct 21, 2016 1:02 pm

Thanks for tip, I'll test with more powerful server in local network with hex. (CPU is loaded for about 30% on strongswan side, but who knows)
P.S. I'm using transport mode, i.e. gre-over-ipsec, hope this could not be an issue.

bedior · Fri Oct 21, 2016 6:01 pm

Please, test it on Softether (l2tp/ipsec) with AES-256. It seems ipsec became very bugged after 6.34.4: http://forum.mikrotik.com/viewtopic.php?t=11071, so 400 mb/s is very sintetic, imho.

godlike · Fri Oct 21, 2016 6:37 pm

I'll do these test either today evening or(more probable) tomorrow morning.

godlike · Fri Oct 21, 2016 11:35 pm

Updated results with GRE-over-IPSec:
1) [strongswan_server]>>>[rb750gr3]>>>[plain_server]

iperf -c <ip_of_plain_server> -B <ip_of_strongswan_server> -t 30 -l 1400
...
[  3]  0.0-30.0 sec   550 MBytes   154 Mbits/sec
iperf -c <ip_of_plain_server> -B <ip_of_strongswan_server> -t 30 -l 1400 -P 10
...
[SUM]  0.0-30.0 sec   518 MBytes   145 Mbits/sec

2) [plain_server]>>>[rb750gr3]>>>[strongswan_server]

iperf -c <ip_of_strongswan_server> -B <ip_of_plain_server> -t 30 -l 1400
...
[  3]  0.0-30.0 sec   743 MBytes   208 Mbits/sec
iperf -c <ip_of_strongswan_server> -B <ip_of_plain_server> -t 30 -l 1400 -P 10
...
[SUM]  0.0-30.0 sec   601 MBytes   168 Mbits/sec

3) Simultaneous bidirectional test:
[strongswan_server]>>>[rb750gr3]>>>[plain_server]
AND at the SAME time:
[plain_server]>>>[rb750gr3]>>>[strongswan_server]

iperf -c <ip_of_plain_server> -B <ip_of_strongswan_server> -t 30 -l 1400
...
[  3]  0.0-30.0 sec   351 MBytes  98.0 Mbits/sec
iperf -c <ip_of_strongswan_server> -B <ip_of_plain_server> -t 30 -l 1400
...
[  3]  0.0-30.0 sec   276 MBytes  77.2 Mbits/sec
iperf -c <ip_of_plain_server> -B <ip_of_strongswan_server> -t 30 -l 1400 -P 10
...
[SUM]  0.0-30.0 sec   348 MBytes  97.1 Mbits/sec
iperf -c <ip_of_strongswan_server> -B <ip_of_plain_server> -t 30 -l 1400 -P 10
...
[SUM]  0.0-30.0 sec   201 MBytes  56.3 Mbits/sec

Conclusions:
1) A little bit better, but Mikrotik's marketing(or synthetic, what do you like) results should be divided by 2
2) In case of uni-directional traffic performance is better FROM RB750Gr3 TO target
3) In case of simultaneous bidirectional traffic performance is better FROM target to RB750Gr3
4) Less parallel streams is generally better

P.S. I'm wondering how could Mikrotik get 400+ Mbits on this chip, when vendor tell us:
http://www.mediatek.com/en/products/con ... /mt7621na/

HW Crypto Engine 200 Mbps IPSec throughput

¯\_(ツ)_/¯

P.P.S.
Hovever, I have one hypothesis.
CPU of RB750Gr3 was loaded by about 48%.
Seems to me, we have another free core which could also be loaded by IPSec, but I'm not sure what do I need - another tunnel or what?

kvic · Sun Oct 23, 2016 8:32 am

P.S. I'm wondering how could Mikrotik get 400+ Mbits on this chip, when vendor tell us:
http://www.mediatek.com/en/products/con ... /mt7621na/
HW Crypto Engine 200 Mbps IPSec throughput
¯\_(ツ)_/¯

P.P.S.
Hovever, I have one hypothesis.
CPU of RB750Gr3 was loaded by about 48%.
Seems to me, we have another free core which could also be loaded by IPSec, but I'm not sure what do I need - another tunnel or what?

I registered to give you a response.

That information could be outdated or MediaTek being conservative. MT7621A integrates EIP-93 crypto core from Authentec. EIP-93 is capable of hitting 300-500Mbit/s with AES-128/SHA1 or 450 Kpps 64-byte packets. So 400+ is perfectly possible if doing it right.

I appreciated your real-world or independent tests. If you could try a plain ipsec tunnel and what mrz suggested, and try to push CPU closer to 100%, that'll be great. Then let us know what throughput you achieve. Note that please repeat the test in both directions - one at a time.

thank you

Jeroen1000 · Sun Oct 23, 2016 12:29 pm

Thank you for your tests. They are most welcome. Maybe try IPsec and L2TP. Just use a Windows 7 or higher client to test. Plain IPsec would require a site-to-site tunnel.

Any thoughts on whether this on is faster than a 850Gx2?

bedior · Sun Oct 23, 2016 2:17 pm

Can you test with Softether VPN on L2TP/IPsec AES-256?

janisk · Mon Oct 24, 2016 11:39 am

tests he made was with AES-256, hence the other poster suggested using MRZ parameters - AES-128 - instead.

bedior · Mon Oct 24, 2016 11:52 am

I more interesting with Softether, how it work with this server, not strongswan.

godlike · Mon Oct 24, 2016 11:57 am

@kvic, thanks for tips, I'll make pure IPSec(tunnel mode) in nearest few days.

@bedior I'll make L2TP+Ipsec test also in nearest few days.

@janisk sorry, but my tunnels are AES-128 - please see my first post with my ipsec config.

dhoulbrooke · Tue Oct 25, 2016 8:20 am

Hi all,

I just did a quick test with 2x RB750Gr3 in tunnel mode and was able to quite happily get ~436Mbps:

Accepted connection from 172.17.19.199, port 49824
[  5] local 172.17.18.199 port 5201 connected to 172.17.19.199 port 49825
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  49.2 MBytes   413 Mbits/sec                  
[  5]   1.00-2.00   sec  52.3 MBytes   438 Mbits/sec                  
[  5]   2.00-3.00   sec  52.2 MBytes   438 Mbits/sec                  
[  5]   3.00-4.00   sec  51.8 MBytes   435 Mbits/sec                  
[  5]   4.00-5.00   sec  52.2 MBytes   438 Mbits/sec                  
[  5]   5.00-6.00   sec  52.3 MBytes   439 Mbits/sec                  
[  5]   6.00-7.00   sec  52.5 MBytes   440 Mbits/sec                  
[  5]   7.00-8.00   sec  52.4 MBytes   440 Mbits/sec                  
[  5]   8.00-9.00   sec  52.5 MBytes   440 Mbits/sec                  
[  5]   9.00-10.00  sec  52.4 MBytes   440 Mbits/sec                  
[  5]  10.00-10.04  sec  2.10 MBytes   443 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.04  sec   522 MBytes   436 Mbits/sec                  receiver

Configuration:

[admin@MikroTik] > ip ipsec export 
# oct/25/2016 18:14:14 by RouterOS 6.38rc15
#
/ip ipsec mode-config
set
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc
/ip ipsec peer
add address=172.17.17.1/32 enc-algorithm=aes-128 nat-traversal=no secret=**********************************************
/ip ipsec policy
add dst-address=172.17.18.0/24 sa-dst-address=172.17.17.1 sa-src-address=172.17.17.2 src-address=172.17.19.0/24 tunnel=yes

Note this was running RouterOS 6.38rc15.

mrz · Tue Oct 25, 2016 11:56 am

Yes, that looks close to the limit. What packet size did you use?

колбаскин · Tue Oct 25, 2016 3:36 pm

Yes, that looks close to the limit. What packet size did you use?

How to be to other users who uses RB 850Gx2
Speed from Mikrotik to Softether didn't rise at me higher than 25megabits l2tp+ipsec aes256
ipsec became very bugged after 6.34.4 http://forum.mikrotik.com/viewtopic.php?t=11071
You try to find ideal conditions to reach the declared characteristics.

mrz · Tue Oct 25, 2016 3:56 pm

@колбаскин you didn't bother to reply to supports request, but still keep complaining over and over again in the forum.

So how can we guess what exactly is wrong in your specific conditions?

BTW this is a RB750Gr3 topic not RB850.

godlike · Tue Oct 25, 2016 6:27 pm

Quickly tested tunnel mode:
Strongswan->Mikrotik direction:

[  3]  0.0-10.0 sec   222 MBytes   186 Mbits/sec

Mikrotik->Strongswan direction:

[  3]  0.0-10.0 sec   193 MBytes   161 Mbits/sec

Firmware: 6.37.1
So, slowness of ipsec could be either firmware version related or Strongswan<->Mikrotik pair related.
May be I'll test with trial CHR.

BTW, one of drawbacks of tunnel mode - fasttrack does not work with ipsec in this case, while in transport mode it works.

колбаскин · Tue Oct 25, 2016 6:34 pm

BTW this is a RB750Gr3 topic not RB850.

I answer your support through mail. Only when hasn't received the help
Many people write that they can't receive the declared speed
Write about problems with ipsec after SW of the version 6.34.4
Whether now there is a question the RB750Gr3 model will have same problems with the hardware encoding as at RB 850Gx2.
If you are not going to solve a problem on old models, purchase of new model can will solve a problem

Я извиняюсь за то что пишут не совсем в нужной теме.
Просто увидел реальные тесты скорости. Но скорость тестировалась между двумя моделями RB750Gr3, на сколько я понял
Вы всё же начали задавать вопрос о размере пакетов. Возможно намекая на то что в другом размере пакета скорость была бы выше. Это попытка сделать идеальные условия для большей скорости.
У меня есть и модель 2011 и RB 850Gx2. Я не могу добиться нормальной скорости через l2tp+ipsec к серверам Softether, по крайней мере на upload.
Многократно писали что после версии софта 6.34.4 ipsec перестал загружать весь процессор, соответственно снизилась пропускная способность.
Были отправлены данные для входа на роутер вашему сотруднику и он меньше минуты просмотрел конфигурацию. После чего был получен ответ что всё нормально.
По форуму много сообщений что скорость далека от заявленной. Что вы предлагаете сделать? Смириться и дальше верить в технические характеристики которые вы предоставляете? В суппорте по почте постоянно просят конфигурационный файл и в итоге ничего конкретного не отвечают. На форуме вы выборочно отвечаете на вопросы.
Будьте добры и внесите ясность в реальную работу аппаратного шифрования ipsec на роутерах. Какие скорость и при каких условиях можно получить.
Надоело тратить своё и ваше время на выяснение деталей работы вашего оборудования.

Tue Oct 25, 2016 9:38 pm

Please use English not Russian. English is the official language for this forum.

колбаскин · Tue Oct 25, 2016 9:59 pm

Please use English not Russian. English is the official language for this forum.

It somewhere is written?
For whom it is necessary, those will understand.

Tue Oct 25, 2016 11:01 pm

A. Search the Forum and you will find the answer.
B. I am so far just asking you to use English.
C. We want all to understand what you are writing.
D. If you want to use Russian language please use eg. http://forum.mikrotik.by.
E. Moderators are expected to look after forum rules.

And finally:
Why you think that writing in Russian is better than English on this forum ?
Could we be sure that you are not spmming our forum with some equivocal content ?

Tue Oct 25, 2016 11:21 pm

interesting to see that improvement of hEX on ipsec is real very good to scale the device for new implementations

patrick7 · Tue Oct 25, 2016 11:34 pm

CPU/IPsec performance looks great, missing switch features doesn't. Cancelled my order today because of this :'(

BlackVS · Wed Oct 26, 2016 9:55 am

Some more tests with RB750Gr3 are here:
https://www.mikrotik-club.in.ua/2016/10 ... s/#more-80
In Russian %) but results readable for English speaking guys.

If shortly:
I tested this device as client VPN router not ipsec.
For case of remote office.
Now RB951G is used as client vpn router and PPTP as connection due to highest bandwidth ( ~50M ) comparing over server-to-client protocols (SSTP, OpenVPN - 2-7M for TCP).
GRE+IPSEC showed much lower results comparing PPTP and right now we havn't global IPs to enable direct IPSEC.
For the case of RB750Gr3 as Internet gateway+VPN client results are higher - 70M for PPTP. For the case of OpenVPN/SSTP - they are higher to... but less 10M...
But when functions of Internet gateway and VPN client were split (RB951G gateway, RB750Gr3 as vpn client) - OpenVPN in TCP test showed... 50-70M!
Hmmmmm...

Later I will test GRE+IPSEC AES-256 - becomes "сuriouser and curiouser"...
PS: and yes, I havn't seen CPU load higher 50% on RB750Gr3 - strange...

bedior · Wed Oct 26, 2016 10:14 am

BlackVS, test, please, LT2P/IPsec with AES-256. About your stranges with CPU, I think this is global firmware bug, I found it at all firmwares after 6.34.4: http://forum.mikrotik.com/viewtopic.php?t=110714. Support say, that all is ok, buy more powerful router. As we see, most very powerful routers has likely bugs, so number of cores and frequency give, because some developers cannot use it.

dhoulbrooke · Wed Oct 26, 2016 11:58 am

Yes, that looks close to the limit. What packet size did you use?

Those first tests were run with iperf's default settings which appear to have an MSS of 1386.

root@iperf1:~# iperf3 -c 172.17.18.200 -V
iperf 3.0.7
Linux iperf1 4.4.19-1-pve #1 SMP Wed Sep 14 14:33:50 CEST 2016 x86_64 GNU/Linux
Time: Wed, 26 Oct 2016 08:41:13 GMT
Connecting to host 172.17.18.200, port 5201
      Cookie: iperf1.1477471273.352359.6942f333580
      TCP MSS: 1386 (default)
[  4] local 172.17.19.200 port 45350 connected to 172.17.18.200 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  53.5 MBytes   449 Mbits/sec   17    322 KBytes       
[  4]   1.00-2.00   sec  52.7 MBytes   442 Mbits/sec    0    426 KBytes       
[  4]   2.00-3.00   sec  51.9 MBytes   436 Mbits/sec    7    344 KBytes       
[  4]   3.00-4.00   sec  52.5 MBytes   440 Mbits/sec    5    338 KBytes       
[  4]   4.00-5.00   sec  53.4 MBytes   448 Mbits/sec    1    319 KBytes       
[  4]   5.00-6.00   sec  52.8 MBytes   443 Mbits/sec    5    208 KBytes       
[  4]   6.00-7.00   sec  51.9 MBytes   435 Mbits/sec    0    348 KBytes       
[  4]   7.00-8.00   sec  53.4 MBytes   448 Mbits/sec    2    326 KBytes       
[  4]   8.00-9.00   sec  52.2 MBytes   438 Mbits/sec    2    300 KBytes       
[  4]   9.00-10.00  sec  53.2 MBytes   446 Mbits/sec    0    409 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   527 MBytes   442 Mbits/sec   39             sender
[  4]   0.00-10.00  sec   525 MBytes   441 Mbits/sec                  receiver
CPU Utilization: local/sender 1.2% (0.1%u/1.1%s), remote/receiver 3.3% (0.3%u/3.0%s)

I spent a bit of time specifying different MSS numbers in iperf3 but most of the results looked much the same. I really need to learn how to use Traffic Generator.

kvic · Wed Oct 26, 2016 12:20 pm

@godlike

I wonder what limits throughput to <200Mbps in your setup. What are the CPU utilisation you see in both cases?

@dhoulbrooke

do you get 400+ throughput regardless of the direction of iperf traffic flow? I assume yes since your test using two RB750r3 - one will be in opposite direction of the other at any time

What CPU utilisation do you see on both RB750r3 when you hit 400+? thanks

dhoulbrooke · Wed Oct 26, 2016 12:32 pm

@dhoulbrooke

do you get 400+ throughput regardless of the direction of iperf traffic flow? I assume yes since your test using two RB750r3 - one will be in opposite direction of the other at any time

Correct. I get identical speeds both ways.

What CPU utilisation do you see on both RB750r3 when you hit 400+? thanks

Typically in the realm of 60%

ipsec.png

npero · Wed Oct 26, 2016 12:54 pm

CPU/IPsec performance looks great, missing switch features doesn't. Cancelled my order today because of this :'(

What's new in 6.38rc19 (2016-Oct-24 11:19):

*) winbox - fixed missing switch menu for mmips devices;

dhoulbrooke · Wed Oct 26, 2016 1:07 pm

What CPU utilisation do you see on both RB750r3 when you hit 400+? thanks

Also forgot to add that CPU load is slightly less on the RX device. Around 40%

ipsec1.png

MstreaM · Wed Oct 26, 2016 2:03 pm

when expected on sale RB750Gr3 ?

kvic · Thu Oct 27, 2016 7:56 am

@dhoulbrooke thanks for detailed feedback. Very impressive numbers.

Mikrotik, good job!

My only complaint about the new hEX is perhaps the 16MB flash..

paulct · Thu Oct 27, 2016 9:04 am

Hopefully a version out with an SFP port.

Thu Oct 27, 2016 12:45 pm

Hopefully a version out with an SFP port.

+1

I feel sorry for Mikrotik, they release a great new product and straight away people are wanting more features !

paulct · Thu Oct 27, 2016 2:41 pm

Hopefully a version out with an SFP port.
+1

I feel sorry for Mikrotik, they release a great new product and straight away people are wanting more features !

Frankly I find to have 1 SFP port a must these days. I can always add a switch to an ethernet.

BlackVS · Fri Oct 28, 2016 8:28 am

BlackVS, test, please, LT2P/IPsec with AES-256. About your stranges with CPU, I think this is global firmware bug, I found it at all firmwares after 6.34.4: http://forum.mikrotik.com/viewtopic.php?t=110714. Support say, that all is ok, buy more powerful router. As we see, most very powerful routers has likely bugs, so number of cores and frequency give, because some developers cannot use it.

RB951G-2HnD gives (Rx/Tx):
(server side is CCR-1009)

L2tp AES-256 CBC
UDP 21/20 CPU 100%
TCP 17/13.5 CPU 100%

L2tp aes-256 ctr
UDP 20/20 cpu 100%
tcp 11.5/14 CPU 60%/100%

RB750Gr3 has:

l2tp aes-256 cbc
udp 85/3..100 CPU 46%/52%
tcp 39/16 CPU 40%/44%

l2tp aes-256 ctr
udp 26/27 CPU 30%/27%
tcp 30/17 CPU 31%/45%

l2tp aes-128 cbc
udp 85/47(or 3..100) CPU 50/25%
tcp 41/18 CPU 40%/45%

Again using Tools/Bandwith test (due to I havn't here stations and can't run iperf).
Direct test between WANs gives stable TCP Rx/Tx=200M/100M.
I.e.:
A) RB750Gr3 wins %).
B) AES-CBC is better optimized
C) decoding (RX) is faster than encoding (Tx) (or assymetric canal issue)

But!
One strange thing for Tx only seen on RB750Gr3 - during test throughout either stable and equal ~50M or periodically jumps between 2-5M and 100M like this:

bedior · Fri Oct 28, 2016 9:43 am

Thank you. Not so fast as needed.

mrz · Fri Oct 28, 2016 12:06 pm

Redo the test without fragmentation and you will see different result.
Slow TCP might be due to CCR reordering packets.

BlackVS · Fri Oct 28, 2016 8:16 pm

Redo the test without fragmentation and you will see different result.

Sure.
But... hm... in such case we will get results not from real world due to I have complex network and real clients send usual ethernet MTU packets.
Of course fragmentation take place and in the case of non-fragmentation we will get better results.
My aim wasn't to test theoretical maximum perfomance of encryption engine.
My aim was to test how much usual users can get using both devices in their usual environment %)
PS: Bandwidth testing tool in WinBox does not allow to set MTU for TCP. Of course exists iperf. But I couldn't use it during my tests. May be I will test using it later in other environment.

Slow TCP might be due to CCR reordering packets.

Life inside Mikrotik world does not become better with this knowledge %).
Because there is no solution how to avoid/compensate this effect.
Except switching to software coded encryption in some cases.

Zorro · Sat Oct 29, 2016 5:41 am

Hopefully a version out with an SFP port.
+1

I feel sorry for Mikrotik, they release a great new product and straight away people are wanting more features !
Frankly I find to have 1 SFP port a must these days. I can always add a switch to an ethernet.

sfp port ? what for ? SFP+ in 10GBps -capable devices make sense for uplink interfaces, but for generic devices - not really any use for.
in such case - simpler to use media converter. unless if you need to use something exotic, like SFP/SFP+ -cased VDSL modem, EPON interface or LTE interface, but thats niche application and solvable w/o SFP usage.
CCR's had SFP and some of RB3011 and RB2011 if you desperately need it

NobodyHUN · Sat Oct 29, 2016 9:34 pm

I am not impressed with the new RB750Gr3. Mainly, a lot of things working differently.
Cannot make static address in DHCP, when address is not in any pool, .
The worst thing, if i change DHCP range on Quick Set page and click on Apply, lost it's ip address and the device will be inaccessible. Then reset, load last working configuration and go on...
Waiting for a new, working release.

msatter · Sun Oct 30, 2016 12:35 am

I am not impressed with the new RB750Gr3. Mainly, a lot of things working differently.
Cannot make static address in DHCP, when address is not in any pool, .
The worst thing, if i change DHCP range on Quick Set page and click on Apply, lost it's ip address and the device will be inaccessible. Then reset, load last working configuration and go on...
Waiting for a new, working release.

Could you try it again with a more current version of Winbox?

NobodyHUN · Sun Oct 30, 2016 9:58 am

I am not impressed with the new RB750Gr3. Mainly, a lot of things working differently.
Cannot make static address in DHCP, when address is not in any pool, .
The worst thing, if i change DHCP range on Quick Set page and click on Apply, lost it's ip address and the device will be inaccessible. Then reset, load last working configuration and go on...
Waiting for a new, working release.

Could you try it again with a more current version of Winbox?

The result is same with Webfig and the latest version of Winbox.

proximus · Sun Oct 30, 2016 2:16 pm

I suspect that your issue is more likely related to 6.38.rc19. I have been trying to build a config on an old RB750GL, in preparation for a new RB750Gr3, and it was giving me fits taking some configuration elements (mostly IPv6 related). The router becomes inaccessible an eventually restarts with completely blank config.

Anyway, can you downgrade the RB750Gr3 to 6.37.1 and try?

paulct · Mon Oct 31, 2016 8:30 am

sfp port ? what for ? SFP+ in 10GBps -capable devices make sense for uplink interfaces, but for generic devices - not really any use for.
in such case - simpler to use media converter. unless if you need to use something exotic, like SFP/SFP+ -cased VDSL modem, EPON interface or LTE interface, but thats niche application and solvable w/o SFP usage.
CCR's had SFP and some of RB3011 and RB2011 if you desperately need it

SFP is just another interface type, we use it extensively for runs where conduits are tight, TV over fibre, distance etc. SFP doesn't just mean FAST speeds. It is merely another form of connectivity. I prefer not installing media converters as it is just another thing that requires power and space. I do like the Hex POE with SFP that came out recently, have not gotten my hands on one yet - but that would fit our installs nicely (except the pasive POE) - hence why a non POE RB750Gr3 with an SFP would be great.

mrz · Mon Oct 31, 2016 11:35 am

Redo the test without fragmentation and you will see different result.
Sure.
But... hm... in such case we will get results not from real world due to I have complex network and real clients send usual ethernet MTU packets.
Of course fragmentation take place and in the case of non-fragmentation we will get better results.

In real world you also want to avoid fragmentation. With TCP it is easily doable by adjusting MSS.

NobodyHUN · Mon Oct 31, 2016 11:43 am

I suspect that your issue is more likely related to 6.38.rc19. I have been trying to build a config on an old RB750GL, in preparation for a new RB750Gr3, and it was giving me fits taking some configuration elements (mostly IPv6 related). The router becomes inaccessible an eventually restarts with completely blank config.

Anyway, can you downgrade the RB750Gr3 to 6.37.1 and try?

Absolutely agree with you, loading configuration from another platform (eg. mips) cause reboot and router will be unstable.

As you recommended, tried to downgrade to v6.37.1 and it seems working fine. But, if i change DHCP range on Quick Set page, it write back the original values after applying config.
It should be the least amount of problems.

mrz · Mon Oct 31, 2016 12:01 pm

I am not impressed with the new RB750Gr3. Mainly, a lot of things working differently.
Cannot make static address in DHCP, when address is not in any pool, .
The worst thing, if i change DHCP range on Quick Set page and click on Apply, lost it's ip address and the device will be inaccessible. Then reset, load last working configuration and go on...
Waiting for a new, working release.

Could you try it again with a more current version of Winbox?
The result is same with Webfig and the latest version of Winbox.

Never use quickset to make some changes if you already started to configure router manually.

G2Dolphin · Tue Nov 01, 2016 9:48 am

Never use quickset

That's enough in 90% times.

proximus · Tue Nov 01, 2016 2:38 pm

Definitely met expectations of better CPU performance. Below we have the my original RB2011 replaced with the RB750Gr3, same config. The hourly CPU spikes are a Talos IP block list updating (currently around 46,000 address list entries). This resulted in a significant and measurable impact on throughput. Now it's not an issue.

I added a Transcend 16GB High Endurance microSD card.

My only disappointment is the 16MB flash. I'm a huge proponent of partitions. There have been times where I have had to revert back after upgrading, due to a bug. Nothing like being able to just boot the other partition! I would be willing to pay the extra $2 to have 32 or 64MB flash.

bajodel · Sat Nov 05, 2016 1:04 am

My only disappointment is the 16MB flash. I'm a huge proponent of partitions. There have been times where I have had to revert back after upgrading, due to a bug. Nothing like being able to just boot the other partition! I would be willing to pay the extra $2 to have 32 or 64MB flash.

Amen .. +1

risk · Sat Nov 05, 2016 10:06 am

My only disappointment is the 16MB flash. I'm a huge proponent of partitions. There have been times where I have had to revert back after upgrading, due to a bug. Nothing like being able to just boot the other partition! I would be willing to pay the extra $2 to have 32 or 64MB flash.
Amen .. +1

I like A-B booting in principle (e.g on Chromebooks), but how would it work in this case, would revert to old partition be automated or manual - how would it be triggered? A software watchdog? A hardware watchdog? A switch on the outside? Would an admin need to confirm new version is alright by issuing an explicit command?

proximus · Sat Nov 05, 2016 2:27 pm

I like A-B booting in principle (e.g on Chromebooks), but how would it work in this case, would revert to old partition be automated or manual - how would it be triggered? A software watchdog? A hardware watchdog? A switch on the outside? Would an admin need to confirm new version is alright by issuing an explicit command?

Auto or manual. Partitions are nothing new: http://wiki.mikrotik.com/wiki/Manual:Partitions Just need to build the box to support it.

DmitryAVET · Wed Nov 09, 2016 12:45 am

Now i have 2 x RB750Gr3 on test.

Configuration:
1) WAN <=> WAN (same Layer 2)
2) Firewall enabled
3) NAT enabled
4) Static routes

In general, typical configuration from REAL life.

Tests with IPSec SHA1/SHA256 and AES128/AES256б, real speed (user data):

512/1400/1518 bytes - 230 Mbits
64 bytes - 135 Mbits
32 bytes - 60 Mbits

Or up to 180-200 Mbits in duplex.

As traffic generator i use iPerf and TCP protocol. Traffic generator - PC on FX-8320 (8 cores, avg load <80%), as iperf server - notebook on Core-i5 (avg load <60%).

Zorro · Wed Nov 09, 2016 1:50 pm

sfp port ? what for ? SFP+ in 10GBps -capable devices make sense for uplink interfaces, but for generic devices - not really any use for.
in such case - simpler to use media converter. unless if you need to use something exotic, like SFP/SFP+ -cased VDSL modem, EPON interface or LTE interface, but thats niche application and solvable w/o SFP usage.
CCR's had SFP and some of RB3011 and RB2011 if you desperately need it
SFP is just another interface type, we use it extensively for runs where conduits are tight, TV over fibre, distance etc. SFP doesn't just mean FAST speeds. It is merely another form of connectivity. I prefer not installing media converters as it is just another thing that requires power and space. I do like the Hex POE with SFP that came out recently, have not gotten my hands on one yet - but that would fit our installs nicely (except the pasive POE) - hence why a non POE RB750Gr3 with an SFP would be great.

technically its "interface for interface or interfaces" with full case/mount for devices in.
SPF+ had other advantages aside speed.
for lower than 10Gbps speed there is no need for SFP/SFP+ port, basically.
surely many "prefer not" many thing, and expect it "as granted" within devices for prices as they are without it. and then other consumers start become whine about their over-pricing or more frequently downperforming, because money saved on essentials from design/production budget, wasted for irrelevant for 90% uses(outside backbone or SMB border of med-size or huge networks), SFP/SFP+ interfaces.
personally i would like routers to be able to make me coffee and tell me fresh anecdotes and help me catch cool chicks and do it for free, of course, but thats not likely to happen. same with SFP within SOHO or most SMB devices.
but talking practically - you may notice that RB 2011/RB3011 and CCR(and some of PPC devices?)have SFP ports, so likely they may design later such model you requested, just within another product line.
likely it would be something like RB2011r2 or something like that. MIkrotik tend to hade several Different models: 1. one most basicmodelwithout wifi or SFP, with slightly downclocked(no need for relevant CPU, happly) and bit smaller RAM(bottleneck, so make much sense too). 2. one model with wifi. 3. one model with wifi and sfp. 4. one model with wifi, sfp 5. and one model with SFP only. but it would be labeled or cased as rb750Gr3 or alike that but separated products, i guess. also as i pointed 4x-core chip on similar core(cost as much as already used, one watt hungrier)also exist and open some options too.

paulct · Wed Nov 09, 2016 2:36 pm

technically its "interface for interface or interfaces" with full case/mount for devices in.
SPF+ had other advantages aside speed.
for lower than 10Gbps speed there is no need for SFP/SFP+ port, basically.
surely many "prefer not" many thing, and expect it "as granted" within devices for prices as they are without it. and then other consumers start become whine about their over-pricing or more frequently downperforming, because money saved on essentials from design/production budget, wasted for irrelevant for 90% uses(outside backbone or SMB border of med-size or huge networks), SFP/SFP+ interfaces.
personally i would like routers to be able to make me coffee and tell me fresh anecdotes and help me catch cool chicks and do it for free, of course, but thats not likely to happen. same with SFP within SOHO or most SMB devices.
but talking practically - you may notice that RB 2011/RB3011 and CCR(and some of PPC devices?)have SFP ports, so likely they may design later such model you requested, just within another product line.
likely it would be something like RB2011r2 or something like that. MIkrotik tend to hade several Different models: 1. one most basicmodelwithout wifi or SFP, with slightly downclocked(no need for relevant CPU, happly) and bit smaller RAM(bottleneck, so make much sense too). 2. one model with wifi. 3. one model with wifi and sfp. 4. one model with wifi, sfp 5. and one model with SFP only. but it would be labeled or cased as rb750Gr3 or alike that but separated products, i guess. also as i pointed 4x-core chip on similar core(cost as much as already used, one watt hungrier)also exist and open some options too.

Ended up getting 50 of the Hex POE units with SFP. Fits our installs nicely - can be powered by 803.at via our netgear switch. Backhaul via fibre via the SFP port. Well done Mikrotik. I am sure they can save on the costs of supplying POE on all ports - and design a new router based on the CPU that is in the RB750gr3 that has an SFP port. We would have gone with the HAP AC - but it was too expensive - especially as we didn't need the onboard wireless. We would have also gone with the Ubiquiti edgerouter X - but hey we like Mikrotik. Either way to me a SFP port in mandatory, I do realize it is not to everyone - but at least we know now the HEX POE can do the job well.

schmeltm · Wed Nov 09, 2016 11:11 pm

Hi all,

i have 2 RB750Gr3 and tried to get VLAN´s via Switch chip working.

I have read the following wiki article: http://wiki.mikrotik.com/wiki/Manual:Sw ... p_Features
But if i configure the vlans to the ports i will be locked out and have to reset the device ...

Somebody who has examples fot switch config on this device?

@Mikrotik: Do you plan to describe the switch cip features somewhere in the wiki?

Br
Markus

Thu Nov 10, 2016 4:12 pm

RB750Gr3 switch chip does not have full VLAN tagging/untagging support yet, it is planned to implement it in future. Currently, you should use RB750Gr3 switch chip only for basic switching.
RB750Gr3 is also powerful enough to handle software based VLANs: http://wiki.mikrotik.com/wiki/Manual:In ... p_examples

schmeltm · Thu Nov 10, 2016 4:45 pm

Thanks for the info!

pe1chl · Thu Nov 10, 2016 5:32 pm

I just received my RB750Gr3 and to my surprise the ports 2-5 cannot be separated, they are always slave of the same master.
This makes it difficult to use it for other things than a basic Internet NAT router...
Normally we use individual ports for separate links in the network and it can be done by removing the master port on an ethernet port.

I have always assumed that internally this is handled by programming the switch to do an untagged VLAN on that port and set
a tagged VLAN with an internally generated VLAN tag on the internal port, configure a VLAN interface at the CPU side, and
tell the user that is the "ether5" port, for example.

Is that how it really works and is it not yet available on the RB750Gr3 for exactly that reason?

schmeltm · Thu Nov 10, 2016 5:57 pm

Hi pe1chl,

i have try it out and i can seperate the interfaces.

/interface ethernet set master-port=none ether5-slave-local

Or set "Master Port" to "none" via winbox on Interface General Tab.

pe1chl · Thu Nov 10, 2016 6:02 pm

Ok thanks I tried it and indeed it works via commandline. The option is not shown in WebFig.
(I never use Winbox)

ingtegration · Wed Nov 16, 2016 10:44 pm

for lower than 10Gbps speed there is no need for SFP/SFP+ port, basically.

Try to do 100M+ on copper!

ingtegration · Wed Nov 16, 2016 11:30 pm

I just did a quick test with 2x RB750Gr3 in tunnel mode and was able to quite happily get ~436Mbps:

Note this was running RouterOS 6.38rc15.

I wonder how you can reach those numbers. I'm testing with ROS 6.38rc31 and i get:

user@IngTegration:~$ iperf3  -c 172.16.1.120
Connecting to host 172.16.1.120, port 5201
[  4] local 192.168.89.224 port 47172 connected to 172.16.1.120 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  41.6 MBytes   349 Mbits/sec   10    382 KBytes       
[  4]   1.00-2.00   sec  40.8 MBytes   342 Mbits/sec    1    362 KBytes       
[  4]   2.00-3.00   sec  41.4 MBytes   347 Mbits/sec    1    348 KBytes       
[  4]   3.00-4.00   sec  40.5 MBytes   339 Mbits/sec    3    329 KBytes       
[  4]   4.00-5.00   sec  40.3 MBytes   338 Mbits/sec   10    335 KBytes       
[  4]   5.00-6.00   sec  41.3 MBytes   346 Mbits/sec    5    318 KBytes       
[  4]   6.00-7.00   sec  39.5 MBytes   332 Mbits/sec   19    311 KBytes       
[  4]   7.00-8.00   sec  40.1 MBytes   337 Mbits/sec    2    293 KBytes       
[  4]   8.00-9.00   sec  39.6 MBytes   332 Mbits/sec    1    277 KBytes       
[  4]   9.00-10.00  sec  40.1 MBytes   337 Mbits/sec    0    372 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   405 MBytes   340 Mbits/sec   52             sender
[  4]   0.00-10.00  sec   403 MBytes   338 Mbits/sec                  receiver

Tested with SHA1 / AES128cbc / modp1024
What i find strange is that i get same throughput using 3DES.

NAT is disabled. I have only 2 firewall rules: accept forward and accept input.

Config (SHA1 / 3DES):

/ip ipsec proposal
set [ find default=yes ] enc-algorithms=3des

/ip ipsec peer
add address=172.16.2.1/32 enc-algorithm=3des nat-traversal=no secret=password

/ip ipsec policy
set 0 disabled=yes dst-address=0.0.0.0/0 src-address=0.0.0.0/0
add dst-address=172.16.2.1/32 sa-dst-address=172.16.2.1 sa-src-address=172.16.2.3 src-address=172.16.2.3/32 tunnel=yes

ingtegration · Thu Nov 17, 2016 2:47 am

Result when adding GRE to the mix (to support OSPF):

user@IngTegration:~$ iperf3  -c 172.16.1.120
Connecting to host 172.16.1.120, port 5201
[  4] local 192.168.89.224 port 49950 connected to 172.16.1.120 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  18.0 MBytes   151 Mbits/sec    3    261 KBytes       
[  4]   1.00-2.00   sec  18.1 MBytes   151 Mbits/sec    2    237 KBytes       
[  4]   2.00-3.00   sec  18.5 MBytes   156 Mbits/sec    0    288 KBytes       
[  4]   3.00-4.00   sec  18.4 MBytes   155 Mbits/sec    1    245 KBytes       
[  4]   4.00-5.00   sec  18.4 MBytes   155 Mbits/sec    0    294 KBytes       
[  4]   5.00-6.00   sec  18.3 MBytes   154 Mbits/sec    7    253 KBytes       
[  4]   6.00-7.00   sec  18.3 MBytes   154 Mbits/sec    0    302 KBytes       
[  4]   7.00-8.00   sec  18.2 MBytes   153 Mbits/sec    9    261 KBytes       
[  4]   8.00-9.00   sec  18.6 MBytes   156 Mbits/sec    2    219 KBytes       
[  4]   9.00-10.00  sec  18.0 MBytes   151 Mbits/sec    0    273 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   183 MBytes   153 Mbits/sec   24             sender
[  4]   0.00-10.00  sec   182 MBytes   153 Mbits/sec                  receiver

Thu Nov 17, 2016 9:42 am

@ingtegration thats a pretty good result!

Hopefully with all the IPSEC changes recently, Mikrotik developers find the time to add IPSEC VTI.

Having VTI support would mean that GRE is not required to run OSPF between the IPSEC endpoints, and in turn remove that overhead.

dhoulbrooke · Thu Nov 17, 2016 12:24 pm

I wonder how you can reach those numbers. I'm testing with ROS 6.38rc31 and i get:

When I'm back in the office I'll give rc31 a test. The only change from the default config was:

/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc

pe1chl · Thu Nov 17, 2016 2:43 pm

Having VTI support would mean that GRE is not required to run OSPF between the IPSEC endpoints, and in turn remove that overhead.

I think there would be no difference in overhead between IPsec in tunnel mode (VTI) and IPIP tunnel over IPsec in transport mode.
GRE tunnel has a few bytes of extra overhead because it can transport other protocols as well, but VTI cannot do that, so compare
it with IPIP not GRE.

mchillinger · Thu Dec 01, 2016 1:02 pm

RB750Gr3 switch chip does not have full VLAN tagging/untagging support yet, it is planned to implement it in future. Currently, you should use RB750Gr3 switch chip only for basic switching.
RB750Gr3 is also powerful enough to handle software based VLANs: http://wiki.mikrotik.com/wiki/Manual:In ... p_examples

I agree that by concept the CPU should be powerful enough to handle software based VLAN tagging and routing BUT there seems to be a problem in all current of RouterOS installable on the RB750Gr3 [6.37.1 through 6.37.3]:

If even a single VLAN interface is added to a bridge device, the entire router will no longer

* work as a DHCP client
* route packets over any bridge

Minimal example code looks like this:

On ether1 there is untagged network traffic as well as tagged network traffic with VLAN tag 5. Both Networks have a working DHCP Server running.

works:

/interface bridge
add name=br-vlan5
/ip dhcp-client
add default-route-distance=0 dhcp-options=hostname,clientid disabled=no interface=ether1

does not work:

/interface bridge
add name=br-vlan5
/ip dhcp-client
add dhcp-options=hostname,clientid disabled=no interface=ether1
/interface vlan
add interface=ether1name=ether1-vlan5 vlan-id=5
/interface bridge port
add bridge=br-vlan5 interface=ether1-vlan5

As covered by an earlier post switch VLAN processing is not available yet. As a result RB750Gr3 seems to be unable to handle VLAN traffic with the current software revision

Can this conclusion be verified or did I miss a major point here? We are unable to replicate setups working with RB750Gr2 on the r3 systems

Thanks a lot guys.

Splash · Thu Dec 01, 2016 1:15 pm

I noticed that the master-port option (ethernet interface) is not visible in winbox where its still configurable within the CLI.

Splash · Thu Dec 01, 2016 1:17 pm

Ive also noticed a problem with the auto-negotiation for 1Gbs. On a number of 1Gbs devices running on the RB750Gr2 work fine, however moving them to a Gr3, they refuse to run 1Gbs and can only work at 100Mbs....

Wlanfr3ak · Fri Dec 09, 2016 3:27 am

Hi,

my 750GR3 is in a BootLoop after i upload the dude-6.38rc45-mmips.npk and want to reboot the RB to install the Dude:

Before i installed the Dude i have updated the RB to routeros-mmips-6.38rc45.npk and make a reboot, after that i updated the Firmware too and reboot again. The Extracted all_packages-mmips-6.38rc45.zip files installed with a reboot but then i want to install The Dude Server ...

Here is the Ouput of Serial/UART Interface:

Rebooting in 1 seconds..Oops[#1]:
Cpu 3
$ 0   : 00000000 00000001 00000000 00000001
$ 4   : 8f449000 811e8700 8f438a00 8021c50c
$ 8   : 803f3394 803f3394 00000001 0000055c
$12   : 8044d2b0 00000001 0000055b 8f438a80
$16   : 8f449000 8040a410 8021bf84 8f438100
$20   : 00000001 00000001 8f438080 00000000
$24   : 00000008 00427ba0                  
$28   : 8fc30000 8fc31db8 7fd99824 8021be88
Hi    : 00000003
Lo    : 00000000
epc   : 8021be80 squashfs_kill_sb+0x14/0x38
    Not tainted
ra    : 8021be88 squashfs_kill_sb+0x1c/0x38
Status: 1100d803    KERNEL EXL IE 
Cause : 80800008
BadVA : 00000000
PrId  : 0001992f (MIPS 1004Kc)
Process init (pid: 1, threadinfo=8fc30000, task=8fc28000, tls=00456460)
Stack : 00000000 00000000 000000a4 80251d3c 8f449000 801bec9c 8f449000 00000001
        8021bf84 8f438100 00000001 8f449000 ffffffea 801bf9fc 80455308 80450000
        801be9a0 803f0000 804552ec 8fcd9480 8040a410 8040a410 8f438100 801c0798
        00000000 801d8b40 ffffffea 8040a410 8f438080 8f438100 8fcd9480 8040a410
        8f438080 801da2a4 8f438080 801d8e34 8ffe3011 00000000 00000001 8f438100
        ...
Call Trace:
{8fc31db8} squashfs_kill_sb+0x14/0x38
{8fc31dd0} deactivate_locked_super+0x84/0xc8
{8fc31df0} mount_nodev+0x68/0xd4
{8fc31e18} mount_fs+0x20/0xe8
{8fc31e40} vfs_kern_mount+0x58/0xd8
{8fc31e70} do_kern_mount+0x44/0xf4
{8fc31ea0} do_mount+0x574/0x6c8
{8fc31ef0} sys_mount+0x94/0xe4
{8fc31f30} stack_done+0x20/0x44


Code: afb00010  8c8201e8  00808021 <0c06f88f> 8c440000  0c06fa95  02002021  8fbf0014  8e0401e8 
---[ end trace d47c101a91ab90ba ]---
Kernel panic - not syncing: Attempted to kill init!
Rebooting in 1 seconds..

Reset and Reset to Netinstall doesnt work and the Output is the same. Over Serial i cannot send Inputs.
Have anybody an Idea ?

pe1chl · Fri Dec 09, 2016 10:41 am

Doesn't a full netinstall including format of the flash fix that?

fragma · Fri Dec 09, 2016 2:50 pm

...after that i updated the Firmware too...

Why did you have to upgrade the firmware, was there a problem ?
[and now you bricked the device ?]

pe1chl · Fri Dec 09, 2016 4:56 pm

...after that i updated the Firmware too...
Why did you have to upgrade the firmware, was there a problem ?
[and now you bricked the device ?]

It does not appear this was caused by the RouterOS and firmware update, but by the install of Dude.
I think the filesystem has been corrupted, maybe it was full (those new devices have that tiny 16GB flash...)
But in that case a full netinstall of RouterOS should fix it.

Wlanfr3ak · Fri Dec 09, 2016 7:15 pm

I have tested with Netinstall but im not running with Windows, so i have to try a other Computer. But the Bootloop is so fast that the Link on the Cable not seem coming before the Board restart, i have to tried that with a Windows Computer and NetInstall again, i hope that this will work :-/

Memodota · Wed Dec 14, 2016 4:01 pm

Hello guys!
Have already trying to config RB750Gr3 for a week. Didn't sleep well for 4 days

But i can't make port forwarding work.
First of all i made 2 WAN Balancing with PCC - works great. Then tried to config port forwarding and got nothing (51413, 80 and 22-23 ports).
add action=dst-nat chain=dstnat dst-port=51413 in-interface=ether1 log=no protocol=tcp to-addresses=192.168.88.100 to-ports=51413
I know about firewall mangle. Made rules for input and forward chains to tag connections and made rules to route traffic into right WAN, It didn't work. So after 2 days of no luck i've tried to make clean config.
1)Made clean config from scratch, configured just WAN, Eth5 (Ubuntu Server) and Eth2 (my Windows PC with Winbox);
2)Made masquerading rule for source address 192.168.88.0/24. Internet works nice on Windows PC;
3)Made no rules in Firewall (so all incoming connections should work);
4)Incoming connection from Internet to Routers Web Page is working;
5) Turned off IP Service (http 80 port), so canceled all connections from previous point.
6) Made rule
add action=dst-nat chain=dstnat dst-port=80 in-interface=ether1 log=no protocol=tcp to-addresses=192.168.88.100 to-ports=80
Aaaand - NOTHING!!!
I see 3 packets incoming on that rule each time i try to connect but no response from the server.
Firewall on Ubuntu server is turned off. And i make connections from mobile phone so there is no hairpin problem.

Memodota · Thu Dec 15, 2016 11:56 am

Problem solved

An error occured on the linux Server. In the network settings were set gateway but this is not enough. Should be set a default gateway.

godlike · Thu Dec 15, 2016 7:18 pm

Thanks guys for fixing CPU profiling on mmips devices.
I hope implementing hw switch functionality will be next

dragon2611 · Sat Dec 17, 2016 2:08 pm

I've been suspicious about VLANs, since Ubiquiti Edgerouter-x doesn't support them either, Mediatek seems to have screwed the pooch on delivering well documented switch drivers to their customers.

The ER-X does support VLANs on the switch, needs EdgeOS 1.85 or later if I remember rightly.

Regarding the RB750Gr3 any idea on the SSTP performance?

Wlanfr3ak · Sat Dec 17, 2016 2:23 pm

I have tested with Netinstall but im not running with Windows, so i have to try a other Computer. But the Bootloop is so fast that the Link on the Cable not seem coming before the Board restart, i have to tried that with a Windows Computer and NetInstall again, i hope that this will work :-/

I tried to test again with a Windows Computer, the same Problem. Link doesnt came up and the RouterBOARD restart to fast, no Chance to press a key or something. This is a really bad taste for a Mikrotik Product

No i have to send it back ...

dragon2611 · Sat Dec 17, 2016 2:48 pm

Try a switch between the RB and the PC

Agromahdi123 · Tue Dec 20, 2016 10:49 pm

So just to confirm what i am reading, Hex v3 to Strongswan ipsec in transport will yeild somewhere around 100Mbits, and in tunnel can yield more? I want to put one of these in front of my 2011 for the aes128 HW encryption wanna make sure i can hit at least 75/75

pimmie · Wed Dec 21, 2016 12:44 pm

So just to confirm what i am reading, Hex v3 to Strongswan ipsec in transport will yeild somewhere around 100Mbits, and in tunnel can yield more? I want to put one of these in front of my 2011 for the aes128 HW encryption wanna make sure i can hit at least 75/75

Finally received a RB750Gr3 yesterday to also replace a RB2011, these are my results for a gre over aes256-sha1-modp4096 ipsec tunnel to strongswan on a 180/30Mbit/s connection (cpu usage ~50%, probably limited to 1 core?):

Ascendo · Wed Dec 21, 2016 1:03 pm

So just to confirm what i am reading, Hex v3 to Strongswan ipsec in transport will yeild somewhere around 100Mbits, and in tunnel can yield more? I want to put one of these in front of my 2011 for the aes128 HW encryption wanna make sure i can hit at least 75/75
Finally received a RB750Gr3 yesterday to also replace a RB2011, these are my results for a gre over aes256-sha1-modp4096 ipsec tunnel to strongswan on a 180/30Mbit/s connection (cpu usage ~50%, probably limited to 1 core?):

Not bad at all. Wish they'd make a "cheap rackmount" version of this too!

pe1chl · Wed Dec 21, 2016 4:42 pm

Not bad at all. Wish they'd make a "cheap rackmount" version of this too!

Maybe someone will make a bracket that can hold 1-3 of these in a 1U panel?
Could be made to fit some other MikroTik models as well...

pimmie · Wed Dec 21, 2016 5:10 pm

Not bad at all. Wish they'd make a "cheap rackmount" version of this too!
Maybe someone will make a bracket that can hold 1-3 of these in a 1U panel?
Could be made to fit some other MikroTik models as well...

Already exists, MaxxWave MW-RA-750-3

Wed Dec 21, 2016 5:53 pm

(cpu usage ~50%, probably limited to 1 core?)

/system resources cpu shows you per core load
/tool profile allows you to see what is taking total CPU time and each core individually what is doing - make sure you run latest 6.38RC, profiler was fixed there

Agromahdi123 · Thu Dec 22, 2016 5:20 pm

So just to confirm what i am reading, Hex v3 to Strongswan ipsec in transport will yeild somewhere around 100Mbits, and in tunnel can yield more? I want to put one of these in front of my 2011 for the aes128 HW encryption wanna make sure i can hit at least 75/75
Finally received a RB750Gr3 yesterday to also replace a RB2011, these are my results for a gre over aes256-sha1-modp4096 ipsec tunnel to strongswan on a 180/30Mbit/s connection (cpu usage ~50%, probably limited to 1 core?):

Not bad at all. Wish they'd make a "cheap rackmount" version of this too!

Thanks for this response! just placed my order, ive decided to just place it in front of my 2011 and just do a dual NAT thing so the 2011 will do most of the routing to network, and all the Hex needs to do is pass route to it.

sallen · Thu Dec 22, 2016 10:41 pm

So just to confirm what i am reading, Hex v3 to Strongswan ipsec in transport will yeild somewhere around 100Mbits, and in tunnel can yield more? I want to put one of these in front of my 2011 for the aes128 HW encryption wanna make sure i can hit at least 75/75
Finally received a RB750Gr3 yesterday to also replace a RB2011, these are my results for a gre over aes256-sha1-modp4096 ipsec tunnel to strongswan on a 180/30Mbit/s connection (cpu usage ~50%, probably limited to 1 core?):

Not bad at all. Wish they'd make a "cheap rackmount" version of this too!
Thanks for this response! just placed my order, ive decided to just place it in front of my 2011 and just do a dual NAT thing so the 2011 will do most of the routing to network, and all the Hex needs to do is pass route to it.

Well that seems slightly silly. Why the double NAT? Just have the Hex do all the routing and turn the 2011 into a switch and AP (if your's is wireless). The Hex has a better CPU and more memory.

pimmie · Fri Dec 23, 2016 12:54 pm

Well that seems slightly silly. Why the double NAT? Just have the Hex do all the routing and turn the 2011 into a switch and AP (if your's is wireless). The Hex has a better CPU and more memory.

I fully agree with this, the Hex is much faster in (almost?) all aspects. What I did was export my config on the rb2011 (in terminal: `/export file=rb2011_20161223.rsc`) and downloaded that file. Then I manually changed the interface/bridge configuration to incorporate the 6 less ports (5ether + sfp) and removed all wireless configuration (as I was not running CAPsMAn yet). Then I uploaded and imported that file to/on the Hex (`/import hex_20161223.rsc`). After that I re-configured my wireless configuration in CAPsMAN and set the correct password for the admin user. Actually I am not using my rb2011 at the moment, I added a hAP ac lite as switch and cap device so I also have 5Ghz wifi.

The only problem I had is that when you import the configuration on the hex and you are changing master ports, your connection is terminated and therefore also the import. So I had to check manually until which line the configuration was imported and then re-run the import with `/import hex_20161223.rsc from-line=XX`

pe1chl · Fri Dec 23, 2016 3:56 pm

This method of transferring configuration must be used with care.
For example, the exported config contains MAC addresses for some interfaces, and you will duplicate
them in the new router this way. When both routers end up being on the same network and the original
place where the MAC address was used is not deleted there, this may cause "interesting" problems later.

Also, note that importing a config does not erase existing config of those same items already present
in the new router. For example, firewall rules will be added at the end but the default rules will not be
deleted.

However, with the proper care for this and other potential problems, it is a quick way of getting everything
setup.

yams · Mon Dec 26, 2016 7:45 pm

Hi

I have a problem with the rsc file. It seems that the RouterBOARD 750G r3 doesn't excute the file when i reset it and execute the file with no default configuration.

i did a test with a minimum of information in the rsc file and it doesn't seems that the file is excuted

/interface ethernet
set [ find default-name=ether2 ] name=ether2-master
set [ find default-name=ether3 ] master-port=ether2-master
set [ find default-name=ether4 ] master-port=ether2-master
set [ find default-name=ether5 ] master-port=ether2-master

if i import the file manually in the console, it does work and i don't have any error in the verbose.

Does anyone have this kind of problems ? Did i miss something ?

pe1chl · Tue Dec 27, 2016 12:17 am

Yes, I have seen this problem before. It is not related to the RB750Gr3, it is related to recent RouterOS versions.
I have no idea what is really causing it and for me too the only workaround is to manually paste new configuration in a terminal session.

yams · Tue Dec 27, 2016 2:18 pm

Hi pe1chl

Thanks for your answers.

I hope it will be fix soon.

nescafe2002 · Tue Dec 27, 2016 2:25 pm

Could be due to (absence of) /system routerboard settings boot-delay feature.

This time is needed to initialize interface etc.

Try adding a delay in your script, e.g.:

# mar/02/2016 04:00:00 by RouterOS 6.35rc15
# software id = XXXX-XXXX
#
:delay 15s
/interface bridge
add name=bridge-lan

yams · Tue Dec 27, 2016 2:30 pm

Could be due to (absence of) /system routerboard settings boot-delay feature.

This time is needed to initialize interface etc.

Try adding a delay in your script, e.g.:
Code: Select all
# mar/02/2016 04:00:00 by RouterOS 6.35rc15
# software id = XXXX-XXXX
#
:delay 15s
/interface bridge
add name=bridge-lan

Hi nescafe2002

I tried that too but it didn't work.

I add like 30 secondes of delay and the router start before this time (Like it didn't take this value)

pe1chl · Tue Dec 27, 2016 7:19 pm

I have been fighting with the "reset with no defaults and import specified script" on another router (RB951)
and I have not been able to get it to work in 6.37 and I think 6.36.
I am sure it worked OK on 6.29 but somewhere after that it has been broken. I have no idea how to debug
this on such a router. On a router with RS232 port I can look at the output there and login to see what is
happening, but on the new devices without RS232 it is not possible to diagnose it.
I have reported it to support and they told me they would look into it, and also took my suggestion of writing
a debug log to a file that we can later download, and to continue processing the script even when a minor
error occurs (as it is now it will stop at the first error). However, as you can see this has not resulted in
improvement yet.

yams · Tue Dec 27, 2016 7:29 pm

Hi nescafe2002

In fact you were right

It solved my problems

I tried before with a delay but my syntax was not good

I took the syntax you wrote in your exemple and it's solved my problems.

thank you

a1x0 · Tue Dec 27, 2016 11:39 pm

As Wlanfr3ak I update ROS to 6.37.3 by installing extra packages and now router loop reboot. Netinstall and configuration reset not work

Raice · Thu Dec 29, 2016 11:44 am

So just to confirm what i am reading, Hex v3 to Strongswan ipsec in transport will yeild somewhere around 100Mbits, and in tunnel can yield more? I want to put one of these in front of my 2011 for the aes128 HW encryption wanna make sure i can hit at least 75/75
Finally received a RB750Gr3 yesterday to also replace a RB2011, these are my results for a gre over aes256-sha1-modp4096 ipsec tunnel to strongswan on a 180/30Mbit/s connection (cpu usage ~50%, probably limited to 1 core?):

Could you share your ROS and StrongSwan config files?

a1x0 · Sat Jan 07, 2017 5:14 pm

Can anybody help?

msatter · Sat Jan 07, 2017 11:43 pm

Can anybody help?

Contact Mikrotik support how to recover your box now normal ways did not work.

Biker111 · Mon Oct 30, 2017 4:44 am

RB750Gr3 switch chip does not have full VLAN tagging/untagging support yet, it is planned to implement it in future. Currently, you should use RB750Gr3 switch chip only for basic switching.
RB750Gr3 is also powerful enough to handle software based VLANs: http://wiki.mikrotik.com/wiki/Manual:In ... p_examples

As a customer, where do we get such information before we buy the product?
Really, really strange guessing game,- on your main product description page there is no such information?

I know, after some time with Mikrotik one learns to follow up and consult pages like https://wiki.mikrotik.com/wiki/Manual:S ... p_Features
But such information should be told the customer directly on the main product page.

soosp · Sun Nov 19, 2017 9:14 pm

RB750Gr3 switch chip does not have full VLAN tagging/untagging support yet, it is planned to implement it in future. Currently, you should use RB750Gr3 switch chip only for basic switching.
RB750Gr3 is also powerful enough to handle software based VLANs: http://wiki.mikrotik.com/wiki/Manual:In ... p_examples

One year has been gone since this "promise" was written. Are ther any development in this question?

rognick · Wed Dec 27, 2017 4:57 pm

Can anybody help?
The problem with VLAN tagging / untagging is fixed with version v6.41

Cybernet1k · Wed Jan 17, 2018 11:09 am

Hello.
The documentation has information that the device can work with disabled switching and enabled switching:
https://i.mt.lv/routerboard/files/RB750 ... 140316.png
https://i.mt.lv/routerboard/files/RB750 ... 152443.png
How disable switching? Is it enough to remove all ports from the bridge and bridge itself?

sebastia · Wed Jan 17, 2018 10:31 pm

That is my understanding too: I've asked support but didn't got conclusive answer.
viewtopic.php?f=3&t=128729

ashpri · Tue Jun 04, 2019 9:23 am

I can confirm that as of today, the HEX (RB750GR3) with v6.44.3 cannot yet implement vlan in switch chip (with hardware offloading).

I have a HAPAC2 with switch chip vlan enabled and the same settings does not work on the HEX.

pe1chl · Tue Jun 04, 2019 10:29 am

I think the first level of VLAN (in hardware) is used to emulate 5 ports on a chip that has only 2.
When you configure ether1, ether2, ether3, ether4 and ether5 you are in fact internally configuring some VLANs that are mapped in the internal switch to exit on that specific port untagged.
Now, when you want to configure tagged VLANs on those ports that is another level of VLAN which is always software because this switch does not support 2 levels of VLAN.
This same issue is present in other MikroTik products as well, but not in all of them. E.g. the more powerful switches use chips that fully support multiple levels of VLAN, and the RB2011 has a better chip on ports 1-5 than on ports 6-10.
The new "filtered bridge" configuration method does not solve this because once you use VLANs on such a bridge, the hardware accel automatically becomes disabled.
The old "switch" configuration method is more powerful in that it can sometimes do VLAN configurations (which then have hardware accel), but not on this model.

Joni · Mon Apr 26, 2021 8:38 pm

RB750Gr3 switch chip does not have full VLAN tagging/untagging support yet, it is planned to implement it in future. Currently, you should use RB750Gr3 switch chip only for basic switching.

Maybe some horizon update of the nearest decade when this might be implemented?

Who is online