Hello,

We are running a 2216 at an internet exchange point. The 2216's cpu running more than %70 percent 7/24 and we would like to replace it.

Since there is nothing greater than 2216 from MikroTik, we think to replace with an x86 system.

To put it simply we are pushing around 17G to a BGP peer(40G Conn soon replaced with 100G) with a default route, incoming traffic from 3x10G(soon replaced with 40G) LAGG. There are some static routing(around 1100 routes) and 10ibgp peer(around 50 prefixes each).

What will be the recommended hardware for that purpose? We also would like to continue to use around 30G BW. Any help appreciated and thanks!

Have you explored the hardware offload feature on the CCR2216?

Hello,

We are running a 2216 at an internet exchange point. The 2216's cpu running more than %70 percent 7/24 and we would like to replace it.

Since there is nothing greater than 2216 from MikroTik, we think to replace with an x86 system.

To put it simply we are pushing around 17G to a BGP peer(40G Conn soon replaced with 100G) with a default route, incoming traffic from 3x10G(soon replaced with 40G) LAGG. There are some static routing(around 1100 routes) and 10ibgp peer(around 50 prefixes each).

What will be the recommended hardware for that purpose? We also would like to continue to use around 30G BW. Any help appreciated and thanks!

Hi

Best hardware gear on lower budget i will recommend to you is R630 server with Dual cpu E5-2699v4 total 44 physical cores + 44 threads

on the mezzanine slot you have 2 flavours.. you can either use mellanox connectx4 card with dual port 40/56gb or you can use Mellanox connectx 4121c 10gb/25 sfp28 dual port..

if you get the chance to get the R630 with the extended PCI riser card, it will allow you to use another 3 PCI cards on the top slots..

here if you need 10G sfp+ ports you can put a combination of dual port 10G or dual port 100G...

i have running a R630 with lower slot Mezzanine Dell R887V ConnectX-4 Lx Dual-Port 10/25GbE

on the top slots i have

1 Chelsio T540-CR 4 ports 10GbE SFP+
1 Mellanox ConnectX4 CX4121A with 2 ports SFP+
1 Mellanox ConnectX-4, MCX416A-CCAT, 100G, with 2 ports 40G/100G QSFP28


The secret that no one will tell you.. either because they have not lost time testing.. is the NIC cards... in order to suceed with Multi-CPU x86_64 on RouterOS v7.15.xx stable version.. is the Network cards... because they are related to IRQ on the CPU..

basically the more IRQs slots available on the NIC in order to connect to various cpus to split the load for processing the packets the better..

and the cards i had best results were Mellanox and Chelsio..

and trust me i have tried several and none had the best IRQ cpu support then Mellanox and Chelsio..

Intel X520da2 very few slots.. only 4 irqs split on 4 cores cpu... (when u set interface queue multi-queue-ethernet-default with queue size lower or higher) you will allways get RX-erros on this.. because on higher traffic over 1ggbps.. the queues fill up on the network card.. its a issue with the hardware of the card and firmware also)

Intel X710da4 its a nice card has 4 ports sfp+ 10G on lower interface mezzanine.. but its horrible for mikrotik because all IRQ queues it as alot.. i creates 88 queues on the IRQ on the server.. but all of them use cpu0 only.. (this gives bottleneck on 1 core only for all traffic packets being processed on that port from the network cards .. again resulting on higher traffic on RX-ERRORS and TX-ERRORS... )
Broadcom BCM57840xxx still waiting on support driver for this cards.. beeing implemented by mikrotik...

Chelsio T540-cr its a great card with 4 ports sfp 10G... it has allot of queues on the IRQ... and splitts traffic even on all cpus cores.. specially on higher traffic with over 500k thousand packets... without giving RX-ERRORS..

But by far the best performance on networking were done with the Mellanox Connectx4 cards... alot of IRQ queues available for all cpu cores.. which splits traffic even on all cpu cores without issues..

below some pics attached for comparison.. i even created support rif tickets.. and were told the same for routerOS v7 multi-cpu .. Intel unfortunatelly most of them are not recommended due to the IRQ support issues..

First picture is of intel x710da4 with 4 ports sfp+ using por1 all IRQ queues from port 1 beeing sent to cpu 0... hundres of thousand of packets.. which obviously generates alot of RX-ERRORS and TX-ERRORS.. simply for the fact the card cannot handle all of it.. even increasing queue size from 50 to 500 to 5000 to 10000 to 20000 still throughs errors and the higher you increase it .. the higher problems you will have with ping and other issues..


and below on the second picture is Mellanox ConnectX4 CX4121A using port1 with double the traffic around 5gbps on BGP server

with no issues and no errors at all


This was a trial and errors.. buying and testing all sorts of cards and Brands..

Broadcom cards on eletrical ports.. and intel pro t 1000 also had good results with IRQ queues balancing all network traffic on all cpus also.. Just the intel i had more problems.. and eventually had to swap them out.

Going back to the hardware... using the Xeon E52699V4 we have acheived great results.. bringing 1 full route BGP up in less then 20seconds..

and with traffic generator using Jumbo frames we have managed to squeeze 8Terabits of full troughput aggregate... so this cpus can take alot.. for the price budget you can find them on the market today...

attached some pics

64b packages
512b packages
1500b packages
9000b jumbo frame package results from the cpu

Attached video...

https://youtu.be/2udqK7ldBlk

The secret that no one will tell you.. either because they have not lost time testing.. is the NIC cards... in order to suceed with Multi-CPU x86_64 on RouterOS v7.15.xx stable version.. is the Network cards... because they are related to IRQ on the CPU..

basically the more IRQs slots available on the NIC in order to connect to various cpus to split the load for processing the packets the better..

and the cards i had best results were Mellanox and Chelsio..

and trust me i have tried several and none had the best IRQ cpu support then Mellanox and Chelsio..

Intel X520da2 very few slots.. only 4 irqs split on 4 cores cpu... (when u set interface queue multi-queue-ethernet-default with queue size lower or higher) you will allways get RX-erros on this.. because on higher traffic over 1ggbps.. the queues fill up on the network card.. its a issue with the hardware of the card and firmware also)

Intel X710da4 its a nice card has 4 ports sfp+ 10G on lower interface mezzanine.. but its horrible for mikrotik because all IRQ queues it as alot.. i creates 88 queues on the IRQ on the server.. but all of them use cpu0 only.. (this gives bottleneck on 1 core only for all traffic packets being processed on that port from the network cards .. again resulting on higher traffic on RX-ERRORS and TX-ERRORS... )
Broadcom BCM57840xxx still waiting on support driver for this cards.. beeing implemented by mikrotik...

Chelsio T540-cr its a great card with 4 ports sfp 10G... it has allot of queues on the IRQ... and splitts traffic even on all cpus cores.. specially on higher traffic with over 500k thousand packets... without giving RX-ERRORS..

But by far the best performance on networking were done with the Mellanox Connectx4 cards... alot of IRQ queues available for all cpu cores.. which splits traffic even on all cpu cores without issues..

Hey @PortalNET, di you notice the existence of the package "extra-nic" on extra packages for ARM64 architecture?

It is just a guess, but it sounds like extra drivers.
The possibility of this being related to IRQ matter you metioned is not-negligible.

The IRQ issue you mentioned on Intel devices is a thing that only in RouterOS I'm not able to deal with.
On a pure Linux scenario, I use intel tools to deal with that.
Maybe what is missing for those NICs stucked on IRQ 0 colud be some tools inside an "extra-nic" package.

This conversation reminded me of a possibility.

Has anyone had the opportunity to test RouterOS directly on high-capacity ARM64 servers with 2X100Gbps NICs or something similar?

Has anyone had the opportunity to test RouterOS directly on high-capacity ARM64 servers with 2X100Gbps NICs or something similar?

Surprised haven't seen anyone try and run ROS on a HoneyComb on here.

...
The secret that no one will tell you.. either because they have not lost time testing.. is the NIC cards... in order to suceed with Multi-CPU x86_64 on RouterOS v7.15.xx stable version.. is the Network cards... because they are related to IRQ on the CPU..

basically the more IRQs slots available on the NIC in order to connect to various cpus to split the load for processing the packets the better..
...

thanks a lot. learnt a load in that
maybe i can convince my SO to buy me some testing equipment for further replacing "big C"

Amazing performance! Thanks for your sharing. Here is my test result. Ros 7.15.2 running on PromoxVE 8.2.7. I got 19.29Mpps throughput without firewall config, tested by Trex 3.5. Hardware platform is E5-2697A V4 with Mellanox MCX4121a NIC. A tiny ITX box.

Hello,
have you ever experienced that the nic installed on the Dell server is not detected in Mikrotik?
I have this problem, where the nic is detected up on the Dell 630 IDRAC but is not on / running on Mikrotik OS version 7.16 or 7.15.3. When disabled on the other side it turns on, only in the Mikrotik OS it is dead.

please help me, any input is greatly appreciated.

Hello,
have you ever experienced that the nic installed on the Dell server is not detected in Mikrotik?
I have this problem, where the nic is detected up on the Dell 630 IDRAC but is not on / running on Mikrotik OS version 7.16 or 7.15.3. When disabled on the other side it turns on, only in the Mikrotik OS it is dead.

please help me, any input is greatly appreciated.

what NIC model is it? have you tried "extra nic" NPK package? have you tried a different NIC which worked?

Hello,
have you ever experienced that the nic installed on the Dell server is not detected in Mikrotik?
I have this problem, where the nic is detected up on the Dell 630 IDRAC but is not on / running on Mikrotik OS version 7.16 or 7.15.3. When disabled on the other side it turns on, only in the Mikrotik OS it is dead.

please help me, any input is greatly appreciated.

what NIC model is it? have you tried "extra nic" NPK package? have you tried a different NIC which worked?

We use Dell Inc. servers. The servers we use are Dell Inc. PowerEdge R730 and 630 with Intel(R) 10GbE 2P X710 NIC and Intel(R) 10G 2P X540-t Adapter.

- On the Intel(R) 10G 2P X540-t Adapter NIC when Mikrotik OS is rebooted, the interface on the NIC is immediately detected by being marked with the running (R) flag on Mikrotik OS.
- On the Intel(R) 10GbE 2P X710 NIC Adapter when Mikrotik OS is rebooted, the interface on the NIC does not immediately run on the Mikrotik side and looks like no cable is plugged in, even though the status in IDRAC is detected on (UP). so, we have to take action by disabling and enabling on the opposite side so that intel(R) 10GbE 2P X710 can be detected on Mikrotik OS.

I tried on Mikrotik-OS version 7.15.xx and 17.16 it's the same

latest NIC firmware installed? those cards sometimes do not initialize when a boot happens

latest NIC firmware installed? those cards sometimes do not initialize when a boot happens

yes, the NIC fimware version is up to date.

maybe a bit of a stretch but tried v7.15?
also with the "extra-nics.npk" from the ISO file? (https://download.mikrotik.com/routeros/7.15/CHANGELOG)

not sure if this is even compatible/possible but maybe worth a shot

maybe a bit of a stretch but tried v7.15?
also with the "extra-nics.npk" from the ISO file? (https://download.mikrotik.com/routeros/7.15/CHANGELOG)

not sure if this is even compatible/possible but maybe worth a shot

I have tried using version 7.15 but there is no “extra-nics.npk” during the installation process. in the x86 extra package there is also no “extra-nics.npk”.

maybe a bit of a stretch but tried v7.15?
also with the "extra-nics.npk" from the ISO file? (https://download.mikrotik.com/routeros/7.15/CHANGELOG)

not sure if this is even compatible/possible but maybe worth a shot

I have tried using version 7.15 but there is no “extra-nics.npk” during the installation process. in the x86 extra package there is also no “extra-nics.npk”.

ok then that .npk might only be available vor AMPERE platform.