Hi guys,
We sell IP transit to some smaller ISPs and also offer High Speed to residential clients and our current Router solution is reaching the limit, I would like to deploy a new Mikrotik Router for 40Gbps of traffic. I don't consider the CCR1072 because it only has 10Gbps ports and I'm considering 40Gbps ports now. It needs to handle 4 full BGP tables and some BGP filters. What solution could you suggest me for that kind of setup?

Thanks,

There are no Mikrotik Routers which has 40gbit QSFP+ ports.
You need to look at somewhere else.

Or build a pc machine for that.

There is no peace of Hardware what can do that from Mikrotik.,
Look at needed Server and use CHR

Look for this PDF or Video from Mum Berlin:
Using MikroTik CHR as a BGP edge router by Kevin Myers (IP ArchiTechs, USA)

They don't even push close to 40 Gbps.

And if I build a x86 with CHR, does RouterOS support any 40Gbps card?

Only your host must support 40G interfaces. CHR uses virtual interfaces.

I would love to see if anyone has it working at close to 40 Gbps.

Only your host must support 40G interfaces. CHR uses virtual interfaces.

What is the Virtualization platform that will offer the best performance in CHR? Xen? Hyper-V? Proxmox? ESXi?

Thanks,

There was a good presentation at EU MUM about CHR performance on different hypervisors
https://youtu.be/xcgdGA1W_0o

Only your host must support 40G interfaces. CHR uses virtual interfaces.

I may be wrong , however I beleive the paravirtual vmxnet3 ethernet interface is NOT limited to 10-Gig maximum throughput.
On two different virtual machines both hosted on the same physical VmWare ESXi server, I've been able to acheive about 38-Gig rates.
This was with multiple transfers at the same time going through the vmxnet3 interface.

Thus, I suspect the limiting factor is NOT the paravirtual vmxnet3 Ethernet interfaces - but rather the physical 40 gig interface talking to the real physical world.

North Idaho Tom Jones

Hi guys,
We sell IP transit to some smaller ISPs and also offer High Speed to residential clients and our current Router solution is reaching the limit, I would like to deploy a new Mikrotik Router for 40Gbps of traffic. I don't consider the CCR1072 because it only has 10Gbps ports and I'm considering 40Gbps ports now. It needs to handle 4 full BGP tables and some BGP filters. What solution could you suggest me for that kind of setup?

Thanks,

Hiya

The best Mikrotik device i would recommend for your needs its called "Juniper MX104" it will suit you just fine up to 160gbps, if you require anything further then 160gbps backhaul i recomend the "Juniper MX240" and be happy.



in the past i have done some testing with with some X64 AMD octa core overclocked 4.0ghzp cpus running Mikrotik RouterOS x86/64 enabled, the max troughput i managed to run on the internal cpu core was abour 25gbps on udp and litle over 11gbps TCP..

not shure how it woudl behave on one Dell PowerEdge 6100 series with 48 cores total from all 8 cpus.. with 192GB RAM.. not shure if it would even be detected correctly as on the PC i runned i had 16GB of ram and only 6GB were detected.. i think there is some old post of mine with the pictures of the benchmark tests.

now putting that aside testing is one thing, real day to day job task , with bgp routing, nat, pppoe and so on.. it another completely differente story.

i just do some benchmarking and single thread performance make a HUGE difference . The best cpu i found so far bang for the buck performance is the 8600k i can push over 70 gbps on hyper-v

The best Mikrotik device i would recommend for your needs its called "Juniper MX104"

How is a "Juniper MX104" a Mikrotik router?

Just thought i'd give some input as i have been wondering about the viability of CHR and 40G interfaces. I wanted to lab up a scenario of replacing my 1072 core routers with CHR/40G.

I have a couple Dell R430 Servers with dual 8 Core Xeon servers (running at 2.1ghz) in my lab along with a Nexus 3064 Switch. I purchased a couple intel XL710 cards and setup a lab. The Dell servers each had a XL710 Card in it connecting to the 40G ports on the nexus switch. the ports on the nexus switch were simply setup as trunk ports. I installed the latest version of VMWARE 6.7 as well as 6.44 CHR using the ova off the mikrotik website. The CHR was given access to all vCPU's totaling 32 (as hyperthreading was enabled) as well as 16GB of memory and a 10G HDD. The CHR was trial licensed for Unlimted.

For my test I wanted to run a full config from my core routers to try and simulate real world performance in my specific case. This included on both routers roughly 16 input firewall rules, 11 forward rules pointing to 25 address lists. There are 32 IP addresses and 24 interfaces participating in OSPF although only one is actually neighbored to the other core router through the switch for this test. These routers were also IPV6 enabled and had roughly 9 input rules pointing to only one address list, but also had an equal amount of OSPFv3 interfaces as ospfv2. The only ipv6 address on the router is the loopback.

As I didn't have any other devices with 40G interfaces the only testing I could do at this time was BT test from one CHR through the switch to the other CHR. This was done with both TCP and UDP, from one loopback address to the other routers loopback address. Meaning it was taking the OSPF route between them through the switch as it's path. The VLAN being used for the OSPF route between the routers I adjusted to try different MTU's. I ran bi-directional BT tests, the max I could try is 8996 as VMWARE only supports up to 9000, minus the vlan equals 8996. My initital results are below:

MTU 700
UDP:3.2G/3.2G
TCP:4.4G/3.7G

MTU 1500
UDP:7G/7G
TCP:7.4G/7.1G

MTU: 8996
UDP: 36G/31G
TCP: 12G/12G

I know my use case and testing methodology might not be very exact, I just had some spare parts and was curious what would happen and thought the result might be helpful to others as I couldn't find much about CHR and 40G. If you guys have any ideas to try testing I should have the lab up for a bit longer. Although I'll be out of town next week.

I'd be curious to see what performance you would get with hyperthreading disabled on the hosts.

disabled hyperthreading and adjusted CHR's to use 16 vCPU and re-ran the same tests. the results are below. In my lab testing i've found hyperthreading to be a benefit even though i've read multiple posts saying to disable it. Obviously my BT test between two routers isn't a real world working scenario which might be the difference. But i dont know how else to simulate real world traffic easily in the lab setup.

MTU 700
UDP:3.2G/2G
TCP:3.5G/3.5G

MTU 1500
UDP:5G/3.5G
TCP:6G/6G

MTU 8996
UDP:16G/16G
TCP:8G/8G

Hi,
you should select proper kernel version for CHR on esxi - 3.x

disabled hyperthreading and adjusted CHR's to use 16 vCPU and re-ran the same tests. the results are below. In my lab testing i've found hyperthreading to be a benefit even though i've read multiple posts saying to disable it. Obviously my BT test between two routers isn't a real world working scenario which might be the difference. But i dont know how else to simulate real world traffic easily in the lab setup.

MTU 700
UDP:3.2G/2G
TCP:3.5G/3.5G

MTU 1500
UDP:5G/3.5G
TCP:6G/6G

MTU 8996
UDP:16G/16G
TCP:8G/8G

You seem to have consistent results with and without hyperthreading (with the exception of the last MTU setting) . I'm wondering if you're not hitting a limitation elsewhere?

You could try using iperf on a couple of VMs?

Hi,
you should select proper kernel version for CHR on esxi - 3.x

I used the ova provided on mikrotiks website to import the VM, I assume there template has the correct kernal version. Currently it's showing as "Other 3.x Linux (64-bit)". What are you suggesting I change it to?

You seem to have consistent results with and without hyper-threading (with the exception of the last MTU setting) . I'm wondering if you're not hitting a limitation elsewhere?

You could try using iperf on a couple of VMs?

I did try that, i installed a centos box on both hosts and gave them a single core and minimal resources to just run iperf. The guest os's run at an mtu of 1500 by default and at that level the VM's got a little less then the bandwidth testing between the routers did, but the CPU on the guest OS was also maxing out. I could give the VM's more resources in theory but then it would take away from the chr running on that host. My guess is that the results would be similar with iperf on a vm to the router BT testing if you account for the loss of overall resources to the CHR running the VM at the same time.

I just did a test between two CHR on one of my ESXi 6.0 servers. From a CHR with a P1 license to one with a P10 license:
Edit - Forgot to add that the destination is 6.44 and the one where the test was run from was 6.44.1. Both have 256MB RAM with two vCPU. VMXNET3 adapters.