What is the fastest CHR platform for BGP & OSPF 10-Gig
Looking for some advice & suggestions

Questions:
For a CHR running BGP (and possibly also running OSPF) with 10-Gig ethernet interfaces , with throughput always greater than 1-Gig (average peak MAX of 6-Gig) ---
What hardware platofrm is the absolute fastest? (SuperMicro ? CPU? (clock speed & CPU cache & cores)
What hyper-visor works best for a high-throughput BGP CHR ? (VmWare , BeHive, Linux-KVM, BSD-KVM, or other hyper-visor)
Depending on the hyper-visor , what settings work best/fastest ?

- My BGP will be doing Full routes
- If /when running OSPF also, full routes from the BGP tables
- I will have a simple queue for 2-Gig burstable to 6-Gig (or 3X faster this year) (I want to manage my purchased bandwidth instead of my upstream Internet provider doing BW on my feed).

FYI - I have a 10-Gig Internet connected network that was static routed to my Cisco T2 sup card. I just switched it over to full BGP on a CHR and now things are around 10 to 30 percent slower under load. So I am willing to purchase the fastest hardware to run a CHR.

I look forward to your suggestions and comments.

North Idaho Tom Jones

I do not use BGP myself in any way that you describe but as I understand that the BGP process is singlethreaded you are most lightly looking for a CPU that can handle singlethreaded performance very well.
Have a look at this table:
https://www.cpubenchmark.net/singleThread.html

If money is not an issue then Intel® Xeon® W-2155 is a good option. Other you can just have a good look at the table in the link to view the singlethreaded performance for each CPU.
Regarding hypervisor my experience is it won't matter that much. Go with what you and CHR can support. It usually gives the best results

Regarding server model I think it probably wont make that much of a difference but make sure you get networks card that play nice with you hypervisor.
I run a CHR under Hyper-V and the broadcom card in that specific server has been a pain in the a** to get to work properly. Cheap broadcom card in that server but it creates a lot of stange TCP issues in the external network. So a good CPU togather with a good network card in the server is probably the most important things for you.

depending on your usage, couple of things to consider:
CPU speed and cores:
Mikrotik 1036 has 36 cpu cores, 1072 has 72 cores - mikrotik Mhz rate is much lower than Xeon CPU but you will have 4-8 Xeon cores unless you budget is really wild.

if the application is couple of servers talking to each other (few connections) then higher CPU rate would be better (CHR)
if the application is lots of connections to multiple locations (mikrotik and more cores) would be better
basically the more things you have (firewall rules, qos, nat, connections) multi core with lower CPU rate will be better.

RouterOS 6 is NOT Multi-threaded:
1 BGP session with full routing table (670k) takes 1 core mikrotik core to 100% load most of the time (I have not tested this on CHR but I would imagine the utilisation would be quite high which means one of your Xeon cores will be dedicated mainly for the BGP)
I've tested couple of days ago 2 1036 routers with 10 Gbit link between them - maximum you can get per connection is 1.2 Gbit (CPU core is 1.2 Ghz) so when you do bandwidth test (1 connection) between both routers max speed tops at 1.2 Ghz and cpu on 1 core goes to 100%. (I would imagine here CHR with Xeon will give you much better performance)

I hope this helps.

I've tested couple of days ago 2 1036 routers with 10 Gbit link between them - maximum you can get per connection is 1.2 Gbit (CPU core is 1.2 Ghz) so when you do bandwidth test (1 connection) between both routers max speed tops at 1.2 Ghz and cpu on 1 core goes to 100%. (I would imagine here CHR with Xeon will give you much better performance)

What you are seeing is BW test tool limitation not connection bandwidth limitation.

Let's assume that money is not an issue for which Intel Xeon CPU.

So now we have a mix of processors we can work with.

So our options come down to

Clock-speed -vs- cores -vs- CPU-cache -vs- #-of-Physical-CPUs

(Intel does not make 8GHz, 1-Gig-cache 100-core CPU yet - but I bet they will be out in less than 10-years)

(and by the way ... I disable hyper-threading, simulated CPUs don't count)

This topic about fastest I/O throughput with a CHR brings up another topic.

I wish there was a CHR ISO boot CD image which would allow a bare-metal install. This would eliminate the need for a hyper-visor to host a guest CHR system.

All hyper-visors (KVM, VmWare ESXi & ESX & BeHive -and- all others)
- Inject overhead
- Have some limitations
- Add propagation delay
- Consume resourses
- Create some CPU cache misses (because the hyper-visor is also doing stuff which auto-cleares some of the CPU cache holding CHR data)
- Inject an upper-layer hyper-visor level of shared system-CPU interrupts (CHR I/O network devices -and- the hyper-visor can all be sharing interrupts - thus more I/O overhead)
- Add an additional level of management
- Add an additional level of security vulnerabilities.
- Intercept CHR I/O (ethernet, disk & ...) and get processed by the hyper-visor prior to getting out or in from the physical network
- Add additional reboots for hyper-visor updates/upgrades
- Hyper-Visors are also subject to lockups, leaks, load-slow-downs & problems

An ISO install on a bare-metal install would eliminate the need for a CHR admin to also deal with the above listed issues

So ....
If we want a balls-out, pedal-to-the-metal, balls-to-the-wall, fastest-of-the-fastest CHR, we also need to consider the limitations of the hyper-visor.

So - back to the original question ,,, what is the fastest CHR hardware solution for BGP (and/or BGP-and-OSPF) ????

Keep in mind while considering the fastest solution:
- 680,955+ network prefix counts (aka BGP routes)
- 680,955+ OSPF routes (from the BGP routes)
Thus, a CHR talking to up-stream providers doing BGP and the CHR also doing OSPF could see 2.7 million routes (as of today - right this moment).
And - the BGP tables are growing every day and will never stop growing !!!

North Idaho Tom Jones

What type of hardware are you using today for your CHR?

What type of hardware are you using today for your CHR?

Currently
SuperMicro (2-years old)
Two Xeon E5-2690 v2 processors , 10-core , 3-Ghz
Hyper-threading disabled
Intel 10-Gig 2-sfp+ network card
VMware ESXi 6.5.x (latest)

This used to be a pretty fast box a few years ago.
Guest hosted CHR/ROS btest to 127.0.0.1 is always faster than 19 Gig