Feature Request - CHR - VPP & ISO version CHR ROS

I've been using thousands of Mikrotik devices and dozens of virtual VM CHRs in my ISP networks for many years now. So far , all of my Mikrotik products have stood up well against the test of time and they have easily handled my ISPs leaps in network throughput growth.

However , I am bumping up against the CHR's upper-end throughput ( which is normally anywhere from 3-Gig to 6-Gig ) where I am unable to achieve 10-Gig+ routing ( I do not see the current CHR having the ability to route at 25-Gig to 100-Gig wire-speed throughputs ).

So , I have two requests I would like to suggest to Mikrotik:
1), Create a CHR ISO , where the CHR can be directly installed on bare-metal hardware ( and/or ISO installed in a hyper-visor ).
2), Create a new CHR ISO image that default utilizes VPP ( Vector Packet Processing ).

Re #1, ISO ; this has been suggested many times by myself and others. This paves the way to install a CHR ISO version on a bare-metal server and eliminates the overhead throughput delays of a hypervisor.

Re #2, VPP ( Vector Packet Processing ) ; This is a new much faster method of processing packets. The routing routines & drivers are re-written to keep software in CPU cache ( greatly increases CPU cache hits by 4 fold and more - where as the current older drivers mostly have CPU cache misses ). VPP is designed to use CPU cache hits - which greatly speeds up throughput and is multiple magnitudes faster.

Note: Other non-Mikrotik VPP software routers ( Vyoxxx, PfSexxxx, Linuxxx ) are now close to or aiming for, or hitting the 100-Gig wire-speed routing throughputs on normal bare-metal x86 Xeon hardware. Recently I have been testing out some of these ISO installable router products - and they are fast ( much faster than my VM CHR routers ). I am now getting ready to start testing out the currently-available various VPP software router products. However , it makes me feel a little sad that for what I know that Mikrotik has zero plans for ISO and VPP router products.

Years-ago , there was 10-Meg, then years later 100-Meg, then years later 1-Gig
Yesterday , I only needed to route at less than 10-Gig
Today , I need to route at 10 to 50-Gig
Tomorrow , I need to route 100-Gig and possibly faster
On the horizon, I see 400-Gig being the ISP's in-house standard in routing

What is Mikrotik planning to support as we move to tomorrow's network throughput demands ?

North Idaho Tom Jones

1. CHR is designed to run in virtual environments and can easily handle Tbps without any issues. But if you really want to run bare-metal, go for ROS x86_64. But why? Properly set up virtual environments are just as fast as bare metal and are way easier to manage.

2. VPP is a user-space solution and doesn't run within an embedded Linux kernel like ROS because of resource constraints.

@ OP / Tom

We are in the same boat as you we are growing but in a much slowly pace as a side effect I think we can still wait at least a few more years to wait for VYoS or Bison Router to mature, any chance you can share other cost and effective solution you are looking into?, I hate to ask this since this is MT forum but it seems like they are not interested in competing with SP space

1. CHR is designed to run in virtual environments and can easily handle Tbps without any issues. But if you really want to run bare-metal, go for ROS x86_64. But why? Properly set up virtual environments are just as fast as bare metal and are way easier to manage.

2. VPP is a user-space solution and doesn't run within an embedded Linux kernel like ROS because of resource constraints.

VPP is being mostly developed for the Linux kernel and has had some headway in those VPP routines being ported to BSD kernels.
As for embedded Linux kernels, VPP is almost the same as other software drivers/routines that can be compiled into a Linux kernel.

As for the current ROS ... can you show me any ROS product that can route and/or nat at 40-Gig to 100-Gig line-speed rates ? There are none. However , in the Linux VPP world , these throughput rates are ( or almost ... ) possible right now.

Re: Properly set up virtual environments are just as fast as bare metal and are way easier to manage.
Well ... almost.
I think what you are saying is more like :
Properly set up non-VPP virtual machines are just as fast as bare metal and are way easier to manage.
IMO
- VPP routing is magnitudes faster than non-VPP routers
- VPP CGN-nat is magnitudes faster than non-VPP CGN-nat routers
- VPP is approaching ASIC line-speed rates and has the potential to outrun some older ASICs.
- VPP is less jittery under heavy load than a non VPP router under heavy sustain loads
- With VPP , there is normally one cache pre-load then run in CPU cache and that same CPU cache is repeatedly used for as long as possible. Without VPP, there are no cache pre-loads that are re-used because the CPU cache is re-loaded with other parts of the driver - which results in CPU cache-misses - which results in much slower CPU throughput.
- With VPP, even if you have a CPU interrupt and the CPU re-writes the cache , when it returns there is one pre-load and the CPU cache is re-used again and again over and over again.

@ OP / Tom

We are in the same boat as you we are growing but in a much slowly pace as a side effect I think we can still wait at least a few more years to wait for VYoS or Bison Router to mature, any chance you can share other cost and effective solution you are looking into?, I hate to ask this since this is MT forum but it seems like they are not interested in competing with SP space

A couple of things for you.
- PfSense currently have a VPP software router ( TNSR on a Linux kernel ). I have heard that it is near 100-Gig wire-speed on good/modern x86 bare-metal hardware with newer PCIe # 100-Gig network interfaces.
- Linux already has VPP options you can package install.
- VyOS already has VPP options you can package install.

When I have some free time , I want to build and test some Linux VPP routers on some newer bare-metal servers. If they are faster than my other software routers, then I might put them into production use at my ISP.

- PfSense currently have a VPP software router ( TNSR on a Linux kernel ). I have heard that it is near 100-Gig wire-speed on good/modern x86 bare-metal hardware with newer PCIe # 100-Gig network interfaces.
- Linux already has VPP options you can package install.
- VyOS already has VPP options you can package install.

This is interesting I hope VyOS will mature much faster it is pretty much streamline for us, rather that rolling out my own linux solution TM, thanks for the input

VPP is user-space software, used in projects like FD.io, DPDK and similar. It's not suitable for embedded network OS environments like ROS.

EDIT:
The above applies to OSes like pfSense, VyOS, BSD, and others.

If I read the point correctly ...

VPP = User space ... and does not fit with in and embedded network OS like ROS.

Why do you think that embedded NOS can't run user space stuff? Running something inside kernel space doesn't make it any faster or smaller or anything ... it's just running with higher privileges (so it can crash the whole device even easier) and that's it. But it's quite much easier to introduce (purpose-built) code which handles some data effectively if it's run in user-land (no need to pursue Linus or one of his "henchmen" to approve code push into kernel source code), kernel space only has to provide API (ABI) which allows user-land code to pass the data left and right.

Well, it won’t work with the current product line. Study the basics and you’ll understand why.

Well, I've read some (general) articles on VPP ... and I still don't get it: why is it orthogonal to embedded NOS such as ROS?

Well, it won’t work with the current product line. Study the basics and you’ll understand why.

re: ... Well, it won’t work with the current product line. ...
Are you referring to it wont work with current CHR ? ( if so , CHR does not run on bare-metal ). CHRs don't have direct access to all of the physical hardware.
Are you referring to ROS on current hardware ? ( well , the current hardware does not have any CPU cache that I know of ).

However some years old x86 Xeon motherboards with VPP enabled Linux are sustaining 100-Gig network routing throughputs ( measured and verified ).

However some years old x86 Xeon motherboards with VPP enabled Linux are sustaining 100-Gig network routing throughputs ( measured and verified ).

are you referring to FRR + VPP or something else care to elaborate more please?

What is Mikrotik planning to support as we move to tomorrow's network throughput demands ?

i think the current path is towards ASIC forwarding as other vendors do since 20 years ago, letting general purpose CPU mostly for control plane

aka L3 Hardware Offloading

under the "right" circunstances (including some limitations on features) is feasible to do 200gbps of forwarding on a CCR2216 using only 1 Rack Unit of physical space and around 100 watt of power consumption at a very reasonable price

with future ASIC's is feasible to expect this forwarding numbers go scale

i understand that for some use cases Forwarding using General Purpose CPU's can be a reasonable choice but supporting the wide spectrum of hardware posibilities and combinations come with its own challenges

Well, I've read some (general) articles on VPP ... and I still don't get it: why is it orthogonal to embedded NOS such as ROS?

To fully utilize VPP (What is VPP?) as a fully-fledged router you need to pair it with a user-space network stack that has all the necessary capabilities using frameworks like FD.io or DPDK and DPDK enabled drivers. This also requires an additional user management interface to orchestrate the control plane similar to how ROS interacts with the Linux network stack (ie netfilter/nftables). Although some people are experimenting with a possible kernel module as a workaround, VPP fundamentally operates in user space to maximize performance by avoiding the overhead of kernel context switches. Anyhow, all these developer frameworks are pretty resource-intensive.

A similar solution in the Linux kernel space is eBPF/XDP which normally has lower resource consumption compared to user space networking.

Re: ... i think the current path is towards ASIC ...

Well crafted ASICs can and do often improve specific types of I/O throughput on specialized hardware.

But what about x86 and CHR ROS on hypervisors and bare-metal machines. There are no router I/O ASICs on common generic motherboards.

IMO, if there are no tik plans to utilize newer faster router I/O software/driver router routines , then that might possibly be an end end-of-life for a product. All hardware and software manufacturers should always be researching and testing methods to improve their products to not only keep up with todays routing throughput demands , but also be working on faster products that can meet the increasing I/O throughput demands for future I/O throughput demands. Most consumers are not willing to stay with products that do not keep up with their current and future needs - the same applies to routers. If an ISP has a demand/need to support 1,000+ customers with 100-Meg to 1,Gig accounts, they will find a way , or that ISP will loose customers to another ISP.

well , just my 2-cents of what I would like to see.

North Idaho Tom Jones

Isn't it counterintuitive for MT to push their hardware sales rather than CHR + VPP that's why it appears they are not interested to make this happen?
I hope this is not the case I think most of the SP guys here including us is willing to pay for a reasonable price just to make this happen anyone care to share their thoughts?

Isn't it counterintuitive for MT to push their hardware sales rather than CHR + VPP that's why it appears they are not interested to make this happen?
I hope this is not the case I think most of the SP guys here including us is willing to pay for a reasonable price just to make this happen anyone care to share their thoughts?

loloski,

I look at it this way:
If you manufacture and sell blue widgets, but customers are also wanting to buy and asking for green widgets - where do those customers go to buy green widgets.

Being in the computer communications industry for almost 50 years now , I've seen many companies slowly loose their customers because somebody is offering a better/cheaper mouse trap.
If somebody wants a better/faster/cheaper mouse trap , then make it and sell it to them and create more loyal customers in multiple markets.
It's better to have more customers in multiple markets than only target specific customers in common narrow vertical markets.

@Tom,

I do agree with you I hope whoever decides on the other side of the aisle also think the same way as you do, but evidently this is not the case here I think and firmly believed they have their own winning formula that they believed to make them thrived and Service Provider oriented product is not among them being on this space I personally feel we are less important for the lack of better word I hope i'm wrong hahaha

VPP is just a frontend towards Intel DPDK which means that you set aside cores from your onboard CPU to not be part of the kernel processing.

This means that alot of the kernel and userspace overhead is removed for these cores which means that they will maximize performance for a single task such as routing packets.

The performance boost is in the range of give or take:

- Regular interruptbased processing: >250kpps per core.

- Pollbased processing: >1Mpps per core.

- Intel DPDK (VPP and the other frontends): >10Mpps per core.

So yes this will add to the size of the image but at the same time the CHR is meant to be runned on x86 without any additional switchchips (Marvell, Broadcom etc) for offloading so having the image increase by another 10-100MB or so shouldnt be an issue (compared to the ROS images used on Mikrotik hardware who have a flash limit of about 16MB).

On the other hand I have always seen Mikrotik as this cheaper alternative to get switchchip offloaded hardware (with RouterOS or SWOS) but for a true software solution I would personally go for VyOS (routing) or OPNsense (firewalling) depending on the task.

I would prefer if Mikrotik fix their current issues (VRF, MLAG etc doesnt work when L3-offloading is enabled along with DNS and logging-services are broken when it comes to using them with VRF) before trying to add time and resources to relase a VPP edition of CHR.

They are going to be slave on fixing does you mentioned issues indefinitely if MT continue on their journey for what appears of not giving much attention to unit testing and somehow luck of leader / visionary for much of the codebase instead they just let the individual programmer to be the king of their own not in concert/symmetry to others that's why the codebase is so fragile if someone change something there's a big chances that other part will not work.

I've seen this situation in other project like asterisk where they get tired of fixing chan_sip that push them to replace it with much better alternative, in MT case they are buried on fixing basic things and having a hard time of innovating and step up their game like this VPP + CHR, I hope i'm wrong honestly it's very hard for me to criticized MT if I had a chance and I don't like it, but we are married to their ecosystem because of the perceived benefit it's true we can't deny that MT is really a compelling alternative to the big guns like juniper and cisco and that's something I have to live and hate.

I'm just speaking with the hard truth I don't mean to antagonize MT I'm just voicing my observation as being an outsider to their company hoping that they improved someday

VPP/FD.io/DPDK/SNABB, etc. = User space networking.
eBPF/XDP = (Semi) kernel space.
CHR = RouterOS for virtual machines in kernel space.

Going with a user-space solution would require some serious iron and building a brand-new control plane to create a versatile router like ROS used in the current product line. That's just not going to happen.

viewtopic.php?p=1088625#p1088148

They can do it for sure they have R & D and programming team that can work hand in hand if they have incentive to do it as a business entity. This is just business decision after all if they gonna do it or not, just my 0.2$

Those customers who buy these customized solutions operate in a completely different market segment than the current product range Mikrotik offers.

Plus, Mikrotik doesn’t have the expertise or resources needed to develop brand a new user-space NOS. Only companies like Cisco have the muscle for that kind of work. Just think about how long it took to migrate from v6 to v7 even with a fairly similar foundation to work with. Developing a completely new control plane and a brand new user-space network operating system isn't realistic, from either a business perspective or a technical standpoint.

... Mikrotik doesn’t have the expertise or resources needed to develop brand a new user-space NOS ...

Am I correct that Mikrotik's ROS is based on Linux ? There are already working available Linux VPP packages. To me it seams like it would be somewhat easy for a tik programmer to spin up a VPP CHR. I suspect the bulk of the work would be in modifying some of the ROS front-ends to properly communicate with the new VPP interface names and new/additional VPP user space. But from what I am reading , after that - almost everything is the same.

I for one would really like to see a ( beta ) VPP-CHR vm version or better yet an ISO VPP-CHR bare-metal version. Then put it out there for some loyal dedicated Tik enthusiasts to test and report back what throughput they experience. Then if things look positive , start to modify some of the most ROS CPU intensive packet processing routines to utilize Vector Packet Processing concepts and target code to utilize CPU cache-hits.

fyi - yesterday , I upgraded some non ROS vm production routers to updated versions which now have a faster control plane ( not VPP enabled yet ). Wow - a 4x throughput improvement in network routing. These are some production routers that I have started using to replace some CHR routers. ( next for me will be to install some VPP packages on those Linux based routers and see if it is faster again ).

Actually I don't think they can't do it, they just have a miss opportunity to hire good engineering talent because they require the developer to know their language as prerequisite I don't think it's a bad thing for them, but that will surely impact the hiring process. Honestly you are right they are doing fine for the current market they want to cater, maybe just maybe I'm one of the few who expect/demand more to them

If pfsense/netgate can do it with their TNSR offering MT can do it too, they are both small and young company as far as i can see another example is bison router which I'm also about to evaluate in a next few months.

re: ... If pfsense/netgate can do it with their TNSR offering MT can do it too ...

VyOS, Debian, Ubuntu, BSD and some others ( including FRR ) also have VPP packages available.
If CHR had it, I would be all-over-it and testing it out in a heart-beat.

@ Tom

I hope if you don't mind asking this question do you have current test setup at least with FRR + VPP how's the performance and any gotcha? I don't mind getting my hands dirty again to rollout pure linux solution as long as they are worth it.

I'm also eyeing for VyOS but as far as i know they are unstable as of this time or I'm just a fool thinking it's not usable at this point?

@ Tom

I hope if you don't mind asking this question do you have current test setup at least with FRR + VPP how's the performance and any gotcha? I don't mind getting my hands dirty again to rollout pure linux solution as long as they are worth it.

I'm also eyeing for VyOS but as far as i know they are unstable as of this time or I'm just a fool thinking it's not usable at this point?

1'st I am a dedicated and loyal Mikrotik customer. I only look to other products when there is no Tik solution for what I want/need to do ( mostly speed ).
That being said , this year , in my ISP , I replaced a dozen CHR routers with Vyos routers with similar router configurations/functions. There was an immediate increase in speed and packet throughput was less jittery and under heavy 6+Gig loads , there was far less packet loss. I am testing out the latest rolling1.5 releases. Everything has been very stable - and not a single gotcha yet.

also - I think they are a much much smaller organization than Mikrotik is.

Did you test VyoS 1.5 rolling + VPP addons how is it? anyway there's no excuse for me now to create a test environment since VPP addons is now available for testing

Re: ... Did you test VyoS 1.5 rolling + VPP addons how is it? ...

I have not played around with the VPP addons yet.
I only updated to the latest rolling release to get the faster control plane. And wow

I will be testing the VPP add-on packages on my lab network test routers b4 putting VPP into production use.

Hey guys, since this is a MikroTik forum would you mind continuing the VPP/VyOS/pfSense lab discussions elsewhere? Maybe on their respective mailing lists, user forums or perhaps Reddit?

Hey guys, since this is a MikroTik forum would you mind continuing the VPP/VyOS/pfSense lab discussions elsewhere? Maybe on their respective mailing lists, user forums or perhaps Reddit?

You are correct re : ... this is a MikroTik forum ...
My posts are intended to be targeted towards asking for a feature request to have VPP included in CHR, and an ISO installable version of ROS.

Re: ... Did you test VyoS 1.5 rolling + VPP addons how is it? ...

@TomjNorthIdaho wrote:

I have not played around with the VPP addons yet.
I only updated to the latest rolling release to get the faster control plane. And wow

I will be testing the VPP add-on packages on my lab network test routers b4 putting VPP into production use.

Well, it doesn't seem like it. Also, if you still don't understand why FD.io/DPDK won't run inside CHR you might need to do some studying as I'm not sure you fully grasp the basic concept yet. Anyway, if you really want to play with FD.io be prepared to spend some serious hours on the initial setup. And remember these are developer frameworks, not plug-and-play runtime modules. Good luck!

https://fd.io/docs/vpp/v2101/usecases/index.html
https://fd.io/community/

Larsa , thanks for the links in your last post.

Wow - I've got some reading to do ....

@ Tom

I hope if you don't mind asking this question do you have current test setup at least with FRR + VPP how's the performance and any gotcha? I don't mind getting my hands dirty again to rollout pure linux solution as long as they are worth it.

I'm also eyeing for VyOS but as far as i know they are unstable as of this time or I'm just a fool thinking it's not usable at this point?

They are very stable.

Their LTS version is currently 1.4.0 while the development edition named "nightly builds" is at 1.5.

Both are based on stable Debian Bookworm (12.x) and using stable FRR (9.x) as routing engine. So you could do your own "VyOS" by installing latest Debian stable and latest FRR stable but what they add is the frontend of vyos-configd (along with a few other features such as persistent directories through overlayfs etc).

The good thing IMHO with VyOS compared to Mikrotik is that they are alot more like a regular traditional NOS as in if you have used Cisco, Arista etc you will quickly get used to the config syntax of VyOS. While with Mikrotik its a bit steeper hill to climb (not to mention all the confusion and differences when it comes to how to configure to get offloading properly working between different hardware models).

The bad thing with VyOS is the current lack of hardware offloading support so their solution is a pure x86 (there exists some ARM images aswell) with some boost from VPP which currently is at alpha/beta stage (the VPP addon that is).

VyOS CLI is pretty much the same with Juniper so I don't have problems with it, I just commissioned a box yesterday doing OSPF + NAT and it's working as expected with light load around 1G, what I'm really interested and want to see how does it perform with VyOS + VPP

They got some metrics available at https://vpp-docs.vyos.dev/performance/ but I think you need to contact them to get access to the VPP addon (I think it will be licensed in future).

1), Create a CHR ISO , where the CHR can be directly installed on bare-metal hardware ( and/or ISO installed in a hyper-visor ).

It is (somewhat) available today, just buy a licence and install x86 ISO on your bare PC and/or server of choice.

ISO installer has been available since day one. What is this discussion even about, sorry?

ISO: https://download.mikrotik.com/routeros/ ... .16rc1.iso
IMG: https://download.mikrotik.com/routeros/ ... .16rc1.zip

ISO installer has been available since day one. What is this discussion even about, sorry?

ISO: https://download.mikrotik.com/routeros/ ... .16rc1.iso
IMG: https://download.mikrotik.com/routeros/ ... .16rc1.zip

It's about customers wanting an accelerated software Data Plane on Mikrotik CHR e.g. VPP+DPDK

And as I’ve tried to explain pretty clearly several times, running a user-space network stack like FD.io/VPP/DPDK in an embedded kernel-based network operating system like ROS just doesn’t work.

But it seems like it’s not really sinking in because people either don’t seem to understand the issues or don’t want to accept the facts, ie running FD.io/DPDK in ROS/CHR is kinda superfluous if you catch my drift! And for the last time, those are developer frameworks!

Sure it works, thats how the others using VPP/DPDK does it.

Edit: But sure, they dont have 16MB of storage for their systems.

Question:
- What are people getting for routing throughput where there are two CHR routers on different servers - where one is on one hypervisor and the other is on another hypervisor ?

On my lab network ( all machines are old 2.10-GHz CPUs with PciE-3 and 40-Gig network cards
When testing with two CHRs ( both CHR's on different physical servers )
Multiple iperf3 test machines on the LAN side of each CHR router
- with CHR routers , I hit about 6 to 8 Gig maximum.
- with a set of some home-brew vm Debian VPP routers , I replaced my CHR routers & hypervisors , I hit about 38-Gig ( I only have 40-Gig network cards - but I do have 100-Gig switches )
my speed testing was using multiple iperf3 clients behind one router to multiple iperf3 servers behind a second router on another box.
* I am not using SR-IOV or special pci passthrough network card configurations


So again , my question: What are people getting for routing throughput where there are two CHR routers on different servers.
Edit - also ... What are people getting for routing throughput where there are two x86-ISO ROS routers on two different bare metal servers.