CHR needs VM-specific stuff. VM tools being a nice example. Or all kinds of virtual drivers. And VMDK disks distributed as SCSI instead of IDE would also be nice.

CHR should run containers, docker or whatever.

We see that the CHR is getting quite popular, so we would like to ask everyone, what new features would you like to see in the future? List your top requests for Cloud Hosted Router (CHR) and virtual machines that we do not have yet.

CHR should run containers, docker or whatever.

CHR should run containers, docker or whatever.

Thats great
+1

Please clarify how you would use it and why

CHR should run containers, docker or whatever.

clarify please. why and how?

(1) A new CHR variant could be added that would be a Docker container. For many people, this would probably be the easiest way to run a CHR variant, given the low overhead and easy setup of Docker containers. This would avoid the requirement to deploy and maintain a more robust (also higher overhead) virtualization system. Docker would also make it easy to mix CHR containers with other network tools, or even applications.

RouterOS in a Docker container- I'm not sure that is at all possible. My understanding is that Docker containers are meant to run a single application, not the whole OS. Containers are self-contained in that they contain an application plus all the required libraries, but not the OS kernel. RouterOS is an OS, not an application.

RouterOS as a Docker host... Come on, guys, we are talking about the CHR here, which is already a virtual appliance. You need Docker host- just run it in a separate virtual appliance. Keeping things simple is always a GoodThing.

RouterOS in a Docker container- I'm not sure that is at all possible. My understanding is that Docker containers are meant to run a single application, not the whole OS. Containers are self-contained in that they contain an application plus all the required libraries, but not the OS kernel. RouterOS is an OS, not an application.

A Docker image doesn't contain the Linux kernel, but it does contain everything beyond the kernel, including the shell and OS binaries. RouterOS does use the Linux kernel, but it has proprietary binaries and an opaque shell on top of it. So it can theoretically be made into a Docker image.

CHR should run containers, docker or whatever.

clarify please. why and how?

In every single setup i put a Mikrotik device on a border

RouterOS in a Docker container- I'm not sure that is at all possible. My understanding is that Docker containers are meant to run a single application, not the whole OS. Containers are self-contained in that they contain an application plus all the required libraries, but not the OS kernel. RouterOS is an OS, not an application.

A Docker image doesn't contain the Linux kernel, but it does contain everything beyond the kernel, including the shell and OS binaries. RouterOS does use the Linux kernel, but it has proprietary binaries and an opaque shell on top of it. So it can theoretically be made into a Docker image.

crowley:~ me$ strings vmlinux | grep '^Linux version'
Linux version 2.6.35 (build@build6) (gcc version 4.5.0 (GCC) ) #15 Fri Oct 12 08:26:11 EEST 2012

crowley:~ me$ strings vmlinux | grep '/build/' | head -1
/home/build/5.21/kernel/linux6/arch/x86/kernel/alternative.c

Make CHR to run on barebone switches. http://www.edge-core.com/productsInfo.p ... 143&id=264

How hard would it be to make a CHR that did not do routing but acted as a synchronization server for Mikrotik routers.
Everyone knows that they can use VRRP however lets say you add some IP's, firewall rules, users etc to the master and then you have to remember to also do it on the slave. It would sure be awesome if it had similar functionality to many other offerings where master config is replicated to slave. Ideally the master would have this ability on its own without a third configuration server however any way that its possible would be welcome. I would even pay for a different license level or additional feature upgrade as I know it would take some development.

P.S I am aware that some have cobbled up some scripting to try and make their own replication. I am talking about an official Mikrotik function.

* BGP multicore support

whenever that is.

How about some basics... open-vm-tools... It seems pretty asinine to me that we have come this far with CHR, and still no basic vm tools.

How about some basics... open-vm-tools... It seems pretty asinine to me that we have come this far with CHR, and still no basic vm tools.

Could you please specify the exact features you would need to be implemented as they are kernel dependent and have to be tailored for the RouterOS.

How about some basics... open-vm-tools... It seems pretty asinine to me that we have come this far with CHR, and still no basic vm tools.

Could you please specify the exact features you would need to be implemented as they are kernel dependent and have to be tailored for the RouterOS.

https://kb.vmware.com/selfservice/micro ... rnalId=340

The parts that would be relevant to RouterOS would be:

Incorrect display of network speed
Provides the ability to take quiesced snapshots of the guest OS
Synchronizes the time in the guest operating system with the time on the host



VMWare Tools source code is available.

• Monitor Health of every CHR deployed
  Centralised Files management (Thinking about hotspot files)
  Centralised firewall custom chains management (jump-to=centrallymanagedfirewallrules)
  Ad-hoc tunnelling system (Controller forwarding / local forwarding like CAPsMAN)
  Centralised Users [AAA and management] (It would be useful to find a connected device through 100 CHRs deployed and check it's status)
  Failover / Backup configuration centrally managed (Avoiding VRRP Configurations)

How about some basics... open-vm-tools... It seems pretty asinine to me that we have come this far with CHR, and still no basic vm tools.

Could you please specify the exact features you would need to be implemented as they are kernel dependent and have to be tailored for the RouterOS.

https://kb.vmware.com/selfservice/micro ... rnalId=340

The parts that would be relevant to RouterOS would be:

Incorrect display of network speed
Provides the ability to take quiesced snapshots of the guest OS
Synchronizes the time in the guest operating system with the time on the host



VMWare Tools source code is available.

Incorrect display of network speed - could you illustrate this with an example?

Provides the ability to take quiesced snapshots of the guest OS

Incorrect display of network speed - could you illustrate this with an example?

The question is exact features for YOU, not ALL features.

And please, make support of SCSI\SATA controller, not IDE

[admin@MikroTik] > /system resource print 
                   uptime: 3m56s
                  version: 6.41rc20 (testing)
               build-time: Aug/29/2017 06:41:04
              free-memory: 14.9MiB
             total-memory: 32.2MiB
                      cpu: Intel(R)
                cpu-count: 1
            cpu-frequency: 1899MHz
                 cpu-load: 0%
           free-hdd-space: 32.1MiB
          total-hdd-space: 63.5MiB
  write-sect-since-reboot: 424
         write-sect-total: 425
        architecture-name: x86_64
               board-name: CHR
                 platform: MikroTik
[admin@MikroTik] > /system resource pci print 
 # DEVICE   VENDOR                                            NAME                                                        IRQ
 0 00:14.0  LSI Logic / Symbios Logic                         53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev: 0)          9
 1 00:07.0  Intel Corporation                                 82371AB/EB/MB PIIX4 ACPI (rev: 8)                             9
 2 00:06.0  Apple Computer Inc.                               KeyLargo/Intrepid USB (rev: 0)                               11
 3 00:04.0  InnoTek Systemberatung GmbH                       VirtualBox Guest Service (rev: 0)                             9
 4 00:03.0  Intel Corporation                                 82540EM Gigabit Ethernet Controller (rev: 2)                 10
 5 00:02.0  InnoTek Systemberatung GmbH                       VirtualBox Graphics Adapter (rev: 0)                         11
 6 00:01.0  Intel Corporation                                 82371SB PIIX3 ISA [Natoma/Triton II] (rev: 0)                 0
 7 00:00.0  Intel Corporation                                 440FX - 82441FX PMC [Natoma] (rev: 2)                         0

And please, make support of SCSI\SATA controller, not IDE

What do you mean with SCSI controller?[/code]

In our build environment we use Qemu tools to convert images to various formats (qemu-img convert), including to VMDK. To convert a VMDK image created by Qemu to ESXi compatible format, VMWare tools (vmkfstools) are necessary. They are provided together with the ESXi server so you can do it on your ESXi. Instructions how to install the CHR on ESXi are available here - https://wiki.mikrotik.com/wiki/Manual:C ... e_ESXi_6.5

In our build environment we use Qemu tools to convert images to various formats (qemu-img convert), including to VMDK. To convert a VMDK image created by Qemu to ESXi compatible format, VMWare tools (vmkfstools) are necessary. They are provided together with the ESXi server so you can do it on your ESXi. Instructions how to install the CHR on ESXi are available here - https://wiki.mikrotik.com/wiki/Manual:C ... e_ESXi_6.5

In our build environment we use Qemu tools to convert images to various formats (qemu-img convert), including to VMDK. To convert a VMDK image created by Qemu to ESXi compatible format, VMWare tools (vmkfstools) are necessary. They are provided together with the ESXi server so you can do it on your ESXi. Instructions how to install the CHR on ESXi are available here - https://wiki.mikrotik.com/wiki/Manual:C ... e_ESXi_6.5

OK you wanted features suggestions.

Ditch all the above and distribute CHR as an ovf like a normal virtual appliance rather than all this total mess you just mentioned.

Guys, I think CHR should be kept clean, as it is now. It's damn small, and you can backup and restore your vm image in disaster recovery scenario quickly.
Thats virtual router, not full blown linux machine. Thats why routeros (beside its bugs, sometimes) is rock solid. Adding too much stuff will end in sw size grow, bugs, attack vectors and much more not predicted situations.
Better focus on networking/drivers/vmtools stuff.

Why you cant just run separate linux with dockers? Since you already running vm environment. It will be even better - you can assign cpu priorities, and in cause of web server or other docker service DoS, you will have just web service down, not whole CHR machine.

From my side, sriov support, vmtools: safe shutdown, memory queiescent.
Plus, maybe some firewall matcher to allow traffic to/from licencing server ( need add static ip now, hope it will not change soon).

Guys, I think CHR should be kept clean, as it is now. It's damn small, and you can backup and restore your vm image in disaster recovery scenario quickly.
Thats virtual router, not full blown linux machine. Thats why routeros (beside its bugs, sometimes) is rock solid. Adding too much stuff will end in sw size grow, bugs, attack vectors and much more not predicted situations.
Better focus on networking/drivers/vmtools stuff.

Why you cant just run separate linux with dockers? Since you already running vm environment. It will be even better - you can assign cpu priorities, and in cause of web server or other docker service DoS, you will have just web service down, not whole CHR machine.

From my side, sriov support, vmtools: safe shutdown, memory queiescent.
Plus, maybe some firewall matcher to allow traffic to/from licencing server ( need add static ip now, hope it will not change soon).

I agree. Many request a swiss army router that can act as a PBX and control their vacuum cleaner, toggle the lights on their fish tank and other things that have no place in a router.

Some perhaps are more into saving money by requesting Mikrotik put everything they need into the software. Seriously if your a business you will have a separate server for PBX and non routing functions.

Some would like it to be more of a UTM appliance which while probably possible would take an amazing amount of development and would require Mikrotik to grow significantly to support such a product and no doubt enter us into the subscription based pricing model to make it worthwhile for them.

Regarding OVF.
You really think there's a "one for all" configuration that will satisfy most of the users?

Regarding OVF.
You really think there's a "one for all" configuration that will satisfy most of the users?

Users can adjust parameters after they installed the OVF. That is not a problem.
MikroTik should be able to define parameters like disk size, amount of memory, required devices etc as they are also
able to build routers according to the RouterOS requirements. When users want to set more memory, they can do so.
The reason that an OVF is required is mainly to get a correct definition of the disk image that is compatible between
RouterOS and ESXi. As it is now, it has to be converted before it works reliably. When this is not done correctly, it
can appear to work but a few days later the router crashes when it tries to write something to disk.

Regarding OVF.
You really think there's a "one for all" configuration that will satisfy most of the users?

Users can adjust parameters after they installed the OVF. That is not a problem.
MikroTik should be able to define parameters like disk size, amount of memory, required devices etc as they are also
able to build routers according to the RouterOS requirements. When users want to set more memory, they can do so.
The reason that an OVF is required is mainly to get a correct definition of the disk image that is compatible between
RouterOS and ESXi. As it is now, it has to be converted before it works reliably. When this is not done correctly, it
can appear to work but a few days later the router crashes when it tries to write something to disk.

Ok, but does it work in the OVA that we have provided earlier (https://www.mikrotik.com/download/share ... S_CHR1.ova)? Can it be imported in the ESXi in the manner that you require without any additional steps?

tested it - works fine, but disk still connected as IDE.
I can change required HW parameters after deploying, no problem for that.

Regarding OVF.

There might be various scenarious how our users would use the CHR. In the OVF there's already specified RAM, HDD, Interface count, etc. - and it might be different for different use cases. We actually have already created such one off template (https://www.mikrotik.com/download/share ... S_CHR1.ova). And in response we got requests to adjust the parameters. You can still take that OVA file, upgrade RouterOS to the newest version and use it.

You really think there's a "one for all" configuration that will satisfy most of the users?

Please check whether this OVA suits your needs and can be deployed on the ESXi without any additional steps:

* It has SCSI drive
* It has been exported from the ESXi

https://www.mikrotik.com/download/share ... R_SCSI.ova

If this suits your needs we'll see how to automate the creation of such OVA file upon RouterOS version releases.

Please check whether this OVA suits your needs and can be deployed on the ESXi without any additional steps:

* It has SCSI drive
* It has been exported from the ESXi

https://www.mikrotik.com/download/share ... R_SCSI.ova

If this suits your needs we'll see how to automate the creation of such OVA file upon RouterOS version releases.

That's it, thanks!

Please check whether this OVA suits your needs and can be deployed on the ESXi without any additional steps:

* It has SCSI drive
* It has been exported from the ESXi

https://www.mikrotik.com/download/share ... R_SCSI.ova

If this suits your needs we'll see how to automate the creation of such OVA file upon RouterOS version releases.

That's it, thanks!

So it works as expected, yes?

Please check whether this OVA suits your needs and can be deployed on the ESXi without any additional steps:
* It has SCSI drive
* It has been exported from the ESXi
https://www.mikrotik.com/download/share ... R_SCSI.ova
If this suits your needs we'll see how to automate the creation of such OVA file upon RouterOS version releases.

@kasparskr
For OVA and ESXI
it don't know if it's possible to be able to select a OVA compatible version, since your ova is generated on a newest version, so who use esxi < 5.5 it is unable to run it

Please check this OVA. HW version has been changed to 10 and should work on the ESXi 5.5. It still works on the ESXi 6.5.

https://www.mikrotik.com/download/share ... 6_40_3.ova

Our procedure is as follows:
* We convert the VMDK file that is currently available on the download page to the the thin format (with vmkfstools on the ESXi machine)
* Create a VMX file with basic parameters
* Register it to the ESXi (vim-cmd solo/registervm)
* Then from a remote machine using 'ovftool' we generate the OVA

For the VM name "displayName" parameter is used.

UPDATE: Adjusted the VMX, added "annotation" field which shows up as "Notes"

Please check this OVA. HW version has been changed to 10 and should work on the ESXi 5.5. It still works on the ESXi 6.5.

https://www.mikrotik.com/download/share ... 6_40_3.ova

Please check this OVA. HW version has been changed to 10 and should work on the ESXi 5.5. It still works on the ESXi 6.5.

https://www.mikrotik.com/download/share ... 6_40_3.ova

This is wonderful! Even if you don't plan to keep doing this process for future versions can you please make sure the link is available on your download page anyway? Install the OVA and upgrade only works if we can find the OVA

The actual version will change over time and this name cannot really be changed anymore.
(you can rename the VM but inside the VM directory the vmdk files will keep the original name - confusing)

Yes. The .ova of course can have a version in its name, but the name suggested in the wizard when importing the .ova is better left generic.
(and the note probably as well, although that can easily be edited afterwards when desired)

Ok, the web interface does not suggest any name, the VM name field is blank when deploying an OVA:

Ok, the web interface does not suggest any name, the VM name field is blank when deploying an OVA:

Is that a change that has been made? Or maybe it depends on VMware version and/or procedure used?
In my case it suggested a name that included the version number.

CHR should run containers, docker or whatever.

Thats great
+1

Please clarify how you would use it and why

CHR should run containers, docker or whatever.

Thats great
+1

Please clarify how you would use it and why

Just leave it here: https://www.youtube.com/watch?v=PivpCKEiQOQ

We see that the CHR is getting quite popular, so we would like to ask everyone, what new features would you like to see in the future? List your top requests for Cloud Hosted Router (CHR) and virtual machines that we do not have yet.

I would like to see a CHR ISO image.
I would like the ability to directly install a CHR ISO onto a server (skipping & not using a hypervisor). Thus a CHR could have direct access to all hardware devices and eliminate the need for a hypervisor.

I would like to see a CHR ISO image.
I would like the ability to directly install a CHR ISO onto a server (skipping & not using a hypervisor). Thus a CHR could have direct access to all hardware devices and eliminate the need for a hypervisor.

How is this different from the normal x86 install that has literally been an option for over a decade? Download: https://mikrotik.com/download (X86 section, right after ARM - the CD image is literally an ISO as you requested) and license info: https://wiki.mikrotik.com/wiki/Manual:L ... nse_Levels (Non-CHR X86 licenses are feature-based, not port-speed-based). CHR was introduced because the basic X86 version assumed it was running on bare metal, and wasn't properly suited for being installed as a VM.

Fist - x86 is a 32-bit system

Fist - x86 is a 32-bit system

I was not aware that the X86 downloads were 32 bit only. Every one of your other points is based on this issue, so it really sounds like what you are asking for is an X86 install that is 64-bit. I understand at this point it may be easier to base this off of the CHR code, but this isn't a feature request for CHR itself. Also, given the "raw image" that is available, can't you just transfer, e.g. with dd, this to a hard drive? ethernet driver support might be a bit limited based on whatever is in the kernel, but at least this way you'd have a 64 bit OS on bare metal?

I would agree that a x86_64 ISO would be useful. I utilize several CHR and I do like them. The license check in I do find annoying. Kinda has a Microsoft feel.

The Elephant in the room is why there is not a CCR that has the balls to handle the more demanding tasks quickly and reliably. I don't think it matters if they have 400 cores if the frequency is only 1GHz or 1.2GHz

Perhaps if they could offload BGP to a FPGA or something. I would imagine that if Tilera GX came in 3GHz multicore we would not be complaining as much and CHR and x86_64 would not be that huge of an issue.

Yea - I've been wanting to transfer a virtual CHR hdd system to a physical system and dump the hyper-visor and try running it on bare metal. I think it would work.

Second - x86 does not have any paravirtual support

Sixth - 64-Bit operating systems also support 64-bit access to memory (faster)
Seventh - 64-Bit systems also can also use more efficient 64-bit compilers and use 64-bit instructions (faster again)
Eighth - I believe 64 bit systems also have additional registers (more registers than 32-bit systems) (again faster again)
Ninth - a fully 64-bit compiled system can process more instructions (some new 64-bit instructions) which can result in fewer CPU clock cycles need to perform a function. (((64-bit hardware can easily out-run 32-bit hardware with both CPUs clocked at the same CPU clock speed.

Eleventh - a 32-bit system running 32-bit code can use twice the CPU/memory read/write clock cycles as a 64-bit system running 64-bit code. (again - lots of these - faster again)

Show me any Tilera / ROS system that can btest udp to 127.0.0.1 and get faster than 20 gig (or even half of that or even half of half of that) - there is no such thing.

We have several places where we run CHR on AWS, both as central location for management/vpn-termination/dude and to use RouterOS for VPN-tunnels instead of having to struggle with the AWS VPN service.
To take this concept further, we would really be helped by the ability to install CHR on KVM and Google Cloud as well.

Make CHR to run on barebone switches. http://www.edge-core.com/productsInfo.p ... 143&id=264

Please support virtio SCSI storage controllers, see viewtopic.php?f=15&t=120413

Currently i have some VM provided by Bandwagon,which use a disk driver so CHR can not run on it.

Currently i have some VM provided by Bandwagon,which use a disk driver so CHR can not run on it.

I guess it will always be possible to craft some environment in which a binary-only distribution cannot run.
The question is if it is worth the trouble to cater for that, or one just has to wait until all the "incompatible" and "difficult" environments just vanish from the market because customers no longer want to put up with those difficulties.

Difficulty level Easy (less than an hour)

Please add Metarouter

Please add Metarouter

VM inside VM? Are you serious?

• Graceful shutdown/reboot on XenServer
• XenMotion between hosts without freezing MikroTik router. (currently will cause MikroTik to lock up)

Small web server, if MikroTik would want to give us something like that, would also make sense for any other RouterOS, not just CHR. And it's also true for pretty much anything else you can think of (any service and such).

CHR needs VM-specific stuff. VM tools being a nice example. Or all kinds of virtual drivers. And VMDK disks distributed as SCSI instead of IDE would also be nice.

Desired features:

• Graceful shutdown/reboot on XenServer
• XenMotion between hosts without freezing MikroTik router. (currently will cause MikroTik to lock up)

I doubt you will see such features. Would it be cool? Absolutely! I would imagine you will grow old and die before you see the requested features

I made a Mikrotik based SD-WAN solution for a client that works quite well. I spent a month in the lab testing various scenarios to get the desired function. I think Pepwave & Mushroom are some possibilities if you are looking for manufactures that currently support SD-WAN solutions.

I doubt you will see such features. Would it be cool? Absolutely! I would imagine you will grow old and die before you see the requested features

I made a Mikrotik based SD-WAN solution for a client that works quite well. I spent a month in the lab testing various scenarios to get the desired function. I think Pepwave & Mushroom are some possibilities if you are looking for manufactures that currently support SD-WAN solutions.

Can you please share with us what you mean with 'I made a Mikrotik based SD-WAN'. Do you mean: A -> some SD-WAN -> Mikrotik router -> some connectivity -> Mikrotik Router -> some SD-WAN -> B . Or you mean some directly integrated solution with roureros / maybe meta router ?

Good to see MikroTik are responsive.
Xen Tools and VM Tools are now available in 6.42RC. The "current" build of 6.42 can't come fast enough for me!

Don't ask about CHR. I think that this product isn't interesting for MT developers as in the beginning of existence.
To bypass your problem, use ISO of your favourite live linux (without installation) and my guide:
viewtopic.php?t=120413

Keep in mind if your cloud provider (ramnode) use current version of KVM, they probably forcing to use VirtIO-SCSI.... and CHR currently doesn't supports it.
viewtopic.php?f=15&t=124905&start=100#p626094
I will be appreciated if you will test it.

MT Support
is so difficult to create an iso version of CHR ?
because some KVM like ramnode dont allow to use virtual disk only iso

ISO is something to make a CD from. Why don't you simply migrate to a cheaper and more modern Cloud provider?

Linode can do it, Hetzner can do it. More powerful machines, SSD disks and cheaper price:

https://www.hetzner.com/cloud

Could you tell us please when is planned to add VirtIO-SCSI boot support?

Could you tell us please when is planned to add VirtIO-SCSI boot support?

*) chr - added "virtio-scsi" driver on KVM installations;

Can we run CHR in Google Cloud now?

What's new in 6.42rc28 (2018-Feb-16 07:02):

*) chr - added "virtio-scsi" driver on KVM installations;
*) chr - added support for Hyper-V ballooning;
*) chr - added support for Hyper-V guest quiescing;
*) chr - added support for Hyper-V host-guest file transfer;
*) chr - added support for Hyper-V integration services;
*) chr - added support for Hyper-V static IP injection;
*) chr - added support for NIC hot-plug on VMware and Xen installations;
*) chr - fixed additional disk detaching on Xen installations;

We would like to receive your comments about these CHR features.