CHR using Apple Virtualization Framework (via UTM)

Amm0 · Tue Feb 20, 2024 5:25 am

.

See post #56 below for UTM CHR installation instructions for the
most streamline way to install RouterOS CHR on Intel-based UTM
viewtopic.php?t=204805#p1125233

While ARM64 is possible using same "packaging into ZIP" approach used above – I just cannot test it easily. The thread below details Apple Silicon macOS approach for UTM and CHR's ARM64 images — which including using QEMU mode and "Use NVMe Interface", but check below for ARM64 details.

chr-utm-screenshot.png

I didn't intend on this thread to have 50+ posts... Which now includes multiple GitHub projects...
CHR re-partitioning: https://github.com/tikoci/fat-chr
UTM packaging to ZIP/URL https://github.com/tikoci/chr-utm

At this point the rest of the content in this post below is now historic:

I'd been experiment using Apple Virtualization Framework (https://developer.apple.com/documentati ... ualization) using the option in MacOS-version of UTM (https://mac.getutm.app).

I normally using VMWare Fusion/ESXi. But UTM (in Apple, NOT QEMU, mode) seems to work okay in my limited testing for a few Linux images. For a lark, I wanted to try CHR using UTM+AppleVMF. Issue is Apple's Virtualization Framework ONLY support UEFI, which CHR – for some unknown reason includes EFI boot files, but the diskpart stuff isn't right.

As it turns out, I somehow got it work. I use CHR on Mac for testing configscripts, so "working" is about all I need. I tried to write up what I did, but post if you try and doesn't work. I do know that CHR does not show the logon prompt on the emulated video display (although the serial port is mirrored as the screen shown below) but serial port and network work fine. /tool/speed-test in a UTM+Apple VM matched a similar bridged network using VMWare Fusion. Since UTM+Apple (e.g. no QEMU) uses less CPU and boot WAY quicker than VMWare Fusion, so far pretty good.

Even more historic information, potentially could be removed:

See #15 post below for more streamlined process to using CHR with Apple virtualization, on Intel-based Macs:
viewtopic.php?t=204805#p1059466

The following is the "manual" way now....

This post had the clue on UEFI issues with CHR: viewtopic.php?p=1025068&hilit=UEFI#p933799
While I did NOT use the script...the post was 100% correct:
For some reason, Mikrotik does actually includes the right bits for UEFI support...but UEFI requires a FAT16 partition — NOT the ext2 that's the boot partition in the CHR .IMG file — so CHR does not work unless some Legacy BIOS is used (which Apple, and other VM platforms, do not offer).

The Mikrotik's help for CHR on Vultr has the big clue to how to avoid needing a script (which will not work on Mac) — boot to the SystemRescure image. See https://help.mikrotik.com/docs/display/ ... stallation

So the Vultr instruction ALMOST works for UTM + Apple Virtualization: e.g. booting UTM with Apple Virtualization enabled to the "SystemRescueCD"... But those need to be COMBINE with the @kriszos to convert the IMG file's partition from ext2 to fat16. Most of the write up below is the process involved in that...

Inside UTM, select:
- hit "+" to add a new VM
- select "Linux" as the type
- check the "Use Apple Virtualization" box
- pick the SystemRescueCD as the "Boot ISO" (after downloading the ISO: https://www.system-rescue.org)
- pick cores/memory as desired
- pick a disk size - RouterOS does not need a big disk... I used 1GB but can be smaller/bigger as desired
- skip shared directory (hit continue)
- now, pick a name for the VM, I used "AppleCHR" and IMPORTANTLY check "Open VM Settings"
- dialog with VM setting will appear:
- under Network, you may to change to use "Bridge" mode (or add more network interface... or less likely, use shared if you really like multiple NATs)
- under Devices section, use "+ New..." to add a "Serial" port (below item network) – default is bring up serial port a new window which is what you want
- reviews other setting, but serial above is about only CRITICAL thing to add
- hit "Save"
- in UTM main window hit Play icon to start the new VM

I'm guessing Vultr is more forgiving than Apple about UEFI. The solution to this is same as @kriszos post above allude: convert the ext2 partition to fat16. And this can be done after the "dd" in SystemRescueCD. The specific steps I used:
- follow other steps from https://help.mikrotik.com/docs/display/ ... stallation
- at same SystemRescueCD's terminal...
- "mountall" after CHR has been extracted/copyed to the /dev/vda disk
- "mkdir /tmp/vda1" to create folder to store RouterOS EFI files in ext2 boot partition
- "cp -r /dev/vda1/* /tmp/vda1" to copy the EFI files
- "umount /dev/vda1" and "umount /dev/vda2" to un-mount the boot and main partitions
- "startx" - to launch X11 (probably some CLI to "parted" work too, but GUI is helpful with disks IMO)
- Hit the icon for "gparted" in the task bar (or run "gparted" from a Terminal in X11 desktop)
- Use the drop-down in upper left to select the /dev/vda disk. You should see two partitions: one ext2 and one ext4.
- Right click on the first one, marked "boot" on right, and select format.
- Pick "fat16" as the format type. This will "cue" the operation.
- Next commit the, use menus or hit the green checkbox in toolbar. Assuming it says success in status window & still marked "boot"...closed gparted.
- Bring up a terminal window from menu/taskbar, and again type "mountall"
- At same terminal, use "cp -r /tmp/vda1 /dev/vda1" to copy the original files back to the re-formatted 1st partition.
- Finally "poweroff"
- CHR should be "installed" and bootable at this point, except you need to remove the SystemRescueCD from UTM – since it will boot to RouterOS at this point.

Note: I only tested on Intel Mac – but I believe UTM with Apple Virtualization framework will work on M1/M2/M3 (ARM) Mac via Rosetta. If someone tries, that LMK if it works.

optio · Tue Feb 20, 2024 5:34 pm

Did you experienced any FS corruption running on UTM with Apple Virtualization Framework? Related to this still open issue for UTM -> https://github.com/utmapp/UTM/issues/4840
I tried with Debian 12.5 (arm64, without Rosetta) which installation is using kernel 6.1 and FS gets corrupted eventually on M1 since some kernel patches for Silicon (M) chips support are made in 6.2, but not sure if these kernel patches are directly related to FS corruption and M chip support or is UTM bug with Apple Virtualization Framework in general (following github issue comments maybe it is only UTM bug).

Larsa · Tue Feb 20, 2024 5:54 pm

@Ammo, thanks for very interesting info! Personally I love Parallels Desktop but for various reasons we are exploring alternative solutions. UTM/VMF might be an option when it becomes stable enough. Will definitely look into it further..

optio · Tue Feb 20, 2024 6:03 pm

For me VFM and VirtioFS seems to be stable using with Docker on M1, have some databases running with it in containers (even some with x86_64 arch using Rosetta), so I guess UTM needs to improve stability and it will be great free VM alternative on Macs.

Amm0 · Tue Feb 20, 2024 9:18 pm

Well, I'm surprised CHR even worked – why I shared. Not really not sure why there isn't a pre-made CHR image that works with UEFI since X86 does. I'm sure there is probably a more optimized path to a working .IMG file than my write up since I just wrote down what I did than figure out the best way to do it.

When you use Apple Virtualization in UTM, I'm not sure UTM does very much other than call the needed Apple API. Since the framework is new, my bet is Apple. For example, the serial console works with Apple VMs, but not the display. I know CHR shows the login prompt on VMWare, but using UTM+Apple Virtualization is video display is blank, but winbox etc via network does so not an issue for me. BUT... without the serial port enabled... it look like it doesn't work...

I just used Ubuntu using Apple Virtualization – that does seem to work, at least for a week. But I don't really stress any of my VMs. I'm trying to ween myself off VMWare overall (Fusion is more useful for me since I use ESXi and the VMs work same on both).

The one problem UTM solve is VMWare Fusion's boot speed ( viewtopic.php?p=1048889&hilit=fusion#p1048889 and others) – it seems to be a recurring problem. UTM+Apple, boots in few seconds.

My only complaint so far is Apple does not support USB passthrough, which is useful to get an LTE modem into CHR.

Amm0 · Wed Feb 21, 2024 10:39 am

I tried this tonight WITHOUT using UTM, only Apple. I used some swift from an Apple sample project that used VZEFIBootLoader() & another sample with the serial console window. And changed the disk image to use same converted CHR disk image (e.g. 1st/boot just changed from EXT2 to FAT16 type, as described in first post) as disk. And CHR still works. Since everyone like icons on Mac included a "teaser":

The real trick was @kriszos finding here about FAT16 being required by UEFI specs – which is NOT how Mikrotik package the CHR image (e.g. they use EXT2):
viewtopic.php?p=1025068&hilit=UEFI#p933799

sirbryan · Thu Feb 22, 2024 12:45 am

I tried this tonight WITHOUT using UTM, only Apple. I used some swift from an Apple sample project that used VZEFIBootLoader() & another sample with the serial console window.

To confirm,

This window is not the VGA graphics from the CHR VM, but a SwiftUI window connecting to the serial port of the CHR?

Amm0 · Thu Feb 22, 2024 5:50 am

I tried this tonight WITHOUT using UTM, only Apple. I used some swift from an Apple sample project that used VZEFIBootLoader() & another sample with the serial console window.
To confirm,

This window is not the VGA graphics from the CHR VM, but a SwiftUI window connecting to the serial port of the CHR?

Correct, it's a serial console in window using https://github.com/migueldeicaza/TermKit to deal with ANSI. Using Swift without UTM, the VGA display is blank – same as UTM.

It's possible Apple is using the GPU variant of VirtIO display (QEMU's virtio-gpu-pci) – which does not have VGA emulation & likely CHR wants VGA... Apple lets you set size not a specific driver (https://developer.apple.com/documentati ... figuration).

stmx38 · Fri Feb 23, 2024 12:34 am

Did a try on mac arm and it doesn't work for me.
1. SystemRescueCD can't boot and I've used Kali Live (Apple silicon ARM64)
2. After performing all the steps VM can't boot with the same error as SystemRescueCD - "EFI Boot Options returned. Exiting VM"
Something was already discussed on GitHub but it was about arm on arm - ARM64 Linux not booting with Apple Virtualization #5517.

Screenshot 2024-02-23 at 00.16.03.png

Emulation works fine, but slightly slow.

Probably AMPERE (coming soon) may solve that?

Amm0 · Fri Feb 23, 2024 12:39 am

Did a try on mac arm and it doesn't work for me.
[...]
Probably AMPERE (coming soon) may solve that?

Yup. Apparently, the specific are Rosetta will help with any X86's running inside Apple Virtualized machine – the actual Linux disto needs to also be same arm. So "Ampere" support is needed. Actually looked it up: https://developer.apple.com/documentati ... th_rosetta
There was a footnote:

Rosetta doesn’t support the bootstrapping or installation of Intel Linux distributions on Mac computers with Apple silicon using the Virtualization framework. Intel Linux distributions can run using the Virtualization framework on Intel-based Mac computers without the need for this translation capability.

kriszos · Wed Feb 28, 2024 11:08 am

I am glad that my work help you resolve your problem. Hopefully MikroTik will provide us with uefi image packaged by them without this ugly workarounds. More and more modern hypervisors stop providing BIOS options at all my own research come from need to run CHR on LXD that didn't supported BIOS at the time.

Amm0 · Wed Feb 28, 2024 6:12 pm

I am glad that my work help you resolve your problem.

Totally @kriszos - it was worth a search for UEFI in the forum.... but I would have never got to the partition scheme in the RAW image was the reason EFI didn't work & given up before trying gdisk

.

I filed a feature request for UEFI support as SUP-144667 after my post here. I'm not sure they got the issue is generically UEFI support in CHR:

We checked our suggestion list, and you are the only person who asked about Apple Hypervisor.
We noted it, but it is not a big chance that it will be implemented soon.

But tried.

sirbryan · Wed Feb 28, 2024 7:26 pm

Should more of us file support/feature requests for generic UEFI support in CHR?

kriszos · Wed Feb 28, 2024 8:58 pm

Should more of us file support/feature requests for generic UEFI support in CHR?

SUP-145276

Amm0 · Thu Feb 29, 2024 7:04 pm

I took @kriszos's "gdisk+bash" script, with small modification to fetch RouterOS in loop, and put it into a GitHub repo "fat-chr" to build FAT UEFI images automatically via GitHub's Action CI.

Since 7.14 stable came out today, I tried out my GH Action script with @kriszos's gdisk+bash script & have 7.14 stable IMG that work without SystemRestoreCD – I just trigger a build in GitHub and download the .IMG. And 7.14 CHR is running without UTM's QEMU, just Apple's.

IMG files built by GitHub and work with Apple (and likely other/newer hypervisors that require proper UEFI support) are here:
https://github.com/tikoci/fat-chr/releases
I would not use these for production - but to spin up a VM for testing, might be helpful.

This greatly simplify the process of adding a CHR to Intel-based UTM using it's "Apple mode".
In UTM, you need to check the "Use Apple" in wizard and still pick an ISO and create some disk... then in the setting BEFORE starting:
- create new VM in UTM "+" > Virtualize > Linux
- check the "Use Apple Virtualization" & pick some file for ISO boot image – going to remove later so doesn't matter what (could be text file whatever)
- click "next" to everything else in UTM setup wizard...and at end check "Open VM Setting"... then in those settings:
- remove the disk the wizard added
- remove the ISO image
- remove the display
- add a serial port
- under disk, use "Import" and pick .RAW image from GitHub
- give it a name if you want (default will be "Linux #")
- then... start it with Play icon... a console window will appear with RouterOS login [via serial —but you won't know – a VGA CLI look the same, so serial saves some bits/complexity if you think about]

Note: The image only fix the disk part of UEFI support. Something else is needed for VGA display. I think Apple's VirtIO driver does not have VGA mode, which is why Display will show nothing since CHR is liking setting VGA mode. Bash nor GitHub can fix that... something in CHR have to change to deal with the newer VirtIO VGA-less display drivers.

optio · Thu Feb 29, 2024 8:34 pm

Great work @Amm0

chr_utm_i.png

Didn't try yet over Rosetta on M, but it could be issues with FS corruption as I mentioned in my previous post (or is just native ARM issue). Too bad there is no USB passthrough support.

Amm0 · Thu Feb 29, 2024 8:54 pm

Rosetta on ARM Mx only helps with X86 AFTER a ARM-based Linux is loaded. The boot process does not go through Rosetta, only user applications. So this won't work for non-Intel Macs.

. Apple docs say:

Rosetta doesn’t support the bootstrapping or installation of Intel Linux distributions on Mac computers with Apple silicon using the Virtualization framework. Intel Linux distributions can run using the Virtualization framework on Intel-based Mac computers without the need for this translation capability.

So still QEMU on M1/M2/... but "UEFI boot" for CHR should be selectable in other hypervisors using these images I think.

Amm0 · Thu Feb 29, 2024 9:11 pm

But Mikrotik does have "AMPERE" on their download page (grayed out). To me, that's Mikrotik messed-up marketing for ARM64 ("aarch64") as ISO coming soon - but dunno. But gives hope for M1+ Macs.

I used VMWare Fusion for a long while (since it come out) but CPU use is ~half with UTM+Apple on my main Intel Mac for a CHR VM. And various problems with VMWare freezing when booting CHR also gone – it boots in few seconds.

optio · Thu Feb 29, 2024 10:03 pm

Ah yes, I forgot about Rosetta is not supporting Linux bootstrapping. Well we will need to wait CHR ROS for aarch64 then...

sirbryan · Tue Mar 05, 2024 4:16 am

I'm excited to see what people come up with now that the Ampere ISO is out. I tried messing around with it but I can't get things to boot.

ayufanpl · Tue Mar 05, 2024 10:17 am

I was able to boot Ampere/ARM64 ISO on Qemu with KVM acceleration: https://github.com/ayufan-research/mikrotik-qemu-arm64. Works just fine. Tested on Raspberry PI 5 and Rock 5B.

Amm0 · Tue Mar 05, 2024 5:44 pm

I'm excited to see what people come up with now that the Ampere ISO is out. I tried messing around with it but I can't get things to boot.

I tried on Equinox Metal, but couldn't get it work. I do not think it's Mikrotik fault... Metal uses iPXE with netboot.xyz for custom OSes. I use VMWare on X86... so I gave up quick since I don't know iPXE stuff (although similar problem as here – what's the kernel & initfs to use for RouterOS ). But kinda did want to see it boot on something before spelunking on ARM Mac. But no success on Metal.

I don't have an ARM Mac handy, but next time I can check I'll see what happens. At the end of the day, Apple Virtualization needs some EFI executable file and a ramdisk with RouterOS file system (e.g. routeros.npk uncompressed AFAIK) — just how it find/pick them may be tricky align.

Amm0 · Tue Mar 05, 2024 6:00 pm

I was able to boot Ampere/ARM64 ISO on Qemu with KVM acceleration: https://github.com/ayufan-research/mikrotik-qemu-arm64. Works just fine. Tested on Raspberry PI 5 and Rock 5B.

Thanks for sharing your scripts. Some working examples always help.

FWIW... this is a likely a good call:

 ... qemu-system-aarch64 ...   
   -vga none -nographic -monitor none 
  -serial chardev:term0 -chardev stdio,id=term0

RouterOS X86 seems to have problems with VirtIO displays (e.g. on Apple, some Hyper-V, etc) — doubt that be fixed in ARM64.

Still... might be worth a check to see if virtual display actually works with "Ampere"+QEMU

sirbryan · Tue Mar 05, 2024 6:13 pm

I'm excited to see what people come up with now that the Ampere ISO is out. I tried messing around with it but I can't get things to boot.
I tried on Equinox Metal, but couldn't get it work. I do not think it's Mikrotik fault... Metal uses iPXE with netboot.xyz for custom OSes. I use VMWare on X86... so I gave up quick since I don't know iPXE stuff (although similar problem as here – what's the kernel & initfs to use for RouterOS ). But kinda did want to see it boot on something before spelunking on ARM Mac. But no success on Metal.

I don't have an ARM Mac handy, but next time I can check I'll see what happens. At the end of the day, Apple Virtualization needs some EFI executable file and a ramdisk with RouterOS file system (e.g. routeros.npk uncompressed AFAIK) — just how it find/pick them may be tricky align.

I have a Studio and MacBook Pro. You've given me a couple ideas to try. There is an efiboot.img on the ISO. It also has all the packages on the ISO. Kernel boots on Apple and on QEMU but doesn't know where init is.

Amm0 · Tue Mar 05, 2024 7:59 pm

I have a Studio and MacBook Pro. You've given me a couple ideas to try. There is an efiboot.img on the ISO. It also has all the packages on the ISO. Kernel boots on Apple and on QEMU but doesn't know where init is.

I'm not expect here, but re-packaging I can do. I added ZIP file output to my GitHub "X86 UEFI CHR builder", while it doesn't have symlinks and stuff. If you extract the ZIP file, you can see how the IMG get's structured without mounting the disk on Mac (since it's cannot read the 2nd partition):
https://github.com/tikoci/fat-chr/relea ... 25-testing

The CHR ARM64 ISO has a very different structure... My wild ass guess is somehow though the sysconfig stuff it mounts routeros-7.15beta4-arm64.npk as the ramdisk. While it ends in .npk, it's still a normal LZMA compressed file (e.g. 7-Zip) that EFI boot loader tries to find.

But I'm really not the expert on UEFI and Linux boot details, especially on ARM64... If someone figures out the magic sauce, I can automate it.

ayufanpl · Tue Mar 05, 2024 8:45 pm

I have a Studio and MacBook Pro. You've given me a couple ideas to try. There is an efiboot.img on the ISO. It also has all the packages on the ISO. Kernel boots on Apple and on QEMU but doesn't know where init is.

This is likely problem of UEFI or HVF accelerator implemented in QEMU on MacOS.

In general if you disable `Use Hypervisor` kernel will boot. The UTM requires to use `NVME` disk for storage, and I was not able to configure CDROM. I cannot force UTM to use `virtio-blk-pci` for CDROM device. Once you install (somehow) the ARM64 MikroTik will boot in non-hypervisor mode via UTM.

sysconfig for x86 is to boot via legacy BIOS.

ayufanpl · Tue Mar 05, 2024 8:56 pm

OK. You can boot/install via UTM:

1. Create new VM => Virtualize => Linux.
2. For Virtualization Engine: Keep `Use Apple Virtualization` unmarked. For Boot Image: Select arm64.iso. Then, continue.
3. Set min 512MB of memory, and at least one CPU core, at least 1GB of disk size. Then, continue a few times.
4. Edit then VM Settings.
5. In QEMU, unselect Use Hypervisor.
6. Go to Display, and click Delete. Then go to Sound and click Delete.
7. Under Devices, add Serial.
8. Go to VirtIO Drive, change Interface to NVMe.
9. Go to USB Drive, and Delete. This remove the iso image.
10. Under Drives, click New. Select VirtIO, click Import, find mikrotik-arm64.iso.
11. You should be able to start and install. After you install, you can remove the ISO drive.

Of course Hypervisor will not work, so the performance will be slow.

sirbryan · Tue Mar 05, 2024 9:26 pm

Sweet. I got it installed and booted.

Now to figure out how to boot it under Apple's Virtualization Framework to get closer to the CPU.

stmx38 · Tue Mar 05, 2024 9:59 pm

Works for me as well, but I almost forgot that ROS is payed software - we need CHR for ARM

With CHR using x86 emulation we can play for free and boot time is similar to that ARM.

Amm0 · Wed Mar 06, 2024 2:09 am

OK. You can boot/install via UTM:

Just for fun, I tried the ARM64 ISO, emulated in UTM on Intel Mac — WORKED.

Instructions almost just work on Intel UTM with QEMU emulation.... Clearly the NVMe disk seem required (tried IDE or VirtIO – neither worked). But for Intel UTM, the ISO image seems to need to be first converted manually to a .qcow2 with VirtIO for the 1st partition to boot before importing it.

qemu-img convert -O qcow2 mikrotik-7.15beta4-arm64.iso mikrotik-7.15beta4-arm64.qcow2

Amm0 · Wed Mar 06, 2024 5:31 am

Now for Apple Virtualization on ARM MacOS... My best guess is the new RouterOS "ARM64 ISO" is missing a needed "virtio_blk.ko" (at least doesn't appear in ISO or any .NPK)
And since VirtIO is only* disk option on Apple Virtualization, kinda problematic (*if there isn't virtio_blk, there isn't NBD in RouterOS, while NBD is supported by Apple's APIs, it's not in UTM with Apple VM set). Explain why QEMU seem requires NVMe emulation to boot in UTM...

I think the "ARM virtio disk problem" is same problem as video one: just another missing different virtio driver...

patrikg · Wed Mar 06, 2024 10:05 am

Sorry

Secure warning !!

Be careful if you download images other than from the official Mikrotik one.

optio · Wed Mar 06, 2024 10:37 am

warning secured...

ayufanpl · Wed Mar 06, 2024 10:54 am

Just for fun, I tried the ARM64 ISO, emulated in UTM on Intel Mac — WORKED.

Instructions almost just work on Intel UTM with QEMU emulation.... Clearly the NVMe disk seem required (tried IDE or VirtIO – neither worked). But for Intel UTM, the ISO image seems to need to be first converted manually to a .qcow2 with VirtIO for the 1st partition to boot before importing it.
Code: Select all
qemu-img convert -O qcow2 mikrotik-7.15beta4-arm64.iso mikrotik-7.15beta4-arm64.qcow2

The step 10. does this conversion (or rather attachment of RAW).

The NVMe device is required as MikroTik requires some Serial Number to be configured to get Hardware ID. The VirtIO does not allow that.

I'm personally running ARM64 variant on my Rock 5B now via KVM virtualization, and it achieves maximum throughput!

Amm0 · Wed Mar 06, 2024 3:32 pm

Code: Select all
qemu-img convert -O qcow2 mikrotik-7.15beta4-arm64.iso mikrotik-7.15beta4-arm64.qcow2
The step 10. does this conversion (or rather attachment of RAW).

Yeah that may be a bug in UTM (or gremlins)... it seem to try to convert it, and then reports in dialog it cannot find mikrotik-7.15beta4-arm64.qcow. Just something I noticed. (Keep in mind, I tested on Intel Mac using emulation, so may not happen on ARM Mac)

The NVMe device is required as MikroTik requires some Serial Number to be configured to get Hardware ID. The VirtIO does not allow that.

Ah yes, the license scheme is different. Which also makes sense why there isn't a virtio blk driver for it either.

I'm personally running ARM64 variant on my Rock 5B now via KVM virtualization, and it achieves maximum throughput!

Nice! There is hope.

sirbryan · Wed Mar 06, 2024 5:38 pm

I'm personally running ARM64 variant on my Rock 5B now via KVM virtualization, and it achieves maximum throughput!

I'll have to try it via KVM on my SolidRun Honeycomb LX2 that's collecting dust. It doesn't support UEFI boot and their UEFI shim is out of date. Otherwise I'd have spent some more time on booting it natively.

ayufanpl · Wed Mar 06, 2024 5:40 pm

I'm personally running ARM64 variant on my Rock 5B now via KVM virtualization, and it achieves maximum throughput!
I'll have to try it via KVM on my SolidRun Honeycomb LX2 that's collecting dust. It doesn't support UEFI boot and their UEFI shim is out of date. Otherwise I'd have spent some more time on booting it natively.

Via KVM it works just fine. No objections there, and especially a better idea for something that will rather be unsupported by MikroTik.

Amm0 · Wed Mar 06, 2024 6:22 pm

I personally think it better to always use a hypervisor. In years of using Mikrotik, never once tried metal X86 before here.

Mikrotik need to just build a ARM64 CHR version of "AMPERE", since it's not just Apple where this comes up.

And the "security warning" in this thread on UEFI image... easily fixed by Mikrotik changing their build to add one more CHR image with FAT as boot. I put GitHub on that task, and since CI runners are free in public project, easy to share. FWIW, the entire build process is observable/auditable, but it's a fair warning. And I'd rather not be involved package building.

optio · Wed Mar 06, 2024 6:53 pm

And it is also possible to compare checksums from original files is someone is concerned...

sirbryan · Thu Mar 07, 2024 5:52 am

I made a tiny bit of headway trying to boot the installer on my Honeycomb LX2.

Using a UEFI image from SolidRun, the ISO boots via USB. It looks for an NVMe drive (SATA are ignored), and then burps because it can't find the CD-ROM.

Based on what we saw with QEMU, I wonder if it expects the CD-ROM to be on the PCIe bus or something. Booting from a SATA drive doesn't make a difference. Going to try two NVMe drives just for fun.

sirbryan · Thu Mar 07, 2024 7:02 am

Huzzah! Just like with QEMU, two NVMe drives works. But it doesn't see any of the NICs or USB interfaces. That would probably explain why the install kernel doesn't see the USB drive after it takes over from UEFI. In System/Resources/PCI it lists just the NVMe drives.

I wonder what cards in Ampere systems they're targeting with this. The onboard NXP ports aren't available (yet).

Amm0 · Thu Mar 07, 2024 7:13 am

But it doesn't see any of the NICs or USB interfaces.

Isn't there some extra-nic.npk you can install – maybe already did – just a thought.

That would probably explain why the install kernel doesn't see the USB drive after it takes over from UEFI.

It does replicate QEMU where the CD-ROM had to be transformed into a disk image. IDK about modern bootloaders, but AFAIK EFI will mount the CD-ROM for an OS install to START. But when the "real" OS boot, it depends on its own driver for CD-ROM. But a physical disk (or emulated one) be same ID/etc in both EFI and RouterOS kernel.

ayufanpl · Thu Mar 07, 2024 10:21 am

The installer does iterate all devices and try to mount the is9660. It does not care about whether this is in a physical CDROM, USB drive, or NVMe drive. So, the drive with a CDROM image needs to be discoverable. Anyway, don't expect USB drivers to work here.

Amm0 · Thu Mar 07, 2024 7:37 pm

All reasons why I'm pro-virtualization. Good news is seems the world is coalescing around VirtIO since Microsoft and even Apple support.... since dealing with device drivers has been a PITA my entire life.

FWIW, I did see @normis report WRT to "AArch64" on the 7.15beta thread:

CHR images are coming in next betas

This be good news for Mac ARM users. I suspect if it work under Apple Virtualization on ARM Mac... it work any other KVM hypervisor using AArch64. e.g. Apple is kinda "worse case" since only supports VirtIO. KVM long offered VirtIO, but most Linux hypervisors do have other supporting functions too... but with Apple it's 100% VirtIO drivers (and clearly a limited subset of them too).

ayufanpl · Thu Mar 07, 2024 9:09 pm

This be good news for Mac ARM users. I suspect if it work under Apple Virtualization on ARM Mac... it work any other KVM hypervisor using AArch64. e.g. Apple is kinda "worse case" since only supports VirtIO. KVM long offered VirtIO, but most Linux hypervisors do have other supporting functions too... but with Apple it's 100% VirtIO drivers (and clearly a limited subset of them too).

Mac does not offer KVM. It does offer HVF and Virtualization framework. Qemu can support anything HVF.

Amm0 · Thu Mar 07, 2024 10:07 pm

This be good news for Mac ARM users. I suspect if it work under Apple Virtualization on ARM Mac... it work any other KVM hypervisor using AArch64. e.g. Apple is kinda "worse case" since only supports VirtIO. KVM long offered VirtIO, but most Linux hypervisors do have other supporting functions too... but with Apple it's 100% VirtIO drivers (and clearly a limited subset of them too).
Mac does not offer KVM. It does offer HVF and Virtualization framework. Qemu can support anything HVF.

True. I just found it telling that even Apple adopted KVM's VirtIO in "HVF", not their own drivers/VMXNET3/something ... Even to the extent that "HVF" is likely a "even more pure" VirtIO-based hypervisor, than even KVM

More a thought that HVF kinda useful for general VM testing.... if OS has poor VirtIO support, you'll get nothing from Apple.

sirbryan · Fri Mar 08, 2024 6:10 am

After messing around with it as bare metal on the Honeycomb (extra-nics was installed), I booted back into Ubuntu and got it working through QEMU/KVM. With 2 or 8 cores, I can get it to receive 1.5Gbps and transmit 700Mbps, so I'm guessing there's a bit of optimization yet to be had. iperf3 on the host can get 8Gbps to/from my Mac Studio using the stock NXP drivers.

Looking forward to ARM64 CHR and hopefully some more optimization.

ayufanpl · Fri Mar 08, 2024 3:09 pm

True. I just found it telling that even Apple adopted KVM's VirtIO in "HVF", not their own drivers/VMXNET3/something ... Even to the extent that "HVF" is likely a "even more pure" VirtIO-based hypervisor, than even KVM

More a thought that HVF kinda useful for general VM testing.... if OS has poor VirtIO support, you'll get nothing from Apple.

This is incorrect:

1. The HVF.framework is similar to KVM: it provides a function to run a second-level isolated hypervisor / virtual machine. It is very bare bones and requires to handle everything. The HVF is used by QEMU to to provide isolation boundary. QEMU for connected devices, can emulate any underlying device type as long as it is programmed, this is why you can attach non-virtio emulated devices to the VM. The HVF.framework was introduced as a way to provide user-space accessible virtualization, similar to how the `/dev/kvm` is exposed on Linux.
2. On MacOS there's also Virtualization.framework. It is Apple's own QEMU-like emulator. The Virtualization.framework does only expose VirtIO-like devices. Underneath the Virtualization.framework uses HVF.framework.

The UTM can use:

1. QEMU that uses HVF, and does emulate various type of devices - all that are supported by the QEMU.
2. QEMU that uses TCG, basically emulate everything, including CPU.
3. Apple Virtualization via Virtualization.framework, very limted to only a VirtIO-like devices.

Amm0 · Fri Mar 08, 2024 5:58 pm

Thanks @ayufanpl. Used VMWare for decades and for a time built iOS apps. Most server/apps/OS supported VMWare, while KVM was hit/miss. Seems the reverse is happening, which is good since VMWare is now pretty expensive. While I know UNIX/Linux well enough, all the KVM/QEMU stuff I'm trying to learn. Seems complicated onion.

But I now get what you mean when you said "HVF" – that QEMU support for Apple's Virtualization.framework. That's nifty – I thought Virtualization.framework was only in UTM actually. But I though you made-up an weird acronym for "Apple's Virtualization.framework" (which has not acronym AFAIK).

UTM with Apple (outside CHR image problem) "just working" got me thinking... So I did try the Virtualization.framework with Apple's Swift sample to directly to boot CHR, which worked: viewtopic.php?p=1061786#p1057843
I kinda liked the generic concept of have an "/Application" that put some nice SwiftUI (perhaps NOT a terminal) around some Linux thing more generically. At some point, I'll put the "MacOS CHR" code/app on GitHub (perhaps, "AHR"

) when I have time. Hopefully Mikrotik will have a ARM64 CHR image by then too.

ayufanpl · Fri Mar 08, 2024 8:26 pm

Thanks @ayufanpl. Used VMWare for decades and for a time built iOS apps. Most server/apps/OS supported VMWare, while KVM was hit/miss. Seems the reverse is happening, which is good since VMWare is now pretty expensive. While I know UNIX/Linux well enough, all the KVM/QEMU stuff I'm trying to learn. Seems complicated onion.

...

I kinda liked the generic concept of have an "/Application" that put some nice SwiftUI (perhaps NOT a terminal) around some Linux thing more generically. At some point, I'll put the "MacOS CHR" code/app on GitHub (perhaps, "AHR" ) when I have time. Hopefully Mikrotik will have a ARM64 CHR image by then too.

If you are coming from VMWare please checkout the Proxmox VE. It is brilliant. Build on top of Debian, QEMU and KVM. And Open Source. The company behind it offers paid support.

Yes, there are number of apps build on top of `Virtualization.framework` to provide GUI. Since, all heavy lifting is done by Apple to provide emulated hardware support so building GUI is significantly easier task. The UTM provides a GUI for both: Virtualization.framework and QEMU.

Amm0 · Fri Mar 08, 2024 9:03 pm

If you are coming from VMWare please checkout the Proxmox VE. It is brilliant. Build on top of Debian, QEMU and KVM. And Open Source. The company behind it offers paid support.

LOL. Yeah I recently just installed Proxmox VE on an old server to test it... It better from a UI POV for sure. And the disk management is just more logical too. Plus seem QEMU has come a long way since I start with VMWare in the 90s. CHR was an addition on ESXi a few years, but it super handy for dealing with all the networking stuff vSwitch can't & avoid need the even more price vSphere stuff. e.g. RouterOS VRRP on two VMWare ESXI does similar HA things to vSphere & easier, and $100/server is a bargain. On Proxmox VE, there is all Linux newer data plane stuff. Haven't gotten there yet with Proxmox VE.

It was the Fusion to UTM part of conversion that got me here. It was nice to be able test a VMX-based VM on Mac Fusion, and just copy files to ESXi (or vise versa). But if want to do same with Proxmox, I need KVM/QEMU on Mac...

One note: @TomjNorthIdaho (who runs an open btest server from his ISP) is in same boat re VMWare. @ayufanpl, if you had any Proxmox advice, might want to share it here: viewtopic.php?t=203953&hilit=vmware

ayufanpl · Fri Mar 08, 2024 10:16 pm

It was the Fusion to UTM part of conversion that got me here. It was nice to be able test a VMX-based VM on Mac Fusion, and just copy files to ESXi (or vise versa). But if want to do same with Proxmox, I need KVM/QEMU on Mac...

One note: @TomjNorthIdaho (who runs an open btest server from his ISP) is in same boat re VMWare. @ayufanpl, if you had any Proxmox advice, might want to share it here: viewtopic.php?t=203953&hilit=vmware

Not really. The disk images are transferable between Proxmox VE and UTM, since both are based on QEMU, both can boot UEFI, both use VirtIO for most of emulated devices. They are super similar on how they configure QEMU and interact with it. The problem with UTM is that it can run amd64 or arm64 natively depending on MacOS arch. The Proxmox VE at least now is only amd64. So, you can transfer UTM <-> PVE only when running on Intel Macs. There are some GitHub repos trying to bring PVE to arm64 with mixed degree of success.

Kartone · Thu Jun 06, 2024 11:51 am

For those who, like me, struggled to have ROS running on Apple Silicon. I'm sharing the config that, for me, solved the issue:
I'm running UTM Version 4.5.3 (99) on Apple M2 Max, Sonoma 14.5.

Hope it helps.

optio · Thu Jun 06, 2024 8:03 pm

That is system emulation with QEMU, you can run any supported architecture like that on any system where you can run QEMU, this topic is related to CHR virtualization using Apple Virtualization Framework.
With Apple VF CHR arm64 raw image won't boot, but x86_64 it does after changing FS type on ESP in image, in arm64 image ESP seeps to be ok, but it won't boot, didn't have time to investigate more.

pcDidakt · Mon Jul 08, 2024 7:55 pm

Hi Kartone and others

Thanks for that! I am new on ARM and I do know there are different types of ARM Cores etc. Do we know if the AMPERE and Apple Silicon are fully compatible on CPU Level? If so, aren't there optimizations in QEMU so that most of the code runs natively? I mean that we need to emulate the IO is not that strange.

Actually I have never used CHRx86 that much on Bare metal so I know if they supported modules with drivers for Network Cards or it was up to the user to buy NE2000, Intel exxx compatible etc?

Keep up the good work and I would LOVE to see a Youtube snippet about how it looks and runs!

/Johan Löfgren
Sweden

Amm0 · Mon Feb 10, 2025 5:43 am

Given the Super Bowl in US... Amm0's container&script superstore is now offering ...

Easiest way to install a RouterOS* — via URL:

utm://downloadVM?url=https://github.com/tikoci/chr-utm/releases/download/v7.17.2/RouterOS.utm.zip

*assuming you have an Intel Mac with UTM installed — To install UTM for Mac, see https://mac.getutm.com .

On Intel macOS, just cut-and-paste URL below into Safari — including the "utm://downloadVM?" which tell UTM to start the process:

utm://downloadVM?url=https://github.com/tikoci/chr-utm/releases/download/v7.17.2/RouterOS.utm.zip

After using URL, UTM will prompt you download it, and will automatically add CHR to the virtual machine assuming you accept the download.

UTM prompt after utm url.png

You can also download the image directly from the "Release" section on the GitHub project that, essentially, ZIP you a directory with a .plist file and .raw image (and icon). If you expand the ZIP, UTM should see it as virtual machine then (as it will have the ".utm" ending when unzip). So same image packaged for UTM (as a ZIP) is under "Releases" from the GitHub tikoci/chr-utm repo:
https://github.com/tikoci/chr-utm/releases

In either case, you should be able to just start it once it installed in UTM — no configuration of serial ports or disks needed.
To start, click the ">" Start icon next to "RouterOS", and a console will appear

CHR started after install using serial console.png

The username is of course "admin", and CHR still uses no password.

Note: RouterOS will use a UTM "shared" (NATed) network by default. With macOS defaults (i.e. macOS firewall enabled), no port should be available beyond you local Mac. You can feel free to adjust any networking as desired — including none if just want to test scripts or commands. But you can add multiple network adapters, or "bridge" CHR to a real interface if desired (i.e. to avoid NAT once it's configured with passwords and a RouterOS firewall). For example, you can add a USB ethernet dongle to Mac, add that interface as "bridge" in RouterOS UTM image "Network" settings. Anyway, there are network setting to adjust in UTM to present more network (say for testing) or use multiple "shared" ones.

Above uses Apple's "App URLs", which get send to UTM app and is same scheme as "UTM Gallery" uses to install images. The image inside ZIP iimage based on the tikoci/fat-chr "no-gdisk" method, and the image is set to use Apple Virtualization which is super fast at bring up CHR.

Finally to use multiple CHR, you'll need to rename any existing ones first named "RouterOS". So if you want to add TWO CHRs, you change "RouterOS" to "RouterOS 1" and then use same URL/download to add another CHR (and perhaps rename that one to "RouterOS 2"). Basically, it won't automatically install a CHR if same already exists.