RouterOS x86, no support for Chelsio T540 VF?

crosswind · Sat Oct 26, 2024 1:24 am

hello,

i have RouterOS x86 running in a virtual machine (bhyve on FreeBSD 15.0). the virtual machine has four Ethernet PCI network devices:

 2 device="00:05.0" name="Virtio network device (rev: 0)" vendor="Red Hat, Inc." category="Ethernet controller" vendor-id="0x1af4" device-id="0x1000" irq=10 memory=0xC000E000-0xC000FFFF io=0x2080-0x20BF 

 3 device="00:05.1" name="Virtio network device (rev: 0)" vendor="Red Hat, Inc." category="Ethernet controller" vendor-id="0x1af4" device-id="0x1000" irq=10 memory=0xC000C000-0xC000DFFF io=0x2040-0x207F 

 4 device="00:05.2" name="Virtio network device (rev: 0)" vendor="Red Hat, Inc." category="Ethernet controller" vendor-id="0x1af4" device-id="0x1000" irq=11 memory=0xC000A000-0xC000BFFF io=0x2000-0x20DF 

 5 device="00:06.0" name="T540-CR Unified Wire Ethernet Controller [VF] (rev: 0)" vendor="Chelsio Communications Inc" category="Ethernet controller" vendor-id="0x1425" device-id="0x5803" irq=0 
   memory=0xC0012000-0xC0012FFF,0xC0000000-0xC01FFFFF,0xC0008000-0xC0009FFF

however, while the "Virtio" devices are detected correctly, the Chelsio VF is not:

[admin@lab1.i.lfns.org.uk] /system> /interface/print
Flags: R - RUNNING
Columns: NAME, TYPE, ACTUAL-MTU, MAC-ADDRESS
#   NAME    TYPE      ACTUAL-MTU  MAC-ADDRESS      
0 R ether1  ether           1500  06:00:00:00:05:08
1 R ether2  ether           1500  06:00:00:00:05:09
2 R ether3  ether           1500  06:00:00:00:05:0C
3 R lo      loopback       65536  00:00:00:00:00:00
[admin@lab1.i.lfns.org.uk] /system>

i assume this is because the kernel is missing a driver for this card. is there some kind of HCL for RouterOS x86 that lists supported drivers? or is it worth asking MikroTik support to add the appropriate driver?

unfortunately i can't use CHR as it doesn't support bhyve (because MikroTik apparently wrongly believe it's paravirtualisation).

kleshki · Sat Oct 26, 2024 1:29 am

A quick google search I've found there's a way to boot on bhyve guide
Can't try myself, but there's a bunch of guides how to boot ROS7, maybe this could solve your problem

crosswind · Sat Oct 26, 2024 1:33 am

A quick google search I've found there's a way to boot on bhyve guide

thanks, i did see that but it seems it involves a lot of low-level work on the CHR image and i'm worried it's going to break when updating. so i might try it, but it would help to know if CHR includes the driver for this card, then :-)

jaclaz · Sat Oct 26, 2024 2:32 am

Amm0 has a dedicated GitHub repository with the updated scripts/mods, AND pre-made corrected images, see here:
viewtopic.php?t=184254

https://github.com/tikoci/fat-chr/releases

You can try one of the pre-made images.
There are no issues when updating, once an image works it means that the partitioning and filesystems are fine and the update process does not change anything related to partitioning or volume formatting/filesystems, it only writes new files.

crosswind · Sat Oct 26, 2024 3:14 am

thanks jaclaz, this works a lot better than the x86 image (if only for improved licensing :-)

unfortunately though, it still does not seem to include the driver for the T540 VF:

[admin@lab1.i.lfns.org.uk] > /system/resource/print
                   uptime: 4m7s
                  version: 7.16.1 (stable)
               build-time: 2024-10-10 14:03:32
         factory-software: 7.1
              free-memory: 291.1MiB
             total-memory: 512.0MiB
                      cpu: AMD
                cpu-count: 1
            cpu-frequency: 3699MHz
                 cpu-load: 0%
           free-hdd-space: 71.2MiB
          total-hdd-space: 89.2MiB
  write-sect-since-reboot: 880
         write-sect-total: 880
        architecture-name: x86_64
               board-name: CHR FreeBSD BHYVE
                 platform: MikroTik
[admin@lab1.i.lfns.org.uk] > /system/resource/pci/print
Columns: DEVICE, VENDOR, NAME, IRQ
#  DEVICE   VENDOR                         NAME                                                    IRQ
0  00:00.0  Network Appliance Corporation  unknown device (rev: 0)                                   0
1  00:04.0  unknown                        unknown (rev: 0)                                          0
2  00:05.0  Red Hat, Inc.                  Virtio network device (rev: 0)                           10
3  00:05.1  Red Hat, Inc.                  Virtio network device (rev: 0)                           10
4  00:05.2  Red Hat, Inc.                  Virtio network device (rev: 0)                           11
5  00:06.0  Chelsio Communications Inc     T540-CR Unified Wire Ethernet Controller [VF] (rev: 0)    0
6  00:1f.0  Intel Corporation              82371SB PIIX3 ISA [Natoma/Triton II] (rev: 0)             0
[admin@lab1.i.lfns.org.uk] > /interface/print
Flags: R - RUNNING
Columns: NAME, TYPE, ACTUAL-MTU, MAC-ADDRESS
#   NAME    TYPE      ACTUAL-MTU  MAC-ADDRESS      
0 R ether1  ether           1500  06:00:00:00:05:08
1 R ether2  ether           1500  06:00:00:00:05:09
2 R ether3  ether           1500  06:00:00:00:05:0C
3 R lo      loopback       65536  00:00:00:00:00:00
[admin@lab1.i.lfns.org.uk] >

i didn't see anything in all_packages that seems relevant either.

kleshki · Sat Oct 26, 2024 4:43 am

Is there a real reason to use passthrough instead of virtio driver or maybe try to use SR-IOV?

crosswind · Sat Oct 26, 2024 5:07 am

Is there a real reason to use passthrough instead of virtio driver

on FreeBSD host, at least, using virtio in the guest with bridge(4) on the host prevents the host from using LRO/TSO which reduces TCP performance.

or maybe try to use SR-IOV?

i am using SR-IOV - that's what creates the PCI device i'm passing through. this is why it shows up as a 'VF' device.

crosswind · Sat Oct 26, 2024 5:31 am

in the mean time i filed feature request SUP-169557 to add the driver, but if anyone has any other suggestions, i'm all ears.

Larsa · Sat Oct 26, 2024 6:36 am

When using CHR, the NIC drivers must be compatible with and managed by the virtual machine host operating system (ie FreeBSD).

So, MikroTik support won’t really help here - you’ll need to check out the FreeBSD Forums instead.

crosswind · Sat Oct 26, 2024 6:39 am

When using CHR, the NIC drivers must be compatible with and managed by the virtual machine host operating system (ie FreeBSD).

sorry, but this is not correct in the situation i have described here.

when passing a PCI device through to a virtual machine, it's up to the virtual machine kernel (e.g., the CHR kernel) to support the device, *not* the host system.

edit: unless you mean that CHR does not support PCI passthrough at all? that would be understandable (although i would find it surprising) but it's still up to MikroTik to fix it.

Larsa · Sat Oct 26, 2024 9:27 am

No, virtual machines support only a limited set of virtual drivers required for the virtual guest to function properly. The host OS drivers are responsible for managing this support.

You need to configure it properly like: https://forums.freebsd.org/threads/sr-i ... est.70653/. If you run into more issues, try posting a question on the FreeBSD forum.

crosswind · Sat Oct 26, 2024 9:35 am

No, virtual machines support only a limited set of virtual drivers required for the virtual guest to function properly. The host OS drivers are responsible for managing this support.

before i write a longer response, can i please ask, do you know what SR-IOV is?

Larsa · Sat Oct 26, 2024 10:44 am

Yep, it’s also up to the host OS drivers to enable SR-IOV support. The virtual machine can check if SR-IOV is available, but if you try to turn it on without driver support, you’ll just get an error or warning. The host OS still needs proper support for everything required by the virtual machine, so if something’s not working, that’s where you should start troubleshooting.

And just to make it crystal clear, you can’t manage any host drivers from within CHR; everything needs to be done on the host OS.

crosswind · Sat Oct 26, 2024 10:57 am

Yep, it’s also up to the host OS drivers to enable SR-IOV support.

correct. host OS has SR-IOV enabled, this is working in all virtual machines on this host except for RouterOS VM.

The virtual machine can check if SR-IOV is available, but if you try to turn it on without driver support, you’ll just get an error or warning.

right. please read my original post, which is about whether RouterOS x86 / CHR has driver for Chelsio T540-CR VF device. i do not get error or warning, but device does not appear in /interface/print.

this is nothing to do with FreeBSD - question is whether RouterOS has driver for this device.

Larsa · Sat Oct 26, 2024 11:07 am

please read my original post, which is about whether RouterOS x86 / CHR has driver for Chelsio T540-CR VF device. i do not get error or warning, but device does not appear in /interface/print. this is nothing to do with FreeBSD - question is whether RouterOS has driver for this device.

This is where you might be misunderstanding things. It’s all about the virtual machine support at the host OS level, not the guest OS.

RouterOS x86 and CHR are completely different. RouterOS x86 is for bare-metal installations, while CHR is designed for virtual machines that require proper support for virtual drivers. If the virtual drivers in the virtual host and its host OS aren’t set up correctly, you won’t be able to use them in CHR.

crosswind · Sat Oct 26, 2024 11:11 am

This is where you might be misunderstanding things. It’s all about the virtual machine support at the host OS level, not the guest OS.

no, it's not. the entire point of PCI passthrough is that the PCI device is directly accessible in the virtual machine, and the VM's kernel is responsible for identifying the device and attaching the appropriate device driver to it.

the *only* thing that needs to be supported at the hypervisor level is to detect the PCI card and attach it to the VM -- which, as i said, is not a problem on FreeBSD at all.

the problem here is that RouterOS does not provide the required device driver for the PCI device.

i'm afraid you simply do not understand what VM PCI passthrough means.

RouterOS x86 and CHR are completely different. blah blah bha...

and neither of them have the driver for Chelsio T540-CR Virtual Function device. which is what my entire post is about.

crosswind · Sat Oct 26, 2024 11:20 am

let me provide an example from a non-RouterOS Linux system, which is a VM with a T540 VF PCI passthrough device.

we can see the device in lspci:

1? 3!media ~% lspci -v 
00:00.0 Host bridge: Network Appliance Corporation Device 1275
        Flags: bus master, fast devsel, latency 0
        Capabilities: <access denied>

00:04.0 Non-Volatile memory controller: Device fb5d:0a0a (prog-if 02 [NVM Express])
        Flags: bus master, fast devsel, latency 0
        Memory at 800004000 (64-bit, non-prefetchable) [size=16K]
        Memory at c000e000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: <access denied>
        Kernel driver in use: nvme

00:05.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem
        Subsystem: Red Hat, Inc. Device 0009
        Flags: bus master, fast devsel, latency 64, IRQ 18
        I/O ports at 2000 [size=512]
        Memory at c000c000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: <access denied>
        Kernel driver in use: virtio-pci

00:06.0 Non-Volatile memory controller: Device fb5d:0a0a (prog-if 02 [NVM Express])
        Flags: bus master, fast devsel, latency 0
        Memory at 800000000 (64-bit, non-prefetchable) [size=16K]
        Memory at c000a000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: <access denied>
        Kernel driver in use: nvme

00:07.0 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]
        Subsystem: Chelsio Communications Inc Device 0000
        Flags: bus master, fast devsel, latency 0
        Memory at c0010000 (32-bit, non-prefetchable) [size=4K]
        Memory at c0000000 (32-bit, non-prefetchable) [size=32K]
        Memory at c0008000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: <access denied>
        Kernel driver in use: cxgb4vf

00:1f.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
        Flags: bus master, fast devsel, latency 0

4!media ~%

'00:07.0 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]' - this is our Virtual Function device, exposed from the host via PCI passthrough.

and you can see lspci says "Kernel driver in use: cxgb4vf" - meaning the Linux kernel in the VM has attached the "cxgb4vf" driver to this device.

and in "dmesg" we can see the Linux kernel has detected this as a network device:

[    1.021179] cxgb4vf 0000:00:07.0: Using assigned MAC ACL: 06:00:00:00:05:a2
[    1.024707] cxgb4vf 0000:00:07.0: eth0: Chelsio VF NIC PCIe MSI-X
[    7.057836] cxgb4vf 0000:00:07.0: eth0: passive DA port module inserted
[    7.057847] cxgb4vf 0000:00:07.0 eth0: link up, 10Gbps, full-duplex, RX/TX PAUSE

and indeed this network device shows up in Linux:

# ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
    link/ether 06:00:00:00:05:a2 brd ff:ff:ff:ff:ff:ff

the problem i'm reporting in this thread is that, apparently, the RouterOS kernel is missing "cxgb4vf" driver, so it cannot detect the PCI device as a network interface.

crosswind · Sat Oct 26, 2024 12:24 pm

after poking around "routeros-7.16.1.npk", the problem is apparently that ROS ships the driver for "cxgb4" (the PF device) but not "cxgb4vf" (the VF device). hopefully, this means it's fairly simply for MT to enable the VF driver -- but we'll see what support says.

Larsa · Sat Oct 26, 2024 2:12 pm

CHR runs fine on FreeBSD as long as you set up the VM with a compatible VF for SR-IOV or configure the standard drivers properly. That said, SR-IOV won’t give you any extra performance unless you’re sharing the NIC with multiple VMs so the easiest way to get started with bhyve with is to bridge a tap interface to virtio-net.

Just a heads-up: as explained previously you can’t manage host drivers directly from within CHR, everything has to be set up on the host OS. PCI passthrough isn’t helpful here because CHR lacks built-in support for the Chelsio T540, so you’re limited to the SR-IOV-based VFs CHR supports (unless you can get MikroTik to add special SR-IOV drivers for the T540, which isn’t likely).

crosswind · Sat Oct 26, 2024 2:25 pm

CHR runs fine on FreeBSD

no, it doesn't, because the CHR image is not compatible with bhyve -- you need to use the third-party image that jaclaz posted earlier. the stock CHR image won't boot on bhyve and is explicitly documented in the CHR documentation as not supporting bhyve. (i believe MT should modify it to support bhyve, but as things stand, it does not.)

as long as you set up the VM with a compatible VF

literally the entire reason i posted this thread was to ask if the Chelsio T540 VF is, or could be a compatible VF -- you're the one who tried to derail it into "the host OS needs to support the network card and the guest OS doesn't matter".

Just a heads-up: as explained previously you can’t manage host drivers directly from within CHR, everything has to be set up on the host OS.

this has nothing to do with what i'm talk about in this thread.

PCI passthrough isn’t helpful here because CHR lacks built-in support for the Chelsio T540

yes, thank you, finally you accept the point i've been trying to make since you first posted here. the entire point of this thread is to ask about support the T540 VF driver in RouterOS. nothing at all to do with the host OS.

edit: however, it seems like RouterOS *does* support the Chelsio T540, it just doesn't support the VF driver -- which should be trivial for MT to add.

Larsa · Sat Oct 26, 2024 2:52 pm

RouterOS as a bare metal x86_64 installation might possibly have drivers for the Chelsio T540, but you need to check with support. If you need help to install CHR on FreeBSD, you can refer to this guide: "Creating a Mikrotik CHR - RouterOS 7 - Bhyve VM in FreeBSD"

Depending on what you are aiming for: if you just want to play with FreeBSD and CHR, you are on your own; don't expect any support from MikroTik. To get it running without SR-IOV, you can simply bridge a tap interface to virtio-net. On the other hand, if you want a decent environment focused on CHR, use one of the supported platforms: "Mikrotik help - Cloud Hosted Router CHR"

crosswind · Sat Oct 26, 2024 2:58 pm

RouterOS as a bare metal x86_64 installation might possibly have drivers for the Chelsio T540

it does not have drivers for the T540 VF, as you could discover by reading the very first post in this thread where i said:

however, while the "Virtio" devices are detected correctly, the Chelsio VF is not

If you need help to install CHR on FreeBSD, you can refer to this guide: "Creating a Mikrotik CHR - RouterOS 7 - Bhyve VM in FreeBSD

we already covered this topic including this exact URL. did you not read the thread?

On the other hand, if you want a decent environment focused on CHR, use one of the supported platforms: "Mikrotik help - Cloud Hosted Router CHR"

could you please describe whereabouts in this page the supported PCI network controllers are documented for use with SR-IOV? i did not see anything about that on a quick read through.

Larsa · Sat Oct 26, 2024 3:04 pm

I did my best to help, but it seems like you’re more interested in semantics and playing the ‘I said, you said’ blame game. Please refer to my previous message regarding supported platforms. Good luck!

crosswind · Mon Oct 28, 2024 3:41 pm

today MikroTik support sent me a development release (7.17_ab236) which adds support for the T540 VF! so hopefully this will be in 7.17.0.

i did notice a strange bug where the router can see itself in /ip/neighbor/print:

[admin@lab1.i.lfns.org.uk] > /ip/neighbor/print
Columns: INTERFACE, ADDRESS, MAC-ADDRESS, IDENTITY, VERSION, BOARD
#  INTERFACE  ADDRESS                MAC-ADDRESS        IDENTITY            VERSION                                       BOARD        
0  ether2     fe80::400:ff:fe00:50a  06:00:00:00:05:0A  lab2.i.lfns.org.uk  7.16.1 (stable) 2024-10-10 14:03:32           CHR          
1  ether3     fe80::400:ff:fe00:50b  06:00:00:00:05:0B  lab3.i.lfns.org.uk  7.16.1 (stable) 2024-10-10 14:03:32           CHR          
2  ether4     fe80::400:ff:fe00:5a3  06:00:00:00:05:A3  lab1.i.lfns.org.uk  7.17_ab236 (development) 2024-10-28 12:06:23  CHR          
3  ether4     fe80::400:ff:fe00:601  74:4D:28:08:81:F6  c1.i.lfns.org.uk    7.16 (stable) 2024-09-20 13:00:27             CRS309-1G-8S+
[admin@lab1.i.lfns.org.uk] >

but i think (based on a recent converstaion with Chelsio support) that this is a hardware bug in the NIC's switch chip, not a RouterOS bug. we'll see how it actually works when i get around to testing it properly.

kleshki · Mon Oct 28, 2024 3:56 pm

Curious if drivers could be added to the image so easily, can MT do some kind of driver repository for CHR, instead of pushing it to the image itself

Larsa · Mon Oct 28, 2024 6:07 pm

@crosswind - I’ll be the first to tip my hat and give you a shoutout (maybe even eat the hat) if you really pulled off getting MikroTik to add SR-IOV for the T540 in CHR running on the unsupported FreeBSD Bhyve in such a short time.

Ps..If that’s the case, I have a bone to pick with MikroTik for not fixing more urgent matters than adding new drivers to FreeBSD.

crosswind · Mon Oct 28, 2024 8:55 pm

if you really pulled off getting MikroTik to add SR-IOV for the T540 in CHR [...] in such a short time

in the past i have often complained, loudly and at length, about my issues with MikroTik support. but this case, i opened the ticket on Saturday, and they provided the new build at 10:30 on Monday. Latvia is two hours ahead of me, so that's less than 4 business hours. i am honestly pretty impressed with that, and i have to say, this matches my experience that MT support over the last few months is a lot better than it was a year or two ago.

i really hope this trend continues.

Ps..If that’s the case, I have a bone to pick with MikroTik for not fixing more urgent matters than adding new drivers to FreeBSD.

but you do realise it's not adding a driver to FreeBSD, right? they added the driver to RouterOS, so now this PCI VF will work in RouterOS on any hypervisor platform - FreeBSD/bhyve, Linux KVM, Windows Hyper-V, etc., etc. - nothing FreeBSD-specific about this change.

Larsa · Mon Oct 28, 2024 11:10 pm

if you really pulled off getting MikroTik to add SR-IOV for the T540 in CHR [...] in such a short time
in the past i have often complained, loudly and at length, about my issues with MikroTik support. but this case, i opened the ticket on Saturday, and they provided the new build at 10:30 on Monday. Latvia is two hours ahead of me, so that's less than 4 business hours. i am honestly pretty impressed with that, and i have to say, this matches my experience that MT support over the last few months is a lot better than it was a year or two ago. i really hope this trend continues.

Yeah, sure! Dream on...

but you do realise it's not adding a driver to FreeBSD, right? they added the driver to RouterOS, so now this PCI VF will work in RouterOS on any hypervisor platform - FreeBSD/bhyve, Linux KVM, Windows Hyper-V, etc., etc. - nothing FreeBSD-specific about this change.

You’re totally misinformed — there’s no such thing as universal driver compatibility between virtual machines! The underlying SR-IOV enables hardware to share virtual network functions (VF), but each host OS and hypervisor has its own setup and driver requirements. To get a hint regarding CHR NICs, read: "Mikrotik help - Usable Network and Disk interfaces on various hypervisors (CHR)"

crosswind · Mon Oct 28, 2024 11:38 pm

how is it possible for someone to get a "Forum Guru" tag while being so confidently and completely wrong about something like this?

the issue i reported was that RouterOS does not include the "cxgbe4vf" driver. that means you could pass through a T540 VF on Linux, or Windows, or any other platform, and RouterOS would not recognise it.

the fix is that Mikrotik now includes the "cxgbe4vf" driver. this means you can now pass through a T540 VF on Linux, or Windows, or any other platform, and RouterOS will recognise it and use it as an Ethernet interface.

nothing here has anything at all to do with FreeBSD. i could have reproduced this issue with a Linux VM host, or a Windows VM host, or any other VM host platform.

i honestly wonder if you're trolling at this point.

MrYan · Mon Oct 28, 2024 11:59 pm

Larsa, you might not like it, but crosswind is correct here. They already have bhyve on FreeBSD configured to initialise the Chelsio PCI VF and pass it through to the VM (RouterOS). There is a FreeBSD driver required for the PF NIC but it's already done its job!

The problem was that the VM (RouterOS) needs a VF NIC driver for the T540 to enable it as an interface within the virtual RouterOS device. Mikrotik have added that and it now works.

Sometimes it's better to take a step back and re-evaluate what you know about SR-IOV and how it works.

Larsa · Mon Oct 28, 2024 11:59 pm

Please stop embarrassing yourselves!

My best advice is to start exploring how VM drivers work and the differences between various hypervisors, particularly in handling IO-SRV and how it compares to regular PCI passthrough variants like ESXi DirectPath.

And once again, all hypervisors have their own drivers; there is no device driver interoperability between, for example, BSD, ESXi, Linux KVM, Windows Hyper-V, etc. Full stop, period.

jaclaz · Tue Oct 29, 2024 1:18 am

how is it possible for someone to get a "Forum Guru" tag while being so confidently and completely wrong about something like this?

No correlation, JFYI.
As an example, I got mine automatically when reaching a certain number of posts, the tag is independent from my knowledge or how much a nice guy I am, it is more like an award for persistence or seniority.

Sorry for the intermission everyone , please continue.

Tue Oct 29, 2024 9:57 am

Yes, passthrough requires a driver on the guest OS, and yes we have added cxgb4vf driver in our latest development branch. v7.17 will have this driver enabled.

Larsa · Tue Oct 29, 2024 10:11 am

Thanks for making that clear once and for all.

And please tell me you didn’t spend an entire weekend just to enable it for FreeBSD!

davis65536 · Tue Oct 29, 2024 11:07 am

And please tell me you didn’t spend an entire weekend just to enable it for FreeBSD!

You do know that the hosting OS has almost nothing to do with it, right?

All your talk about getting drivers for FreeBSD (or the host in general) suggests you're mixing up what connecting a VF to a VM via SR-IOV actually does. The proof is that the resolution was changing the RouterOS release to one that actually included the required driver, rather than any work at all on the host OS. Nothing would've needed to change on the host, at all.
In fact I'm willing to bet that nothing has changed on the host between the non-working and the working state.

Larsa · Tue Oct 29, 2024 12:32 pm

If you're interested in some technical details on SR-IOV, down below are some solid explanations from Red Hat and Intel about different PCI hardware abstraction layers. I’d especially recommend Red Hat’s intro, which covers PCIe Physical Functions (PFs) and PCIe Virtual Functions (VFs), and Intel’s section 1.2, Scalable I/O Virtualization.

Also, check out how SR-IOV needs upstream implementation in both the host OS device driver and the VM hypervisor’s internal driver to set it up and pass it to the guest OS. Each host OS driver and hypervisor differs in how it handles SR-IOV and there is no driver interoperability between for example, BSD, ESXi, Linux KVM, Windows Hyper-V. It’s similar for the guest OS. For example, the cxgb4vf driver would be implemented differently for a different guest OS.

https://docs.redhat.com/en/documentation/red_hat_virtualization/4.0/html/hardware_considerations_for_implementing_sr-iov/index
https://cdrdv2-public.intel.com/671403/intel-scalable-io-virtualization-technical-specification.pdf
https://www.altaro.com/hyper-v/hyper-v-virtual-hardware-emulated-synthetic-and-sr-iov/

EDIT:
The offer regarding my hat still stands.

Tue Oct 29, 2024 1:05 pm

I do not understand the point you are trying to make from these posts.

OP stated that the device is passed, but not recognized correctly by ROS. So you can already assume that all drivers and config for passthrough is set up correctly on the host system. Right? Right.

ROS now has enabled the cxgb4vf driver and boom magic, the device is recognized.

Larsa · Tue Oct 29, 2024 2:58 pm

Well, @mrz, purely educational as a reply to a previous post. But why do you care? This is a user forum, right?!

On a more serious note, though, I genuinely (really!) don’t understand Mikrotik’s priorities here. Why address a FreeBSD request for the unsupported Bhyve when there are more pressing issues to focus on?

Tue Oct 29, 2024 3:09 pm

We have to say this every time, but, we have many developers, each team works on different things.

Larsa · Tue Oct 29, 2024 3:15 pm

Like addressing a FreeBSD request for the unsupported Bhyve? Why not fix the UEFI Boot issue instead? You guys never stop surprising me!

jaclaz · Tue Oct 29, 2024 4:35 pm

Maybe it is just a case of serendipity, one of the good Mikrotik developers had this driver/fix/whatever ready in his drawer but never found the time/opportunity to add it to ROS, and crosswind's request just triggered the mechanism that allowed adding it to the next release.

About the UEFI booting issue, that one is not a (possibly marginal, useful to only a few users) missing feature.
It is two-fold:
1) the base is utterly wrong
2) additionally the latest images are malformed in the GPT part
#2 is most likely an accidental mistake, #1 is a (IMHO serious) conceptual error, UEFI booting, like it or not, wants the /EFI/BOOT/BOOTX64.EFI on a FAT volume on most (if not all) hardware, physical or virtual.

Anyway, now at least mrz and normis know that the issues exist, so hopefully they will be fixed, before or later.

davis65536 · Tue Oct 29, 2024 5:05 pm

If you're interested in some technical details

I'm at least fairly familiar with SR-IOV and VFs, but thanks.

https://docs.redhat.com/en/documentation/red_hat_virtualization/4.0/html/hardware_considerations_for_implementing_sr-iov/index

I'm pleased you linked this doc; I know it fairly well. Can you quote any part of it which mentions drivers for the Host OS? I really and truly believe you have misunderstood the layer at which the driver matters.

The offer regarding my hat still stands.

I can recommend some sauce.

mkx · Tue Oct 29, 2024 6:13 pm

Like addressing a FreeBSD request for the unsupported Bhyve?

Perhaps they could do it in no time because the issue is not about supporting anything specific to FreeBSD host (it's been said multiple times but you fail to get this through) but rather about including a hardware driver which already existed in linux kernel codebase but was left out of compiled driver set until this opportunity. I can imagine that CHR doesn't include every driver available in linux kernel due to various reasons, one of them (perhaps the least important) could be concern about disk space requirements.

Larsa · Tue Oct 29, 2024 7:47 pm

Can you quote any part of it which mentions drivers for the Host OS?

I haven't had time to check the actual references, but if you mean the drivers for PCIe IO-SRV support, you might for example checkout the vfio-pci driver for Linux, the FreeBSD ppt driver or the Microsoft Windows Driver Model (WDM) using DDA/VFIO in Hyper-V

Microsoft also has a pretty detailed and comprehensive walkthrough of the different layers for the Windows:

I can recommend some sauce.

Well, thank you! That was nice of you!

But I'm still not convinced someone fixed it that quickly. I'll need an official statement from MT that it actually happened during the weekend just to please the OP.

Znevna · Tue Oct 29, 2024 8:11 pm

When staff intervenes to CONFIRM that to get this working they didn't touch any FreeBSD bits, but Larsa keeps insisting that this was FreeBSD-related.

Larsa, for the love of all Latvian deities, please understand that this had nothing to do with the host operating system.
We can try to make a pretty drawing if it helps you.
They already had cxgb4 included and working, they only added cxgb4vf (vf stands for "Virtual Function") as an extra thing.
It probably happened Monday morning, "hey let's enable this too since we support the card anyway", compiled, and boom it went for testing to the user.
Get over it.
He might as well have had this running under ESXi (and will obviously work now there too) by your logic they worked over the weekend with VMware/Broadcom to fix this too?

Larsa · Tue Oct 29, 2024 9:35 pm

I suspect you might have missed something, misunderstood, or simply skipped over some posts, so I’ll graciously ignore the grumbling tone.

Mr. Znevna, just tell me how I can help you improve your, let’s say, ‘hat-wagering’ skills. Or perhaps you have anything more intriguing to say besides the whining? I’m all ears!

Znevna · Tue Oct 29, 2024 10:24 pm

Nah, I'm good.

davis65536 · Wed Oct 30, 2024 12:28 pm

I haven't had time to check the actual references, but if you mean the drivers for PCIe IO-SRV support, you might for example checkout the vfio-pci driver for Linux

I doubt you meant to link to the MS docs for the vfio on linux driver, but since you mentioned vfio, let's go look at the linux documentation.

You'll note here that vfio is a way of presenting devices to e.g. VMs or (less frequently, IME) user-space software. The way this is typically done is to unbind the current driver (if any) and then use the device-generic vfio interfaces to connect it to your VM or user-space software. As the OP is talking about a VM, the only device-specific driver will be on the VM. The most vfio is doing there is simply presenting the device to the VM, and offering control surfaces for e.g. user permissions.

The bit I'm actually going to quote is

By unbinding the device from the host driver and binding it to a VFIO driver,

It says "unbinding the device from the host driver".... Here, it's because there's an assumption that a host driver may have claimed the PCI device already, and in order to the VFIO part, you actually want no host driver involved, hence the unbind instruction. Were no driver claiming it, this unbind step would be a no-op.

I'll repeat it, again: Once you've connected the VF to the VM (however that's achieved) any host-level drivers here are irrelevant. The change in the VM software that fixes the issue proves that. The number of people pointing out you've misunderstood it agree with it, and the documentation you (attempted to) refer to undermines your position.

The only argument you've offered is that we're "embarrassing ourselves" and saying we don't understand it. Please, by all means, quote something from the documentation you've pointed us to that shows host-level drivers being in any way relevant.

Of course, the other possibility is that you might simply be wrong about how it works, which is fine; we've all been there.

Who is online