I purchased the Mikrotik CCR2004-1G-2XS-PCIe and would like to use it in my Proxmox server. The card is parameterized via the management port via Winbox. The SFP28-1 and 2 are connected to a Mikrotik CRS309-1G-8S+ with two XS+DA0001 cables.

The hardware consists of an MSI B550-A PRO motherboard and an M.2 hard drive as well as an Nvidea Quadro A2000 PCIe slot 1 and the Mikrotik CCR2004-1G-2XS-PCIe in slot 2 are connected to the PCIe bus

Initially I always lost the Winbox connection to the management port Ether1 when the XS+DA0001 cables were plugged in. In order for me to reconnect, I had to disconnect the DAC cables again.

Now that I have updated the router OS to 7.14 and the firmware to 7.14, I can no longer connect the SFP28 ports with passthrough on ether-pcie. The Winbox closes for 5 seconds and then the passthrough settings are gone again.

Under Proxmox the network adapters are listed with the designation enp33s0f0 - enp33s0f3

There were also errors in the Proxmox system:

dmesg | grep atl1c
[ 1.354453] atl1c 0000:21:00.0: can't ioremap BAR 0: [??? 0x00000000 flags 0x0]
[ 1.354468] atl1c 0000:21:00.0: cannot map device registers
[ 1.354504] atl1c: probe of 0000:21:00.0 failed with error -5
[ 1.354546] atl1c 0000:21:00.1: can't ioremap BAR 0: [??? 0x00000000 flags 0x0]
[ 1.354551] atl1c 0000:21:00.1: cannot map device registers
[ 1.354579] atl1c: probe of 0000:21:00.1 failed with error -5
[ 1.354616] atl1c 0000:21:00.2: can't ioremap BAR 0: [??? 0x00000000 flags 0x0]
[ 1.354621] atl1c 0000:21:00.2: cannot map device registers
[ 1.354648] atl1c: probe of 0000:21:00.2 failed with error -5
[ 1.354687] atl1c 0000:21:00.3: can't ioremap BAR 0: [??? 0x00000000 flags 0x0]
[ 1.354692] atl1c 0000:21:00.3: cannot map device registers
[ 1.354720] atl1c: probe of 0000:21:00.3 failed with error -5

Settings like these two:
# GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=on pci=realloc pci=noaer"
or
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

led to problems such as:

users
[ 0.545844] pci 0000:21:00.0: [1969:1083] type 00 class 0x020000
[ 0.545867] pci 0000:21:00.0: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[ 0.545922] pci 0000:21:00.0: [Firmware Bug]: disabling VPD access (can't determine size of non-standard VPD format)
[ 0.545988] pci 0000:21:00.0: PME# supported from D0
[ 0.546044] pci 0000:21:00.0: 31.504 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:00:02.1 (capable of 63.008 Gb/s with 8.0 GT/s
PCIe x8 link)
[ 0.546134] pci 0000:21:00.1: [1969:1083] type 00 class 0x020000
[ 0.546156] pci 0000:21:00.1: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[ 0.546210] pci 0000:21:00.1: [Firmware Bug]: disabling VPD access (can't determine size of non-standard VPD format)
[ 0.546270] pci 0000:21:00.1: PME# supported from D0
[ 0.546387] pci 0000:21:00.2: [1969:1083] type 00 class 0x020000
[ 0.546410] pci 0000:21:00.2: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[ 0.546463] pci 0000:21:00.2: [Firmware Bug]: disabling VPD access (can't determine size of non-standard VPD format)
[ 0.546523] pci 0000:21:00.2: PME# supported from D0
[ 0.546642] pci 0000:21:00.3: [1969:1083] type 00 class 0x020000
[ 0.546664] pci 0000:21:00.3: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[ 0.546718] pci 0000:21:00.3: [Firmware Bug]: disabling VPD access (can't determine size of non-standard VPD format)
[ 0.546778] pci 0000:21:00.3: PME# supported from D0

There are 3 devices in the PCI bus but according to the MSI manual, slot 1 with 16x can be used for the graphics card, the first M2 slot for a hard drive and the 2nd slot for another PCIe card. I use amd_iommu=on to pass the graphics card to a VM

I'm completely desperate and wanted to reinstall Proxmox, but during installation I get error messages again when initializing the network adapters.

On the card only one of the management ports is provided with a DHCP client and otherwise everything is as it was after the reset to "no default config"

Good evening,

at the moment I have managed to get the ether-pcie1 and 2 to show up as running. When I try to use the passthrough mode on SFP+1 the card crashes and the log shows the following two messages:
- router was rebooted without proper shutdown, probably kernel failure,
- kernel failure in previous boot


If I try to downgrade the card from RouterOS 7.14 or install the current version again, I get no connection to the card and after a reboot of the server the following message:
- kernel failure in previous boot

Hi! I was experimenting with such a card, but my plan was to use it with FreeBSD, which has never worked. I did test however after my failed attempts with Proxmox, but that was sthg like a year ago, so with maybe ROS 7.4-7.8. It worked on Proxmox without any issue, however I did not really needed in that machine such a card, so since then it´s in a box.
Maybe you are trying to use PCI Passthrough? The card does not support that, afaik. Enabled IOMMU should not be a problem, but you should maybe try and disable IOMMU from BIOS? Or maybe try disable resizable BAR also in BIOS?
Sorry I don´t really have a solution for you, but I´m very interested if you figure it out!

What's new in 7.15beta6 (2024-Mar-08 08:23):
*) sfp - improved system stability for CR2004-1G-2XS-PCIe (introduced in v7.14);

What's new in 7.15beta6 (2024-Mar-08 08:23):
*) sfp - improved system stability for CR2004-1G-2XS-PCIe (introduced in v7.14);

My CR2004-1G-2XS-PCIe became very stable after upgrading to 7.14.
It cannot even boot with a SFP module installed.
It also crashed when I tried to upload a ros package.
The worst of all... I cannot either downgrade to factory version or upgrade to 7.15beta6 Testing... (tried winbox/webfig)

I had the same issue after upgrading to v7.14. I had to use Netinstall and the reset switch on the card to downgrade back to 7.13.5 to get stability back under Proxmox.

Thank you for the solution with the 7.14, I have now upgraded to the 7.15beta6 but had to do this via Netinstall because the card could not be downgraded or upgraded in any other way.

Now the problem remains that I still cannot get a connection via the SFP* modules. I have a CCR309 switch with SPF+ and would like to achieve 10GBit there. I have connected the CCR2004 and the CCR309 with the Mikrotik via two XS+DA0001. Since the CCR2004 can do 25Gbit and the CR309 only 10GBit what settings do I have to make there since auto negotiation does not work?

Greetings Dennis

i have 7.14 installed and boot only without sfp modules.
I tried to install the new 7.14.1, no luck, get stuck, fan getting crazy. Is really a disaster at the moment. Got two of this card and both broken ...
Log say

"kernel failure in previous boot"

I had the same issue after upgrading to v7.14. I had to use Netinstall and the reset switch on the card to downgrade back to 7.13.5 to get stability back under Proxmox.

How did you put the card in netinstall mode ?
Really have to open the server and press the non existent reset button, with screwdriver and hope is booting ... that's really not good design this time - sorry I m mikrotik fans but ...

I have now solved all the problems, thank you very much for your help.

To put the card into boot mode you have to hold the "button" until the card appears in the netinstall menu.

i think as long as the flashing of the sfp led's stops again.

may i ask please, if your computer reboots when you restart the card - or that happens only with HP/DELL like servers?
Thanks!

I had the same issue after upgrading to v7.14. I had to use Netinstall and the reset switch on the card to downgrade back to 7.13.5 to get stability back under Proxmox.

I got exactly the same issue. If I upgrade to a version later than 7.13.5, I get weird crashing issues under the latest proxmox 8.1.10 (and bare metal ubuntu). Tried the card in various machines (from Dell servers to ordinary PCs). The most simple setup, using 1 bridge with sfp28-1 (dac cable) and ether-pcie1 under ROS and making 1 bridge under proxmox for host/vm access with ether-pcie1 in it causes the crash if I try to put significant amounts of data through it (speedtest etc). Doesn't matter if I do that from the host or from a vm. Under 7.13.5 I got zero issues, so it seems an issue with later ROS versions.

Already made a ticket with support but I was wondering if anyone else has the same problem or a solution?

Initially I always lost the Winbox connection to the management port Ether1 when the XS+DA0001 cables were plugged in. In order for me to reconnect, I had to disconnect the DAC cables again.

Ever got to the bottom of this?
Having the same issue with Proxmox or just bare Debian 12 with KVM+QEMU+Libvirt and XS+DA0001

Regarding the "no link" issue, set the RX and TX flow control on the interface to Auto instead of off.
I mean, Auto is still off, but for some reason this fixes the issue for me.

Did anybody get a response from the support? I have the same issue and whatever I do my card keeps crashing this also happens when I have no sfp+ nic connected. I tried upgrading to 7.15rc4 but I guess the issue was not fixed there.

It would be really good to know if this is an issue that they are working on.

I have installed version 7.15.3 and on sfp2 I didn't manage to get link up.
It say all time, no link.
I connect it to a CSR305 with a module and cable. This side say link ok, cable of and I got RX, CCR2004 side say no link and no communication.
There is outthere somody managed to get it running, both sfp ? Which firmware version.
It look like for me as is a gambling ...

@mikrotik_support some statement from your side ?

Thanks

At least on 7.17, the card honors what you ask it to advertise as part of auto negotiation. If you are getting no link when pairing with a 10G device, remove the 25G options from what the interface on the CCR2004-1G-2XS-PCIe advertises and it will link.

At least on 7.17, the card honors what you ask it to advertise as part of auto negotiation. If you are getting no link when pairing with a 10G device, remove the 25G options from what the interface on the CCR2004-1G-2XS-PCIe advertises and it will link.

I m running now 7.16.2 and i still not rebootet since 7.17 got stable ... anyway the link is now since weeks stable. For mikrotik XS+DA0003 i set FEC off and no auto negotiation, just 50G baseCR. Two CCR2004 in two servers, connected by XS+DA0003.
I don't know if I will get back to test again ... but tnx for the update.