New exciting features for storage

Fri Feb 07, 2025 8:26 am

Those of you who are testing 7.18 might have noticed, but most of you probably did not notice yet.
There are plenty of new features in one specific category - storage.

Those of you with devices that support any kind of external storage, like the RB1100, X86, CHR systems or even devices with USB hubs with external drives attached, it would be great if you could test things like:

In default package: https://help.mikrotik.com/docs/spaces/R ... 3346/Disks
- Disk formatting and partitions
- Storage on RAM (tmpFS) when you don't have storage space, but have free RAM
- Swap
- Mount file as block device

In ROSE package https://help.mikrotik.com/docs/spaces/R ... SE-storage
- RAMdisk (slightly different from tmpFS)
- Mounting SMB or NFS onto your router from some NAS and storing files there
- RAID

- BtrFS https://help.mikrotik.com/docs/spaces/R ... 9711/Btrfs

and more

Why are all these features there? Well, soon you will find out, but if you can test some of them, that would help, thanks

teslasystems · Fri Feb 07, 2025 10:45 am

I want to share my feedback about SMB client (/disk add type=smb ...) and about working with large amount of files.

After connecting a new SMB share with about 50 files, a file tree is loaded pretty quickly, all files are visible in WinBox, but then RouterOS starts doing something weird. It creates a huge load on a network and eats a lot of traffic from SMB server. Within about 30 minutes it has downloaded about 4.5 GB of traffic from the server. And this never stops until you dismount your SMB disk. What the hell is it doing? (TILE, 7.18beta4)
On the Dude graph it looks like this:
.

SMB_traffic.png

--------------------------------------------------------------------------------------------------------------------------------------------------

Regarding woking with large amount of files, I often hear that you are improving the behavior for this scenario. But it will never work normally with large amount of files in current implementation. The reason is simple: RouterOS scans a whole file structure completely. If you have 10K files, 100K files, it will uselessly scan all these thousands of files, creating an unneeded load on network and server. And this process will begin over and over again after each reboot.

To work with large amount of files, file management mechanism should be completely revised:

1. RouterOS should never scan a whole file structure, only content of root directories on the disks.
2. In WinBox, 'File List' should be replaced by 'File Browser' with ability to navigate between directories like it's done in any OS.
3. CLI file commands should also change their behavior. If you type /file print, it should show the contents of root directory only. If you need to show the contents of some subdirectory, you may specify it in 'where' parameter like /file print where name~"subdir/". Sometimes, it may required to show the contents including subdirecories, in such case an additional parameter could be added, like "/file print include-subdirs" for example.
In Linux there is an 'ls' command, so it shouldn't be a problem to translate file commands into 'ls' command accordingly.

It may sound wild for those who get used to current behavior, but without this, any adequate behavior with large amount of files is impossible.

--------------------------------------------------------------------------------------------------------------------------------------------------

P.S. It's still unclear if SMB client feature is incuded in ROSE-storage package, because it works even without this package.

Fri Feb 07, 2025 10:51 am

Which version is this?

teslasystems · Fri Feb 07, 2025 10:53 am

Which version is this?

I have specified it, 7.18beta4

eworm · Fri Feb 07, 2025 11:24 am

Current situation is a disaster for scripts. I have reported my issue here:
viewtopic.php?t=214071#p1120688

... but did not yet open an issue for support. Looks like the implementation is racy, so all scripts that handle files are randomly crashing.

Sometimes even file is available for find, but remove fails anyway, something like:

/file/remove [ find where name="file/with/path" ];

Fri Feb 07, 2025 12:00 pm

Changelog tells you about how to do it, use the new parameters (not a bug)

*) file - allow printing specific directories via path parameter;
*) file - added "recursive" and "relative" parameters to "/file/print" for use in conjunction with "path" parameter;

jaclaz · Fri Feb 07, 2025 12:01 pm

Meh, this shows how old (and grumpy) I am getting , personally I don't feel particularly excited, the whole stuff appears to be at the moment more wishful thinking than anything else, and anyway it all sounds to me like, to make the usual automotive comparison:
"Hey! We added round wheels, an exciting new development from the square ones we had till now".
(the triangular ones, one bump less, were tested but abandoned earlier)

Ullinator · Fri Feb 07, 2025 12:03 pm

I totally agree eworm.

hc_483.jpg

this sccrenshot was made on a rb5009 with ROS 7.18beta4.
The failed script in the sceenshot downloads FW blocklists, merge them and create list entries.
To do this, some files (size <200kb) are downloaded, stored in "Files" and after creation of the "Address List" in the FW menu these files were deleted.
It runs fine on every ROS before ROS7.18Beta2 and now Beta4.
The script itself is provided by eworm.
(please see at https://git.eworm.de/cgit/routeros-scripts/about/)
Please look at it, thank you!

eworm · Fri Feb 07, 2025 12:22 pm

Changelog tells you about how to do it, use the new parameters (not a bug)

Wait... What!?

So find is aware of the file and returns an id. But remove still can not remove it. Does that require a path now, for a unique id?

:local FileID [ /file/find where name="file/with/path" ];
/file/remove path="file/with" $FileID;

🤪

That's bullshit, no?

Please have another look at that.

Fri Feb 07, 2025 1:35 pm

We can't replicate what you are saying, but actually it would be better and more correct to do it this way:

/file/remove file/with/path

and not

/file/remove [ /file/find here name="file/with/path" ]

eworm · Fri Feb 07, 2025 2:24 pm

All that fails the same way. Not every time, but way too often.

As this is racy you need to remove a file that's not too old. Perhaps I should open an issue and provide more details... The problem is that behavior on racy things changes all the time, so there is no perfect reproducer.

Fri Feb 07, 2025 2:47 pm

yes, please make a ticket with details how to replicate, how many files you have, which file you removed etc.

itimo01 · Fri Feb 07, 2025 3:14 pm

- Swap

Swap appears to be working great since I'm now able to spin up a minecraft server on my ax3 :D
While previously trying to get it working, the router would crash with an OOM condition.

container/envs/add key=EULA name=mcsrv value=TRUE
container/envs/add key=MEMORY name=mcsrv value=512M

container/add remote-image=itzg/minecraft-server:latest interface=veth1 envlist=mcsrv root-dir=/usb1-part1/containers/store/mcsrv comment=mcsrv logging=yes
container/start [find comment=mcsrv]

ip/firewall/nat/add action=dst-nat chain=dstnat disabled=yes dst-address=192.168.41.1 dst-port=25565 protocol=tcp to-addresses=172.17.0.2 to-ports=25565

Sadly i cant upload my screenshot right now. (linked it externally)

Valerio5000 · Fri Feb 07, 2025 3:23 pm

Why are all these features there? Well, soon you will find out, but if you can test some of them, that would help, thanks

I'll take a guess..you want to enter the NAS market? tell the truth :D

Anyway it's always nice to see the MK team developing and proposing new features ;)

dang21000 · Fri Feb 07, 2025 9:01 pm

Exciting features... ? I don't understand why other networks products, forti, xtrem, cisco do like you !
Minecraft on container on network security gateway... it's a joke ?

And during this time, with evolution of routerOS, their more and more constraints about device mode, functions before allowed are now locked and need the physical button press to be changed. When router are far, it's a great feature, for sure ! But it's better to transform routerOS as a NAS, gaming server....

Fri Feb 07, 2025 9:28 pm

Who said you have to run Minecraft? One forum user did it for testing?
You can run Greylog or Grafana.

itimo01 · Fri Feb 07, 2025 9:37 pm

Minecraft on container on network security gateway... it's a joke ?

Yes, and you can definitely run more useful things on a router if you so desire.

teslasystems · Sat Feb 08, 2025 5:22 am

need the physical button press to be changed. When router are far, it's a great feature, for sure !

Don't want to sound rude and sorry for offtopic, but having hardware on remote location without having additional devices (like GSM-sockets or something similar) that allow to remotely power-cycle your hardware is very strange at least. Anything could happen and power-cycling often helps. In case of device-mode changing it would also solve the problem. So, as to me, having such devices is a mandatory thing and I don't know how to explain when people don't have them. May be they like to drive (or even fly) to their remote sites each time something happens...

eider · Sat Feb 08, 2025 6:26 am

This is RouterOS, not StorageOS. If you desire to add file management and make a NAS out of a router then just spinoff ROS into another branch that is developed separately. Storage device does not need BGP and MPLS and router/switch does not need NFS, Btrfs support, RAID, NVMe over TCP or fancy UI for collaborative file sharing. You already have strong base with basic networking, just cut unnecessary slack, integrate rose-storage into base and sell it as separate OS to be ran on new class of devices.

The naming of ROSE is also pretty poor, "RouterOS Enterprise" - 100% enterprise storage, 0% enterprise networking.

---

You can run Greylog or Grafana.

I wouldn't dare any more than you would dare to run it on Junos :> Possible? Yes, absolutely! Good idea? In hell.

sirbryan · Sat Feb 08, 2025 7:03 am

I, for one, welcome our Storage overlords.

Those who are poo-pooing things haven't been paying close attention to the hyper-convergence of functions the last decade or two. Cisco built a virtual router for VMware 15 years ago, and VMware created vSAN shortly thereafter. Now you have Proxmox VE + Ceph, and with a CHR (or two) in the mix, or FRR on the bare metal, customers can build (and have) practically anything on commodity hardware.

Look at what TrueNAS has done with TrueNAS Scale. You put the server on a decent PC, spin up your ZFS pool, go download a bunch of free apps, and/or put a VM on the box. MSP's love this kind of stuff.

It looks like MikroTik is trying to get their hardware and software into this market. You can do a barebones PC, or you can buy a beefy router (RB5009, CCR2004, CCR2x16 or CHR + Ampere) with simple storage, and, like with TrueNAS scale, load up apps (micro services) on your router.

When ROSE came out, I put an NVMe to SATA adapter in a 2116, punched a hole, ran some thin SATA cables out, and hooked up six SATA drives in a bay the size of a CDROM. It works pretty well.

The advantage MikroTik has over a PC is wirespeed switching/routing (on Marvell platforms). I run a bunch of containers on my 2116. Going to have to add the Minecraft one (didn't know it existed for ARM64).

Rox169 · Sat Feb 08, 2025 8:43 am

Mikrotik NAS?

noradtux · Sat Feb 08, 2025 9:29 pm

How I see it 'Tik are just making the existing features of the kernel accessible to the user. I welcome that and I do use containers and storage in private and intent to use it in enterprise.

Larsa · Sat Feb 08, 2025 9:40 pm

@sirbryan, that’s not gonna happen. ROS is designed as an embedded NOS with its own limitations. When it comes to running ROS as CHR, there are way better options. Plus, MT lacks the skill set and experience, and ROS is too unreliable for storage solutions like hyper-convergence.

Kentzo · Sun Feb 09, 2025 2:51 am

I think it makes a great SoHo router though. A couple of SMB shares and SWAP for memory heavy containers is all I need. Finally can retire an rpi and run Homebridge directly on my AX3.

eider · Sun Feb 09, 2025 5:34 am

SWAP for memory heavy containers is all I need

SWAP over USB is a horrible idea and recipe for kernel deadlocks and freezes - definitely not something you want on a device that is also processing your network traffic. Will it work? Yes, and its also likely you won't see any issues... until you hit them hard and waste days debugging it.

Kentzo · Sun Feb 09, 2025 5:46 am

I got an impression that their SWAP implementation is only for containers and otherwise unused (and isolated) from network functionality.

Sun Feb 09, 2025 3:59 pm

...
Why are all these features there? Well, soon you will find out, but if you can test some of them, that would help, thanks

My bet: fast stable shared resource to provide true HA.

teslasystems · Sun Feb 09, 2025 4:33 pm

Oh, and regarding this,

- Storage on RAM (tmpFS) when you don't have storage space, but have free RAM

Please read
viewtopic.php?t=214077

eider · Sun Feb 09, 2025 9:13 pm

I got an impression that their SWAP implementation is only for containers and otherwise unused (and isolated) from network functionality.

Since containers share kernel with host, there is no distinction here. Only consider it for devices which have PCI-E lanes exposed over physical slot (RBM33G does for example, but it's hardly a good base for it anyway because of MMIPS and 16M of storage).

anav · Sun Feb 09, 2025 11:08 pm

Zerotrust cloudflare on a storage device??? ;-PP

guipoletto · Mon Feb 10, 2025 5:08 am

Regarding tmpfs:
i ran a script to create "/tmp" on several production devices:

/disk/add slot=tmp type=tmpfs;

this should create a ramdisk with "upto" 50% of RAM

it failed on all "64M" devices (RB2011iL, RB2011iLS, RB750r2, RB433)
for these, the default of 50% should probably be reviewed
they accepted up to 30M for tpmfs, even when "free-memory" was around 20-24mb

this leads to another observation:
a Netpower 16P reports 183mb free (out of 256), and also failed to create the ramdisk, with the same message as the 64m ones;
"failure: too much memory requested for tmpfs/ramdisk"

another Netpower 16P with 199mb "free" successfully created the ramdisk with the same parameters

this is weird, as 128mb are clearly free

and also leads to my main question:
when ramdisk is created near boot-time (when ram is mostly free and unfragmented),
what would be the expected behahviour when system-ram usage grows, and the ramdisk already exists with "upto" 50% of ram, but is empty / near-empty?

teslasystems · Mon Feb 10, 2025 7:05 am

@guipoletto
It seems, that there is some "reserved" memory. My TILE router with 2 GB of RAM, goes to reboot with out-of-memory if free RAM goes down to about 150 MB.

eworm · Mon Feb 10, 2025 5:40 pm

yes, please make a ticket with details how to replicate, how many files you have, which file you removed etc.

Decided against. While this breaks my scripts, I can not find a really good reproducer. This drove me crazy - every time I though I am near it changed behavior. 🤪

Instead I modified my scripts. I bumped the required version to RouterOS 7.15 - the new code for directory creation (

/file/add type=directory name=$dir;

instead of

/file/add name=$dir/file; /file/add name=$dir/file;

) made it less likely to hit the issue. Also introduced a global function for removal - there I have a central point to catch and ignore the error. 🤪
The only downside is that stale files may be around... But as I put everything into a tmpfs this is not much of a concern - a reboot does clean up.

toxicfusion · Mon Feb 10, 2025 8:29 pm

This is RouterOS, not StorageOS. If you desire to add file management and make a NAS out of a router then just spinoff ROS into another branch that is developed separately. Storage device does not need BGP and MPLS and router/switch does not need NFS, Btrfs support, RAID, NVMe over TCP or fancy UI for collaborative file sharing. You already have strong base with basic networking, just cut unnecessary slack, integrate rose-storage into base and sell it as separate OS to be ran on new class of devices.

The naming of ROSE is also pretty poor, "RouterOS Enterprise" - 100% enterprise storage, 0% enterprise networking.

---

You can run Greylog or Grafana.
I wouldn't dare any more than you would dare to run it on Junos :> Possible? Yes, absolutely! Good idea? In hell.

I 1000% agree with you and echo this. MikroTik, please LISTEN -- Cut this bullshit with storage on RouterOS as a network appliance hardware used for routing. Move it to StorageOS and let it be a fork and then let people test the features.

Since RouterOS 7.17 with introduction of device-mode and then "ROSE", we're leaving MikroTik -- moving onto greener pastures. This is after 10+ years. Perhaps we have matured, but we cannot risk our company name and customer satisfaction.

Storage protocols do NOT belong on a router. SMB, NFS, NVME drivers, RAID, etc. WTF? Please STOP. We're done -- we will only use MikroTik for 60Ghz connectivity, that is the only current stable wireless product [besides Cube 60 water ingestion]. Perhaps when MikroTik restructures themselves and they also grow up -- we will circle back and become again interested. Focus should be on better web interface, Winbox development, CAPsMAN management [sync], dynamic routing fixes, or cloud management... get with the times. There are better products out there. MikroTik can be so much more in 2025 and stifle competition with better investment within themselves. Perhaps better leadership guidance?

MikroTik releasing "enterprise" hardware for consumer niche market to tinker. We enjoyed MikroTik when we could "route the world" and felt satisfaction.

teslasystems · Mon Feb 10, 2025 9:48 pm

@toxicfusion If you hate storage features so much, just don't install ROSE package and don't use any of these features, no one forces you to use them. And if you moved to other "better" solutions - what's you problem? Leave silently without posting this bullshit.

toxicfusion · Mon Feb 10, 2025 11:31 pm

network storage protocols dont belong on a router, or in this case RouterOS -- even if installed as an additional package. "enterprise my ass" [Apple joke]. This would be fine for an open source project - or for kid tinker toys. MikroTik putting "Tik in Tinker" as of lately.

Introducing storage protocols on a router, home lab or dev -- that then becomes someones office production network..... is horrifying. The security risks alone should make this be a completely separate operating system by Mikrotik. Some new IT intern or "Green" IT staff would see this as cost saving measure to do SMB on their MikroTik router and put into production.. scary.

MikroTik fork the project; create your bootable linux MikroTik operating system with basic TCP/IP stack and the ROSE package loaded. Otherwise, people just install real operating system to do the job. TrueNAS these days is child's play - just works. If MikroTik was to fork this project to create MikroTik NAS -- they better hire a better engineer or web designer. Winbox4 or the new Webfig is not going to cut it. Its a sad state.

I get the "appeal" for "all-in-one". But why...... Only makes sense for a hobbyist or home lab..... This serving production workloads or REAL workloads is laughable.

Even if there was a possibility to attach high speed storage [USB-C] via JBOD external enclosure -- there are better solutions.

toxicfusion · Mon Feb 10, 2025 11:49 pm

Further -

I'd MUCH rather see continued development of containers on RouterOS. Other vendors do this, run containers. IE: Cisco has run containers on their hardware with great success. Also ARISTA... etc.

Or look at what Palo Alto does. Their management plane on completely separate partition and CPU [How you interact and commit changes], all else runs on other available CPU's and on the ASICS - data and control plane. Now this would be better investment in resources at MikroTik.

But yet MIkroTiks marketing team loves to "market" some of the CCR hardware as enterprise ready -- IMO there appears to be a shift happening right now with all that is broken and lack of focus. [Winbox4 is a mess, 7.17 device-mode]. We're still griping about having to physically touch customer deployed hardware. If we're having to make customer physically power cycle or we have to truck roll, we will just be replacing with other vendor.

guipoletto · Tue Feb 11, 2025 12:36 am

network storage protocols dont belong on a router

So long as these features are modularized, i don't see the problem.

toxicfusion · Tue Feb 11, 2025 8:13 pm

Ok -- but it is a waste of CPU cycles and processing power.... When can instead be used to process packets, and firewall rules. Not all the MikroTik switch chips are that great. As they dont all have real ASICS.

Storage technology/features does not belong on a router. Its for tinker, hobbyist, or homelab. MikroTik is uncertain of their identity. Are you enterprise, or are you home-lab? If you're both -- there needs to be a clear separation in product lines.

noradtux · Wed Feb 12, 2025 7:51 am

Why not be the one vendor that fits in home lab _and_ enterprise? For me it was exactly this how they even got noticed by us. First used in home lab, than slowly migrating into enterprise. Give users the choice, you don't need to run those features.

eworm · Wed Feb 12, 2025 10:49 am

yes, please make a ticket with details how to replicate, how many files you have, which file you removed etc.
Decided against. While this breaks my scripts, I can not find a really good reproducer. This drove me crazy - every time I though I am near it changed behavior. 🤪

As this still broke with my workarounds in place I digged deeper. I think I have a reproducer now... Please have a look at SUP-179200 for details. Thanks!

Wed Feb 12, 2025 10:51 am

We found the issue you describe, it was triggered if the file was in a subdirectory, looks like the next release will have a fix

eworm · Wed Feb 12, 2025 11:44 am

Eagerly waiting for that then. 😊

teslasystems · Wed Feb 12, 2025 1:16 pm

What about issue with SMB that I desribed at the beginning?

Wed Feb 12, 2025 1:29 pm

RouterOS file subsystem will soon have improvements for listing large amount of files, but normally you should not be using Winbox as a file manager. Use SMB to browse the files.

I did not fully understand, you also have issues with SMB listing, if Winbox is closed?

teslasystems · Wed Feb 12, 2025 2:06 pm

@normis, Listing is OK. But, it's consuming a lot of traffic even if WinBox is closed. No difference, if it's opened or closed. But even if it's opened, what is it doing? I'm not downloading any files from a share, just connected an SMB disk, it has quickly downloaded a list of 50 files and then it's eating 9 GB per hour for no reason.

Regarding large amount of files, I understand, that WinBox is not a file browser, but what will happen, if you have 100K files? Doesn't matter if it's SMB, or just a USB HDD. RouterOS is collecting a whole file structure and this will happen over and over again after each reboot. This is the key problem for using it with large amount of files.

sikkim87 · Wed Feb 12, 2025 10:47 pm

Idea for a useful and interesting feature (which shouldn't be very difficult to implement); when a device has little RAM but enough storage (NAND, SATA, NVME or USB), it would be nice to have a native swap feature (like in standard Linux where a partition/file is used as a swap when RAM runs out)

/system disk add type=swap size=8192 file=/swapfile

I'm thinking of my RB1100 Dude which is rebooting me due to out of memory condition even though I have 256GB SATA SSD and 1GB RAM

PS : Yes, I know, I could migrate to CHR my Dude... but that's not what this post is.

eworm · Wed Feb 12, 2025 10:55 pm

The latest beta does have support for swap. 😉

guipoletto · Wed Feb 12, 2025 11:06 pm

The latest beta does have support for swap. 😉

Swap is evil, and in a ROUTER, largelly indicates device-abuse
exposing some swap for *containers*, and the applications hosted therein is one thing
if those crash, who cares.

but if the host-system (which is a ROUTER) is forced into swapping ,
then trough device-abuse and feature-creep it's no longer a router and the network topology needs to be reviewed... (or a beefier router with actual RAM needs to be purchased)

sikkim87 · Wed Feb 12, 2025 11:35 pm

That's right. I didn't see this subtle line in changelog 7.18 beta.- However, I'll wait for it to arrive in the stable channel.

Let's hope it also works for system processes (such as the Dude) as this suggests!

Thanks a lot! :)

PS: It's not really the debate whether a swap is good or not- I have a 2 year old device with 1GB of RAM. It was enough for Dude/RouterOS v6... Not enough anymore for RouterOS v7. I'm not going to throw away a $400 device, redundant power supply & with a level 6 license just because it no longer has enough RAM. Swap is a linux kernel feature that should (if possible) be implemented and - at the user's option - should be usable. In my opinion, it's a more interesting feature than DLNA. :)

Thu Feb 13, 2025 9:31 am

The latest beta does have support for swap.
Swap is evil, and in a ROUTER, largelly indicates device-abuse
exposing some swap for *containers*, and the applications hosted therein is one thing
if those crash, who cares.

but if the host-system (which is a ROUTER) is forced into swapping ,
then trough device-abuse and feature-creep it's no longer a router and the network topology needs to be reviewed... (or a beefier router with actual RAM needs to be purchased)

Swap is not for the host system. Please read the manual about it before complaining: https://help.mikrotik.com/docs/spaces/R ... -Swapspace

ConradPino · Thu Feb 13, 2025 9:45 am

Swap is not for the host system. Please read the manual about it before complaining: https://help.mikrotik.com/docs/spaces/R ... -Swapspace

@normis, thank you and please thank Druvis Timma for making that update Feb 07, 2025 10:17. IMO a super addition.

eworm · Thu Feb 13, 2025 10:35 am

We found the issue you describe, it was triggered if the file was in a subdirectory, looks like the next release will have a fix

This still happens with RouterOS 7.18beta6. Is that version supposed to have the fix?

teslasystems · Thu Feb 13, 2025 11:07 am

@normis,
Seems, I have found, what’s happening with SMB. I’ve captured SMB packets with WireShark and see, that RouterOS is reading the contents of each file (only first 256 KB). And this repeats again and again.

On the graph at the beginning of the topic, each peak is one reading cycle. I.e., RouterOS reads the contents (first 256 KB), then pauses for about 8 seconds; reads again, pauses for 8 seconds. And so on, cyclically. Moreover, it reads each file multiple times within the same cycle, usually 3 times in each cycle.

Thats why it’s consuming so much traffic. It’s not normal.

Duplicated this to SUP-158277.
.

SMB_Cap1.png

SMB_ScansPerCyle.png

optio · Thu Feb 13, 2025 5:13 pm

Swap is not for the host system. Please read the manual about it before complaining: https://help.mikrotik.com/docs/spaces/R ... -Swapspace

This is not how is mentioned in documentation:

... This is useful when using containers on RouterOS to be able to run containers that require much more RAM than you RouterOS device has. ...

It stated as "useful when using containers" - ROS will not go OOM if containers consume more memory than is physically available, but is still can confuse reader if for other system processes memory pages are swapped or not.
If is really dedicated only for containers, it should be stated like that, I guess then this is implemented with Linux kernel Per-cgroup swap file only for container ROS process.

eider · Thu Feb 13, 2025 8:14 pm

Swap is not for the host system. Please read the manual about it before complaining: https://help.mikrotik.com/docs/spaces/R ... -Swapspace

Please explain how do you achieve separation of SWAP usage between host and containers, given that both share same kernel via namespaces and cgroups. For container to use SWAP this space must be physically present to kernel on host. Similarly, you can not adjust vm.swappiness value for container without affecting host.

If is really dedicated only for containers, it should be stated like that, I guess then this is implemented with Linux kernel Per-cgroup swap file only for container ROS process.

I don't see how this would work given ROS documentation since there is no ability to configure hard memory limit for container, just soft limit. There also does not seem to be any way to dedicate specific swapfile to given container, so this would exclude per-cgroup swapfile too. My best guess so far is that any SWAP created is simply attached to host and is shared with containers in same way host memory is.

optio · Fri Feb 14, 2025 3:46 am

It will be nice that they explain how did achieve this and state directly that swap is used only for containers. Here is interesting reading for Open Containers regarding containers memory management and comparison between cgroup1 and cgroupv2.

Fri Feb 14, 2025 9:28 am

Swap must be explicitly turned on. If you don't like how it affects your host (if it does), just do not turn it on.

Cha0s · Sat Feb 15, 2025 10:02 pm

Cool stuff!
I'll move all my NAS data over to my router now, and I'll convert my NAS into a working BGP router.
How I didn't think of this before? Genius!

ConradPino · Sat Feb 15, 2025 10:20 pm

@Cha0s LOL

infabo · Sat Feb 15, 2025 10:50 pm

Better late than never. 😉

massinia · Thu Feb 27, 2025 7:18 pm

What's new in 7.18 (2025-Feb-24 10:47):

*) disk - allow to add swap space without container package;

https://help.mikrotik.com/docs/spaces/R ... -Swapspace

Sorry, it's not clear to me if I can use the SWAP partition only for containers or also with ROS.
I'm asking because I have several hAP ac2 that can't use SMB (router freezes and restarts itself when transfers are large) and containers due to low RAM.
Before changing disk partitions i would like to make sure it is useful...

Thanks

EDIT: I found the answer, Normis wrote:

Swap is not for the host system. Please read the manual about it before complaining: https://help.mikrotik.com/docs/spaces/R ... -Swapspace

infabo · Thu Feb 27, 2025 9:52 pm

https://btrfs.readthedocs.io/en/latest/ ... -practices

RAID56 - at latest btrfs version - is not considered stable for production.

Larsa · Thu Feb 27, 2025 10:38 pm

Even vanilla Btrfs RAID 1 can be a real headache, for example, if a disk intermittently disconnects or fails for some reason and then gets marked as unreliable. Restoring a Btrfs RAID 1 is a pretty complicated process and requires expert knowledge, as @Petch1 pointed out in another thread. In other words, Btrfs RAID 1 is nothing like a normal RAID 1 with automatic resync. There are plenty of examples of this online.

If any kind of Btrfs failure happens with a ROSE Data Server, you’re pretty much out of luck since RouterOS doesn’t provide any Btrfs tools for repair or recovery. On top of that, there’s no proper backup and restore solution, which is absolutely essential for quickly getting back up and running if something goes wrong or if an important file accidentally disappears.

For MT to succeed, it must invest significantly more effort into developing a robust software solution to make the RDS2216 a viable storage server for business-critical use. But since this isn’t Mikrotik’s area of expertise, and building a storage server on 'standard' RouterOS is fundamentally flawed, the whole approach seems pointless.

To be bluntly honest, I think MT should either kill the data server product line or spin off the business into a completely separate and independent subsidiary.

The RDS2216 hardware looks great though.

Refs:
viewtopic.php?p=1129645#p1129506
viewtopic.php?p=1129645#p1129574
viewtopic.php?p=1129657#p1129601
viewtopic.php?p=1129657#p1129614
viewtopic.php?p=1129657#p1129633

infabo · Thu Feb 27, 2025 11:43 pm

When comparing the official BTRFS man page with MikroTik’s BTRFS help page, I couldn’t help but notice some more info on official docs:

- https://btrfs.readthedocs.io/en/latest/man-index.html
- https://help.mikrotik.com/docs/spaces/R ... 9711/Btrfs

While I have no real-world experience with BTRFS (other than trying it once out of curiosity), I’ve been working with Linux for over 20 years, and it seems like some essential BTRFS tools might be missing in ROSE. However, I may be mistaken, and perhaps certain functionalities are abstracted away. Even if these tools aren't needed for everyday use, their existence suggests they serve an important purpose.

Considering that the new RDS comes at a price of around $2000 - and with disks, the total cost could reach several thousand dollars - it certainly requires a fair amount of confidence to invest as an early adopter.

loloski · Fri Feb 28, 2025 12:37 am

This is a new cool toy in the block, If you could beat the reliability WAFL filesystem and have snap mirror functionality which only transfer the delta update at the block level then I can easily convince the management to upgrade at least 10 ageing filer that we have which runs 24x7x365 downside (very pricy disk if fails :) )

This product will be disruptive in the market if MT can proved this product is _reliable_ and _comparable_ to other system out there, proved us wrong as naysayers the argument of some folks here is valid and reasonable, don't take this as negative but instead use this argument to your advantage to iron out some wrinkles on the product offering, the hardware and price point of the product is reasonable in my opinion, best of luck MT

Larsa · Fri Feb 28, 2025 8:32 am

Well, this new ”cool toy” RDS2216 isn’t even playing the same sport as WAFL, let alone competing in 24x7x365 business-critical operations.

I’d say MikroTik is in way over its head on this one and in a different galaxy than NetApp. 😉

markonen · Fri Feb 28, 2025 9:57 am

To me, the RDS2216 is like a CCR2216 except:
- twice the RAM
- a more diverse network port mix (fewer SFP28 ports, but 4x SFP+ and 2x copper 10G compensate)
- interesting if early storage features
- $845 cheaper list price(!)

So it's not like you're paying a premium for the storage stuff. If anything, the opposite is true.

Larsa · Fri Feb 28, 2025 10:51 am

Hmm, I’d say the RDS2216 sits somewhere between the CCR2216 and CCR2116 in terms of both specs and price. But yeah, the RDS2216 has a pretty attractive price for that configuration, plus, like you said, you get some storage as well (though personally I wouldn’t use it as a business-critical storage solution at this point)

code

Model			Price	Strg-MB	RAM-GB	 Ports
========================================================================
CCR2116-12G-4S+		 $995	128 	16	 12xGbE, 4x10GbE SFP+
RDS2216-2XG-4S+4XS-2XQ	$1950	128 	32	 4x25G SFP28, 2x100G QSFP28, 4xSFP+, 2x10GbE 
CCR2216-1G-12XS-2XQ	$2795	128 	16	 12x25G SFP28, 2x100G QSFP28

loloski · Fri Feb 28, 2025 10:54 am

Well, this new ”cool toy” RDS2216 isn’t even playing the same sport as WAFL, let alone competing in 24x7x365 business-critical operations.

I’d say MikroTik is in way over its head on this one and in a different galaxy than NetApp. 😉

They won't call this enterprise storage for no reason let them cook this to their heart desire and let the early adopter cast the fate of this product, who knows one of these days when they are done experimenting they figured they would rather go back and focus on routing and switching again

Larsa · Fri Feb 28, 2025 1:27 pm

Yeah, but why waste valuable resources on this experiment? I mean, it’s not even close to offering basic functionality for the SMB market like DSM or QTS do, so IMO the whole concept is dead on arrival and just a total waste of money. Why not just call it what it is: a router with some extra storage capabilities.

Honestly, those resources would’ve been way better spent on developing the new controller or a proper next-gen network management and monitoring tool.

nmt1900 · Fri Feb 28, 2025 2:14 pm

Honestly, those resources would’ve been way better spent on developing the new controller or a proper next-gen network management and monitoring tool.

When talking about "next-gen" things in a corporate space then any third-party network/security audit has one sure question in it - "What UTM/NGFW solution is in use and does it have HA?"

I rest my case about "next-gen". While customers are reading newsletters about Wifi 7 solutions from other vendors, it makes me want to rest my case about wireless networking as well. After this it is futile to try market this "server" to any of these customers...

Fri Feb 28, 2025 5:07 pm

Larsa, you are confusing this with a home NAS. It's not

toxicfusion · Fri Feb 28, 2025 6:21 pm

@Normis.

This product IS DOA at this point. The RDS [Also, RDS == Remote Desktop Services -- Microsoft]. does NOT have readily available features of existing SMB home NAS... QNAP and Synoloy home NAS already has these features, and ability for User management, LDAP/AD integration...

When will there be proper disk management
LED blink to identify disks
BTRFS is dead/non-starter -- Unless using Linux mdadm for RAID..
How we monitor disk health, scrub tasks, snapshot viewing / snapshot tree / restore functions?
User management for SMB shares -- AD/LDAP integration. Or MikroTik planning to instead use the UserMan package for this?

The Unifi Dream Machines** are more capable than this RDS hardware at this point.

RDS is cool concept, but not ready for prime time.

This is the problem -- MikroTik Hardware == capable and quite good. It is always the hardware released before software is ready. We wait year+ for development to catch up.

Is MikroTik NOT taking the time to research what competition is doing, and how they're doing it?
IE: Explore Synology, QNAP [QT-OS / QTHero], Unifi storage, TrueNAS [SCALE].

toxicfusion · Fri Feb 28, 2025 6:26 pm

We're still waiting for bug fixes in V7 for ROUTEROS....

Still waiting on proper fixes for CAPsMAN [config sync for HA].
Cambium and Unifi wiping the floor in wireless for access point management, deployment, and performance...... CAPsMAN becoming overly complex and constantly tweaking to get proper wireless performance expected out of commodity wireless chipsets.

Sure-up software quality and we will be happier and perhaps less vocal toward MikroTik with our feedback.

Larsa · Fri Feb 28, 2025 6:26 pm

Larsa, you are confusing this with a home NAS. It's not

Normis, does indeed not! 😉

Just to be clear, I'm comparing the RouterOS v7 special ROSE edition, running on the ROSE Enterprise Data Server RDS2216 ("designed for enterprise environments. Secure, scalable, and under your control"), with SMB entry-level NAS solutions like DSM or QTS. From that perspective, the special ROSE edition doesn’t even come close. Do you want me to put together a comparison table of the missing key features in RouterOS v7 special ROSE edition?

Have a nice weekend!

P.S.
According to the product page, you can still run virtual machines on the RDS2216 ("Run VMs, automate workloads...") and take advantage of "mission-critical workloads with near-instant failover and future-proof scalability as NVMe technology advances." 😁

toxicfusion · Fri Feb 28, 2025 6:31 pm

Larsa, you are confusing this with a home NAS. It's not

Normis, does indeed not! 😉

Just to be clear, I'm comparing the RouterOS v7 special ROSE edition, running on the ROSE Enterprise Data Server RDS2216 ("designed for enterprise environments. Secure, scalable, and under your control"), with SMB entry-level NAS solutions like DSM or QTS. From that perspective, the special ROSE edition doesn’t even come close. Do you want me to put together a comparison table of the missing key features in RouterOS v7 special ROSE edition?

Have a nice weekend!

P.S.
According to the product page, you can still run virtual machines on the RDS2216 ("Run VMs, automate workloads...") and take advantage of "mission-critical workloads with near-instant failover and future-proof scalability as NVMe technology advances." 😁

I concur with you..... I also just posted my gripes of missing features.

This product is great concept... but "eggs in one basket" scenerio.

How can this be mission-critical? Only will be with DUAL HA-APPLIANCES! [How even work, when config sync is MISSING]

you perform maintenance on the RDS appliance, install new RouterOS release... BOOM -- entire network goes Down [Routing, and STORAGE]. This is risk with all-in-one. This is OKAY for those in small networks, home, or lab....

It is better if the RDS appliance ran a stripped DOWN version of RouterOS along-side the ROSE package.

Leave routing to a dedicated routerOS appliance.

toxicfusion · Fri Feb 28, 2025 6:52 pm

If MikroTik like to evaluate and R&D how to implement ZFS into RouterOS -- talk to Jorgen Lundman of OpenZFS. He can probably help you develop ZFS for RouterOS -- MikroTik's own linux OS. Given that ZFS cannot be a kernel module on a linux Distro only part of BSD*

floaty · Sat Mar 01, 2025 3:10 am

.
... just ingested the february news-cake ! ! ! ! !
.
... this is ... is ... a bold offer !
.

Larsa, you are confusing this with a home NAS. It's not

.
lot of people will ... both ways ...
.
and as an old MOTD-wizzard wisely said:
.
great power, comes with great responsibility

floaty · Sat Mar 01, 2025 3:31 am

... don't get me wrong *) ... I'm flashed ...
.
*) ... cause you always get me wrong

CGGXANNX · Sat Mar 01, 2025 7:07 am

Even vanilla Btrfs RAID 1 can be a real headache, for example, if a disk intermittently disconnects or fails for some reason and then gets marked as unreliable. Restoring a Btrfs RAID 1 is a pretty complicated process and requires expert knowledge, as @Petch1 pointed out in another thread. In other words, Btrfs RAID 1 is nothing like a normal RAID 1 with automatic resync. There are plenty of examples of this online.

They have updated the docs and change the file system in the RAID examples to ext4
https://help.mikrotik.com/docs/pages/di ... ersions=91

toxicfusion · Sat Mar 01, 2025 7:49 am

lmfao, this cant be real -- they changed from BTRFS to ext4 , in real-time... what about the marketing for snapshots?............

Yes, ext4 mdadm is much preferred and stable over BTRFS. I'd actually like to see XFS -- but you lose snapshot ability.

Synology and QNAP have their own implementation of RAID5/6 of BTRFS -- which I doubt MikroTik has yet to research or solidify...

ZFS would be ideal here, but would take significant development to work with their own linux distro and the kernel version they're using.

In my eyes - This is a very early product, beta. Hardware was released WAY to early; software QA needs to catch up.

We still need a way to comfortably manage the disks, scrub tasks, Smart tests, blink LEDS. etc. Alot is lacking at this point. No one will want to use MikroTik CLI to configure their "enterprise storage", when it can be done BETTER and with more peace of mind from other vendors. Until MikroTik adds ease of disk management and all the needed elements into Winbox or WebFig... this is DOA.

RDS should be running its own MikroTik storageOS, with its own management interface.

markonen · Sun Mar 02, 2025 7:45 pm

I have zero interest in NASes, or in MikroTik making a NAS, let alone a NAS OS. I did, however, immediately order two RDS2216s (to start), specifically because they are routers that run RouterOS.

I think if you currently use high end MikroTik gear like the CCR2216 in your organization you are already well off the beaten path in terms of making choices nobody ever fired an IT manager for. So for folks like you, incrementally adding some storage features to your network infrastructure might enable some interesting use cases, and if not, at least you got a weird green CCR on the cheap.

MikroTik is eating the elephant one bite at a time. Hopefully in a couple of years you can do more, with a more polished experience.

But people who say that MikroTik should have chosen to spend years more, developing storage software in the dark, not shipping any hardware, know nothing about how products in general or software in particular is made.

It's equally tiring to hear capital-letter IT Professionals who would only ever use MikroTik at home chime in anyway. Please don't enumerate the ways the RDS2216 is worse than your seven-figure storage install in terms of your specific ISO 27001 compliance regimen. Literally nobody cares. You can laugh it up with your NetApp or Nutanix rep the next time you're wined and dined pre-renewal.

sszbv · Mon Mar 03, 2025 5:56 pm

Now with the nice new ROSE server surprise, I decided to test the rosestorage package.
Like, when MikroTik dares to store files on RouterOS, then I can do it too right?
Well, it works great!!!
So I'm, running a 4core 3ghz PC 16gb ram, 4x4tb harddisk in raid5 + 2x 500gb ssd in raid1.
I've put a few million files on it, exact duplicate of my windows fileserver.
This works very good, I'm surprised. And it's pretty easy to setup too!

Except that there is no real way to see which files are on the disks.
The current 'Files' option in winbox tries to download the complete list of files first, then creates a treeview of it.
But with millions of files, this takes ages and I've never actually seen it finish.
In the CLI, something similar happens, you just get a dump of all the files.

I suppose you (MikroTik) experience this as well.
The obvious solution is to change this behavior into getting a list of files/dirs in the current directory first, then when you click a plus sign, it will get the list of that folder, etc. Like in a real file-explorer on macos or windows.
I hope this will become a reality in the future.

In the meantime I can just use an SMB client to view the files.

Speaking of SMB, I noticed that when you create a share with uppercase characters, there will always be a second share visible with lowercase characters only. Don't know if that is a linux thing, but for now I just changed all sharenames into lowercase.

BTW Good job on the rosestorage package and the cool green box, awesome.

teslasystems · Mon Mar 03, 2025 6:13 pm

The current 'Files' option in winbox tries to download the complete list of files first, then creates a treeview of it.
But with millions of files, this takes ages and I've never actually seen it finish.
In the CLI, something similar happens, you just get a dump of all the files.

I suppose you (MikroTik) experience this as well.
The obvious solution is to change this behavior into getting a list of files/dirs in the current directory first, then when you click a plus sign, it will get the list of that folder, etc. Like in a real file-explorer on macos or windows.
I hope this will become a reality in the future.

I've mentioned it multiple times and no one cares. It will never work normally without fully changing approach to work with files.

sszbv · Mon Mar 03, 2025 6:17 pm

This is offtopic, but I've mentioned it multiple times in other topics and no one cares. It will never work normally without fully changing approach to work with files.

Maybe now is the time for them to start taking this issue seriously?
It makes more sense now, right?

teslasystems · Mon Mar 03, 2025 6:19 pm

This is offtopic, but I've mentioned it multiple times in other topics and no one cares. It will never work normally without fully changing approach to work with files.
Maybe now is the time for them to start taking this issue seriously?
It makes more sense now, right?

+

Larsa · Mon Mar 03, 2025 6:46 pm

Mon Mar 03, 2025 6:56 pm

Post about ROSE by sszbv moved to the right topic.

sszbv · Mon Mar 03, 2025 8:09 pm

Post about ROSE by sszbv moved to the right topic.

Sorry for posting it in the wrong topic, I searched for a rose topic, but couldn't easily find it.
Second best was Winbox 4 in my opinion.

I see here that others also mentioned the file tree being, well, not very useful with many files.

On cli it is sort of workable with the new path option.
Like:
/file print path=<tab>
sort of gives you the possibility to browse trough the files

sszbv · Mon Mar 03, 2025 11:40 pm

I, for one, welcome our Storage overlords.

When ROSE came out, I put an NVMe to SATA adapter in a 2116, punched a hole, ran some thin SATA cables out, and hooked up six SATA drives in a bay the size of a CDROM. It works pretty well.

The advantage MikroTik has over a PC is wirespeed switching/routing (on Marvell platforms). I run a bunch of containers on my 2116. Going to have to add the Minecraft one (didn't know it existed for ARM64).

+1 for your idea!
No mercy on the case, just put a hole in it!
Anyway, I think I'll try that too, just to see what happens.
I allready tried the CCR2004 PCIE card on the CCR2116 with a m2 to pcie converter :)
That works fine

Tue Mar 04, 2025 12:13 am

Post about ROSE by sszbv moved to the right topic.
Sorry for posting it in the wrong topic, I searched for a rose topic, but couldn't easily find it.
Second best was Winbox 4 in my opinion.....

Not a problem. I should have written "Moved to dedicated to ROSE topic.".

markonen · Tue Mar 04, 2025 9:51 am

I allready tried the CCR2004 PCIE card on the CCR2116 with a m2 to pcie converter :)
That works fine

At the risk of being off topic, could you describe that further? Works fine in terms of just getting power, or does it also appear as network interfaces on the CCR2116? Any details on the physical setup?

sszbv · Tue Mar 04, 2025 11:22 am

I allready tried the CCR2004 PCIE card on the CCR2116 with a m2 to pcie converter :)
That works fine
At the risk of being off topic, could you describe that further? Works fine in terms of just getting power, or does it also appear as network interfaces on the CCR2116? Any details on the physical setup?

It shows up in the interfaces. Just use a m2/nvme to pcie 16x slot converter from aliexpress + powersupply.

sszbv · Tue Mar 04, 2025 11:26 am

Even vanilla Btrfs RAID 1 can be a real headache, for example, if a disk intermittently disconnects or fails for some reason and then gets marked as unreliable. Restoring a Btrfs RAID 1 is a pretty complicated process and requires expert knowledge, as @Petch1 pointed out in another thread. In other words, Btrfs RAID 1 is nothing like a normal RAID 1 with automatic resync. There are plenty of examples of this online.

They have updated the docs and change the file system in the RAID examples to ext4
https://help.mikrotik.com/docs/pages/di ... ersions=91

I used normal raid first and then used BTRFS as a filesystem, because I thought it would be safe enough. But now that the example has changed...
Should I be worried and redo the whole system with ext4?

infabo · Fri Mar 07, 2025 11:08 am

I wonder if MikroTik has really thought about worst-case scenarios when developing RDS and ROSE. It’s easy to assume everything will work perfectly, but reality is often more complicated. That’s why setting up a NAS on a standard Ubuntu system isn’t simple - there’s a reason why special distributions exist for this.

Also, a full Linux system offers many recovery options. I would be concerned. As an RDS admin, if I run into an unexpected Btrfs issue that MikroTik didn’t plan for, I can’t spend weeks sending *supout.rif* files back and forth with support. My data needs to stay available and online.

massinia · Fri Mar 07, 2025 4:09 pm

What's new in 7.18 (2025-Feb-24 10:47):

*) disk - allow to add swap space without container package;
https://help.mikrotik.com/docs/spaces/R ... -Swapspace

Sorry, it's not clear to me if I can use the SWAP partition only for containers or also with ROS.
I'm asking because I have several hAP ac2 that can't use SMB (router freezes and restarts itself when transfers are large) and containers due to low RAM.
Before changing disk partitions i would like to make sure it is useful...

Thanks

EDIT: I found the answer, Normis wrote:

Swap is not for the host system. Please read the manual about it before complaining: https://help.mikrotik.com/docs/spaces/R ... -Swapspace

I'm starting to think that SWAP partition is not only used by containers

swap.jpg

I am not using containers and I don't even have the container package installed

infabo · Fri Mar 07, 2025 5:04 pm

Even Druvis mentions the SWAP can be used not only for containers: https://youtu.be/wJw50I7STck?t=131

grusu · Fri Mar 07, 2025 5:35 pm

On my RBAHx4 Dude Edition I tried setting up a swap file but it doesn't seem to work:
/disk add type=file file-path=sata-part1/swapfile file-size=500M swap=yes

massinia · Fri Mar 07, 2025 5:36 pm

Even Druvis mentions the SWAP can be used not only for containers: https://youtu.be/wJw50I7STck?t=131

Thanks, I hadn't seen the video...

I tried setting up a swap file

SWAP file it's much slower (I only tried with hAp ac2), if you can add a small partition to use as SWAP.

grusu · Fri Mar 07, 2025 5:42 pm

I just wanted to try it. It doesn't work.

massinia · Fri Mar 07, 2025 6:06 pm

It seems that for the SWAP file you need to install rose

/disk add type=file file-path=usb1/swapfile file-size=1G swap=yes

failure: not supported without rose-storage package

grusu · Fri Mar 07, 2025 7:07 pm

rose-storage is present.

CTassisF · Tue Mar 18, 2025 5:38 pm

I'm preparing an external SSD for use as storage (ext4) and swap in RouterOS.
What’s the correct way to set up the swap partition? Should I format it as Linux swap (gdisk type 8200) and run mkswap?
I'm confused because RouterOS doesn’t seem to recognize swap partitions.

itimo01 · Wed Mar 19, 2025 1:01 am

I'm preparing an external SSD for use as storage (ext4) and swap in RouterOS.
What’s the correct way to set up the swap partition? Should I format it as Linux swap (gdisk type 8200) and run mkswap?
I'm confused because RouterOS doesn’t seem to recognize swap partitions.

Just format the partition as "wipe" on ROS and use something like:

disk/set usb1-part2 swap=yes

Josephny · Wed Mar 19, 2025 2:32 am

Can someone explain what the end-user usefulness or benefit of ROSE (or any directly attached) storage would be to an MT router?

Maybe in the form of: Just like with a smartphone I have access to multiple forms of communications, information, location apps, and so many other other things with me at all times, by adding 100MB to 10TB of directly accessibly storage by the ROS, I can <------->.

anav · Wed Mar 19, 2025 3:10 am

Dunno, I thought it was so Larsa can fly around and sniff sweet smelling MT equipment. ;-P

ROSE (RouterOS Enterprise) package adds data center functionality to RouterOS - for supporting disk monitoring, improved formatting, RAIDs, rsync, iSCSI ,NVMe over TCP, NFS. This functionality currently is supported on arm, arm64, x86 and tile platforms.

Josephny · Wed Mar 19, 2025 3:16 am

I can understand and even relate easier to Larsa's sniffing switches easier than I can understand the usefulness of 1TB directly connected to a router.

Is the idea that a router with ROSE and storage becomes NAS?

sirbryan · Wed Mar 19, 2025 3:31 am

Think low-cost HCI (hyper-converged infrastructure). Most hardware sits there spinning its wheels (figuratively), so why not give it other tasks to do?

For smaller routers, it doesn't make much sense, but if you have a USB3 port on the router, you can share an external hard drive with all the computers/phones/tablets in your home without hardly any effort. With enough RAM and some disk space, you can run containers, which are like adding apps to your router. Popular containers include Pi-Hole and other ad blockers, Home Assistant, Uptime Kuma, Nextcloud, and so on. ARM machines in particular are much cheaper to run than an old spare PC, and quiet too.

The big boys (CCR2116,2216,RDS2216) have room for internal storage, ample cores and RAM to load even more containers (I run about 7 on my office 2116). Imagine them being used as branch office devices where a business can deploy an all-in-one box that routes Internet traffic, provides VPN access, serves files, runs an internal file or document server, etc. Or for home labs where you might want to tinker with the other hardware, but need something that just stays put (like a router). Put the drives in, set up storage, and the router/storage box just does its job in the background.

In the ISP/MSP side, these could be useful as content stores, edge caches, etc.

Larsa · Wed Mar 19, 2025 6:47 am

Managing storage requires administrative tools for restructuring, error handling, and backup/restore. All of these are missing in ROS. In other words, this solution is not suitable for mission-critical applications.

grusu · Wed Mar 19, 2025 8:32 am

Maybe it's work in progress. They will probably be implemented in the future.

grusu · Wed Mar 19, 2025 8:37 am

A roadmap would be interesting. It would be useful for customers to be able to plan their future purchases based on what implementations they are following.
But Mikrotik doesn't work that way...

Wed Mar 19, 2025 10:01 am

Managing storage requires administrative tools for restructuring, error handling, and backup/restore. All of these are missing in ROS. In other words, this solution is not suitable for mission-critical applications.

Can you clarify what you mean by "restructuring" and "error handling"?

- you can monitor SMART stats in /disk print stats for errors
- we have full support for rsync for backups/restores
- backups with btrfs snapshots, snapshot transfers
- btrfs builtin data replication across multiple disks with separate data/metadata profiles on live filesystem
- automatic error handling in RAID with parity/mirror disks
- any number of hot spares to automatically rebuild parity/mirror disks upon failure
- manual replacing and of parity/mirror disks and rebuild

Larsa · Wed Mar 19, 2025 5:27 pm

Normis, thanks for the reply and details on existing features. But let’s be real here—while these are basic storage functions, they’re nowhere near the administrative tools that actual enterprise (or even small office/home office) NAS solutions provide.

Restructuring – More than just RAID

Restructuring isn’t just about manually swapping disks in a RAID array. Enterprise NAS solutions allow volume resizing, dynamic storage pools, and flexible LVM management.
With QNAP, Synology, and even budget NAS devices, you can easily expand storage, migrate volumes, and allocate resources dynamically—without manually juggling RAID arrays.
In ROS, if you want to restructure storage, it looks like the only option is “start over”—not really enterprise-friendly.

Error Handling – What happens when things go south?

Sure, you can check SMART stats, but that’s only helpful before a failure. What about real-time, automated health monitoring that predicts failures and prevents downtime?
Enterprise NAS systems don’t just log errors; they have self-healing file systems, proactive failure notifications, and automated data recovery.
When a file system corruption occurs (which it inevitably will at some point), what’s the recommended ROS solution? Send the disk (or whole server) to Mikrotik support and put the business on hold?

Backup & Restore – Rsync alone isn’t enough

Rsync is great—for manual backups. But real NAS solutions provide scheduled, incremental, and versioned backups with built-in policy management.
Enterprise-grade NAS solutions have seamless cloud/hybrid backups and integration with industry-standard tools like Veeam, CommVault, or even Synology Hyper Backup.
Btrfs snapshots are nice, but without a user-friendly way to manage, restore, and automate them, they’re just another manual workaround. And if Btrfs fails, you’re out of luck.
A proper backup solution must be able to back up the entire unit, including containers and configurations. A cold restore should be able to recover everything from scratch, not just some random files when everything is up an running.

The Big Picture

Right now, MikroTik’s approach to storage feels more like a DIY Linux server setup with all the admin tools stripped out than an enterprise storage solution. If ROS is meant to compete with similar SMB tech as QNAP or Synology, then it needs actual administrative tools, not just CLI commands and basic RAID functions.

The hardware is solid, but without proper management features, this isn’t a realistic option for mission-critical environments. Are there any plans to add a real admin interface, more automation, and enterprise-level storage tools in future updates? Because that’s what’s missing.

Otherwise, for now, I’m sorry to say this is IMO more of a “tinkerer’s NAS” than a serious storage solution.

infabo · Wed Mar 19, 2025 6:18 pm

the drive behind RDS: https://youtu.be/LGAwubEU92A?si=pkeK0dBYeN_u6Uii

Larsa · Wed Mar 19, 2025 7:35 pm

Putting lipstick on a pig but since someone called it "Kermit the NAS," I guess it’s more like putting lipstick on Miss Piggy! 😉

infabo · Wed Mar 19, 2025 8:30 pm

As I understood it, the product idea started with:

Genomics researchers: we need a network storage.
Mikrotik: challenge accepted! When you have a ROS - everything is a router.

But I like the idea/intention to combine router/storage and save on energy. And as Druvis said, maybe more useful as edge storage or something that is not as criticial.

fouinix · Thu Mar 20, 2025 12:19 am

Is there any hope we will be able to use external storage to offload local storage ? I think about routers with only 16MB.

bratislav · Thu Mar 20, 2025 3:57 pm

Is there any hope we will be able to use external storage to offload local storage ? I think about routers with only 16MB.

I am not sure about that, but it seems that adding all these feature increased the RouterOS package substantially so many of us wont be able to use 16MB devices at all...

toxicfusion · Thu Mar 20, 2025 6:10 pm

MikroTik is completely missing our points, they dont care and will not listen...... they're misguided and heads in sand. This product IS DOA. Engineers had fun and were excited with the new features available in the Linux kernel they're now using.

MikroTik does not yet have the software / tools ready for us to properly manage the disks. Why would we want to use mdadmin or rely on RAID for "enterprise" storage. We need a full storage management solution from MikroTik - as @Larsa beautifully pointed out. BTRFS is garbage for this use-case.

Innovation would been MikroTik taking the time to implement openZFS as a .npk add-on along with their "ROSE".

MikroTik will just be "Tinker Toy" for this product. This will only be for those select few, very niche market of home-labs and those interested in using this platform for storage needs. As the product ships, It is TOO complicated to manage. For us having to rely on command line to build the array, manage disks and script out storage checks -- non starter. Others would rather build their own storage appliance as "roll their own solution".

MikroTik has lost their focus and has stopped innovating on the networking side. They give us puzzle pieces, but never finalize the software to solid solutions. The documentation is even crap.

Sorry, but we've been slowly moving our customers to another vendor who is doing it better [Routing, Wireless] for similar cost. The real focus should be adding IPS/IDS, Web Filtering - or ability to add third-party subscriptions, and device management solution [Controller] -- a better portfolio.

UniF**k is now circling back and has been innovating their networking side. Their new "enterprise" aggregation switches and new software releases are showing great steps forward. They are also allowing for switch stacking. MikroTik -- are you awake?

toxicfusion · Thu Mar 20, 2025 6:14 pm

If MikroTik releases the tools needed to manage the storage within Winbox or another software.. then this product would be "usable". But we will wait for 2+years for that and the product will be dead. Using the terminal to setup the NVME RAID...... wtf.

We need disk management:
Volume resizing, dynamic storage pools, and flexible LVM management.
expand storage, migrate volumes, and allocate resources dynamically
Pool quota assignment, snapshots, ability to manage the snapshot tree, manage the snapshot file-system and restore.

Larsa · Thu Mar 20, 2025 9:48 pm

Normis, if Mikrotik is still serious about developing a storage solution, it might be more practical to offer it as an add-on "install package" using one of the well-established open-source storage platforms rather than building a proprietary system from scratch. I strongly advise against developing a proprietary solution, as Mikrotik lacks the necessary knowledge, experience, and resources for it.

Mikrotik’s strengths lie in networking, not storage, and developing a robust, enterprise-grade storage stack requires deep expertise, years of refinement, and extensive testing—which competitors like TrueNAS, OpenMediaVault, and Ceph have already done. Instead of reinventing the wheel, why not leverage an existing solution and integrate it tightly with RouterOS?

This could be split into two approaches:

SMB market: Something simple like OpenMediaVault or TrueNAS Core for small businesses.
Enterprise/high-performance storage: Solutions like Ceph or BeeGFS for scalable, clustered storage with high availability.

A Mikrotik appliance running one of these as an officially supported option would be far more appealing than a homegrown, feature-limited alternative. Since these platforms already have integrated web-based administration, you also avoid the hassle of integrating your own proprietary storage system into WinBox.

Here’s an comparison table with some of the most popular open-source and Linux based NAS/storage solutions and their key features:

+---------------+------------+------------+------------+-------+------+----------------+-----------------+
| Solution      | Filesystem | HA Cluster | Docker/K8s | iSCSI | RoCE | Backup Options | Best Suited For |
+---------------+------------+------------+------------+-------+------+----------------+-----------------+
| TrueNAS Scale | ZFS        | Limited    | Yes        | Yes   | No   | Snapshots, S3  | SMB, SOHO, Pros |
| OpenMediaVault| ext4, XFS  | No         | Yes (addon)| Yes   | No   | Rsync,Snapshots| SMB, Home Users |
| Ceph          | CephFS     | Yes        | Yes        | Yes   | Yes  | Multi-layered  | Enterprise, HPC |
| BeeGFS        | BeeGFS     | Yes        | Limited    | No    | Yes  | External Tools | HPC,AI Workloads|
| Rockstor      | Btrfs      | No         | Yes        | Yes   | No   | Snapshots      | SMB, SOHO       |
+---------------+------------+------------+------------+-------+------+----------------+-----------------+

Some key considerations:

TrueNAS Scale is a great option for SMBs with ZFS, Docker, and iSCSI support.
OpenMediaVault is the easiest to use but lacks HA clustering.
Ceph & BeeGFS are ideal for high-performance environments but require more complex setups.
Rockstor & XigmaNAS are good lightweight alternatives with Btrfs/ZFS but limited HA.

Here are some articles that review and compare open-source NAS solutions:

ConradPino · Fri Mar 21, 2025 12:27 am

@Normis I believe @Larsa makes some good points. I believe MikroTik strengths are:

hardware design and manufacturing at excellent pricing value points
prior history delivering reliable and stable networking software

but recentl RouterOS 7 development has unmasked incompetance managing software development process; in particular:

number of RouterOS networking function regressions introduced as development proceeds are excessive overall
RouterOS documentation quality is degrading from insufficent resources to stay current in sufficient detail
insufficent testing new feature releases to limit regressions is turning all upgrading users into alpha testers
not maintaining a parallel regression free lagging features release for users wanting networks that just work

Please focus software development onto providing reliable software first and new features introduced as distinct release which users can adopt or not.

I see @Larsa suggestion as best chance MikroTik has to salvage RDS2216 development investment. Opening RDS2216 for hosting existing ready today solutions @Larsa identified allows returning limited software developer resources back onto the core networking competency.

Amm0 · Fri Mar 21, 2025 1:45 am

@Normis I believe @Larsa makes some good points.
[...]
I see @Larsa suggestion as best chance MikroTik has to salvage RDS2216 development investment. Opening RDS2216 for hosting existing ready today solutions @Larsa identified allows returning limited software developer resources back onto the core networking competency.

While 100% agree on the release management and docs, which is frustrating. And the current docs on ROSE storage features border on engineering malfeasance. I do like the green, and like the work of their graphics artist/firm — but they needed a doc writer, not a designer.

I did watch @dru's video with RDS and some YouTuber. Once again ton of good information that should have been written down in docs - not in 30 minute video. But it good to hear Druvis seems to acknowledge the UI needed improved — which is just an absolute mess and very poorly organized. But improving PCI bus was also discussed — that was annoying to hear — since agree focus on making the software usable and well tested before even thinking about making more "storage" models – the speed will be what it is & one seem enough to test the concept.

Now I agree with other commentary that with storage losing data is unforgivable sin & the lack of attention to detail doesn't give one the "warm fuzzies" their critical data will be safe.

But I think WAY too early to suggest the need to "salvage" their investment. And I actually don't think they invested much... which may part of the complaints I suspect. To me, it looks like largely wrapped the Linux drivers/apis for "storage" into the RouterOS schema & called it ROSE. And MikroTik has good supply chain for the hardware.

And for the internal needs like Dude, UM, or /containers... snapshots and RAID0/1/5 seems like the main feature one would need beyond what was already. They seem to like btrfs - I don't know enough to suggest that's good/bad but it support snapshots which I think was the only thing the "old" ext4-only ROSE was missing for "basic" needs. What I think be bad is trying to support MULTIPLE "high end" volume management schemes. If they want to focus on making btrfs work, that seem like a good idea than worrying about ZFS vs XXXFS.

Plus if one ignores the over-focus on "storage" in the RDS. And look at more like low-cost, network-centric ARM server... it doesn't seem stupid. it does seem like a good deal as a next step up if your beyond a couple RB5009 or needed disks for containers or even home media server/NVR. And I've seen plenty of Dell/IBM/HP/etc with a shit ton of unused drive bays, and RDS likely end up same in 90% of cases in future be my bet. But folks, including me, can easily justify "well we don't know if we'll need more storage".

Anyway, if one ignore usage as a classic "NAS" / storage server, and rather a RouterOS device that has a ton of disk available, and they already have features that use storage... Now if you do want to use it as a generic storage server for critical applications in a data center, it's clearly not designed for that nor should it be for all the reason above about MT's core capacities. But wrapping LVM-like API into RouterOS does seem within their wheelhouse.

Fri Mar 21, 2025 2:50 am

IMO , I will always be for --> classic "NAS" / storage servers.

I prefer a ( router be a router) and a (switch be a switch) and a (NAS be a NAS) and a (HyperVisor be a HyperVisor) and virtual machines VMs should not be inside containers.
I guess I'm just old school in my opinions for what works best on what I maintain.
If you put all of the above in a blender and make a do-everything-box , then you have anything and everything that might give you problems.

toxicfusion · Fri Mar 21, 2025 8:55 pm

I 100% agree and echo what @Larsa has laid out and recommended.

MikroTik strengths are hardware design and development. Software aspect is shit, sorry MikroTik - you have lacked refinement over the years. Your software takes years to compliment the hardware you release.

You're better off NOT re-inventing the wheel. But instead release dedicated .npk add-on packages that will easily deploy onto the MikroTik hardware and take full advantage of it. This would give you higher marks for product reviews, level of interest and perhaps assist in market share for hardware sold.

TrueNAS or XigmaNAS [This was a fork of FreeNAS] would be great options.

Otherwise, MikroTik --- you would have to work to develop an .npk add-on package for OpenZFS. Not to mention all the software development to TACK ON - bolt on for storage management into Winbox. This would be a bloated disaster. WebFig is garbage for this use-case and will put bad taste into end-users mouths that have prior experience with other vendors. The 'coolness" will wear-off within 1-week. What will happen when a customer has a hardware failure, disk failure(s) and the RAID volume gets corrupted. What tools will there be for drive management, etc?

RouterOS === Routing...

Stay in your lane and focus on what your business has been already built upon. Innovate where you will benefit the most.

As it stands, the RDS and "ROSE" is only suited for Homelab and a few NICHE use-cases for branch offices or small business owners. Otherwise -- sticking to slinging hardware to third-world countries.

toxicfusion · Fri Mar 21, 2025 9:09 pm

@Normis I believe @Larsa makes some good points.
[...]
I see @Larsa suggestion as best chance MikroTik has to salvage RDS2216 development investment. Opening RDS2216 for hosting existing ready today solutions @Larsa identified allows returning limited software developer resources back onto the core networking competency.
While 100% agree on the release management and docs, which is frustrating. And the current docs on ROSE storage features border on engineering malfeasance. I do like the green, and like the work of their graphics artist/firm — but they needed a doc writer, not a designer.

I did watch @dru's video with RDS and some YouTuber. Once again ton of good information that should have been written down in docs - not in 30 minute video. But it good to hear Druvis seems to acknowledge the UI needed improved — which is just an absolute mess and very poorly organized. But improving PCI bus was also discussed — that was annoying to hear — since agree focus on making the software usable and well tested before even thinking about making more "storage" models – the speed will be what it is & one seem enough to test the concept.

Now I agree with other commentary that with storage losing data is unforgivable sin & the lack of attention to detail doesn't give one the "warm fuzzies" their critical data will be safe.

But I think WAY too early to suggest the need to "salvage" their investment. And I actually don't think they invested much... which may part of the complaints I suspect. To me, it looks like largely wrapped the Linux drivers/apis for "storage" into the RouterOS schema & called it ROSE. And MikroTik has good supply chain for the hardware.

And for the internal needs like Dude, UM, or /containers... snapshots and RAID0/1/5 seems like the main feature one would need beyond what was already. They seem to like btrfs - I don't know enough to suggest that's good/bad but it support snapshots which I think was the only thing the "old" ext4-only ROSE was missing for "basic" needs. What I think be bad is trying to support MULTIPLE "high end" volume management schemes. If they want to focus on making btrfs work, that seem like a good idea than worrying about ZFS vs XXXFS.

Plus if one ignores the over-focus on "storage" in the RDS. And look at more like low-cost, network-centric ARM server... it doesn't seem stupid. it does seem like a good deal as a next step up if your beyond a couple RB5009 or needed disks for containers or even home media server/NVR. And I've seen plenty of Dell/IBM/HP/etc with a shit ton of unused drive bays, and RDS likely end up same in 90% of cases in future be my bet. But folks, including me, can easily justify "well we don't know if we'll need more storage".

Anyway, if one ignore usage as a classic "NAS" / storage server, and rather a RouterOS device that has a ton of disk available, and they already have features that use storage... Now if you do want to use it as a generic storage server for critical applications in a data center, it's clearly not designed for that nor should it be for all the reason above about MT's core capacities. But wrapping LVM-like API into RouterOS does seem within their wheelhouse.

Great points. Yes, I also do feel MikroTik has a good relationship with their hardware supply chain - and have good engineers designing the hardware.

However, I feel MikroTik did not do their homework. They failed to see what other vendors are doing, how they're doing it and how MikroTik can be different and do better. Software is lacking. UniF**k is doing it better with their UDM and Dream Machines.

Focus needs to be on innovation for networking, of their current network stack of hardware products. We're still lacking switch stacking, CAPsMAN HA-controller config sync, etc.

They'll just keep flooding market with low-cost hardware until this current business model fails and then the CEO will start asking questions.

If MikroTik's RDS hardware appliance can do it better than UniF**k's -- then I will eat my words. But it is still alpha release in my eyes.

Druvis mentioned this in the Youtube interview..... They will release and obtain feedback from their distribution partners and adjust accordingly. How about listen to your loyal customers, MUM attendies, WISP providers and certified professionals?

Distributors just sell the hardware and at times will work through RMA process.....

If MikroTik had a dedicated reseller / partnership program - there would be a much better feedback loop..

hajde · Sun Mar 23, 2025 5:53 am

Must to have is own/mikrotik ssd. Offcourse compatibility list of it.

Zfs at all.

sirbryan · Sun Mar 23, 2025 7:14 pm

Some thoughts after playing with the RDS2216 this weekend.

Wow. Ugh. Oof. WIP. Fingers crossed.

(My background: 25 years of working in the WISP & telco/ISP industry in general, with lots of time spent managing small data centers and networks, as well as writing mostly web-based applications running on those servers. I have personally deployed hundreds of MikroTik radios, routers, and switches in the last five years alone.)

The RDS2216 kit itself is great. Inexpensive, slim design, lots of drive density, and good smattering of ports on the back. Adding drives (physically) was a piece of cake, although the drive trays are pretty weak. While a careless tech could break them pretty easily, I guess once they're in, you forget about them. It's loud, but not SuperMicro or Dell/HP loud. I will definitely look for some Noctua fans if this thing stays in the lab much longer (vs. getting deployed).

The management of the drives/arrays/filesystems/shares is all over the place. I get it that you guys needed to fit it within your existing 20+-year-old design paradigm, but in order for the future ROSE series to go primetime, a new set of management tolls is needed for the storage side of things. While a CLI is great for doing advanced things, it will hinder adoption and makes managing large arrays very difficult. People need to see things visually. (TrueNAS SCALE's GUI is pretty good at that, for example.)

While I have gained some experience with ROSE over the least year or two (since it came out), I spent way too long trying to build and rebuild different MDRAID array configurations. And because I wasn't doing it right, RouterOS crashed/rebooted at least a dozen times. This was in both the RDS and the ARM64 CHR machine I'm comparing it to. There are some serious error handling problems in RouterOS 7.

Once I got the mdraid-based arrays working properly, I tried to use the GUI tools (Webfig & Winbox) to build a BTRFS array. That went absolutely nowhere, so I had to drop to the CLI, and that only worked because there are some examples in the documentation.

(One thing RouterOS's GUI's lack is any form of documentation/help/guidance. I can't imagine it would be that hard to embed some kind of unique identifier with each element that the GUI renderer could use to generate a URL that links to online documentation related to the item they're creating or editing. )

So, now with drives actually doing something (formatted, mounted, shared), I ran some tests.

ROSE SMB is still bad. 100Mbps max read/write to macOS.
The documentation is missing macOS NFS mounting instructions (we have a forum post where a few of us figured it out)
A Samba container and ROSE NFS on both machines produced better results: roughly 2.5Gbps (RDS) and 3.5Gbps (CHR/ARM64) write speeds and 6-8Gbps read speeds to two RAID1 Samsung enterprise NVMe drives (drive specs claim 9Gbps sequential write and 18Gbps sequential read on Gen 3 x4).
The throughput limitations are tied to single-thread performance; the 3GHz of the CHR ARM64 outperformed the 2GHz of the RDS2216 (both had 16 cores, 32GB of RAM)

I got NVMe over TCP to work between the CHR and RDS, but didn't do any testing yet. Ideally, I'd like to export the drives (or assembled arrays) as NVMe or iSCSI block devices and see if TrueNAS or Proxmox can ingest individual drives for use by ZFS, or arrays for use by hypervisors.

More to come.

Larsa · Sun Mar 23, 2025 8:28 pm

Thanks for great feedback and for sharing your hands-on experience! Looking forward to seeing what you come up with next.

infabo · Sun Mar 23, 2025 9:13 pm

+1. Very valuable for Mikrotik.

ConradPino · Sun Mar 23, 2025 11:23 pm

+1 @sirbryan Thank you!!

sirbryan · Mon Mar 24, 2025 12:47 am

Some more quick tests with Blackmagic Design Disk Speed Test over the network.

Ampere ARM64 server booted into RouterOS 7.18.2 bare metal (64 out of 80 x 3GHz cores, 128GB RAM):

2x Samsung MZQLW960HMJP-00003 enterprise NVMe drives via SlimSAS connections, in a BTRFS RAID1 configuration
- Samba Container: 700-800MB/s writes & reads
- NFS: 490-500MB/s write, 980MB/s reads (saturating the 10Gbps of my Mac Studio)

2x Samsung NVMe drives, exported from RDS2216 to Ampere machine using NVMe over TCP (25Gbps), RAIDed on Ampere box in a BTRFS RAID1 configuration, shared via NFS to Mac:
- NFS: 260MB/s write, 980MB/s read
- Samba container: 717MB/s write, 800MB/s read (with weird pauses at the end waiting for SYNCs to complete

I observed that traffic was flowing in both directions from the Ampere to the 2216, suggesting that the host was actually sending data to the drives and wasn't caching everything (see attached graph).

I have yet to determine the optimal configuration of drives + array configurations + services, but so far running Samba on the Ampere box has provided consistent 7+Gbps results in both directions. I'll do more testing and put it all in a video.

Initial conclusions: the 2216 has the ability to utilize the throughput of the disks and put it out on the network, it's just a matter of the best way to tap into it. With Ampere processors behind ROSE (as hinted by Sept. 2024's press release), MikroTik's HCI push could become an even bigger deal. But they need a better way to manage storage and containers.

Larsa · Mon Mar 24, 2025 2:18 pm

Would be interesting to see the same tests run directly on the RDS2216. Any chance you could give that a try?

spippan · Mon Mar 24, 2025 2:42 pm

Would be interesting to see the same tests run on the RDS2216. Any chance you could try that?

do you mean from RDS2216 <-> RDS2216 ?

Larsa · Mon Mar 24, 2025 2:52 pm

I just realized I wasn’t clear enough. I meant running a disk speed test directly on the RDS2216 inside a container, if that’s possible.

sirbryan · Mon Mar 24, 2025 8:47 pm

FIO tests run in container, with two of the Samsung NVMe drives as BTRFS RAID1 mounted in the container:

Random writes = 179MB/s, or 1.432Gbps on the network:

/Shared/tests # fio --name=fiotest --ioengine=sync --rw=randwrite --bs=4k --numjobs=1 --size=5G --runtime=1m --time_based
fiotest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
fiotest: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=192MiB/s][w=49.1k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=365: Mon Mar 24 18:28:07 2025
  write: IOPS=43.6k, BW=170MiB/s (179MB/s)(9.99GiB/60001msec); 0 zone resets
    clat (usec): min=12, max=1003, avg=16.56, stdev= 2.66
     lat (usec): min=13, max=1003, avg=16.88, stdev= 2.67
    clat percentiles (nsec):
     |  1.00th=[13632],  5.00th=[14144], 10.00th=[14528], 20.00th=[15168],
     | 30.00th=[15552], 40.00th=[15936], 50.00th=[16192], 60.00th=[16512],
     | 70.00th=[17024], 80.00th=[17792], 90.00th=[18816], 95.00th=[20352],
     | 99.00th=[22656], 99.50th=[23680], 99.90th=[27520], 99.95th=[31360],
     | 99.99th=[43776]
   bw (  KiB/s): min=99888, max=226752, per=100.00%, avg=197555.43, stdev=17473.05, samples=105
   iops        : min=24972, max=56688, avg=49388.86, stdev=4368.26, samples=105
  lat (usec)   : 20=94.18%, 50=5.82%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=17.38%, sys=79.70%, ctx=248524, majf=0, minf=17
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2618329,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=170MiB/s (179MB/s), 170MiB/s-170MiB/s (179MB/s-179MB/s), io=9.99GiB (10.7GB), run=60001-60001msec

Random reads = 35MB/s, or 280Mbps on the network (surprisingly bad for RAID1):

/Shared/tests # fio --name=fiotest --ioengine=sync --rw=randread --bs=4k --numjobs=1 --size=5G --runtime=1m --time_based
fiotest: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=35.0MiB/s][r=8967 IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=384: Mon Mar 24 18:30:31 2025
  read: IOPS=8528, BW=33.3MiB/s (34.9MB/s)(1999MiB/60001msec)
    clat (usec): min=86, max=10078, avg=108.36, stdev=27.63
     lat (usec): min=87, max=10078, avg=108.68, stdev=27.64
    clat percentiles (usec):
     |  1.00th=[   92],  5.00th=[  100], 10.00th=[  101], 20.00th=[  102],
     | 30.00th=[  102], 40.00th=[  103], 50.00th=[  103], 60.00th=[  106],
     | 70.00th=[  117], 80.00th=[  119], 90.00th=[  120], 95.00th=[  122],
     | 99.00th=[  125], 99.50th=[  126], 99.90th=[  133], 99.95th=[  176],
     | 99.99th=[ 1483]
   bw (  KiB/s): min= 7248, max=35968, per=100.00%, avg=35300.31, stdev=2654.96, samples=115
   iops        : min= 1812, max= 8992, avg=8825.08, stdev=663.74, samples=115
  lat (usec)   : 100=5.47%, 250=94.49%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=4.20%, sys=27.12%, ctx=511865, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=511745,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=33.3MiB/s (34.9MB/s), 33.3MiB/s-33.3MiB/s (34.9MB/s-34.9MB/s), io=1999MiB (2096MB), run=60001-60001msec

The following sequential tests match what I was seeing with Blackmagic Design's Disk Speed Test from my Mac over NFS to the RDS2216 directly.

Samba's better write results indicate it's caching inbound traffic, making writes appear faster, but then it stalls while waiting for syncs to complete. NFS more closely matches actual disk behavior and doesn't have the sync stall at the end of the test.

Sequential write = 223MB/s, or 1.784Gbps on the wire:

/Shared/tests # fio --name=fiotest --ioengine=sync --rw=write --bs=4k --numjobs=1 --size=5G --runtime=1m --time_based
fiotest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=234MiB/s][w=60.0k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=406: Mon Mar 24 18:35:09 2025
  write: IOPS=54.5k, BW=213MiB/s (223MB/s)(12.5GiB/60001msec); 0 zone resets
    clat (usec): min=12, max=994391, avg=13.79, stdev=551.02
     lat (usec): min=12, max=994392, avg=14.12, stdev=551.02
    clat percentiles (nsec):
     |  1.00th=[12736],  5.00th=[12736], 10.00th=[12864], 20.00th=[12864],
     | 30.00th=[12992], 40.00th=[12992], 50.00th=[12992], 60.00th=[13120],
     | 70.00th=[13248], 80.00th=[13376], 90.00th=[14144], 95.00th=[17024],
     | 99.00th=[17792], 99.50th=[18304], 99.90th=[20352], 99.95th=[21376],
     | 99.99th=[24960]
   bw (  KiB/s): min= 2960, max=254616, per=100.00%, avg=239736.44, stdev=42952.28, samples=108
   iops        : min=  740, max=63654, avg=59934.13, stdev=10738.07, samples=108
  lat (usec)   : 20=99.88%, 50=0.12%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%
  lat (msec)   : 4=0.01%, 100=0.01%, 1000=0.01%
  cpu          : usr=19.13%, sys=78.71%, ctx=104546, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3267120,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=213MiB/s (223MB/s), 213MiB/s-213MiB/s (223MB/s-223MB/s), io=12.5GiB (13.4GB), run=60001-60001msec

Sequential Read = 600MB/s, or 4.8Gbps on the network:

/Shared/tests # fio --name=fiotest --ioengine=sync --rw=read --bs=4k --numjobs=1 --size=5G --runtime=1m --time_based
fiotest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=732MiB/s][r=187k IOPS][eta 00m:00s] 
fiotest: (groupid=0, jobs=1): err= 0: pid=425: Mon Mar 24 18:36:44 2025
  read: IOPS=147k, BW=573MiB/s (600MB/s)(33.5GiB/60001msec)
    clat (nsec): min=1506, max=5270.1k, avg=4363.37, stdev=105224.35
     lat (nsec): min=1613, max=5270.2k, avg=4481.51, stdev=105224.40
    clat percentiles (nsec):
     |  1.00th=[   1800],  5.00th=[   1848], 10.00th=[   1864],
     | 20.00th=[   1912], 30.00th=[   1928], 40.00th=[   1944],
     | 50.00th=[   1976], 60.00th=[   2008], 70.00th=[   2024],
     | 80.00th=[   2096], 90.00th=[   2224], 95.00th=[   2320],
     | 99.00th=[   2608], 99.50th=[   3120], 99.90th=[  12352],
     | 99.95th=[  31104], 99.99th=[4751360]
   bw (  KiB/s): min=637600, max=770048, per=100.00%, avg=748540.22, stdev=27540.44, samples=93
   iops        : min=159400, max=192512, avg=187135.05, stdev=6885.13, samples=93
  lat (usec)   : 2=59.32%, 4=40.29%, 10=0.17%, 20=0.16%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.05%
  cpu          : usr=17.64%, sys=82.19%, ctx=246, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=8794147,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=573MiB/s (600MB/s), 573MiB/s-573MiB/s (600MB/s-600MB/s), io=33.5GiB (36.0GB), run=60001-60001msec

sirbryan · Mon Mar 24, 2025 9:43 pm

Same tests as previous post, but with a twist.

Container is running on Ampere machine running ROS7, talking to two NVMe drives that the 2216 has exported via NVMe over TCP. The Ampere box then puts them in a BTRFS RAID1 configuration and mounts the drives.

Random writes = 348MB/s, or 2.784Gbps on the network (exactly double the previous test... curious):

/Shared/fiotests # fio --name=fiotest --ioengine=sync --rw=randwrite --bs=4k --numjobs=1 --size=5G --runtime=1m --time_based
fiotest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
fiotest: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [w(1)][100.0%][eta 00m:00s]                         
fiotest: (groupid=0, jobs=1): err= 0: pid=307: Mon Mar 24 19:22:08 2025
  write: IOPS=84.9k, BW=332MiB/s (348MB/s)(20.0GiB/61733msec); 0 zone resets
    clat (usec): min=2, max=6470.0k, avg= 9.69, stdev=4865.35
     lat (usec): min=2, max=6470.0k, avg= 9.76, stdev=4865.35
    clat percentiles (nsec):
     |  1.00th=[ 3312],  5.00th=[ 3568], 10.00th=[ 3696], 20.00th=[ 3792],
     | 30.00th=[ 3920], 40.00th=[ 4016], 50.00th=[ 4128], 60.00th=[ 4192],
     | 70.00th=[ 4320], 80.00th=[ 4448], 90.00th=[ 4704], 95.00th=[ 5024],
     | 99.00th=[ 6368], 99.50th=[ 7200], 99.90th=[11072], 99.95th=[11584],
     | 99.99th=[14016]
   bw (  KiB/s): min=    8, max=884432, per=100.00%, avg=665762.54, stdev=299519.35, samples=63
   iops        : min=    2, max=221108, avg=166440.63, stdev=74879.84, samples=63
  lat (usec)   : 4=36.00%, 10=63.79%, 20=0.21%, 50=0.01%, 100=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : >=2000=0.01%
  cpu          : usr=6.78%, sys=46.42%, ctx=6386, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5242881,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=332MiB/s (348MB/s), 332MiB/s-332MiB/s (348MB/s-348MB/s), io=20.0GiB (21.5GB), run=61733-61733msec

Random reads = 23MB/s, or 184Mbps on the wire:

/Shared/fiotests # fio --name=fiotest --ioengine=sync --rw=randread --bs=4k --numjobs=1 --size=5G --runtime=1m --time_based
fiotest: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=20.4MiB/s][r=5227 IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=375: Mon Mar 24 19:32:25 2025
  read: IOPS=5674, BW=22.2MiB/s (23.2MB/s)(1330MiB/60001msec)
    clat (usec): min=128, max=3369, avg=171.98, stdev=35.87
     lat (usec): min=128, max=3369, avg=172.13, stdev=35.88
    clat percentiles (usec):
     |  1.00th=[  141],  5.00th=[  149], 10.00th=[  149], 20.00th=[  151],
     | 30.00th=[  157], 40.00th=[  165], 50.00th=[  167], 60.00th=[  176],
     | 70.00th=[  178], 80.00th=[  188], 90.00th=[  194], 95.00th=[  204],
     | 99.00th=[  277], 99.50th=[  318], 99.90th=[  367], 99.95th=[  392],
     | 99.99th=[ 1516]
   bw (  KiB/s): min= 5120, max=26320, per=100.00%, avg=22910.17, stdev=2371.42, samples=118
   iops        : min= 1280, max= 6580, avg=5727.54, stdev=592.86, samples=118
  lat (usec)   : 250=98.64%, 500=1.32%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.04%, 4=0.01%
  cpu          : usr=0.89%, sys=8.22%, ctx=340549, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=340496,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=22.2MiB/s (23.2MB/s), 22.2MiB/s-22.2MiB/s (23.2MB/s-23.2MB/s), io=1330MiB (1395MB), run=60001-60001msec

Sequential Write = 491MB/s, or 3928Mbps on net:

fio --name=fiotest --ioengine=sync --rw=write --bs=4k --numjobs=1 --size=5G --runtime=1m --time_based
fiotest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=430MiB/s][w=110k IOPS][eta 00m:00s] 
fiotest: (groupid=0, jobs=1): err= 0: pid=442: Mon Mar 24 19:38:23 2025
  write: IOPS=120k, BW=469MiB/s (491MB/s)(27.6GiB/60359msec); 0 zone resets
    clat (usec): min=2, max=549718, avg= 6.96, stdev=739.55
     lat (usec): min=2, max=549718, avg= 7.03, stdev=739.55
    clat percentiles (nsec):
     |  1.00th=[ 3216],  5.00th=[ 3376], 10.00th=[ 3536], 20.00th=[ 3696],
     | 30.00th=[ 3792], 40.00th=[ 3888], 50.00th=[ 4016], 60.00th=[ 4128],
     | 70.00th=[ 4256], 80.00th=[ 4512], 90.00th=[ 4704], 95.00th=[ 4960],
     | 99.00th=[ 6112], 99.50th=[ 6688], 99.90th=[ 9024], 99.95th=[10048],
     | 99.99th=[18560]
   bw (  KiB/s): min= 3224, max=909424, per=100.00%, avg=512688.71, stdev=187898.83, samples=113
   iops        : min=  806, max=227356, avg=128172.16, stdev=46974.77, samples=113
  lat (usec)   : 4=48.49%, 10=51.46%, 20=0.04%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (msec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (msec)   : 750=0.01%
  cpu          : usr=7.56%, sys=58.17%, ctx=1413, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7241729,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=469MiB/s (491MB/s), 469MiB/s-469MiB/s (491MB/s-491MB/s), io=27.6GiB (29.7GB), run=60359-60359msec

Sequentlal read = 579MB/s, or 4.632Gbps on the network:

/Shared/fiotests # fio --name=fiotest --ioengine=sync --rw=read --bs=4k --numjobs=1 --size=5G --runtime=1m --time_based
fiotest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=477MiB/s][r=122k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=509: Mon Mar 24 19:40:46 2025
  read: IOPS=141k, BW=552MiB/s (579MB/s)(32.3GiB/60001msec)
    clat (nsec): min=480, max=37171k, avg=5864.05, stdev=118007.43
     lat (nsec): min=520, max=37171k, avg=5938.61, stdev=118018.51
    clat percentiles (nsec):
     |  1.00th=[    524],  5.00th=[    524], 10.00th=[    564],
     | 20.00th=[    564], 30.00th=[    564], 40.00th=[    604],
     | 50.00th=[    604], 60.00th=[    604], 70.00th=[    604],
     | 80.00th=[    644], 90.00th=[    724], 95.00th=[    884],
     | 99.00th=[   3152], 99.50th=[  37120], 99.90th=[2768896],
     | 99.95th=[2899968], 99.99th=[3981312]
   bw (  KiB/s): min=48000, max=950928, per=100.00%, avg=580760.56, stdev=183028.69, samples=116
   iops        : min=12000, max=237732, avg=145190.10, stdev=45757.14, samples=116
  lat (nsec)   : 500=0.44%, 750=90.74%, 1000=4.50%
  lat (usec)   : 2=3.18%, 4=0.19%, 10=0.25%, 20=0.10%, 50=0.21%
  lat (usec)   : 100=0.10%, 250=0.06%, 500=0.02%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.05%, 4=0.12%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=5.07%, sys=22.61%, ctx=114176, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=8476501,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=552MiB/s (579MB/s), 552MiB/s-552MiB/s (579MB/s-579MB/s), io=32.3GiB (34.7GB), run=60001-60001msec

Interesting that writes improved almost by double, but reads remained the same.

This was actually an accident; I had hope to run the test on the Ampere's local drives, but it turns out the container had the remote drives mounted. I'll run tests with the same drives in the Ampere box locally.

sirbryan · Mon Mar 24, 2025 10:07 pm

FIO tests on Ampere ARM64 ROS 7.18.2 native, same Samsung drives as in RDS2216 in BTRFS RAID 1.

Random writes = 425MB/s, or 3.4Gbps on network:

/Shared/fiotests # fio --name=fiotest --ioengine=sync --rw=randwrite --bs=4k --numjobs=1 --size=5G --runtime=1m --time_based
fiotest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
fiotest: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=581MiB/s][w=149k IOPS][eta 00m:00s] 
fiotest: (groupid=0, jobs=1): err= 0: pid=75: Mon Mar 24 19:45:27 2025
  write: IOPS=104k, BW=405MiB/s (425MB/s)(23.7GiB/60001msec); 0 zone resets
    clat (usec): min=3, max=1920.2k, avg= 7.36, stdev=1410.73
     lat (usec): min=3, max=1920.2k, avg= 7.43, stdev=1410.73
    clat percentiles (nsec):
     |  1.00th=[ 4048],  5.00th=[ 4384], 10.00th=[ 4576], 20.00th=[ 4832],
     | 30.00th=[ 5024], 40.00th=[ 5216], 50.00th=[ 5472], 60.00th=[ 5664],
     | 70.00th=[ 6112], 80.00th=[ 6560], 90.00th=[ 7328], 95.00th=[ 8032],
     | 99.00th=[10176], 99.50th=[11584], 99.90th=[15808], 99.95th=[19328],
     | 99.99th=[34048]
   bw (  KiB/s): min=    8, max=763840, per=100.00%, avg=528835.61, stdev=210957.88, samples=93
   iops        : min=    2, max=190960, avg=132208.90, stdev=52739.47, samples=93
  lat (usec)   : 4=0.63%, 10=98.24%, 20=1.09%, 50=0.04%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
  lat (msec)   : 500=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
  cpu          : usr=8.49%, sys=72.24%, ctx=201540, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,6220993,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=405MiB/s (425MB/s), 405MiB/s-405MiB/s (425MB/s-425MB/s), io=23.7GiB (25.5GB), run=60001-60001msec

Random read = 41.2MB/s, or 329Mbps:

/Shared/fiotests # fio --name=fiotest --ioengine=sync --rw=randread --bs=4k --numjobs=1 --size=5G --runtime=1m --time_based
fiotest: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=39.8MiB/s][r=10.2k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=142: Mon Mar 24 19:47:56 2025
  read: IOPS=10.1k, BW=39.3MiB/s (41.2MB/s)(2357MiB/60001msec)
    clat (usec): min=75, max=320, avg=96.94, stdev=17.98
     lat (usec): min=75, max=321, avg=97.04, stdev=17.98
    clat percentiles (usec):
     |  1.00th=[   80],  5.00th=[   89], 10.00th=[   89], 20.00th=[   90],
     | 30.00th=[   90], 40.00th=[   91], 50.00th=[   91], 60.00th=[   92],
     | 70.00th=[  105], 80.00th=[  106], 90.00th=[  108], 95.00th=[  109],
     | 99.00th=[  131], 99.50th=[  255], 99.90th=[  293], 99.95th=[  302],
     | 99.99th=[  314]
   bw (  KiB/s): min= 9424, max=41128, per=100.00%, avg=40572.75, stdev=2893.22, samples=118
   iops        : min= 2356, max=10282, avg=10143.19, stdev=723.31, samples=118
  lat (usec)   : 100=66.00%, 250=33.44%, 500=0.56%
  cpu          : usr=1.21%, sys=11.20%, ctx=603539, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=603499,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=39.3MiB/s (41.2MB/s), 39.3MiB/s-39.3MiB/s (41.2MB/s-41.2MB/s), io=2357MiB (2472MB), run=60001-60001msec

Sequential write = 757MB/s, or 6Gbps:

/Shared/fiotests # fio --name=fiotest --ioengine=sync --rw=write --bs=4k --numjobs=1 --size=5G --runtime=1m --time_based
fiotest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=982MiB/s][w=251k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=209: Mon Mar 24 19:52:57 2025
  write: IOPS=185k, BW=721MiB/s (757MB/s)(42.3GiB/60001msec); 0 zone resets
    clat (usec): min=2, max=47866, avg= 3.63, stdev=14.39
     lat (usec): min=2, max=47866, avg= 3.70, stdev=14.39
    clat percentiles (nsec):
     |  1.00th=[ 2992],  5.00th=[ 3088], 10.00th=[ 3120], 20.00th=[ 3248],
     | 30.00th=[ 3312], 40.00th=[ 3440], 50.00th=[ 3600], 60.00th=[ 3696],
     | 70.00th=[ 3760], 80.00th=[ 3856], 90.00th=[ 4048], 95.00th=[ 4384],
     | 99.00th=[ 5600], 99.50th=[ 6368], 99.90th=[ 7904], 99.95th=[ 8160],
     | 99.99th=[ 9152]
   bw (  KiB/s): min=39648, max=1051880, per=100.00%, avg=885232.73, stdev=229704.93, samples=99
   iops        : min= 9912, max=262970, avg=221308.14, stdev=57426.23, samples=99
  lat (usec)   : 4=87.88%, 10=12.12%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (msec)   : 50=0.01%
  cpu          : usr=11.85%, sys=87.56%, ctx=93042, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,11082206,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=721MiB/s (757MB/s), 721MiB/s-721MiB/s (757MB/s-757MB/s), io=42.3GiB (45.4GB), run=60001-60001msec

Sequential read = 1546MB/s, or 12.4Gbps on network:

/Shared/fiotests # fio --name=fiotest --ioengine=sync --rw=read --bs=4k --numjobs=1 --size=5G --runtime=1m --time_based
fiotest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=1168MiB/s][r=299k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=276: Mon Mar 24 19:54:58 2025
  read: IOPS=377k, BW=1474MiB/s (1546MB/s)(86.4GiB/60002msec)
    clat (nsec): min=480, max=2754.5k, avg=1644.97, stdev=35804.10
     lat (nsec): min=520, max=2754.5k, avg=1695.82, stdev=35804.16
    clat percentiles (nsec):
     |  1.00th=[    524],  5.00th=[    564], 10.00th=[    564],
     | 20.00th=[    604], 30.00th=[    604], 40.00th=[    604],
     | 50.00th=[    604], 60.00th=[    644], 70.00th=[    644],
     | 80.00th=[    644], 90.00th=[    684], 95.00th=[    724],
     | 99.00th=[   1080], 99.50th=[   3696], 99.90th=[ 191488],
     | 99.95th=[ 493568], 99.99th=[1597440]
   bw (  MiB/s): min=   42, max= 1969, per=100.00%, avg=1635.02, stdev=552.67, samples=107
   iops        : min=10778, max=504266, avg=418564.39, stdev=141483.76, samples=107
  lat (nsec)   : 500=0.06%, 750=95.36%, 1000=3.33%
  lat (usec)   : 2=0.44%, 4=0.41%, 10=0.14%, 20=0.07%, 50=0.04%
  lat (usec)   : 100=0.02%, 250=0.04%, 500=0.03%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.05%, 4=0.01%
  cpu          : usr=17.03%, sys=74.23%, ctx=61584, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=22640469,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1474MiB/s (1546MB/s), 1474MiB/s-1474MiB/s (1546MB/s-1546MB/s), io=86.4GiB (92.7GB), run=60002-60002msec

These drives are rated at 1800MB/s read, 930MB/s write (sequential).

sirbryan · Tue Mar 25, 2025 2:36 am

Looking through those FIO results can be really daunting, but I figured I'd leave those posts there for the few individuals who like to scour data.

There are four Samsung drives in the RDS2216 and two in the Ampere box, and six Micron drives in the Ampere box on an LSI card. All are used enterprise drives. (I ran the tests on the SATA drives as a sanity check.)

Some things to note:
- These are tests with 4K-sized blocks, so the "Big numbers" aren't going to be there
- On the RDS2216, the drives show connected as PCIe 2x8 GT/s
- On the Ampere box, the drives show connected as PCIe 4x8 GT/s
- [Updated] The PM963 drives are rated for up to 2000MB/s read and 1200MB/s write, and 430K IOPs read, 40K IOPs write (4KB)
- It seems that exporting drives from the RDS2216 via NVMe mainly improves write throughput at the expense of random reads
- NVMe over TCP Sequential writes are burstier than the reads; test results are an average of the throughput in both directions

Here are some charts and data (all as pictures, sorry about that).

Screenshot 2025-03-24 at 6.33.29 PM.png

Screenshot 2025-03-24 at 6.33.20 PM.png

Screenshot 2025-03-24 at 6.57.50 PM.png

Screenshot 2025-03-24 at 6.33.12 PM.png

ConradPino · Tue Mar 25, 2025 5:02 am

@sirbryan my hero! Thank you!!

Tue Mar 25, 2025 10:00 am

Thank you for the effort, but about the SMB, something is wrong in your setup. Here is me just copying one file to the RDS SMB share, no tricks. Mostly limited by the MacBook network adapter, around 10Gbit at the moment.

Screenshot 2025-03-25 at 09.58.54.png

Tue Mar 25, 2025 1:17 pm

disk test provides similar functionality as dd utility in linux.
when testing locally CPU may be a bit capped due to generation of data (random or zeroes)
with 6 nvme disks (entry level U.2/U.3 drives Micron_7400 - 960GB):

/disk test nvme1,nvme2,nvme3,nvme4,nvme5,nvme6 block-size=4K direction=read thread-count=2 type=device
AVG TOT  54.9Gbps of disk traffic over few minutes

on small bursts speed can reach close to 70Gbps

close to same speeds can be achieved over network as well:

both 100Gbps interfaces are connected to x86 RouterOS or another RDS or multiple devices,
rise MTU, set up IP addresses (on both ends):

/interface/ethernet/set qsfp28-1-1 l2mtu=9570 mtu=9000 
/ip address add address=172.16.3.2/24

On RDS export disks as nvme-tcp

/disk
set nvme1 nvme-tcp-export=yes
set nvme2 nvme-tcp-export=yes
set nvme3 nvme-tcp-export=yes
set nvme4 nvme-tcp-export=yes
set nvme5 nvme-tcp-export=yes
set nvme6 nvme-tcp-export=yes

import them on other end:

/disk
add nvme-tcp-address=172.16.3.2 nvme-tcp-host-name=MikroTik nvme-tcp-name=nvme1  type=nvme-tcp
add nvme-tcp-address=172.16.3.2 nvme-tcp-host-name=MikroTik nvme-tcp-name=nvme2  type=nvme-tcp
add nvme-tcp-address=172.16.3.2 nvme-tcp-host-name=MikroTik nvme-tcp-name=nvme3  type=nvme-tcp
add nvme-tcp-address=172.16.3.2 nvme-tcp-host-name=MikroTik nvme-tcp-name=nvme4  type=nvme-tcp

run tests -

/disk test block-size=64K direction=read type=device disk=nvme-tcp-172-16-3-2-nvme1,nvme-tcp-172-16-3-2-nvme2,nvme-tcp-172-16-3-2-nvme3,nvme-tcp-172-16-3-2-nvme4,nvme-tcp-172-16-3-2-nvme5,nvme-tcp-172-16-3-2-nvme6 thread-count=4

speeds should be around 55Gbps or higher in shorter bursts

Larsa · Tue Mar 25, 2025 2:13 pm

Just a quick note on the short bursts: those are most likely hitting cached data rather than reflecting sustained transfer speeds from storage to user space.

The comparison to dd might be useful for Linux users, though both tools are fairly synthetic and don’t always reflect real-world I/O patterns. Also, your point about the CPU potentially being a bottleneck during data generation is actually very important—especially on the RDS2216, where CPU headroom can become a limiting factor when pushing high throughput with multiple threads or interfaces. It’s probably worth highlighting that more clearly in the docs for anyone running tests and wondering about unexpected limits.

The NVMe-TCP throughput is actually pretty impressive with the 100G interfaces and a performant x86 server. That said, real-world results will likely depend heavily on tuning, available CPU headroom and network conditions (e.g. mtu=9000). It might help to add a bit more context here, so others don’t assume they’ll see the same results on something like the RDS2216.

And lastly, just a gentle suggestion: the documentation as it stands is quite dense and assumes a fair bit of background ROS knowledge. For any organization that isn’t already deep into ROS or networking, it could really benefit from a clearer structure and more explanatory steps to make it broadly usable.

sirbryan · Tue Mar 25, 2025 2:28 pm

Thank you for the effort, but about the SMB, something is wrong in your setup.

Show me your config and I'll show you mine.

/ip smb users
add name=bscott
/ip smb
set enabled=yes
/ip smb shares
add directory=nvme-raid name=Shared valid-users=bscott

/disk
set nvme5 raid-master=nvme-raid raid-role=0
set nvme6 raid-master=nvme-raid raid-role=1
add media-interface=*2000000 raid-chunk-size=128K raid-device-count=2 raid-type=0 slot=nvme-raid type=raid

I copied 6GB over from my Mac to the share. It took 3-4 minutes. It finally surged to as much as 3Gbps at the end for a few seconds.

Screenshot 2025-03-25 at 6.12.25 AM.png

Mac Studio M1 Max, Sequoia 15.3.2, onboard Ethernet into MikroTik FTC11XG -> CRS354 2xSFP+ -> CRS326-24S+ MLAG pair 4xSFP+ -> RDS2216, all at Layer 2. MTU 1500, L2MTU on all switches set at 9000 or higher.

iperf3 tests between a container on the RDS2216 (232) and the Mac (24):

/ # iperf3 -c 192.168.3.24
Connecting to host 192.168.3.24, port 5201
[  5] local 192.168.3.232 port 36242 connected to 192.168.3.24 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.38 Gbits/sec   18   1.38 MBytes       
[  5]   1.00-2.00   sec  1.09 GBytes  9.32 Gbits/sec   58    850 KBytes       
[  5]   2.00-3.00   sec  1.06 GBytes  9.12 Gbits/sec  108   1.30 MBytes       
[  5]   3.00-4.00   sec  1.09 GBytes  9.39 Gbits/sec    4   1.16 MBytes       
[  5]   4.00-5.00   sec  1.09 GBytes  9.38 Gbits/sec   32   1.33 MBytes       
[  5]   5.00-6.00   sec  1.09 GBytes  9.37 Gbits/sec   21    925 KBytes       
[  5]   6.00-7.00   sec  1.09 GBytes  9.38 Gbits/sec   23   1.34 MBytes       
[  5]   7.00-8.00   sec  1.09 GBytes  9.37 Gbits/sec   18   1.28 MBytes       
[  5]   8.00-9.00   sec  1.09 GBytes  9.38 Gbits/sec   24   1.11 MBytes       
[  5]   9.00-10.00  sec  1.09 GBytes  9.37 Gbits/sec   49   1.36 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.35 Gbits/sec  355             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.35 Gbits/sec                  receiver

iperf Done.


/ # iperf3 -c 192.168.3.24 -R
Connecting to host 192.168.3.24, port 5201
Reverse mode, remote host 192.168.3.24 is sending
[  5] local 192.168.3.232 port 36250 connected to 192.168.3.24 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   746 MBytes  6.25 Gbits/sec                  
[  5]   1.00-2.00   sec   604 MBytes  5.07 Gbits/sec                  
[  5]   2.00-3.00   sec   854 MBytes  7.16 Gbits/sec                  
[  5]   3.00-4.00   sec   632 MBytes  5.30 Gbits/sec                  
[  5]   4.00-5.00   sec   906 MBytes  7.60 Gbits/sec                  
[  5]   5.00-6.00   sec   901 MBytes  7.56 Gbits/sec                  
[  5]   6.00-7.00   sec   634 MBytes  5.32 Gbits/sec                  
[  5]   7.00-8.00   sec   879 MBytes  7.37 Gbits/sec                  
[  5]   8.00-9.00   sec   659 MBytes  5.53 Gbits/sec                  
[  5]   9.00-10.00  sec   870 MBytes  7.30 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  7.51 GBytes  6.45 Gbits/sec                  sender
[  5]   0.00-10.00  sec  7.50 GBytes  6.45 Gbits/sec                  receiver

iperf Done.

sirbryan · Tue Mar 25, 2025 2:46 pm

disk test provides similar functionality as dd utility in linux.
when testing locally CPU may be a bit capped due to generation of data (random or zeroes)
with 6 nvme disks (entry level U.2/U.3 drives Micron_7400 - 960GB):

.....

speeds should be around 55Gbps or higher in shorter bursts

That's fine and all, but these disks are rated for much higher numbers and I'm not getting anywhere near that when running fio directly on the RDS2216 itself, where there should be just raw PCIe speeds on sequential reads and writes.

This is marketed as a machine that, among other things, can host containers that manage your files, like Nextcloud and other open source alternatives to online services. Yet, a very simple container (Alpine Linux with Samba, fio, and iperf3 on it) seems to have access to but a fraction of the throughput the drives offer.

Running your test on the two NVMe drives in a striped RAID showed promising results; how do we get the apps or other machines on the network to benefit?

[admin@rds2216-office] /disk> test nvme-raid block-size=4K direction=read thread-count=2 type=device    
Flags: R - RUNNING
Columns: SEQ, RATE, IOPS, BYTES, DISK, THREAD, TYPE, PATTERN, DIR, BSIZE, STATE
  SEQ  RATE         IOPS  BYTES      DISK       THREAD  TYPE    PATTERN     DIR   BSIZE  STATE
R TOT  9.2Gbps   282 511  171.4GiB   nvme-raid  0       device  sequential  read   4096  run  
R TOT  9.4Gbps   287 560  174.4GiB   nvme-raid  1       device  sequential  read   4096  run  
  TOT  18.6Gbps  570 072  345.8GiB   nvme-raid  TOT     device  sequential  read   4096

Tue Mar 25, 2025 3:12 pm

Just a quick note on the short bursts: those are most likely hitting cached data rather than reflecting sustained transfer speeds from storage to user space.

/disk test utility clears caches before running tests.

The NVMe-TCP throughput is actually pretty impressive with the 100G interfaces and a performant x86 server. That said, real-world results will likely depend heavily on tuning, available CPU headroom and network conditions (e.g. mtu=9000). It might help to add a bit more context here, so others don’t assume they’ll see the same results on something like the RDS2216.

Larger MUT is to lower CPU usage for a bit and in direct device to device connection there is no reason not to increase MTU, for 50Gbps+ transfers over nvme-tcp CPU use will be around 30% with larger MTUs and around 25-40 with 1500, so there is enough headroom for background router tasks.

sirbryan · Tue Mar 25, 2025 3:12 pm

I ran a disk test from the Ampere ROS 7 to the RDS2216, with two disks exported. The results are exactly half of what I get when running it locally.

[admin@rocket80-ros] /disk> test block-size=4K direction=read thread-count=2 type=device tcp-raid-1       
Flags: R - RUNNING
Columns: SEQ, RATE, IOPS, BYTES, DISK, THREAD, TYPE, PATTERN, DIR, BSIZE, STATE
  SEQ  RATE         IOPS  BYTES      DISK        THREAD  TYPE    PATTERN     DIR   BSIZE  STATE
R TOT  4.7Gbps   144 506  86.5GiB    tcp-raid-1  0       device  sequential  read   4096  run  
R TOT  4.7Gbps   144 773  86.7GiB    tcp-raid-1  1       device  sequential  read   4096  run  
  TOT  9.4Gbps   289 279  173.3GiB   tcp-raid-1  TOT     device  sequential  read   4096

Tue Mar 25, 2025 3:36 pm

I ran a disk test from the Ampere ROS 7 to the RDS2216, with two disks exported. The results are exactly half of what I get when running it locally.

Increase thread-count to give device more parallel streams, each stream is single core bond.

i just tried testing from container to store - with 6 disks in raid and bare alpine image with 20 parallel dd streams:
it wrote 196GB in 1m 20.02s.
and reads 196GB in 34.75s

Amm0 · Tue Mar 25, 2025 3:59 pm

Has anyone tried packet sniffer to [local] disk with a high network load on the RDS?

i.e. could you use it as a switch that had a rolling buffer of packet traces

Larsa · Tue Mar 25, 2025 4:12 pm

/disk test utility clears caches before running tests.

Just a quick follow-up on the cache clearing part: even if /disk test flushes some caches before the run, that doesn't necessarily eliminate caching effects—especially in short burst tests where you're still likely to hit RAM-level buffers or controller cache.

And when using NVMe-TCP, I’m curious how cache clearing works on the remote/exporting side. Is it coordinated across both ends? Because unless you're bypassing all caching layers, including hardware controller cache, some of those numbers will still be optimistic.

To be fair, the throughput looks great, but without more clarity on what exactly is being measured (device vs. network vs. user space performance), it’s tricky to interpret results with confidence. Would be awesome if the docs explained a bit more about the mechanics behind /disk test, especially in remote scenarios.

Right now it feels a bit like a black box—and I think many of us would appreciate some insight into what’s really going on under the hood. 🙂

sirbryan · Tue Mar 25, 2025 6:56 pm

OK, I've been doing some math.

If these PM963 drives are capable of up to 1800-2000MB/s (14-16Gbps) with 4 PCIe lanes, that means that I should see roughly 7-8Gbps in straight reads from one drive with 2 lanes on the RDS2216.

The disk test to a single drive on Ampere (4xPCIe):

[admin@rocket80-ros] /disk> test block-size=4K direction=read thread-count=1 type=device nvme4 
  SEQ  RATE         IOPS  BYTES      DISK   THREAD  TYPE    PATTERN     DIR   BSIZE  STATE
R TOT  24.1Gbps  738 172  1289.7GiB  nvme4       0  device  sequential  read   4096  run

Same disk test to a single drive on RDS2216 (2xPCIe):

[admin@rds2216-office] /disk> test block-size=4K direction=read thread-count=1 type=device nvme1 
R TOT  11.4Gbps  349 301  704.9GiB   nvme1       0  device  sequential  read   4096  run

Why does ROS's test show 160-170% of rated read speeds?

CGGXANNX · Tue Mar 25, 2025 7:14 pm

PM963 are PCIe 3.0 x4. With two lanes data transfer at about 2 GB/s is still possible. See: https://en.wikipedia.org/wiki/PCI_Expre ... ison_table

Tue Mar 25, 2025 7:18 pm

If these PM963 drives are capable of up to 1800-2000MB/s (14-16Gbps) with 4 PCIe lanes, that means that I should see roughly 7-8Gbps in straight reads from one drive with 2 lanes on the RDS2216.

You should be able to get 12-16Gbps on RDS - PCIe connection is just to NVMe controller, not memory chip - it does not mean that its automatically half speed.

on Ampere observe SEQ, it seems that there is some overflow happening that has messed TOT math.
edit: not sure why are you getting such good results on ampere, but test seems to be running correctly, is it same 960GB disk? larger ones usually have much better performance.
1.92TB disk usually has double performance than 960GB counterpart.

sirbryan · Tue Mar 25, 2025 11:22 pm

edit: not sure why are you getting such good results on ampere, but test seems to be running correctly, is it same 960GB disk? larger ones usually have much better performance.
1.92TB disk usually has double performance than 960GB counterpart.

I bought 8 960's. Two are in the Ampere and six in the RDS.

Wed Mar 26, 2025 9:30 am

there is not much to configure for SMB:

/ip smb users
add name=normis
/ip smb
set enabled=yes
/ip smb shares
set [ find default=yes ] valid-users=guest
add directory=raid1 name=shrek valid-users=normis

/disk print
23  BM   raid1      raid1        RAID6 2-parity-disks

make sure your macOS computer has correct MTU for the adapter you are using, I know macOS has a bug where you can't set MTU above 8000 and if your adapter says it needs 9000, it will glitch in various ways. I have a Sonnettech Twin25 and have set MTU to 8000

Wed Mar 26, 2025 10:17 am

I bought 8 960's. Two are in the Ampere and six in the RDS.

i have same hardware, i will check whats going on, on RDS test seems to be correct and provides expected results.

Amm0 · Wed Mar 26, 2025 12:05 pm

there is not much to configure for SMB:
[...]
I know macOS has a bug where you can't set MTU above 8000 and if your adapter says it needs 9000, it will glitch in various ways. I have a Sonnettech Twin25 and have set MTU to 8000

Where might one find that information in your documentation? If it's bug in macOS, do you have an open case with Apple about the issue?

Wed Mar 26, 2025 12:16 pm

I think this bug applies to all users and Apple is aware for some 10 years, it's not related to MikroTik in any wya. There are many reports of MTU setting not staying as set, but i'm not sure if that's even related to this. Just experiment with MTU size and lower it slightly

sirbryan · Wed Mar 26, 2025 2:40 pm

make sure your macOS computer has correct MTU for the adapter you are using, I know macOS has a bug where you can't set MTU above 8000 and if your adapter says it needs 9000, it will glitch in various ways. I have a Sonnettech Twin25 and have set MTU to 8000

MTU is 1500. But that doesn't appear to be the problem in this case.

I tested the built-in SMB server on the RDS2216 and Ampere box, both running 7.18.2. It was awful (10-20Mbps). So on a hunch I tested it on my CCR2116 (7.16.2) with a 4TB NVMe drive. Writes were ~250MB/s, reads about 600-700MB/s, meaning it works fine on that version.

So I downgraded the Ampere box to 7.16.2, and I get 500-600MB/s writes to the Samsung NVMe drive RAID 0 array and 940MB/s reads (saturating the 10G interface).

I downgraded the RDS2216 to 7.17.2 and now I'm getting 245MB/s writes to a RAID0 NVMe (same setup as Ampere) and 900MB/s reads. Writes are similar to the CCR2116, reads are slightly better. I attribute that to them having the same CPU. The Ampere is getting double the write throughput though, and could probably do more on reads with a 25Gbps adapter on the Mac.

So something seems to be wrong with stock SMB in 7.18.2.

toxicfusion · Wed Mar 26, 2025 7:38 pm

Can you instead test NFS performance over SMB. At times, Windows SMB has its own performance limitations.

The issue is with the RAID creation and management in the current form [relying on CLI]. This is all non-starter. MikroTik still needs to develop a proper tool stack. Or they get their head out of the sand, get off their EGO and deploy container of already production ready software we can deploy on the hardware. Why reinvent the wheel?

Unless MikroTik puts heavy investment into R&D, testing and Software QA -- this is DOA for another 2+ years while we wait for software and tools to mature.

We need better Software QA from MikroTik.....

Also, bullshit on MacOS jumbo frames. I myself am using it on an M1 Mac Mini with 10Gbe NIC -- serving as my Plex Server, also running OpenZFS with a OWC ThunderBay 8+ enclosure attachced via USB-C. This is my file server/NAS........

JumboFrames.
https://forums.macrumors.com/threads/ho ... 1.2370303/

sirbryan · Wed Mar 26, 2025 9:56 pm

Can you instead test NFS performance over SMB. At times, Windows SMB has its own performance limitations.

I tested it earlier and I believe I posted some results. It works similarly (i.e. works fine), depending on which machine I'm connected to and which set of drives I'm testing.

grusu · Thu Mar 27, 2025 10:02 am

Hi,
I have a m.2 SATA SSD left over from a laptop that no longer works and I tried to install it in the RB1100AHx4 Dude Edition router. I encountered the following problem: I can't format it because the fileman service is trying to index the files in the former windows partition which are in the order of hundreds of thousands. When I try to format the SSD it tells me that there are files opened by fileman and the processor has a load of up to 75%. I don't have any device left in which to put the m.2 sata SSD. Is there any solution?

Thanks,
Geo

Thu Mar 27, 2025 10:52 am

Make sure your Winbox FIles menu is not open. You can reboot the router, do not open Winbox at all, log into SSH and then format.
This is a known issue and fix is coming.

grusu · Thu Mar 27, 2025 11:22 am

I tried this option too but it doesn't work.
I also tried running a script at startup to eject or umont without success.

Thu Mar 27, 2025 12:02 pm

but what happens when you try to format the disk from command line, after reboot, when winbox is not open? can you post the output of your commands?

grusu · Thu Mar 27, 2025 12:07 pm

I get the same error, meaning there are files opened by fileman service. RouterOS version: 7.18.2

toxicfusion · Thu Mar 27, 2025 5:23 pm

Why doesnt MikroTik investigate creating a partnership with the folks at "unRAID" and work to create a solution...... Deploy "unRAID" as a container or managing overlay for the RDS storage system. Be ALOT less work then re-inventing the wheel for the required tool stack. Are you listening, MikroTik? Or are you dead-set on creating your own. learning as you go? Not the right approach....

https://unraid.net/community/apps

Another forum member has also mentioned this. Create options for two(2) different installs. "unRAID" for those home-lab niche users, and perhaps look into a way to deploy 'TrueNAS SCALE". Gives us options and flexibility. This would put your storage appliance hardware right in-line with other storage vendors in this market space [45Drives. IXSystems]

Also -- unRAID has ALOT of docker support for apps. This would be a sweet home-lab win for the RDS appliance.

Ability to run iSpy / Agent-DVR docker, and store all your camera NVR recordings right onto the RDS storage appliance storage pool/volume..... Or run PlexMedia server in a container, and it have direct access to the storage pool.

Open your brains :)

itimo01 · Thu Mar 27, 2025 5:33 pm

Why doesnt MikroTik investigate creating a partnership with the folks at "unRAID" and work to create a solution...... Deploy "unRAID" as a container or managing overlay

Unraid is a subscription, which would go against the "you buy it, you own it" way of mikrotik.

toxicfusion · Thu Mar 27, 2025 6:10 pm

"You buy it, you own it" hardware. Let the consumer decide for add-on subscription choices. Otherwise, MikroTik needs ALOT of R&D for storage. Unless they've recently hired on some storage engineers to help aid them with this. Highly doubt it.

Anyways, I'll rant and get off topic on what their focus should be for innovation. They're going to put bad taste in mouths of consumers, leaving their flagship network hardware waiting for supporting software and features. Other companies are full steam ahead with continued innovation and adding new features.....

sirbryan · Fri Mar 28, 2025 6:59 am

TrueNAS doesn't have an ARM64 variant. Openmediavault might be an option, but at the moment, you can't pass the raw disks through, so at best you'd be giving a container an already assembled array, which defeats the purpose of using one of those solutions as it is.

The door is open, however, for someone to build a "pretty" container that leverages the MikroTik API to manage disks and arrays, taking some of the complexity and rawness away. I myself am considering doing just that. I wouldn't be surprised if MikroTik hasn't also considered that approach.

toxicfusion · Sat Mar 29, 2025 5:32 am

The only other possibility is MikroTik developing a dedicated .npk addon package that will bring OpenZFS. There is one for ARM64, or well - atleast for Apple Silicon (I’m running M1 mini with OpenZFS with Thunder Bay 8 enclosure attached). They can work with the OpenZFS developers - or reach out to Lundman.

They’d still need to develop a full tool stack to bolt on and integrate into Winbox. At that point, Winbox will become more than just a routerOS management utility….. it will feel bloated and no cohesive. People deploying MikroTik routers in enterprise production or customer facing as CE’s may feel awkward seeing all the storage abilities on their routers that are suppose to…..route…. Or only becomes visible with the ROSE .npk loaded

Wed Apr 02, 2025 1:44 am

I'm still old school where a router is a router and a switch is a switch and storage is storage and a nas is a nas ...

I've run 24+ TrueNAS servers for a very long time now.
On my faster TrueNAS servers , I can hit speeds of almost 100-Gig writes to my NFS TrueNAS system when many machines are writing at the same time - as in full-nightly backups of everything everywhere from more than a dozen plus Proxmox servers & Veeam ( all NFS mounts ).
* Notes: 100-Gig network cards & 1-TB of ram and solid-state disks ( some are also SAS ) & normally 1/2 Peta-Bytes of storage on each NAS.
* After my 1-TiB ZFS write/read cache fills up , then I drop down do a slow 30-to-40 Gig of sustained write speeds to my NAS server(s) while everything is backing up to my TrueNAS system(s).

I've never had a problem in 10+ years with my TrueNAS servers. When a physical disk dies in my ZFS-3 LZ4 file system , the system continues just fine, and I've have always been able to replace physical drives in real-time and re-silver in real-time.

IMO , Mikrotik might have a great new storage system - but I prefer industry standard proven hardware with Xeon CPUs that can easily run a better NAS Operating System if/when something better comes out.
So , when Mikrotik has a stable, popular and easily expandable storage server that can keep up with my I/O demands , then at that time I may try a Mikrotik storage solution at that time.

IMO - just a bit of advice.
Always get at least four storage systems. - Two for local redundancy and two more where each of the remaining two are at remote sites. - I use multiple remote locations.
And always use at least two or three different types of backups of your data ( such as Veeam and Rsync full backups and SnapShots ). If one backup method fails ( or failed and was not working for a long time (month or more) , then you have your second and third types of backup methods to restore all of your data from.
And - very important here - Your NAS network should never be fully exposed to workstations. If/when there is a full network encryption ransom attack , your backup storage systems are not reachable , because your data/server backups are off-line at all times ( air-gap ) and not part of your normal workstation/server network. ( no single workstation or server or network should have access to all of your backup systems ).

Wed Apr 02, 2025 9:04 am

I'm still old school where a router is a router and a switch is a switch and storage is storage and a nas is a nas ...

It's always good to listen to a wise and experienced person.

If your functionality is spread across multiple devices and one stops working, the rest will still work. No nightmares and pissed off users.

sszbv · Wed Apr 02, 2025 4:20 pm

I'd like to add some of my experiences:

On x86 hardware with RouterOS, the SMB sharing works very good.
Setting up disks, raid, filesystems, shares and users is easy.
Performance is great.

Drawbacks:

With millions of files, the 'Files' function in winbox does not work well.
There is no way to locally copy files from one place to another.

Workaround:

Create a container with alpine linux, create a mount for each filesystem (a mount to / doesn't give you the right permissions, so don't use that).
Now you can use the container to manage files.
Copying files is very fast, no problems at all.

I like it!

Now going to test what happens when disks fail, I've got some bad disks for that purpose.

Amm0 · Wed Apr 02, 2025 10:26 pm

If your functionality is spread across multiple devices and one stops working, the rest will still work. No nightmares and pissed off users.

Well... for the RDS, one random idea be to have some small-form versions of the CCR2004-1G-2XS-PCIe full-sized card (perhaps different specs / switch / CPU / etc offerings), to be one of RDS "disks" on the PCIe bus. i.e. RDS acts a "chassis" for a bunch of individual router (or switch) "line cards".