MikroTik

Hi,

I got my workstation (with Mellanox Connectx-2) connected per DAC to the CSS326. WIthin SWOS it tells me "10G", but if I try to copy something to the servers (1Gbit) I only get 30-180mbit traffic speed. If I connect the workstation with a 1Gbit cable instead, I get around 980 mbit/s. If I connect the Workstation directly to one of the servers per DAC, then I get the full 10gbit link. What's the problem?

Edit: I made some tests:
It doesn't matter which Server/PC is using the Mellanox as sending-device (I tested both cards): it will only be between 20-400mbit/s, depending on the OS (linux is faster than windows in sending). Receiving is about 980mbit/s (full Gbit; I only have one DAC right now).
I found someone in a different forum who has the same problem with 2* Mellanox Connectx-2 and the CSS326 with DAC.

edit: downgrade to 2.3 solved the problem

Thanks for sharing your experiences.

I'm also suffering some problems with CSS326 and SwOS 2.5. Downgrade OS to solve problem? That sounds wired. I was expecting SwOS are rock solid like RouterOS, but seems I'm totally wrong.

Seems Mikrotik official not care much about SwOS. Is CSS326 ready for production usage? What OS version should I use?

Yes, I'm not the only one with SFP-> 1Gbit transfer problems. Downgrading solved the problem. 2.3 and 2.4 are running fine. So far it is a very good switch!

same Problem here
please look my Performancepicture
Downgrade to 2.4 or 2.3 fix this Problem.
https://abload.de/img/mikrotikqbu3c.png

Same problem here as well.
While TCP traffic between 1G <-> 10G is reasonably Ok - UDP traffic has huge losses and out-of-order issues.
Streaming in LAN (NVIDIA GameStream) is unusable with SwOS 2.5 and 2.6.

Downgraded to 2.4 and everything worked smoothly.

(I am also using Mellanox Connectx-2)

I just received an answer from Mikrotik support that the 10G<->1G performance issue will be fixed in the next release (current is 2.6).

When will 2.7 be released? Also the fan problem on CRS-317 has to be fixed asap

Please can someone help me on how to use SwOS/CSS326 sfp as trunk. The switch is running on a v2.7

Please can someone help me on how to use SwOS/CSS326 sfp as trunk. The switch is running on a v2.7

Maybe this example will help you: vlan configuration example !

Hi,

We are using two CSS326-24G-2S+ with both firmware 2.7 and we can't get above 170mbps any ideas?
Traffic path:
Server -> switch -> CSS326-24G-2S+ -> SFP+ -> SPF+ -> CSS326-24G-2S+ -> Switch - > Server

Kind regards,

Marcel

We’ve recently purchased two CSS326-24G-2S+ switches.
They have a 10gbps connection between the two switches / sites. (two datacenter sites)
The CSS326-24G-2S+ switches are connected by 1gbps coper to our switch infrastructure in either site.
When we generate traffic from one site through the other site so:

Server -> switch -> CSS326-24G-2S+ -> SFP+ -> SPF+ -> CSS326-24G-2S+ -> Switch - > Server

We get stuck at 170mbps.
This is very strange.
What do we need to do to be able to get around 1gbps of traffic through from 1 site to the second site?

Yes, just noticed the same bad SFP+ -> 1GB performance bottleneck... SMB performance from Server 2016 -> Windows 10 even had as low as 600 KB/s from the 10GB-Link.

It's also on current 2.7 firmware. Downgraded to 2.3, speed returned to normal.

Yes, just noticed the same bad SFP+ -> 1GB performance bottleneck... SMB performance from Server 2016 -> Windows 10 even had as low as 600 KB/s from the 10GB-Link.

It's also on current 2.7 firmware. Downgraded to 2.3, speed returned to normal.

I sometimes see the same thing, but it's intermittent for me. Sometimes the speeds are just fine.
MikroTik, let me know if you need log files etc. to help your investigations.

Hello, since v2.7 we have had difficulties in repeating your reported performance issues on CSS326.
Do all of you experience performance problems only with SMB transfers on CSS326 with SwOS 2.7?
Does any other bandwidth testing method also give the same bad results?

For us it's usually SMB, we have had to return to 2.3.
We thought 2.7 had fixed the issues of 2.5 and 2.6, but 2.7 drops to back crawling without any rhyme or reason, so we had to go back to 2.3.

For us it's usually SMB, we have had to return to 2.3.

I can confirm the performance problem since 2.5. We read half the speed from a 10GB port than we can write. E.g. reading is about 50MByte/s, writing is > 100MByte/s.
This is independend on the protocol. Appears on SMB, FTP, SFTP.

According to Mikrotik support the buffer handling changed since 2.3/2.4.
SwitchOS 2.3 is not the solution as it brings the switch into an endless reboot loop if 10G DAC are connected while boot.

Unfortunately the 317/326 switches are not applicable for us because of the performance reduction.

Could the performance issues be caused by bursty traffic and too small buffers in the switch? So the traffic is below 1 Gb/s on average, but comes in short 10 Gb/s bursts that overflow a too small buffer resulting in packet loss? The issue is not theoretical, I've seen it already a few times in cases like traffic enters a gigabit switch port and leaves through a 100 Mb port (or a 300 Mb microwave link). Software queues in RouterOS are usually large enough, but hardware (such as a switch chip) is limited, and higher speeds need larger buffers (but not too large - bufferbloat is bad too). Rule of thumb for the best throughput of a TCP flow is the RTT multiplied by the data rate. So the issue gets worse when traffic comes over the Internet and/or over wireless networks (often much longer RTT compared to a wired LAN). What is the hardware buffer size in CRS/CSS 3xx series switches? CPU in a CRS is too weak to handle much traffic, so larger queues possible in RouterOS don't help. This is important, the switch buffer size should be known and also be tunable. Good example of (more expensive) hardware that got it right is NEC iPasolink microwave links - default queue of 64 KB is too small for a 300 Mb/s link, but can be increased to 1024 KB which works much better when moving traffic at such speeds from the Internet to a few hundreds of customers on my network (mostly wireless).

Hello, cisco 4948 - S+DA0003 cable - CRS317-1G-16S+RM switch - 10G sfp link - CRS317-1G-16S+RM

Since we upgraded the connection between the CRS317 switches to 10G, any of the customers connected to any of the CRS317-1G-16S+RM switches would have 10-12-13 Mbps download speed and 500 Mbps-900 Mbps upload speed (depending of the speedtest server). I've noticed that in a speedtest, the further (geographically) the test server, the more that the speed on download degrades. When the speedtest server is closer (200 Km away for example), the download speed is 100 Mbps vs 850 Mbps normally, and further more, at 2000 km away, is 12 Mbps vs 750 Mbps. If we make a speedtest in the US, the speed would degrade even more, would be for example 5 Mbps instead of 500 Mbps. I don't understand how could this happen and how the distance could determine the speed of a connection through such a switch. I attached some examples.

Stupid question: where can i find the 2.3 swos firmware? On the Mikrotik website there is only an archive for routeros versions, not for the swos. And google finds only mikrotik's website which has no archive for older swos

I don't understand how could this happen and how the distance could determine the speed of a connection through such a switch. I attached some examples.

Stupid question: where can i find the 2.3 swos firmware? On the Mikrotik website there is only an archive for routeros versions, not for the swos. And google finds only mikrotik's website which has no archive for older swos

Speed through Internet depends on minimum available peering bandwidth along the path. You should never test a switch speed like that.

Go to the officail MT download page: https://mikrotik.com/download, copy the link of the latest version for the device you are interested in and edit it to reach a former version.
Eg: https://download2.mikrotik.com/swos2/cs ... 26-2.7.bin -> https://download2.mikrotik.com/swos2/cs ... 26-2.3.bin

Can confirm anything after SwOS 2.4 does not work well at all when receiving data on a 10G port and sending it out on a 1G port.
There is significant packet re-ordering, using default settings, both with and without flow control enabled link partners. I don't expect all that much at this price point but delivering packets in order is a basic requirement for a switch.

on 1G port: iperf3 -s
on 10G port: iperf3 -c <IP of 1G client> -i 1

If you do a packet capture the packets are all jumbled up (very visible in wireshark)

Note: the actual performance impact of packet re-ordering varies between operating systems, configurations and RTT of the TCP connection. And some benchmarks may not "see it" if they use a ton of paralell TCP connections.

Hello, since v2.7 we have had difficulties in repeating your reported performance issues on CSS326.
Do all of you experience performance problems only with SMB transfers on CSS326 with SwOS 2.7?
Does any other bandwidth testing method also give the same bad results?

Two new CRS328-24P-4S+RM on SwOS 2.7 with a pair of S+85DLC03D, no matter the SFP port (tried all combinations) only gets, consistently 170Mbit. Unfortunately I can't downgrade to 2.4 as the CRS328 ships with 2.7.

The fact that a downgrade seems to fix this speaks to a software issue, and I really hope the issue magically fixes itself via an upgrade one day.

I'm suffering the same problem on CRS326 with RouterOS even with v6.43rc.

mikrotik switches are disasters, It has been almost a year, no fix, no nothing.

Not sure if that could help someone, but reading the spec sheet of the Marwell, the 10Gb ports are rated as uplink for stacking or cascading providing up to 48 non blocking ports. So I think it was designed to let pass 10x 1Gb without much issue but not to provide decent downscaling support from 10Gb to 1Gb. To improve slightly this you might have to adjust the tcp windows on both client/server. Now I am still puzzle why people connected at 10Gb all the way get poor performance as well. Is there some jittering, poor SFP+, poor cabling ? Does it work any better with competitor switches ?

I just upgraded my pair of CRS328-24P-4S+RM to SwOS 2.8 and retried my pair of S+85DLC03D 10G multi-mode fiber links. I didn't bother with a throughput test this time because I was seeing ~80% packet loss instead D:

I decided to scrounge up some 1G mm fiber transceivers to see what happened and they worked flawlessly. No packet loss, ~950Mbit throughput. I swapped the 1G for the 10G back and forth a few times and consistently saw packet loss on the 10G, flawless on the 1Gs. The 1Gs were just some FiberLux generic transceivers.

Something to note about the 10G to 1G thinking. All of my testing is for simply gigabit speeds. I only have one 10G client, but I test from it and other 1G clients. I am successfully running with a single S+RJ10 (copper 10G) link across my pair of CRS328's without any throughput issues from any of these clients. When I try to use fiber, SwOS 2.7 was ~170Mbit, but now SwOS2.8 is insane packet loss.

The saga continues. Based on the change in behavior between software revs, I lean toward these issues with the S+85DLC03D transceivers being in software.

Upgrading to v2.9 has corrected my issues with the S+85DLC03D 10G multi-mode fiber transceivers. I'm no longer seeing insane packet loss, nor am I seeing terrible throughput. I now see the same throughput on the 10G fiber link as a 10G copper link; note that I still lack 10G clients on both ends, but I can at least max out a 1G link.