Hi all,

I have 2 x CRS317 switches connected with 10gbps BiDi sfp (10km). Rx Signal on Switch A is -18dbm, Rx on Switch B is -20dbm. Tx on both is around -1dbm.

When I do a bandwidth test From SwA to SwB, download speed is normal eg. 380Mbps (due to cpu), but upload is around 1Mbps. The inverse is the same when doing test from SwB. Strange thing is, when doing trest using udp, both sides speed test fine up and down. Using tcp, is when the issue crops up.

Has anyone ever experienced this or maybe can assist with the issue?

Ps. When forcing link to 1Gbps, all test run normally.

Thanks in advance.

I'm guessing you're performing tests using iperf ... so are there any significant number of dropped packets shown (receiver's stats is relevant)? When performing UDP tests these are normal if sender emits traffic at rate higher than the bottleneck (in your case that might be any of switches due to problems on SFP link), but with TCP tests dropped packets should be none if the only bottleneck is low throughput on some link (TCP can throttle down Tx rate to what receiver seems to be capable of handling). However dropped packets when running UDP test with throughput lower than what test devices (and tested devices) are capable of indicate problems on physical layer (e.g. faulty cables).

Known problem when there's change of link speeds between different L2 segments (e.g. between switches) is that on the switch between faster and slower link some buffering might happen because some frames can arrive via faster link and can not leave via slower link. Normally switches buffer those, but buffering is limited to a few frames to avoid buffer bloat. So it can happen that switch drops a frame now and then (which affects TCP throughput big time because some packets have to be retransmitted and that means introduction of large delay while colapsing speed). In theory ethernet frow control (for both Tx and Rx) should help because switch which starts to buffer data can indicate partner who is delivering buffered data to temporarily stop transmitting (and that flow control can eventually pause original transmitter). Since this happens on L2 and frequently, things are supposed to improve significantly. However flow control implementation is sometimes less than ideal (and has some conceptual problems as well), thus results are sometimes even worse than running network with flow control disabled. And MT seems to be on the worse side with this regard.

[edit] Note: running speedtest on the ROS devices them selves is prone to give weird (if not invalid) results because speedtest process is very CPU intensive and can falsely show problems which don't exist in reality. So if you're running speedtest on both CRSes, try to perform tests using iperf on connected PCs to verify you get comparable results.

Thanks mkx,

Will do a few test and report back.