CCR2004-1G-12S+2XS SFP+ Upload issues

FinlayDaG33k · Tue Jul 06, 2021 8:52 pm

Hii there,

I just got in my CCR2004-1G-12S+2XS to replace my CRS326-12G-12S+RM as a router (and make it just a switch) and went to hook it up.
Once I was done, however, I ran into a pretty big issue.
Any direct uplinks (to server and my desktop) made through my SFP+ are rather bad.

This is the upload from my desktop to speedtest using SFP+:

Oddly enough, when I disable my SFP+ port on my desktop and use the gigabit port to my switch that is *also* hooked up using SFP+, everything is just fine.

Additionally, when I try to upload a file to my NAS over SFP+, it can barely outperform a gigabit connection to that same server (and it isn't the server nor desktop, because previously, it could hit 10Gbit transfers just fine).

This is how I hooked up my gear:
- CCR2004 ---SFP+---> Desktop
- CCR2004 ---SFP+---> Server
- CCR2004 ---SFP+---> CRS326
- CRS326 ---UTP-----> Desktop

Everything else is the same really...
The same modules and fibre cables were used in conjuction with my CRS326 (I basically just took them out, put them in the CCR2004 and connected the cables back up again) and everything worked perfectly fine on that.

FinlayDaG33k · Wed Jul 07, 2021 12:27 pm

So I've done some deeper diving and found that the 10Gbit speed to my server is due to the thing lacking the switching performance to do this.
Though it doesn't explain why I can't hit my full upload to speedtest while with a switch between it I can :|

StubArea51 · Wed Jul 07, 2021 3:55 pm

Have you tried setting the queue type to mq-pfifo on the CCR2004?

What version of ROS are you using?

FinlayDaG33k · Wed Jul 07, 2021 6:19 pm

I have set my queue type to an `mq-pfifo` resulting in this:

 # INTERFACE                                        QUEUE                                          ACTIVE-QUEUE                                      
 0 ether1                                           only-hardware-queue                            only-hardware-queue                               
 1 ;;; To Modem
   sfp-sfpplus1                                     multi-queue-ethernet-default                   multi-queue-ethernet-default                      
 2 ;;; To Main PC (sfp+)
   sfp-sfpplus2                                     multi-queue-ethernet-default                   multi-queue-ethernet-default                                
 3 ;;; To Isoroku (sfp+1)
   sfp-sfpplus3                                     only-hardware-queue                            only-hardware-queue                               
 4 ;;; To Isoroku (sfp+2)
   sfp-sfpplus4                                     only-hardware-queue                            only-hardware-queue                               
 5 sfp-sfpplus5                                     only-hardware-queue                            only-hardware-queue                               
 6 sfp-sfpplus6                                     only-hardware-queue                            only-hardware-queue                               
 7 sfp-sfpplus7                                     only-hardware-queue                            only-hardware-queue                               
 8 sfp-sfpplus8                                     only-hardware-queue                            only-hardware-queue                               
 9 sfp-sfpplus9                                     only-hardware-queue                            only-hardware-queue                               
10 sfp-sfpplus10                                    only-hardware-queue                            only-hardware-queue                               
11 ;;; To Switch (sfp-sfpplus1)
   sfp-sfpplus11                                    only-hardware-queue                            only-hardware-queue                               
12 ;;; To Switch (sfp-sfpplus2)
   sfp-sfpplus12                                    only-hardware-queue                            only-hardware-queue                               
13 sfp28-1                                          only-hardware-queue                            only-hardware-queue                               
14 sfp28-2                                          only-hardware-queue                            only-hardware-queue                               
15 ;;; To Switch
   bonding1                                         no-queue                                       no-queue                                          
16 bridge1                                          no-queue                                       no-queue

But this sadly doesn't change anything.
Most of the router settings are basically just defaults (not using quickset).
The version is 6.48.3.

Cablenut9 · Wed Jul 07, 2021 6:29 pm

The CCR2004 is notoriously bad at switching, so you probably need to use a real switch instead.

FinlayDaG33k · Wed Jul 07, 2021 6:50 pm

Yea, I already found that elsewhere too, but it doesn't explain why the upload using speedtest.net is so bad...

Cablenut9 · Wed Jul 07, 2021 7:21 pm

I had a similar problem and the issue was the SFP+ not autonegotiating to 1 gigabit, so it stayed on 10 gigabit and kept trying to push that kind of signal through a 1 gigabit interface in the other end.

FinlayDaG33k · Wed Jul 07, 2021 8:32 pm

The link to the modem is 1gbit.
The download from the ISP is 300mbit and my upload is 30mbit.
I can reach the 300mbit download pretty fine through the SFP+ but the upload is only 1/3rd of what it should reach.
So even if it didn't negotiate down, it doesn't explain why my upload is so poor.

mducharme · Sun Jul 11, 2021 5:30 am

The link to the modem is 1gbit.

Can you show the "Status" tab for the sfp-sfpplus port that goes to the modem/ONT?

FinlayDaG33k · Sun Jul 11, 2021 11:35 pm

Yes I can :)
Though it doesn't seem like anything weird is going on there?

The "rate" is 10Gbit because I use an SFP+ module but the link to the modem itself is 1Gbit (the modem only has 1Gbit ethernet ports on it).
Also, considering that if I use the 1gbit RJ-45 connection (which goes through my switch first, then router, then modem) my upload using speedtest.net is just fine, I wonder whether that'll be the issue anyways.

biomesh · Mon Jul 12, 2021 12:21 am

You might want to try rate limiting on your modem port (probably egress) to close of what your internet speed is. When you have a 10 gigabit link but the actual connection is much slower you will get tons of retransmits and the speed will be much slower.

By rate limiting, you can normally get the speeds you are looking for while still being able to route traffic on the other ports at the correct speeds.

It's worth a shot to try.

mducharme · Mon Jul 12, 2021 6:05 am

Though it doesn't seem like anything weird is going on there?

Nothing weird going on there? Your module is SFP+ but it should be negotiating to 1Gbps if that is all that your modem supports. I haven't had experience with this module to understand how it shows the auto negotiation for lower speeds practically speaking, but I can certainly see an issue if you have an ethernet link where one side of the cable is transmitting at 1Gbps and the other at 10Gbps.

I assume you are using the S+RJ10 module. Looking at the Mikrotik wiki I see the following warning

The latest revision of S+RJ10 contains "/r2" by the end of serial number. It comes with following improvements:

    Jumbo frames up to 10218 Bytes at 2.5G, 5G and 10G speeds;
    Actual link speed reporting;
    DDM monitoring (Supply Voltage, Module temperature).

I believe that "Actual link speed reporting" means that the original S+RJ10 (without the /r2) did not report the actual link speed in the status tab (i.e. the negotiated rate), but the one with r2 should. Which version of the module do you have?

FinlayDaG33k · Mon Jul 12, 2021 4:09 pm

You might want to try rate limiting on your modem port (probably egress) to close of what your internet speed is.

Tested it, setting it *slightly* under what I should be able to get (300/30 is what I should get, set it to 250/25) but this only lowers download without affecting upload at all (just stays around being far too low).

I assume you are using the S+RJ10 module.

No, I'm using a module from FS.com (vendor part number "SFP-10G-T")
I've already manually set it to 1000M but then it just doesn't work all together.

But it also doesn't explain the issue where with the switch, it works perfectly fine? The link to the modem is the same in that case but somehow it isn't affected by the issue?
PC -> Switch -> Router -> Modem: Can use full download and upload perfectly fine
PC -> Router -> Modem: Can use full download, can't use full upload.

mducharme · Mon Jul 12, 2021 8:17 pm

But it also doesn't explain the issue where with the switch, it works perfectly fine? The link to the modem is the same in that case but somehow it isn't affected by the issue?

The SFP+ module could be autonegotiating correctly to 1Gbps in the switch but not the router.

FinlayDaG33k · Tue Jul 13, 2021 4:34 pm

But it also doesn't explain the issue where with the switch, it works perfectly fine? The link to the modem is the same in that case but somehow it isn't affected by the issue?
The SFP+ module could be autonegotiating correctly to 1Gbps in the switch but not the router.

That is definitely an option yes.
Though I guess I'll see about performance at the end of the month since I'll be ordering a switch to fix the well... switching issue.
Maybe that'll also solve the issue with the upload speeds?

FinlayDaG33k · Sun Aug 01, 2021 1:56 am

Sadly, adding an additional switch (CRS317-1G-16S+) didn't work, for fixing the upload speed
.It did, however, fix the LAN speed, so that's worth something.

FinlayDaG33k · Wed Aug 18, 2021 6:01 pm

Hi hi, hi-de-ho~

So I've been fiddling around and discovered the issue wasn't with the network itself, it was my Windows PC.
Apparently, I had to disable the "Large Send Offload V2 (IPv4)" setting for my NIC in the device manager.

Durr.

CCR2004-1G-12S+2XS SFP+ Upload issues

CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues

Re: CCR2004-1G-12S+2XS SFP+ Upload issues