I am trying to troubleshoot slow transfer speeds between two VLANs using iperf. This only happens when one side is set to Gig speed. Anyone seeing the same thing or suggestions?
Simple layout:
IMAC1 (VLAN100) -> CRS125 -> IMAC2 (VLAN101)
Test 1:
all ports and adapters set to auto-negotiate. Confirmed all ports and adapters are negotiating at 1000M Full
Results: 5.41 Mbits/sec
Test 2:
set 1 port to only allow 100M Full (negotiate down the speed on one machine only, the other is still 1000M Full)
Results: 77.4 Mbits/sec
Test 3:
If I put both MAC1 and MAC2 in VLAN100 and negotiate all ports and adapters to 1000M Full
Results: 918 Mbits/sec
This tells me the ports, cables, adapters, iperf command is working as expected
Re: Slow speeds when setting CRS125 ports to Gig
Well, anything going between two VLANs is being routed rather than switched, unless there is something I'm missing... Is the CRS the default route between the two devices, or is traffic having to go somewhere else in order to route between those VLANs?
Re: Slow speeds when setting CRS125 ports to Gig
Yes it is only routing within the CRS. I'm not too worried about the 77.4 Mbps speed but 5.41 Mbps when the interface is set to Gig, that's the issue.
Re: Slow speeds when setting CRS125 ports to Gig
Is the switch doing anything else at the time?
What ports are you using on the CRS?
I ask because anything that goes up to Layer3 presumably involves the CPU, and the CPU only has a single 1gbps link back to the internal switch. Check the block diagram (PDF).
What ports are you using on the CRS?
I ask because anything that goes up to Layer3 presumably involves the CPU, and the CPU only has a single 1gbps link back to the internal switch. Check the block diagram (PDF).
Re: Slow speeds when setting CRS125 ports to Gig
Before doing the tests, I did a configuration reset and put in just the minimal VLAN example. I didn't have any ports activated except the two workstations. CPU on the switch didn't go about 10% the entire time so I don't think it's a CPU bottleneck.
Interesting enough, the iperf was defaulting to a 64k TCP window size when producing the poor results (3mbps). If I switch the window size to 10k performance goes all the way up to 160 Mbps. I'm by no means a TCP expert, but I thought a larger window should increase performance unless you have alot of drops on the network. I'm seeing 0 collisions or TX errors.
I'm thinking it's a bug in the way the CPU handles routing for large packet sizes on Gig interfaces. I'm going to try the same tests on older firmware and see if it changes before I log a support ticket.
Interesting enough, the iperf was defaulting to a 64k TCP window size when producing the poor results (3mbps). If I switch the window size to 10k performance goes all the way up to 160 Mbps. I'm by no means a TCP expert, but I thought a larger window should increase performance unless you have alot of drops on the network. I'm seeing 0 collisions or TX errors.
I'm thinking it's a bug in the way the CPU handles routing for large packet sizes on Gig interfaces. I'm going to try the same tests on older firmware and see if it changes before I log a support ticket.
Re: Slow speeds when setting CRS125 ports to Gig
Heard back from support, changing the queue type from hardware to ethernet-default has fixed it. Speeds went from 3Mbps to 180Mbps just by running the command below. Seems to be this should be set as default or better documented.
/queue interface set [find] queue=ethernet-default
/queue interface set [find] queue=ethernet-default
Re: Slow speeds when setting CRS125 ports to Gig
I had the same problem. I applied your command line and my cable speed went up from 12up/50down Mbps to 12up/112down Mbps as it should be. Thanks, this article helped.
Re: Slow speeds when setting CRS125 ports to Gig
Well.. I have a similar issue.. I'll explain the topology below.
CRS125 -- Gigabit Ethernet - SXTAC -=-=-=-Wireless -=-=-=- SXTAC --Gigabit Ethernet -- RB2011UiAS2Hnd
The CRS 125 is the side with Internet access, but in my test, its not being used for internet access.
All Ethernet ports have been se to " Ethernet - Default " Queues.
the CRS has most, but not all ports in switch config. The SXT AC has a bridge to bridge the wireless to Ethernet, it is a PtP link, The Rb 2011 is set up as Ether 1 as master and ether 2 as slave, the SXT connects on Ether 1 via Gig Ethernet. Eth 2 connects to another device, a UBNT wireless product that connects to a farther location.
This is my own private network, it is all a flat network, ie, all on same subnet.
CRS is set up for NAT, with minimal rules. basic home type setup.
My tests are as follows,
PC connected on Ethernet of CRS, port 8, in the switch with the rest including the ether for the first SXT.
- using BTEST, I ran a test SEND to the far RB 2011, I get speeds at about 35-40 Mbit UDP
- Using Bandwidth test on CRS, send to far RB2011 I get the same 30-40 Mbit UDP
- Test from the first SXT to the far RB 2011 I get 300+ Mps UDP This is verified by seeing the data on the Ethernet port on the RB 2011
So, here is the weird part, Those test were done one at a time... SO I tried this....
Tried BTEST from PC to SEND, 40 Mbit, left test running.
Started Bandwidth test on CRS to send to rb 2011, roughly same 40 Mbit ----- Btest on PC still running at roughly same speeds. Ethernet interface for the pC was confirming inbound 40 mbit, and total of about 80 Mbit out Ethernet 4 which connects to the first SXT. Total data received at RB2011 Ether 1 port, roughly 80 Mbit.
To confuse me even more, I tried run Bandwidth test from the first SXT to SEND to the same RB 2011, 180-190 Mbit on that, WHILE the other two tests were running, and the other two tests remained roughly at their 40 mbit.
When I try BTEST on the PC for RECIEVE from the RB 2011 on the far end, I get roughly 250 Mbit
If I do that test, receive to CRS from the RB 2011, oddly this time I am getting 360 Mbit to 410 Mbit
My conclusion is that something is going on in the CRS that is slowing down the traffic, BUT, the CPU load is around 20%. I do not understand why it will allow each stream of data to stay the same.
Funny, as I was typing, I used another test, a Netmetal 5, that is connected to ether 3 ( master port) on the CRS, It is a link to another location, but when I try the SEND speed test to that same RB2011, I am getting 280 - 300 Mbit, and that goes through the CRS .. In ether 3, master, out ether 4 slaveto3, to SXT, wireless, SXT Ethernet to Ether 1 on RB 2011.
I am at a loss.
CRS125 -- Gigabit Ethernet - SXTAC -=-=-=-Wireless -=-=-=- SXTAC --Gigabit Ethernet -- RB2011UiAS2Hnd
The CRS 125 is the side with Internet access, but in my test, its not being used for internet access.
All Ethernet ports have been se to " Ethernet - Default " Queues.
the CRS has most, but not all ports in switch config. The SXT AC has a bridge to bridge the wireless to Ethernet, it is a PtP link, The Rb 2011 is set up as Ether 1 as master and ether 2 as slave, the SXT connects on Ether 1 via Gig Ethernet. Eth 2 connects to another device, a UBNT wireless product that connects to a farther location.
This is my own private network, it is all a flat network, ie, all on same subnet.
CRS is set up for NAT, with minimal rules. basic home type setup.
My tests are as follows,
PC connected on Ethernet of CRS, port 8, in the switch with the rest including the ether for the first SXT.
- using BTEST, I ran a test SEND to the far RB 2011, I get speeds at about 35-40 Mbit UDP
- Using Bandwidth test on CRS, send to far RB2011 I get the same 30-40 Mbit UDP
- Test from the first SXT to the far RB 2011 I get 300+ Mps UDP This is verified by seeing the data on the Ethernet port on the RB 2011
So, here is the weird part, Those test were done one at a time... SO I tried this....
Tried BTEST from PC to SEND, 40 Mbit, left test running.
Started Bandwidth test on CRS to send to rb 2011, roughly same 40 Mbit ----- Btest on PC still running at roughly same speeds. Ethernet interface for the pC was confirming inbound 40 mbit, and total of about 80 Mbit out Ethernet 4 which connects to the first SXT. Total data received at RB2011 Ether 1 port, roughly 80 Mbit.
To confuse me even more, I tried run Bandwidth test from the first SXT to SEND to the same RB 2011, 180-190 Mbit on that, WHILE the other two tests were running, and the other two tests remained roughly at their 40 mbit.
When I try BTEST on the PC for RECIEVE from the RB 2011 on the far end, I get roughly 250 Mbit
If I do that test, receive to CRS from the RB 2011, oddly this time I am getting 360 Mbit to 410 Mbit
My conclusion is that something is going on in the CRS that is slowing down the traffic, BUT, the CPU load is around 20%. I do not understand why it will allow each stream of data to stay the same.
Funny, as I was typing, I used another test, a Netmetal 5, that is connected to ether 3 ( master port) on the CRS, It is a link to another location, but when I try the SEND speed test to that same RB2011, I am getting 280 - 300 Mbit, and that goes through the CRS .. In ether 3, master, out ether 4 slaveto3, to SXT, wireless, SXT Ethernet to Ether 1 on RB 2011.
I am at a loss.
Re: Slow speeds when setting CRS125 ports to Gig
Are you using Switch VLAN features, or are you using regular software VLANS??
All configuration need to be done under switch menu and everything works wire speed then.
All configuration need to be done under switch menu and everything works wire speed then.
Re: Slow speeds when setting CRS125 ports to Gig
Not using any VLAN, the ether ports are eth 3 master, 4 - 24 are slaves to it. Eth 1 is on its own as the WAN uplink, Eth 2 disabled and was supposed to be WAN2, but not set up yet
Re: Slow speeds when setting CRS125 ports to Gig
CRS125 CRS109 have exactly the same performance as RB2011, with fastrack hitting almost 1Gbps throughput real traffic (torrents, linux distributions of courseNot using any VLAN, the ether ports are eth 3 master, 4 - 24 are slaves to it. Eth 1 is on its own as the WAN uplink, Eth 2 disabled and was supposed to be WAN2, but not set up yet
)I do not suggest testing small cpu boards like this, to and from is bad idea, you need to test through
Re: Slow speeds when setting CRS125 ports to Gig
I understand that, and I read that same suggestion that everyone puts on forum posts.
The problem is that using that same RB 2011 at the far end, and the CRS on this end, when testing speeds, its slower at the PC connected at the CRS side, than if you use the CRS itself to do the bandwidth test.
The expected behavior would be that the CRS CPU is busy running the test, so the throughput would suffer. Here in my situation it is different, its actually faster by a bit, when using BTEST on the PC the speeds are a bit slower, not much, but they are.
The wireless link in between the isn't even considered, nor the lack of cpu from the rb 2011.
The point is that if I can test the speed from the first SXT in that set up over the wireless and to the RB2011 and get around 300 Mbit, why is it when I test from the CRS it slows down that much, even skipping the CRS and running the software on the PC has the slow speeds AND to make it even more interesting, running multiple BTEST send streams, will ironically get double the speed
SO, if the speed bottle neck was the CPU then how does it get the double speed, ie 35 +35 Mbit from the PC running Btest or if I run one Btest stream, it stays at about 35 as well, when I run 3 streams I get around 85-90 Mbit
so the cpu of either the CRS and 2011 is not the issue here
The problem is that using that same RB 2011 at the far end, and the CRS on this end, when testing speeds, its slower at the PC connected at the CRS side, than if you use the CRS itself to do the bandwidth test.
The expected behavior would be that the CRS CPU is busy running the test, so the throughput would suffer. Here in my situation it is different, its actually faster by a bit, when using BTEST on the PC the speeds are a bit slower, not much, but they are.
The wireless link in between the isn't even considered, nor the lack of cpu from the rb 2011.
The point is that if I can test the speed from the first SXT in that set up over the wireless and to the RB2011 and get around 300 Mbit, why is it when I test from the CRS it slows down that much, even skipping the CRS and running the software on the PC has the slow speeds AND to make it even more interesting, running multiple BTEST send streams, will ironically get double the speed
SO, if the speed bottle neck was the CPU then how does it get the double speed, ie 35 +35 Mbit from the PC running Btest or if I run one Btest stream, it stays at about 35 as well, when I run 3 streams I get around 85-90 Mbit
so the cpu of either the CRS and 2011 is not the issue here
Re: Slow speeds when setting CRS125 ports to Gig
Do you mean the windows executable btest.exe?The expected behavior would be that the CRS CPU is busy running the test, so the throughput would suffer. Here in my situation it is different, its actually faster by a bit, when using BTEST on the PC the speeds are a bit slower, not much, but they are.
It only uses 1 TCP stream where btests from the routers default to 20. That always shows slower IME
Re: Slow speeds when setting CRS125 ports to Gig
Yes, I mean BTEST.exe and yes, I know the limit, and yes, I tried multiple tests at the same time, but the max was around 90 Mbit, but bypassing the crs125 yields 300+ mbit. Its all above in the notes.
Re: Slow speeds when setting CRS125 ports to Gig
It's posts like this that keep me squarely in the Cisco camp when it comes to switches.
You could get a used 3750 for $50 on the Internet, default the configuration, and plug it in where the CRS is, and not have any of these problems.... no need to worry about interface queue types and stuff.
Or if Cisco is just too.... well, let's just say I hate Cisco's firmware policies for instance..... there are other switches like Adtran, HP, Dell, etc. that can do the job. It's 2016. Getting a switch to push wire speed traffic shouldn't require forum posts.
You could get a used 3750 for $50 on the Internet, default the configuration, and plug it in where the CRS is, and not have any of these problems.... no need to worry about interface queue types and stuff.
Or if Cisco is just too.... well, let's just say I hate Cisco's firmware policies for instance..... there are other switches like Adtran, HP, Dell, etc. that can do the job. It's 2016. Getting a switch to push wire speed traffic shouldn't require forum posts.