I have a weird one here. I’ve searched and can’t find anyone or any mention of this type of issues elsewhere, but if I’ve missed it please let me know.

I have a CCR1036 that I use as our Core router. Recently we’ve been noticing slowdowns everywhere. So I’ve ran some speed tests using the built in MT Bandwidth Test. I could only get 50-60 Mbps from the CCR to any other router, even a RB493G that’s directly connected via cat5 and linked at 1Gbps Full.

If I run a speed Bandwidth Test from the RB493G to the CCR1036 I can pull 140 Mbps (TCP), but the other way 58Mbps max.

From here I wasn’t sure I could trust the built in Bandwidth Test, so Loaded up iperf on a laptop and a server directly connected to the CCR1036. Same result, if I go through the CCR1036 I get 50-60Mbps.

I’ve checked Queues to ensure they aren’t the culprit. Checked firewall rules, mangle, queue tree, etc.

I’m at a lost.

The CCR is running 6.18 The firmware is 3.09 but it looks like I can upgrade to 3.18. I’m unable to find a change log for the firmware, and I’m unsure rather that would have any effect on this. I’m also considering upgrading to 6.20. Unfortunately with this being our core router I can’t just run these upgrades. I’m hoping someone else know what’s change on the firmware side and might have a better idea if it would solve this.

Any help, ideas, suggestions on this would be appreciated!

-Lance

A true Bandwith test can only run through the routers.
Then you can expect nearly the most possible bandwith
When you test from router to router the CPU is the limiting problem espacially TCP

e.g.

CCR1 or PC with IPERF ->Gigabit link <-CCR-test -> Gigabit link<-RBXXX -> Gigabit link <- CCR2 or PC with IPERF

Correct. That's why I loaded iperf on a server and my laptop and conducted another test.

<Laptop> <eth3 CCR> (software bridge) <eth7 CCR> <Server>

At the time of the test the total bridge throughput was 120Mbps

Is the CPU on the CCR maxed out during the tests?

Also, in Winbox, under Interfaces, add the columns Tx Drops, Rx Drops, Tx Errors, and Rx Errors if they aren't there already. During your testing, check to make sure these are all holding 0.

Let us know how it goes.

Zeros all across. The CPU on the CCR is holding 2-3%. A bandwidth test from the CCR to another router doesn't increment the CPU more than 1% overall and usually just one core raises to 40-50% which in my mind rules out traffic rate being limited by CPU. The receiving router reaches about 65% CPU.

I'm starting to think flow control or something along those lines. However I have flow control turned off on the interfaces. I half wonder if pause frames aren't seeping in from elsewhere on the bridge.

I have a scheduled maintenance window planned for the wee hours to upgrade to 6.20 and update the firmware. I will update after the window.

The upgrades had no effect.

The problem appeared to be a weird flow control issue. I had previously turned flow control off because of some lockup issues we had with a ubiquiti switch. I had also turned flow control off on the CCR. But apparently the ubiquiti switch misbehaves even with flow control turned off and still controls the flow. I finally turned flow control on for the CCR interface to the ubnt switch and on within the ubnt switch for the interface facing the CCR and all my throughput issues went away.

I believe what I was seeing was a ubiquiti bug. I do however wonder why it was effecting traffic that didn't or shouldn't have flowed through the ubnt switch.

-LRL

Hello lrl!

I have nearly the same problem with CCR1036 (ROS 6.20) with bridging performance. (aprox 150 Mbit) You wrote, that you could fix it with activating flow control on Interface to ubnt switch.

Where is this Swich in your test scenario, when you testet the following:

Iperf Server < --> CCR1036 <--> Iper Client

Did you have the same issue here?

Thanks for your reply.

run the mikrotik bandwidth test from your PC and at least 10 active sessions at the same time. The problem with the CCR is that each connection/link is only given up to 1 core to run.

I use a CCR1036 with bridge and doing a bunch of other stuff but i realised that to get full performance you need many connections/links.

The internal bandwidth test and the one core per session could affect this, but in my testing I was still able to push into the mid 200's before I reached the limit imposed by one core.

In my situation the UBNT switch was off of int3 and the IPERF server off of int4 and the IPERF client off of int6. I'm not sure why the ubnt switch was causing these issues when it wasn't involved in the test traffic. My hypothesis is the pause frames were passing through the CCR bridge and causing the interfaces on the server or client to pause. Because the nic in the server didn't allow flow control options it was the unknown.

It's a weird one that ultimately got down to a finger pointing game.