Hello,

I've been seeing this topic repeated several times on the forum with no sufficciently good explanation or solution, so don't blame me for reposting, I just want this working.

The problem is the TCP performance over various mikrotik routers.

I've already been solving this once, when I saw that our rb1100's can't really support TCP connection faster than around 5Mbit (UDP went well), it got solved by enabling the correct interface queueing method, as seen here: http://forum.mikrotik.com/viewtopic.php?f=2&t=62377

Some other is here: http://forum.mikrotik.com/viewtopic.php?f=2&t=59830

Our problem is that the same is now happening on slower/simpler machines, especially rb711's and rb433(AHs), where the fix with queue limits doesn't work or even apply (as there's no multiqueue).

For example, we have this setup:

rb433AH ~cable~ rb711 ~wifi~ rb711 ~cable~ rb433AH

UDP bandwidth test between the rb433s measures around 80Mbit (matches the actual wifi performance), single-connection TCP stays at around 15Mbit.

I wanted to rule out the problem with CPU on the bandwidth-measuring machine going 100%, so I connected:

linux box ~cable~ rb711 ~wifi~ rb711 ~cable~ linux box

and again measured only around 15Mbit of single-connection TCP.
I also successfully recreated the problem without any actual WiFi in the way (only a rb433AH forwarding the stuff). Also noticed that when more routers are added to the chain, performance gradually decreases to around 4Mbit.

Fixing and setting the Interface Queues to either sane or anyhow extreme values doesn't help in this case, no matter how it helped with rb1100's and similars.

For the questions:

• Is there some good way to have good TCP single-connection performance on such routers?
  Does it look like a regression in v5 only for me, or others also see similar thing?
  Is there any good explanation of what is happening that the TCP congestion mechanisms get triggered?
  Is there any good debug info I can provide so this can finally get solved&closed&buried?

Thanks
-exa

EDIT: ofcourse everything that would hog CPU is turned off, no bridge, no firewall, no conntrack. No queues except for interface queues. No other traffic. Low route count (only those necessary for testing).

same problem.

no firewall, no conntrack.

Real Internet downloads has similar results. Please mikrotik to comment on the issue. This is a big problem.

Thanks for image, hapi!

The similarity between your and my measured speeds is actually striking me.

I can only add that NV2 doesn't make the problem, it's the same with whichever variant of 802.11a/b/g/n and/or nstreme.

The similarity between your and my measured speeds is actually striking me.

Yes, I'm surprised that we have the same results. It is not a coincidence.

I can only add that NV2 doesn't make the problem, it's the same with whichever variant of 802.11a/b/g/n and/or nstreme.

I fully agree. It makes no difference.

I am curious if it is only problem of v5 ROS, or it is also in lower versions...

If Mikrotik wont take care of serious problems, people find alternative which in a few months will be ubnt edgerouter with fully functional debian inside and possibilities of installing what you want - and is in linux enviroment.....

Hello,

I´ve got same problem as MT users above...
Question for MT support: Could you help with this issue?

I have the same problem with 7 mtik on x86 machine in chain all are P4 single and some are dual core processors and PTP links in it to the end of tha chain. All are on N standards with nstream enabled and. ccq are around 100% and links are not longer than 5km each...

I have the same problem with 7 mtik on x86 machine in chain all are P4 single and some are dual core processors and PTP links in it to the end of tha chain. All are on N standards with nstream enabled and. ccq are around 100% and links are not longer than 5km each...

Same problem ...

where is MK support? .. Help.. !!

Hello,

I´ve got same problem as MT users above...
Question for MT support: Could you help with this issue?

Me too ...

same problem.

no firewall, no conntrack.

Real Internet downloads has similar results. Please mikrotik to comment on the issue. This is a big problem.

Your problem is the first wireless link. First RB711 get only 40mbps because you tested with internal btest so cpu max out. The problme is 15mbps in the first link. Did you tested with internal btest also? Btest is not very reliable, especialy if it run's on windows. Solve first wireless link. And why do you test with only one TCP connection? It's like you forced your car to work with just one cylinder. Event a simple web page have more than one connection...

Do you get these results in both directions?

I have similar returned value in more connected places usualy resolved by only 1 TCP connection (like downloading ex. from web) This result aren't only BT, but in real downloading.

same problem.

no firewall, no conntrack.

Real Internet downloads has similar results. Please mikrotik to comment on the issue. This is a big problem.

Your problem is the first wireless link. First RB711 get only 40mbps because you tested with internal btest so cpu max out. The problme is 15mbps in the first link. Did you tested with internal btest also? Btest is not very reliable, especialy if it run's on windows. Solve first wireless link. And why do you test with only one TCP connection? It's like you forced your car to work with just one cylinder. Event a simple web page have more than one connection...

Hmm, but exa have same results on this: linux box ~cable~ rb711 ~wifi~ rb711 ~cable~ linux box
Problem is in 1 TCP connestion. Thats uses speedtests or downloads.
we tested it on 40HMhz 802.11N Duplex line with the UDP throughput in approximately 210Mbps, but 1 TCP only 15-20Mbps (the same speedtest on non-shaped line)

Try this for testing TCP connection between PC on windows station maybee more usefull for TPC than Bandwidth test MT. On Gigabit ethernet get more throughput than BT on PC.

http://www.roadkil.net/program.php/P5/CommTest

Is the radio link in the diagram above configured as nstreme dual? If not then it is a half duplex link which will potentially interact with TCP's ACK timing window.

If nstreme dual has not been tried it might be interesting to see if it can affect the 40/15 Mbps differential above.

Do you get these results in both directions?

Yes, there's practically no difference (test lab setup is symmetrical anyway.)

same problem.

no firewall, no conntrack.

Real Internet downloads has similar results. Please mikrotik to comment on the issue. This is a big problem.

Your problem is the first wireless link. First RB711 get only 40mbps because you tested with internal btest so cpu max out. The problme is 15mbps in the first link. Did you tested with internal btest also? Btest is not very reliable, especialy if it run's on windows. Solve first wireless link. And why do you test with only one TCP connection? It's like you forced your car to work with just one cylinder. Event a simple web page have more than one connection...

no, no, you mistake. 14Mbit test is normal? no!!!

Please mikrotik to comment on the issue.

Real downloading from the Internet is identical to btest.

Each piece of this network is affected. Not only for me.

UDP: over 80Mbit
TCP: under 10Mbit
This is not good. This is a disaster. Customers calling them is slow downloads.

Good day to everyone. Today I made following setup to investigate this issue:
linux box <-cable-> SXT5HPnD <----wi-fi---> SXT5HPnD <-cable-> linux box
I used RouterOS 5.21 with new bootloader. Between SXTs was just few meters. I used nstreme as wireless protocol. FTP download and TCP BTest (both in one tcp connection) showed nearly 80Mbit/s bandwidth. Maybe this issue is affected by hardware revisions in conjunction with ROS version or something?

(.....) Maybe this issue is affected by hardware revisions in conjunction with ROS version or something?

Could you post the versions of your firmware/bootloader please?

(btw. it's most probably not about RouterOS version, I had the latest 5.21 on everything)

Thanks,
-exa

Could you post the versions of your firmware/bootloader please?

(btw. it's most probably not about RouterOS version, I had the latest 5.21 on everything)

Thanks,
-exa

I've tested it with firmware version 3.0.

So have anyone new information about this?

So have anyone new information about this?

No, and never will be, becouse in "The Lab" everything is ok. So there is no problem. Subject closed. Back off.

So have anyone new information about this?

No, and never will be, becouse in "The Lab" everything is ok. So there is no problem. Subject closed. Back off.

OK so that Lab results show in numbers, please!

First of all - DO NOT run traffic to or from RouterOS devices, - routers are all about throughput.

IF you wanna test single TCP connection performance - easy.

Take 2 linux x86 boxes with SSD and fast HDD and configure on one FTP client and on other FTP server, take large enough file and try to download it.

I had to make similar setup that involved chain of RB1100Ahx2 routed like this -

My laptop with Linux <--
<-- ether1 RB1100AHx2 ether10 <--
<-- ether1 RB1100AHx2 ether10 <--
<-- ether1 RB1100AHx2 ether10 <--
<-- ether1 RB1100AHx2 ether10 <--
<--Linux FTP server

One 4,2GB file was downloaded via single FTP data connection with ~850Mbps average speed
same file, from same server only with ftp client on windows7 running on the same laptop ~420Mbps!!!

It is easy to try it yourself!

First of all - DO NOT run traffic to or from RouterOS devices, - routers are all about throughput.

IF you wanna test single TCP connection performance - easy.

Take 2 linux x86 boxes with SSD and fast HDD and configure on one FTP client and on other FTP server, take large enough file and try to download it.

I had to make similar setup that involved chain of RB1100Ahx2 routed like this -

My laptop with Linux <--
<-- ether1 RB1100AHx2 ether10 <--
<-- ether1 RB1100AHx2 ether10 <--
<-- ether1 RB1100AHx2 ether10 <--
<-- ether1 RB1100AHx2 ether10 <--
<--Linux FTP server

One 4,2GB file was downloaded via single FTP data connection with ~850Mbps average speed
same file, from same server only with ftp client on windows7 running on the same laptop ~420Mbps!!!

It is easy to try it yourself!

OK and where is wireless on RB1100AHx2 (802.11 - halfduplex), discuse begin here tell about wireless behind wireless, what i seen, yes?
Powerfull source and destination? Yes, of course.

My laptop with Linux <--
<-- ether1 RB1100AHx2 ether10 <--
<-- ether1 RB1100AHx2 ether10 <--
<-- ether1 RB1100AHx2 ether10 <--
<-- ether1 RB1100AHx2 ether10 <--
<--Linux FTP server

We are talking about wireless TCP problems.

We are talking about wireless TCP problems.

Wireless or not - you can't get proper results if you run test to or from router - you need to run it trough.
That was the point i was trying to get across.

I do not have test setup handy, but im also pretty sure that our clients have high speed TCP connections all the time - i had to make special qos for that.

Problem solved. The problem caused a switch at the top left in the picture. Gigo transition to fast ethernet. TL-SG1016 exchanged for RB250GS. Maybe flow control that could not be turned off and maybe small buffer in a switch that mishandled tcp window size in burst.

If the connections on the switch were not negotiating correctly you have have had one side of the connection running half duplex and the other side running full duplex. That will kill throughput. That condition usually shows up as a spike in late collisions and poor throughput under load.

If the connections on the switch were not negotiating correctly you have have had one side of the connection running half duplex and the other side running full duplex. That will kill throughput. That condition usually shows up as a spike in late collisions and poor throughput under load.

OK that's mean is recomended set autonegotating OFF?

I would only set it off if it was regularly failing or causing problems. If an interface sets itself to half-duplex you can usually see that and it is worth checking.

e.g. if two devices *negotiate* 100 Mbps and full duplex then all is well, but if the negotiation fails and one side *senses* the other side using 100 Mbps it may well configure itself at 100 Mbps *half-duplex*. If the other end is running full-duplex then misery will ensue...

More info here:

http://en.wikipedia.org/wiki/Duplex_mismatch

although I don't think it mentions the condition leading to a mismatch that I outlined above but it does happen. e.g. Cisco switches will select half duplex on 100 Mbps ports if the auto-negotiation failed and they sense the line speed.

If the connections on the switch were not negotiating correctly you have have had one side of the connection running half duplex and the other side running full duplex. That will kill throughput. That condition usually shows up as a spike in late collisions and poor throughput under load.

I did not write anything about the half-duplex connection!! Do not twist my words.

If the connections on the switch were not negotiating correctly you have have had one side of the connection running half duplex and the other side running full duplex. That will kill throughput. That condition usually shows up as a spike in late collisions and poor throughput under load.

I did not write anything about the half-duplex connection!! Do not twist my words.

I did not twist your words. I was pointing out that a Gigabit interface which ends up running at 100 Mbps because it *senses* that speed rather than negotiates it may well set itself to half-duplex. I have no idea if that actually happened in your case but it is a useful possibility to bear in mind.

Problem solved. The problem caused a switch at....

Hey guys! this is not solved, I don't have any badly configured switches in the way (particularly, absolutely no switches!)

Anyway, I'm also starting to think that this is at all times a weird combination of several unpleasant issues (signal, configuration, bad weather, switches, cables, slower routerboard, ...) than a general error. It just still "triggers" this much annoyingly, and the explanation doesn't really fit the lab setup... Any general point where to start debugging such stuff would be great.

What we have here performs great:

Linux machine -->
Netgear managed switch -->
*Juniper SRX210 router -->
Netgear managed switch -->
*x86 machine with RouterOS 6.0rc6 and AR9380 wifi -->
(wireless)
*RB600A with RouterOS 5.22 and AR9220 wifi -->
*RB450G with RouterOS 5.1x-->
RB250GS -->
Linux server -->
Linux VM

*=routing through OSPF.

We get speeds of 185mbit/s single TCP session using iperf.
And with Windows filesharing about the same speeds.

Hi,

i have same problem on 3 x RB433L. can't pass over 10 mbps TCP single connection over ethernet. UDP is 100 Mbps. Any solution?

Hi,

i have same problem on 3 x RB433L. can't pass over 10 mbps TCP single connection over ethernet. UDP is 100 Mbps. Any solution?

All Mikrotik customers waiting for new wireless drivers for routeros included in new version MT?

I had a very similar problem: UDP Bandwith test was super, but TCP test quite bad. Problem was a poor network cable connection. Check the statistics-page of the 100MBit FDS (the right RB711G). This was mine: Leo.

this can be bufferbloat, or rb433 series problem....

I had a very similar problem: UDP Bandwith test was super, but TCP test quite bad. Problem was a poor network cable connection. Check the statistics-page of the 100MBit FDS (the right RB711G). This was mine:

Code: Select all

/interface ethernet print stats
name:            ether1-local
rx-fcs-error:    234
rx-align-error:  37
rx-fragment:    1
rx-length-error:8
rx-jabber:       306
rx-drop:         579
tx-too-short:  16 466

One day later: This was not the cable. I changed the POE splitter and power supply. (Not sure which of both was the source).

Leo.

Anyone ever figure out a good explanation for this? I'm getting odd tcp performance issues. I'm not using any mikrotik wireless, they are strictly routers. Udp and tcp with multiple connections = full speed. Tcp with one connections t uns anout an 8th of the speed but only one direction.

We are also seeing issues with single stream performance with tcp. Multiple connections and udp full speed single connection tcp testing a small fraction of what it should be. Have not identified problem yet...

A post with almost one year duration and no opinion of mikrotik support? My clients are going to other ISPs, and I'll have to go to another provider routers?

If you have a specific problem I suggest that you describe it in detail in a new thread. This thread contains references to a number of different problems and the post 2 was confirmed fixed by replacing (non-Mikrotik) switch.

@hapi
have youe tried to sihft the frequency on the wireless link 5-10MHZ on up or down?
Do the test from your big xeon machine TCP 1 send to the Omnitik and post.

We had a simmilar link 10km on 5500Mhz NV2 2ms. Good RX/TX -67 both chains 98-100% CCQ
At 5500 TCP 1 connection was only arround 20Mbit/s - I twas not enough for us, so we tied to shift the frequency down and up.
The best Connection was on 5505Mhz TCP raisd to 65mBit/s (CPU maxed out)

let us know if it helps -;)

Hi all,

we've had similar problems and it turned out to be a bufferbloat problem on one of our licensed links.
Our Setup looked like this:

[Server] <----GigE----> [C2960G] <----GigE----> [Licensed_150Mbps_Link] <----GigE----> [C2960G] <----FastE----> [Ubnt_RocketM5] <----300/300 Mbps----> [Ubnt_NB5M] <----FastE----> [PC]

The 150Mbps-Link was connected over GigE to the Switch, but since it was not able to transfer data at GigE wirespeed, it started to buffer as much data as it could => bufferbloat. When we switched the Interface of the Server to 100Mbps, our Problems went away, because the 150MBps-Link was able to transmit at FastE wirespeed so there was nothing to buffer.

The solution was to enable Flowcontrol on the 150Mbps-Link and the C2960G.


I don't know if this applies to your Setup but maybe it helps to find the solution.



- Mat

If you have a specific problem I suggest that you describe it in detail in a new thread. This thread contains references to a number of different problems and the post 2 was confirmed fixed by replacing (non-Mikrotik) switch.

My problem is the same of this topic subject: TCP performance

RB433AH <-> switch 3com gigabit <-> Rocket M5 -- WIRELESS -- Rocket M5 <-> switch 3com gigabit <-> RB433AH

Btest

protocol: UDP
bandwidth: 60Mbps

Protocol: TCP
bandwidth: 12Mbps

Similar results testing between PC stations at the ends, using the FTP protocol, in both directions.

People often expect that UDP and TCP performance should be the same but TCP performance is affected by factors such as latency and half duplex links so the fact that TCP and UDP performance differs doesn't automatically mean that there is a problem in RouterOS.

People often expect that UDP and TCP performance should be the same but TCP performance is affected by factors such as latency and half duplex links so the fact that TCP and UDP performance differs doesn't automatically mean that there is a problem in RouterOS.

Please, something new.

If you have a specific problem I suggest that you describe it in detail in a new thread. This thread contains references to a number of different problems and the post 2 was confirmed fixed by replacing (non-Mikrotik) switch.

My problem is the same of this topic subject: TCP performance

RB433AH <-> switch 3com gigabit <-> Rocket M5 -- WIRELESS -- Rocket M5 <-> switch 3com gigabit <-> RB433AH

Btest

protocol: UDP
bandwidth: 60Mbps

Protocol: TCP
bandwidth: 12Mbps

Similar results testing between PC stations at the ends, using the FTP protocol, in both directions.

Please, do not even try to use TCP Btest on non-x86 hardware. It is misleading!
What speeds do you get when testing directly connected to the Rocket M5 AP's? What is the latency on the link?

Hi!

Please, do not even try to use TCP Btest on non-x86 hardware. It is misleading!

As informed, also tested at two x86 in the ends, FTP protocol, same result.

What speeds do you get when testing directly connected to the Rocket M5 AP's?

Network Speed Test:

66.39Mbps

ps.: The 'Network Speed Test' from the ubnt is UDP protocol.

I have not tested via FTP directly connected to the radio from x86, will perform this test.

What is the latency on the link?

Five Packet Count:

2.46 ms 64
2.73 ms 64
2.86 ms 64
2.61 ms 64
3.5 ms 64

I have five locations with the same equipment in the arrangement I mentioned, all with the same behavior.

I'm thinking of removing the switch and make new tests.

ps1.: the rb are running pppoe server
ps2.: sorry for my bad english.

Please, something new.

Sarcasm doesn't solve such problems.

Please, something new.

Sarcasm doesn't solve such problems.

Quote the "people", and not me.

I see this effect, too.

Sometimes it is due to some packet loss. A single TCP-streams shows this problems much earlier than a udp connection.

Sometimes it stays unknown. E.g. I have a RB433AH on one side of a wireless link. Performance on the link is fine (bw test from the RB433AH to the other side), performance to the RB433AH from the ethernet side is fine.
Connections through the RB433AH over the link shows slow tcp-speed. Looks like a forwarding problem within the RB.

I see this effect, too.

Sometimes it is due to some packet loss. A single TCP-streams shows this problems much earlier than a udp connection.

Sometimes it stays unknown. E.g. I have a RB433AH on one side of a wireless link. Performance on the link is fine (bw test from the RB433AH to the other side), performance to the RB433AH from the ethernet side is fine.
Connections through the RB433AH over the link shows slow tcp-speed. Looks like a forwarding problem within the RB.

What is at the other end if the radio link in this case and how is the link set in terms of protocol, duplex etc?

I see this effect, too.

Sometimes it is due to some packet loss. A single TCP-streams shows this problems much earlier than a udp connection.

Sometimes it stays unknown. E.g. I have a RB433AH on one side of a wireless link. Performance on the link is fine (bw test from the RB433AH to the other side), performance to the RB433AH from the ethernet side is fine.
Connections through the RB433AH over the link shows slow tcp-speed. Looks like a forwarding problem within the RB.

What is at the other end if the radio link in this case and how is the link set in terms of protocol, duplex etc?

Rb411ah nv2 ap/station routed r52n cards.

Hello all.

What is the best method for identify the real throughput of a path?

The real traffic in the network usually are TCP not UDP.

Infact the your problem is the difference of results between TCP and UDP tests.

Hello all.

What is the best method for identify the real throughput of a path?

The real traffic in the network usually are TCP not UDP.

Infact the your problem is the difference of results between TCP and UDP tests.

There is no "real throughput". If you do torrent UDP-Speed is what you get. If you do ftp you use a single tcp-connection for data. If you use a browser it depends on how much connection this specific browser uses. Some do prefetches.

speedtest uses different amount of tcp-connections depending on what performance it gets.

Hi,
don't forget TCP react to network congestion and ajust is speed depending on window size in ack it received.

But, I must admit something strange in ros v5.x using NV2 wireless link.

router A<-wifi NV2-> router B <-ether Gigabit -> router C <-wifi NV2-> router D

ALL TCP Btest use only one connection:

Btest TCP A to B 35 Mbps
Btest UDP A to B 220 Mbps
Btest TCP A to C 35 Mbps
Btest UDP A to C 220 Mbps
Btest TCP A to D 15 Mbps
Btest UDP A to D 220 Mbps

You might think this is normal result and wifi link between link C and D is bad or congested, but here we go with a BTest between C and D:
Btest TCP C to D 60 Mbps
Btest UDP C to D 220 Mbps

As you can see throughput over one wireless link goes well but over two, TCP throughput fall dramatically.

Upgrading to 6.5 allow more throuput over TCP:
Btest TCP A to B 60 Mbps
Btest UDP A to B 220 Mbps
Btest TCP A to C 60 Mbps
Btest UDP A to C 220 Mbps
Btest TCP A to D 30 Mbps
Btest UDP A to D 220 Mbps

Almost double performance...

I think there is something to do with the NV2 queueing... Anyway do sommeone as opened ticket with MT support regarding this issue ?

Regards,
Michael

Data Center RB100AH - Fiber- RB2011 Ethernet - RB711GA -Wireless - RB711GA - Ethernet - RB433AH

I have recently had speed issues. Not sure what was the cause of the problem as the link used to work 30Mbps from the RB11AH to the RB433AH.

TCP bandwidth testing only getting 5Mbps. Just tried upgrading the two RG711GA with ROS 6.24

I am not saying I solved the problem, but when I changed the Wireless Link from NV2 to Nstream, the bandwidth test increased to 20Mbps.

Any one else had the same experience?

Cheers

Matt

Oh.. YOU have this issue too? I am reviving this because I had it as well, and after a bunch of time wasted, I know what is doing this, but don't understand why.

My Setup:

Main Router:
CRS125-24G-1S-2HnD -- *Gig Ethernet* -- SXT G5HPacD --Wireless-- SXT G5HPacD -- *Gig Ethernet* -- RB2011UiAS-2HnD

So, my UDP and TCP tests from the CRS125 24G to the Rb2011 would get no more then 108 Mbit on UDP,a nd around 30Mbit on TCP

Testing from the first SXT to the RB2011 and over the wireless, I would get 300Mbit UDP, and 50+ Mbit TCP * Noting the 100% Cpu.

Testing each segment of the link, everything would get faster then 100 on UDP, EXCEPT from the CRS125-24G to the SXT over the wired gigabit connection.

Puzzled, I thought it was a bad port on the SXT. I used another SXT in a bench test environment, connected to another Port on the CRS125-24G. Result, on UDP was 160 Mbit. ** , checked this new SXT in the same port Ether 4 on the CRS125 now, slow speeds again. I scratched mey head, thought that this was a messed up ether 4 on the CRS125. Looked at the config closer... Found that the port I checked originally was ** NOT ** in switch configuration, its "Master port" was set to NONE.

Changed the Ether 4 from Master port = Ether 3, to NONE, and then put ether 4 onto the Bridge-LAN so it could communicate with the other devices.

SUPRISE, checked the speed test again and then the UDP rated went up from the max 100 I was getting up to 200 - 300 Mbit over the Ethernet gigabit.

SO yes, I know that the processor I used for when a port is not in switch configuration, but I find it strange that this was this issue. If the speed is faster using the bridge method, then why would you want a switch? I am assuming the port to port data would be faster over the switch chip, but from the processor to the switch chip there seems to be a bottle neck.

I will have to test this again just to make sure, but it was re-producible last night a few times.

Anyone try this setup?

I had a very similar problem: UDP Bandwith test was super, but TCP test quite bad. Problem was a poor network cable connection. Check the statistics-page of the 100MBit FDS (the right RB711G). This was mine:

Code: Select all

/interface ethernet print stats
name:            ether1-local
rx-fcs-error:    234
rx-align-error:  37
rx-fragment:    1
rx-length-error:8
rx-jabber:       306
rx-drop:         579
tx-too-short:  16 466

One day later: This was not the cable. I changed the POE splitter and power supply. (Not sure which of both was the source).

2 years later I solved another issue: the "Auto Negotation" setting of the ethernet interface was different on both sides, on a omnitik is was on, on the Sextant-G it was fixed to 100MBit full duplex. You see similar faulty ethernet stats as above; Bandwith-Test-UDP over the wireless link show maximal speed, ping show no loss, bandwith test UDP just over the ethernet cable (Sextant to Omnitik) show 97MBit,
but real TCP transfer rates are low (faulty link:800kybtes/s, after fix: 4MBytes/s).

Changing both sides to auto-negotiation solved the problem!

I learned (after several days of searching!): In case you have faulty wireless connections check for ethernet failures too - even if it seems to be good (with ping or bw test)!

Leo.

older versions than 6.19 had problems with 1gbit ports connected to 100mbit/s...it has to be manually selected 100mbit/s in eth. interface, new version seems to solve it...so this was bug of RouterOS
- another point, after ros 6.19 bandwitch test for tcp is completely not working correctly, it shows bad throughoutput and mostly sinusoid type...but this is caused that mikrotiks gives a fuck on bandwitch test and stared to push traffic generator but there it is not able to test tcp....
- it is interesting that old ubnt hardware M5 has ability to test tcp using iperf and it is precious, if you want to use you must be logged to device using ssh (in web page there is only udp test and this is for nothing, just shows max possible throu... but not real because it doesnt take into account noise...)

- another point, after ros 6.19 bandwitch test for tcp is completely not working correctly, it shows bad throughoutput and mostly sinusoid type

TCP Test has never been working correctly. The CPU-load goes to 100% and is limiting the bandwidth. Often the link ist not the limit. For TCP-Tests I put Raspberries on our towers!

Leo.

- another point, after ros 6.19 bandwitch test for tcp is completely not working correctly, it shows bad throughoutput and mostly sinusoid type

TCP Test has never been working correctly. The CPU-load goes to 100% and is limiting the bandwidth. Often the link ist not the limit. For TCP-Tests I put Raspberries on our towers!

Leo.

exactly this same ideas I had.....
what maximum speeds can raspberry pi generate for you?

For TCP-Tests I put Raspberries on our towers!

exactly this same ideas I had.....
what maximum speeds can raspberry pi generate for you?

wget http://localhost/bigfile.bin -O /dev/null
100%[======================================>] 20.463.616 33,4M/s in 0,6s

- localhost is apache2 and reading from SD card is also limiting; I get the 33MBytes/s after a second call! (first call only 14.8M)

One wireless example: (just wget)
- I get 4.3Mbytes/s over a link of 7km (QRT-Sxt) and NV2 set to 1ms
- (Mikrotik UDP-BW test 56MBit/s)
- I get 2.1Mbytes/s over a link of 7km (QRT-Sxt) and NV2 set to 2ms (sometimes it goes up to 3Mbytes/s)
- (Mikrotik UDP-BW test 80MBit/s)

iperf examples:
iperf -s &
iperf -c localhost
[ 5] local 127.0.0.1 port 5001 connected with 127.0.0.1 port 41550
[ 3] local 127.0.0.1 port 41550 connected with 127.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 403 MBytes 338 Mbits/sec
[ 5] 0.0-10.0 sec 403 MBytes 337 Mbits/sec


It seems nobody from the Mikrotik team is realising the discrepanies between real world TCP traffik and onboard bandwidth tests (or academic traffic generators) , because they have no working TCP test tools in their routers.

M.

Same problem here. We have a dedicated 200mbps line, and a 7km PtP

In nv2, the customer gets about 15-20mbps (unstable) in speedtests. If I run btest with 20 connections, or UDP traffic, I can pass over 110mbps

I've configured the link in nstreme... and test shows about 80mbps (no matter if TCP or UDP) . It's a lot better... but is 1/3 of original bandwidth.

By now, the link seems to be stable. There's netmetal5 in both sides, and nstreme with 20mhz and both chains enabled.

I've tried also replacing it with rocket M5 (same mANT30). 1 TCP connection gives about 30mbps

Conclusion: if you need to give "moderate" internet access (about 10-15mbps per customer), you can go with nv2. We've other link with about 300 customers (nv2/20mhz/ac) and we see traffic over 110mbps daily

If you need to give 50-60mbps to one customer, you must go with nstreme, and not too much hops.

If you need to give more bandwith, you must go to FDD solution.

I think is not a MikroTik problem. With these bandwidths, latency plays against.

Ubiquiti has solved it with FDD links (Airfiber)

MikroTik, why you don't resurrect nstreme-dual? It would be amazing with the new ac cards.

Same problem here. We have a dedicated 200mbps line, and a 7km PtP

In nv2, the customer gets about 15-20mbps (unstable) in speedtests. If I run btest with 20 connections, or UDP traffic, I can pass over 110mbps

I've configured the link in nstreme... and test shows about 80mbps (no matter if TCP or UDP) . It's a lot better... but is 1/3 of original bandwidth.

By now, the link seems to be stable. There's netmetal5 in both sides, and nstreme with 20mhz and both chains enabled.

I've tried also replacing it with rocket M5 (same mANT30). 1 TCP connection gives about 30mbps

Conclusion: if you need to give "moderate" internet access (about 10-15mbps per customer), you can go with nv2. We've other link with about 300 customers (nv2/20mhz/ac) and we see traffic over 110mbps daily

If you need to give 50-60mbps to one customer, you must go with nstreme, and not too much hops.

If you need to give more bandwith, you must go to FDD solution.

I think is not a MikroTik problem. With these bandwidths, latency plays against.

Ubiquiti has solved it with FDD links (Airfiber)

MikroTik, why you don't resurrect nstreme-dual? It would be amazing with the new ac cards.

You cant go with nstreme in many cases as it unstable with light interference.
Bigger sliding windows in newer OSes gives high tcp speed even with higher latency. So even non FDD links should be able to show good performance. We dont see this problem with (expensive) PTP600. It is a matter of implementation. We'll test Mimosa soon to see if it does better.

I think is not a MikroTik problem. With these bandwidths, latency plays against.

I think one main problem of NV2 is the jitter of the latency. TCP frames arrive asynchronly to the NV2 TDMA time slot, sometimes they get through within 1ms mostly within 3ms, but sometimes with 15ms (one out of 100). Maybe - just a idea - it would help to delay the incoming TCP frames of a PtP link by a certain time - so that over the link - the ping time is always 15ms. This would be a big benefit for the TCP timing algorithms in the client computers.

The idea as example: (ROUTER A and B have a synchronised time base t, I write this time count in front of each of the following lines)
(37) - ROUTER A get packet from ethernet interface and measure the packet timestamp: ta=t (ta=37)
(37) - some delay...
(38)- ROUTER A send this packet
( )- wireless link with max delay tmax (e.g. 15)
(42)- ROUTER B get packet and timing information ta
(42)- ROUTER B queue the paket for the time ta+tmax = 52
(42)-ROUTER B queue waits until timestamp reaches t==52
(52)-ROUTER B will send packet out over the ethernet interface

Can this "ping equalizing" be done with current mikrotiks? Can I mark and timestamp packets and send this packet-marks to another router?
Can this be done with linux (e.g. rasyberrys)? Maybe it is sufficient to synchronise both clocks over the links (statistically over a longer time) or use GPS modules with PPS output...

Leo.

I think is not a MikroTik problem. With these bandwidths, latency plays against.

I think one main problem of NV2 is the jitter of the latency. TCP frames arrive asynchronly to the NV2 TDMA time slot, sometimes they get through within 1ms mostly within 3ms, but sometimes with 15ms (one out of 100). Maybe - just a idea - it would help to delay the incoming TCP frames of a PtP link by a certain time - so that over the link - the ping time is always 15ms. This would be a big benefit for the TCP timing algorithms in the client computers.

The idea as example: (ROUTER A and B have a synchronised time base t, I write this time count in front of each of the following lines)
(37) - ROUTER A get packet from ethernet interface and measure the packet timestamp: ta=t (ta=37)
(37) - some delay...
(38)- ROUTER A send this packet
( )- wireless link with max delay tmax (e.g. 15)
(42)- ROUTER B get packet and timing information ta
(42)- ROUTER B queue the paket for the time ta+tmax = 52
(42)-ROUTER B queue waits until timestamp reaches t==52
(52)-ROUTER B will send packet out over the ethernet interface

Can this be done with current mikrotiks? Can I mark and timestamp packets and send this packet-marks to another router?
Can this be done with linux (e.g. rasyberrys)? Maybe it is sufficient to synchronise both clocks over the links (statistically over a longer time) or use GPS modules with PPS output...

Leo.

Exactly, the problem must be jitter. Cambium ePmP has higher latency, but is stable. And it don't have this problem. If latency fluctuates, TCP can't adjust the window size.

You cant go with nstreme in many cases as it unstable with light interference.
Bigger sliding windows in newer OSes gives high tcp speed even with higher latency. So even non FDD links should be able to show good performance. We dont see this problem with (expensive) PTP600. It is a matter of implementation. We'll test Mimosa soon to see if it does better.

Of course PtP600 works. It has rock solid low latency... but is insultingly expensive. Here you can buy a licensed link for half the price of PtP600

We have some iPasolink links, and TCP and UDP speed is exactly the same, using 1 or 100 connections

I think is not a MikroTik problem. With these bandwidths, latency plays against.

I think one main problem of NV2 is the jitter of the latency. TCP frames arrive asynchronly to the NV2 TDMA time slot, sometimes they get through within 1ms mostly within 3ms, but sometimes with 15ms (one out of 100). Maybe - just a idea - it would help to delay the incoming TCP frames of a PtP link by a certain time - so that over the link - the ping time is always 15ms. This would be a big benefit for the TCP timing algorithms in the client computers.

The idea as example: (ROUTER A and B have a synchronised time base t, I write this time count in front of each of the following lines)
(37) - ROUTER A get packet from ethernet interface and measure the packet timestamp: ta=t (ta=37)
(37) - some delay...
(38)- ROUTER A send this packet
( )- wireless link with max delay tmax (e.g. 15)
(42)- ROUTER B get packet and timing information ta
(42)- ROUTER B queue the paket for the time ta+tmax = 52
(42)-ROUTER B queue waits until timestamp reaches t==52
(52)-ROUTER B will send packet out over the ethernet interface

Can this be done with current mikrotiks? Can I mark and timestamp packets and send this packet-marks to another router?
Can this be done with linux (e.g. rasyberrys)? Maybe it is sufficient to synchronise both clocks over the links (statistically over a longer time) or use GPS modules with PPS output...

Leo.

Exactly, the problem must be jitter. Cambium ePmP has higher latency, but is stable. And it don't have this problem. If latency fluctuates, TCP can't adjust the window size.

You cant go with nstreme in many cases as it unstable with light interference.
Bigger sliding windows in newer OSes gives high tcp speed even with higher latency. So even non FDD links should be able to show good performance. We dont see this problem with (expensive) PTP600. It is a matter of implementation. We'll test Mimosa soon to see if it does better.

Of course PtP600 works. It has rock solid low latency... but is insultingly expensive. Here you can buy a licensed link for half the price of PtP600

We have some iPasolink links, and TCP and UDP speed is exactly the same, using 1 or 100 connections

I took PTP600 as example that it can be done with TDD radios. Radwin does not show this problem either. The "cheap" MT und UBNT TDMA implementations show this behavior.

mikrotik 6.38.7 - bugfix only firmware (misbe). udp bandwidth test working correctly (95mbit\sec), but tcp wrong (8mbit\sec). Solution is to start bandwidth test from server side (direction send, mikrotik to which you want calculate tcp speed), monitoring cpu load, mtu size, etc. In this case test working correctly.

Helooo

Same problem via VDSL link.

LINUX -(1G)- BR3011-eoiptunnel -(VDSL)- eoiptunnel_RB3011 -(1G)- LINUX

UDP single - fast 218mb/s
TCP single - slow 33 mb/s

SOLUTION: in LINUX(data source) change TCP congenstion to ILLINOIS
https://en.wikipedia.org/wiki/TCP-Illinois

HOW TO change TCP illinois in Mikrotik ?
Linux: /etc/sysctl.conf:net.ipv4.tcp_congestion_control = illinois
MIKROTIK: ?????????????????????

Thanks Havrla

Hello guys,
I approve this issue too in v6.44.6 long-term on both x86 and Mikrotik 750G r3
UDP : ~950mbps
TCP: ~512mbps
I hope this could be solved as soon as possible

Try to enable a mp-pfifo for the wlan-interface. Try it first at the Client - this improved my tcp-download-rate dramatically. (however you can also add this to the accesspoint). (mq-pfifo is available on newer releases only).

Leo.

Try to enable a mp-pfifo for the wlan-interface. Try it first at the Client - this improved my tcp-download-rate dramatically. (however you can also add this to the accesspoint).

Code: Select all

/queue type
add kind=mq-pfifo mq-pfifo-limit=200 name=mq_pfifo
/queue interface
set wlan1 queue=mq_pfifo

(mq-pfifo is available on newer releases only).

Leo.

Unfortunately, this didn't do any improvement in my case, I use ethernet ports and multiple MT routers, and the problem is between 2 Mikrotik routers.