Hello,

See http://forum.mikrotik.com/viewtopic.php ... 55#p520164 for solution

Using RB1100AHx2 with RoS 3.34.
No NAT, just routing with public IPs.

Problem.
Downloads from a server behind a RB1100AHx2 using HTTP on port 80 to another host on the outside are limited to ~15MBps.
Testing is done using a 1.5 GB .rar file.
The weird part is that ONLY HTTP(S) downloads are affected. Both 80 & 443 TCP ports.
Downloads through SSH & FTP are almost wire speed at ~ 910 Mbps
If I test with Ookla mini speed test on the same server, it reports 500+ Mbps download & upload.
I have tried changing the HTTP server from Apache to nginx on port 80, no change.
I have tried changing port from 80 to 8182, no change.
I have tried changing the test server with a Synology NAS connected to the same MT router, no change again.
I have tried stopping all HTTP servers and running iperf3 on port 80 to rule out my ISP throttling some ports. Using iperf3, the speed is ~900 Mbps, so no luck again.
I disabled all firewall rules on MT router. No change.
If I test on the same machines behind the MT router, connected to the same MT router, one on port 2 and another on port 6 (different switches), HTTP downloads are topping 110 MBps


No QoS enabled on either server or MT router. HTTP servers do not have any QoS mods enabled.
No http(s) inspection or other devices between the test servers and MT router.
All interfaces negotiate at 1 Gbps on MT router.
My ISP took a look at the problem and told me they do not limit any port. Cannot be too sure that they actually did, but what can I do.

I am out of ideas.
Is there any setting beside "QoS" on MT routers that can cause such behavior?
Please throw some ideas at me.


A simple flow diagram

This is a problem we can pontificate on forever (and I have a few ideas) but the quickest way and the absolute surest way to see what is going on is with a wireshark trace. Setup the trace as follows:

• *Filter on tcp 80, tcp 443, udp 53, icmp
• *Capture 256 bytes per packet (Header plus a little payload)
• *Shutdown all browser sessions before starting wireshark trace because capturing the three-way handshake is critical for a few things, most notably TCP windowing
• *Start wireshark capture
• *Start http file copy
• *Capture about 50-100MB worth of capture data

Assuming the file isn't your plans to take over the word and none of the IP stuff is sensitive if you drop me a pm with a link to the capture file I can take a look real quick and see what I see (I do this for a living).

probably antivirus software acting as proxy.

Good chance you are right

Hello,

Thank you for the replies and ideas. Will try them and get back with details.

Believe me or not, I spent all day today trying to get to the bottom of this.

Long story short: using Ubuntu 14, or 15 Desktop, HTTP downloads are always full speed.
By full speed I mean ~90-100 MBps, depending on server load at the time.
Even on an Android 6 phone (LG G4) connected to an AC network, I can get ~30 MBps while downloading the same file from the same server.

Now comes the weird and longer story.

On Windows 7 I tried 4 different NIC drivers for my Intel Gigabit adapter. All drivers behaved the same.
Tried all tweaks found on the web and even some I cooked up by myself regarding TCP optimization under Windows 7.
I did a clean install of Windows 7 just to rule out 3rd party software problems. No Antivirus installed. Firewall and Defender deactivated.
I even tried testing on Windows XP, which resulted in speeds of ~10-13 MBps.

To exclude a browser problem, I used wget (under cygwin) to test the speed.
With wget, most of the times the speed would be between 15 & 20 MBps, but, and this is weird, once in a while the speed will jump to ~100 MBps. Consequent retries will return to 15-20 MBps until I will get a lucky break and jump again at ~100 MBps.

Even weirder, downloads over HTTP, like an Ubuntu ISO from my ISP mirror http://ubuntu.mirrors.linux.ro/releases ... -amd64.iso are downloading most of the time at 30-33 MBps and sometimes, jump to ~100 MBps. Why the difference from 15-20 MBps to 30-33 MBps beats me.

From a Linux terminal using wget, the same Ubuntu mirror or my server will provide ~100 MBps speeds all the time.
Under Windows, 99% of the times, the speed will be 30-33 MBps for Ubuntu mirror and 15-20 MBps for my server, with some lucky retries going full speed.
Both servers have 1-2 ms latency from my test location.

This is really frustrating to say the least.
At the first glance, it seems Windows is somehow limiting HTTP downloads while Ubuntu provides full speed.
I have to state again that ONLY HTTP(S) downloads seem to be affected. SFTP or just plain FTP are working full speed under Windows.

I will repeat myself when I say you can play around at the application layer all day long but until you know whats going out the wire you are fumbling around in the dark. 5 minutes with Wireshark will be more informative then a whole day playing around with drivers and OS's.

I will mention one setting to check under the Intel NIC driver and that is "Flow Control". Make sure that is DISABLED! Ethernet flow control plays VERY POORLY with TCP and will sabotage TCP flow control though it would also need enabled on your switch/router and it looks like Mikrotik disables it by default.

I am happy to report that I found the reason for slow HTTP downloads in Windows 7 (Windows 2008 server also seems to be affected)

Straight from the horse's mouth: https://support.microsoft.com/en-us/kb/2675785
In plain english:

If between location A and location B you have very good latency (1-8 ms), you will notice slower than normal download speeds.
I know, crazy, right? One would think, better latency, better speed. Not with a default Windows install.

To fix this, one needs to apply KB2675785 from Microsoft. This KB will not be downloaded automatically and will not appear as an optional update.(as far as I can tell)
Direct link to KB (64 bit version): https://www.inteq.ro/web/files/patches/ ... 85-x64.msu
Or request it by email from Microsoft at https://support.microsoft.com/en-us/hot ... kbln=en-us
I have to mention also that this is a problem on all Windows 7 and Windows 2008 R2 Server installs without the KB.
I have tested on several other computers with Windows 7 and all of them suffer from slow download speeds.

The reason. I think, why so few people notice this problem is the lack of high speed gigabit internet connections available for residential use.
In a server environment, while using Linux, you will not have this problem.

After applying the KB and a restart: Full speed all the time (depending of course on the server load)

Thank you nspitzer for offering your help.

I hope this will help some people not going through all the hoops with netsh and other useless Windows tweaks in vain.

Wow... That's actually a terrible bug that the description totally minimizes the impact of. Thanks for sharing the solution.