Hi. Since Covert lock down, our max download has almost double. Should get a second fiber in two weeks to resolve this problem but wanted to implement a queue tree QOS system to prioritize pings and browsing. Also added speed test as priority 2 to test the effect of the Queue tree. Implemented this on the Wan interface and it does limit the download speed to the users. Speed test data is definite prioritized and function perfect, however pings just get worst. We have a 400Mb/s fiber and can see the traffic coming in stay at 395Mb/s and i am sure this is why the pings suffer. Have to limit the the Max download speed of parent QOS to about 300Mb/s, to see an improvement on the pings, which has the higher priority. Interesting enough, speed test still function perfect, except for higher pings, even at 300Mb/s. I read and tried every possible QOS implementation, but came to the conclusion this is not possible or i am missing something very basic. My feeling is that, though the TCP window sizing would control TCP download speed because packets is dropped, UDP packets will just keep on coming, because there is no way to tel the upstream provider to limit the amount of traffic, coming in on the WAN. Would appreciate if somebody with knowledge could confirm this?

My feeling is that, though the TCP window sizing would control TCP download speed because packets is dropped, UDP packets will just keep on coming, because there is no way to tel the upstream provider to limit the amount of traffic, coming in on the WAN. Would appreciate if somebody with knowledge could confirm this?

This is correct. You can throttle the TCP download by throttling the delivery of the received TCP packets to the clients, so they respond later and the source accommodates to this by lowering the sending rate, but this doesn't work for majority of UDP traffic (QUIC may be one of the possible exceptions, as it implements similar mechanisms to TCP on application level).
As for the pings, are you talking about loss or increase of round-trip delay? Queueing the traffic automatically means latency, and if the upstream ISP uses QoS per customer, without discriminating by traffic type, it is clear that the ping responses wait in the same queue with the rest of the traffic there so when the queue is long, the latency grows.

Hi. Thanks for the response. Yes talking about increase of round trip time and understand what you are saying. Without congestion, pings would normally be around 25ms and would increase to around 80ms if we hit 395 Mb/s. This is however what is confusing me. If i implement QOS at 360Mb/s max download, the prescribed 10% down from 400Mb/s, it will take much longer before hitting 80ms even though we are running at 390Mb/s and if i limit the max download speed to 300mb/s the pings will test 25ms again. Wan will still be around 393Mb/s and lan side 300Mb/s. Doing a speed test which has lower priority than pings, will run up to 40Mb/s, whether limiting max download at 360Mb/s or 300Mb/s, indicating to me that QOS is functioning as expected. If i lower speedtest priority to 8, the speed test drop to around 4Mb/s.

If i implement QOS at 360Mb/s max download, the prescribed 10% down from 400Mb/s, it will take much longer before hitting 80ms even though we are running at 390Mb/s and if i limit the max download speed to 300mb/s the pings will test 25ms again.

I would think that this is just because you prioritize the ICMP traffic also in the download direction (i.e. from the internet to clients), which means you use different queues for it than for the TCP traffic, and because you throttle the TCP one. The ISP's queues are long because they prefer low loss to latency so that they could smooth spikes in the download (from your perspective) traffic without dropping packets (as drops mean retransmissions so more traffic to handle for them). Hence their queues are empty most of the time (except those spikes) as long as your end does the throttling before they'd have to.

Yes do agree, but i have to drop around 100Mb/s of 400Mb/s data to be able to start controlling this. Have watched quite a bit of videos and tutorials on QOS and all indicated around 10% From our discussion, dropping TCP traffic will eventual decrease the download data from the upstream provider. Dropping UDP data will have no effect on this?

Well, I was wondering since the beginning where those "prescribed 10%" come from. Unless these were videos from some event organised by a network vendor (Mikrotik, Cisco, Juniper, whatever), I'd be careful taking them literally, but even if from people knowledgeable enough to be allowed to present on a vendor-organised event, the 10% is an experience from some particular network situation, which may be completely different at your place and time - Covid is locking people at home watching videos on Youtube which uses QUIC which wants to be "a much better TCP so it has to use UDP to avoid the TCP stacks doing it all wrong" or doing videoceonferences, some of which may also use UDP (never bothered to check while in a videoconf myself, but e.g. Skype for Business does prefer UDP where it gives better results - you may guess what are the network settings when it does ).

And as said, no matter how you throttle the "ordinary" UDP (i.e. one which doesn't carry some application protocol inside, which adjusts the sending rate according to some feedback, like QUIC does) at your side, it will not decrease the source rate, so the queues at the ISP will still be filled with that UDP traffic. So use a mangle rule at the top of the prerouting chain to just count download UDP traffic:
/ip firewall mangle print chain=prerouting where !dynamic
/ip firewall mangle add place-before=0 chain=prerouting action=passthrough in-interface=your-wan-interface-name protocol=udp
Then, let it show the traffic volume every 100 seconds:
/ip firewall mangle print stats interval=100s where chain=prerouting protocol=udp action=passthrough
And then calculate the delta between each pair of subsequent printouts, take an average, divide it by 100 (seconds) and mulitply it by 8 (bits per byte), and you'll get the real UDP download bandwidth, in bits per second, now and here (provided that you throttle the TCP enough so that the UDP is not dropped at the ISP before starting to measure, of course).

FYI, just finished a videoconference on Skype for Business - it ran over UDP as expected.

Thanks.

You are asking a lot in one post. I'll respond to your conversation points.

Have more:
It is always ideal to have a faster router and a bigger pipe to manage incoming packets. If you can, always have more available to you than what will ever be sent to you. If applications don't play by the rules, then it is trivial to overflow your equipment. Imagine what would happen if 1M computers started sending you packets and did not stop even if you ask them to? Imagine if 10M computers did so! Only when senders follow the rules, can you QoS anything.

With that in mind, be sure to read my QoS implementation in my signature. Now, for some tips.

UDP:
It is true that pure UDP traffic, that is exceeding your bandwidth, is a hopeless situation for you. You'll need to buy more bandwidth. If you drop them, they will simply keep coming. Understand that many TCP packets or little UDP packets, that are out to get you, are going to win. You need applications playing by the rules to QoS. Thankfully, more and more UDP traffic is coming from applications designed to be QoS'd. So, in 2020, you can treat some UDP this way. The trick is to identify UDP applications and treat them accordingly. Drop the ones that play nice but don't drop DNS, VoIP and so on. Anything you can't identify, goes in low priority catchall queues. If any traffic type is overflowing your network, and you can not account for it, then you are under attack.

Headroom at 10%:
This is a starting point and is dependent upon your CPU speed, total simultaneous connections, and total bandwidth. If you have thousands of incoming packet flows, the CPU has to loop over them all and bring them under control. It must do this all before the ISP starts buffering (because the router is not processing fast enough). If 10% is not enough headroom for you, you may need a faster router, more bandwidth, or verify why you have so many connections.

Hi PCunite

Thanks so much for the response. Had to read through your original QOS post regarding this and in the end i made changed to my QOS in according yours and bharrisau testing and setup in post 139. Every thing is working flawless now. Have a 400Mb/s fiber that max out at 395Mb/s. With new setup, if max limit is set to 10%, which is around 360Mb/s, my ping stay at 22ms. Pushed the max limit to 370Mb/s which increased ping to 23 ms. This is of cause on the lan side. On the Wan side my speed average around 385Mb/s. I am happy with this results.

That said. I absolute dislike shaping traffic and agree with you having a big enough pipe is the best. Unfortunately have been caught of guard by our country's lock down and subsequent doubling of average max download speeds. Fortunate we did apply for a second fiber in Jan and should be up and running in two to three weeks time.