Simple queue going over max-limit

mikegleasonjr · Thu Jun 02, 2022 10:18 pm

Hi,

Something really weird is happening. Fasttrack is disabled. The simple queue works fine for all traffic except when there's a steam download.

Whenever I do speed tests (even by multiple clients at once) or I download large files, the queue barely go over its max limit. But when I start a steam game download, the queue overshoots almost always and sometimes up to 18 mbps (I have a 100mbit connection, so it's almost up to 20%). When I go to the connections tab of the firewall, I see that steam opens up multiple TCP connections. They're not fasttracked.

What's odd and indicates a bug, sometimes when I play the queue settings, it stop overshooting. But when I reboot the router (RB4011), it starts happening again.

My wan is pppoe over a VLAN on a fiber ONT (bridged) with the Mikrotik router.

In the screenshot below, you can see I lowered the max-limit of the queue to 80M but still my pppoe traffic overshoots to 96M.

Here's my full config:


/interface bridge
	add admin-mac=XX:XX:XX:XX:XX:XX auto-mac=no frame-types=admit-only-vlan-tagged name=bridge1 vlan-filtering=yes

/interface vlan
	add comment=lan interface=bridge1 name=vlan10 vlan-id=10
	add comment=guest interface=bridge1 name=vlan20 vlan-id=20
	add comment=voip interface=bridge1 name=vlan30 vlan-id=30
	add comment=cams interface=bridge1 name=vlan40 vlan-id=40
	add comment=iam-wan interface=bridge1 name=vlan881 vlan-id=881

/interface pppoe-client
	add add-default-route=yes disabled=no interface=vlan881 max-mru=1492 max-mtu=1492 name=pppoe-iam user=*****

/interface list
	add name=WAN
	add name=LAN
	add name=GUEST
	add name=VOIP
	add name=CAMS

/ip pool
	add name=vlan10 ranges=*********
	add name=vlan20 ranges=*********
	add name=vlan30 ranges=*********
	add name=vlan40 ranges=*********

/ip dhcp-server
	add address-pool=vlan10 interface=vlan10 lease-time=1d name=vlan10
	add address-pool=vlan20 interface=vlan20 lease-time=1d name=vlan20
	add address-pool=vlan30 interface=vlan30 lease-time=1d name=vlan30
	add address-pool=vlan40 interface=vlan40 lease-time=1d name=vlan40

/queue type
	add cake-flowmode=dual-srchost cake-nat=yes cake-overhead=36 kind=cake name=cake-out
	add cake-diffserv=besteffort cake-flowmode=dual-dsthost cake-nat=yes cake-overhead=36 cake-wash=yes kind=cake name=cake-in

/queue simple
	add bucket-size=0.002/0.002 max-limit=80M/47500k name=wan queue=cake-in/cake-out target=pppoe-iam

/interface bridge port
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether1
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether2
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether3
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether4
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether5
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether6
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether7
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether8
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether9
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether10
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=sfp-sfpplus1

/ip neighbor discovery-settings
	set discover-interface-list=LAN

/interface bridge vlan
	add bridge=bridge1 tagged=bridge1,ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9,ether10,sfp-sfpplus1 vlan-ids=10
	add bridge=bridge1 tagged=bridge1,ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9,ether10,sfp-sfpplus1 vlan-ids=20
	add bridge=bridge1 tagged=bridge1,ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9,ether10,sfp-sfpplus1 vlan-ids=30
	add bridge=bridge1 tagged=bridge1,ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9,ether10,sfp-sfpplus1 vlan-ids=40
	add bridge=bridge1 tagged=bridge1,ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9,ether10,sfp-sfpplus1 vlan-ids=881

/interface list member
	add interface=pppoe-iam list=WAN
	add interface=vlan10 list=LAN
	add interface=vlan20 list=GUEST
	add interface=vlan30 list=VOIP
	add interface=vlan40 list=CAMS

/ip address
	add address=******* interface=vlan10 network=******
	add address=******* interface=vlan20 network=******
	add address=******* interface=vlan30 network=******
	add address=******* interface=vlan40 network=******

/ip cloud
	set ddns-enabled=yes

/ip dhcp-server network
	add address=***** dns-server=******** gateway=******** ntp-server=*******
	add address=***** dns-server=******** gateway=******** ntp-server=*******
	add address=***** dns-server=******** gateway=******** ntp-server=*******
	add address=***** dns-server=******** gateway=******** ntp-server=*******

/ip dns
	set allow-remote-requests=yes cache-max-ttl=0s servers=********

/ip firewall address-list
	add address=******* comment=asterisk list=VOIP-WHITELIST
	add address=******** comment=traefik list=DSTNAT-WHITELIST
	add address=******* comment=scale list=DSTNAT-WHITELIST

/ip firewall filter
	add action=accept chain=input connection-state=established,related,untracked
	add action=drop chain=input connection-state=invalid
	add action=accept chain=input protocol=icmp
	add action=accept chain=input dst-port=13231 protocol=udp
	add action=accept chain=input dst-address=127.0.0.1
	add action=accept chain=input dst-port=123 in-interface-list=!WAN protocol=udp
	add action=jump chain=input dst-port=53 jump-target=dns protocol=tcp
	add action=jump chain=input dst-port=53 jump-target=dns protocol=udp
	add action=drop chain=input in-interface-list=!LAN
	add action=accept chain=forward connection-state=established,related
	add action=accept chain=forward in-interface-list=LAN
	add action=accept chain=forward in-interface-list=GUEST out-interface-list=WAN
	add action=accept chain=forward in-interface-list=VOIP out-interface-list=WAN src-address-list=VOIP-WHITELIST
	add action=accept chain=forward connection-nat-state=dstnat dst-address-list=DSTNAT-WHITELIST in-interface-list=WAN
	add action=drop chain=forward
	add action=drop chain=dns in-interface-list=WAN
	add action=drop chain=dns in-interface-list=CAMS
	add action=drop chain=dns in-interface-list=VOIP src-address-list=!VOIP-WHITELIST
	add action=accept chain=dns

/ip firewall nat
	add action=masquerade chain=srcnat comment=wan ipsec-policy=out,none out-interface-list=WAN
	add action=dst-nat chain=dstnat comment=******* dst-port=**in-interface-list=WAN protocol=tcp to-addresses=***** to-ports=**
	add action=dst-nat chain=dstnat comment=******* dst-port=** in-interface-list=WAN protocol=tcp to-addresses=***** to-ports=**
	add action=dst-nat chain=dstnat comment=******* dst-port=** in-interface-list=WAN protocol=tcp to-addresses=***** to-ports=**
	add action=dst-nat chain=dstnat comment=******* dst-port=** in-interface-list=WAN protocol=udp to-addresses=***** to-ports=**

/system clock
	set time-zone-autodetect=no time-zone-name=********

/system identity
	set name=rb4011

/system ntp client
	set enabled=yes

/system ntp server
	set enabled=yes manycast=yes

/system ntp client servers
	add address=*******
	add address=*******
	add address=*******
	add address=*******

/system package update
	set channel=testing

/tool bandwidth-server
	set enabled=no

/tool mac-server
	set allowed-interface-list=LAN

/tool mac-server mac-winbox
	set allowed-interface-list=LAN

ivicask · Thu Jun 02, 2022 10:30 pm

viewtopic.php?p=931315#p931315

I would also like some answers on this but even Mikrotik support I contacted few times about this never gave any useful info..

mikegleasonjr · Thu Jun 02, 2022 10:47 pm

viewtopic.php?p=931315#p931315

I would also like some answers on this but even Mikrotik support I contacted few times about this never gave any useful info..

wow ok I thought I was crazy. Thing is, I am glad we finally narrowed down something that is happening (IMHO) for a long time. Probably a bug in the queues.

While fiddling with the settings, sometimes it starts to shape the traffic correctly. It happened two times over the course of two days for me. Then I revert all my changes and go back to the initial config. Then everything keeps working fine, until I reboot the router.

The common denominator I see is that we're on pppoe over fiber.

ivicask · Thu Jun 02, 2022 10:52 pm

viewtopic.php?p=931315#p931315

I would also like some answers on this but even Mikrotik support I contacted few times about this never gave any useful info..
wow ok I thought I was crazy. Thing is, I am glad we finally narrowed down something that is happening (IMHO) for a long time. Probably a bug in the queues.

While fiddling with the settings, sometimes it starts to shape the traffic correctly. It happened two times over the course of two days for me. Then I revert all my changes and go back to the initial config. Then everything keeps working fine, until I reboot the router.

The common denominator I see is that we're on pppoe over fiber.

I have 100% same issue as you show in screen and I noticed this issue over year now and it's super easy reproducible by almost any major game downloaders like Steam, Epic games, Adobe programs etc..

I did very dirty and very complicated workaround by making queue which matches such downloaders with multi connections and then I set speed almost half from desired one.

So queue is set to 30mb in order to actually download at 60mbs and not to kill my internet..

mikegleasonjr · Thu Jun 02, 2022 11:23 pm

I did very dirty and very complicated workaround by making queue which matches such downloaders with multi connections and then I set speed almost half from desired one.

That's true I noticed it does the same thing with the Epic Games client.

What's the general route you've taken ? You're marking packets coming from IPs you're constantly keeping an eye to every time you start a download ?

ivicask · Thu Jun 02, 2022 11:30 pm

I did very dirty and very complicated workaround by making queue which matches such downloaders with multi connections and then I set speed almost half from desired one.

That's true I noticed it does the same thing with the Epic Games client.

What's the general route you've taken ? You're marking packets coming from IPs you're constantly keeping an eye to every time you start a download ?

Very big list of adresses and script which dynamically collects them from dns and adds more + some extra packet marks by connection number + rate limits..

It's very hard to tune and get it to work properly but for now it solved my problems.

Zoxc · Sun Jun 05, 2022 2:42 pm

Do you see the same behavior when using CAKE's shaper? That should be preferred if you're matching a slow ISP link. Make the simple queue unlimited and enter your bandwidth in the CAKE queue types to give it a try.

ivicask · Sun Jun 05, 2022 3:19 pm

Do you see the same behavior when using CAKE's shaper? That should be preferred if you're matching a slow ISP link. Make the simple queue unlimited and enter your bandwidth in the CAKE queue types to give it a try.

70mbit isn't exactly slow link, and queue doesnt matter, it completly chokes dsl line and triggers bufferbloat. I did try setting cake with limit, and it still doesn't respecet even that limit and goes way over..

Zoxc · Sun Jun 05, 2022 7:15 pm

I'd like to see a bandwidth measurement from the device doing the downloading too, to see if the queue works, without relying on measurements on the router.

ivicask · Sun Jun 05, 2022 7:24 pm

I'd like to see a bandwidth measurement from the device doing the downloading too, to see if the queue works, without relying on measurements on the router.

If I run speedtest or simple driver download from nvidia it's respecting set limit exactly as set in queue as we mentioned already.

mikegleasonjr · Tue Jun 07, 2022 2:43 pm

I'd like to see a bandwidth measurement from the device doing the downloading too, to see if the queue works, without relying on measurements on the router.

See previous answer.

But also the fact that my buffer bloats shoots up tells me the measurements on the router are accurate.

mikegleasonjr · Tue Jun 07, 2022 2:45 pm

Do you see the same behavior when using CAKE's shaper? That should be preferred if you're matching a slow ISP link. Make the simple queue unlimited and enter your bandwidth in the CAKE queue types to give it a try.

Cake comes later in the equation, so if the queue gives too much packets already (more than your internet connection speed), cake can't do nothing about it. It improves things but still buffer bloats shoots up because more packets are pushed in a pipe that can't handle it

fragtion · Tue Jun 07, 2022 2:48 pm

I started the other thread (linked in post 2). Unfortunate that there are duplicate threads now but on the bright side it's a good testament that the bug is becoming more noticed and prevalent and should be taken more seriously by support.

Besides for game downloaders as mentioned by @ivicask, another easy way to trigger this is with torrents. For some reason torrents easily exceed mikrotik qos as well.

I created a support ticket about this issue, SUP-81506, but the support assistant (Arturs C) requested a supout which has stalled the ticket (I don't see why an elaborate supout is needed to reproduce an issue that is well described and attested on the forums... I've already wasted enough time on this issue and am sure that the supout won't reveal anything of interest so I haven't bothered to do that yet)

If someone has time to reproduce this bug on a clean config and send evidence to mikrotik support along with the pesky supout file which they refuse to proceed on the ticket without, then that would be great. I don't have the time right now but if I manage to do this in future I will update this post accordingly.

At first I thought it was incoming UDP traffic causing the problem, but it happens with TCP also. It would be helpful to test and rule out if this happens only with pppoe over ethernet, or if it is a general problem with qos which could affect any connection type

ivicask · Tue Jun 07, 2022 3:10 pm

I started the other thread (linked in post 2). Unfortunate that there are duplicate threads now but on the bright side it's a good testament that the bug is becoming more noticed and prevalent and should be taken more seriously by support.

Besides for game downloaders as mentioned by @ivicask, another easy way to trigger this is with torrents. For some reason torrents easily exceed mikrotik qos as well.

I created a support ticket about this issue, SUP-81506, but the support assistant (Arturs C) requested a supout which has stalled the ticket (I don't see why an elaborate supout is needed to reproduce an issue that is well described and attested on the forums... I've already wasted enough time on this issue and am sure that the supout won't reveal anything of interest so I haven't bothered to do that yet)

If someone has time to reproduce this bug on a clean config and send evidence to mikrotik support along with the pesky supout file which they refuse to proceed on the ticket without, then that would be great. I don't have the time right now but if I manage to do this in future I will update this post accordingly.

It will be a great day when this issue is fixed... nothing more frustrating than QoS failing dismally to do what it was implemented for.

I also gave them supout and it was useless, i gave them full details and easy way to reproduce the issue, but they didint care much i think.

BTW i have no issue with torrents, its 100% respecting limits for me, torrents use multiple connections with different ports while this problematic downloaders are opening multiple connections to same port like 443, 80 or 8080..

fragtion · Tue Jun 07, 2022 3:30 pm

BTW i have no issue with torrents, its 100% respecting limits for me.

It could be something to do with the torrent client (local or remote), or the number of seeds? In my case with qBittorrent and a torrent with thousands of seeders, I did experience the issue with torrents too. But that was with a simple queue, not queue tree marking torrent packets, so maybe that changes the outcome somehow for torrents?

ivicask · Tue Jun 07, 2022 3:39 pm

BTW i have no issue with torrents, its 100% respecting limits for me.
It could be something to do with the torrent client (local or remote), or the number of seeds? In my case with qBittorrent and a torrent with thousands of seeders, I did experience the issue with torrents too. But that was with a simple queue, not queue tree marking torrent packets, so maybe that changes the outcome somehow for torrents?

I also tested with qbit, and it opens also 200+ connections but both simple and queue tree are respecting limits for me in this case.

Try lowering bucket sizes to 0.001 and try codel queue with lowered packet and quantum limits, thats how i have maybe that helps you.

fragtion · Thu Jul 07, 2022 5:38 am

I think I've been able to "work around" this issue somewhat, but unfortunately the technique only works for queue tree, not simple queues

Basically the idea is to create a new "Queue Type" with "pfifo" (or bfifo) kind, and set a higher than normal "Queue Size". In my case anything more than around 1000 packets or 5120000 bytes seems to be effective, but I suppose YMMV if this even works for you?

So in my case the main queues are fq_codel, then I assign this custom queue type to all the children queues in the tree, something like this:

looksllikethis.png

My pings have never been more stable under full load...torrents blazing, I can browse or play games very responsively but I was hoping to achieve all of this with a single simple fq_codel or cake queue targeting the pppoe interface so I suppose that will remain a pipedream for the time being

I'm guessing the bug has something to do with not enough packets being processed by the queues by default? Or failing to shape some kind of transport overhead? hopefully more users can chime in with some feedback on this

erkexzcx · Wed Nov 27, 2024 11:13 pm

For anyone stumbling upon this thread - here is the issue: CAKE fails to shape traffic if one device has hundreds of connections and tries to download/upload at max speed.

My ISP is 100M up and 100M down, my torrents server is really putting CAKE to a test lol. When I configure everything correctly - everything works. Start downloading 200-300 torrents without any limits (no limits in any connections, no limits in any amount of torrents etc) and CAKE simply stops working. Take a look at this queue tree:

# Download
/queue tree add bucket-size=0 max-limit=95M name=down parent=br0 queue=cake-download
/queue tree add bucket-size=0 max-limit=95M name=lan-down packet-mark=packet_lan,packet_server parent=down priority=2 queue=cake-download
/queue tree add bucket-size=0 max-limit=95M name=torrents-down packet-mark=packet_torrents parent=down priority=3 queue=cake-download

# Upload
/queue tree add bucket-size=0 max-limit=95M name=up parent=ether1 queue=cake-upload
/queue tree add bucket-size=0 limit-at=60M max-limit=95M name=server-up packet-mark=packet_server parent=up priority=2 queue=cake-upload
/queue tree add bucket-size=0 limit-at=35M max-limit=95M name=lan-up packet-mark=packet_lan parent=up priority=2 queue=cake-upload
/queue tree add bucket-size=0 max-limit=95M name=torrents-up packet-mark=packet_torrents parent=up priority=3 queue=cake-upload

Basically my goal is to give torrents only leftover internet - LAN devices have full internet speed, while torrents gets only leftover. E.g. I download steam game at full speed, and torrents gets literally zero (0 bytes per sec) speed. This is my goal.

This setup works like a charm if qbittorrent downloads 1 file. Not so many connections, maybe 20-30 max - it has no impact, everything works as expected.

Now pump 300 torrents at max speed without any limits and see what happens - any device on the lan gets only 1-20mbps of internet, even tho it has higher priority.......

As a workaround (suggested by comment above), it has something to do with Mikrotik queue settings - maybe overhead is different etc. So turns out, after random experimenting, THIS WORKS:

/queue type add cake-diffserv=besteffort cake-flowmode=dual-dsthost cake-mpu=84 cake-overhead=38 cake-overhead-scheme=ethernet cake-rtt=33m20s kind=cake name=cake-download
/queue type add cake-diffserv=besteffort cake-flowmode=dual-srchost cake-mpu=84 cake-nat=yes cake-overhead=38 cake-overhead-scheme=ethernet cake-rtt=33m20s kind=cake name=cake-upload

Notice the "cake-rtt=33m20s", or in other words "RTT Scheme = none" and "RTT = 2000000" (max I was able to set). I can't believe it works, but it works lol. My internet is again usable on LAN devices.

Sometimes it feels like it doesn't fully work, sometimes works like you would expect CAKE to work, but it's definitely day & night improvement.

EDIT: It works 100% as expected now (with above crazy rtt value). Go to torrent node and set appropriate limits - do NOT disable torrenting limits (torrents, connections etc), because if you have 100mbps up/down connection, you will hardly get any improvement by disabling those limits. Leave them default, and if your download connections are stable - reduce them even more. Not this will result in better networking CAKE and router wise, but also your disks random IO access would be reduced and less stress on your storage (and CPU).

So after all, CAKE isn't that magical (but it's still best thing I have on my Mikrotik router).

EDIT 2: Quote from Mikrotik CAKE documentation on "interplanetary" selection:

interplanetary: This disables Active Queue Management (AQM) actions, because the RTT is so long (3600 seconds). It's named "interplanetary" because the distance from Earth to Jupiter is about one light-hour. Example: This is not typically used in standard networking situations, but might be useful in extremely high latency situations, such as experimental long-distance communication scenarios.

Probably torrents cause SUPER heavy congestion and this is basically the only way to go forward?

Simple queue going over max-limit

Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Re: Simple queue going over max-limit

Who is online