MikroTik

kevinb361

It was working freaking great for me in simple queue's. Granted, that was with a 100/20 link.

Anyhow.. I changed the interface queue for my WAN interface to cake with a limit of my upload speed. It appears to be working well. With a few quick tests.. my flent tests appear about the same as before.

kevinb361

RB5009 arrived. Here's some brief testing of cake. ISP: Aussie Broadband Technology: Fibre To The Premise (FTTP) Down/Up: 1000M/50M /queue type add cake-diffserv=besteffort cake-nat=yes kind=cake name=cake-default add cake-ack-filter=filter cake-bandwidth=45.0Mbps cake-diffserv=besteffort cake-nat=...

kevinb361

Apologies for the late reply.. I am still tinkering with my brothers setup. Without any queue.. I can get a gig download.. but any kind of queue, it drops in half.. I thought it was the 2.5g port per other peoples responses.. which in fact even without a queue, I cannot get over roughly 500mbit down...

kevinb361

Well, interestingly.. we never upgraded the switch at his place.. and there was a ton of updates in the changelogs.. it was v6.47 or something like that.. anyhow.. all the way up to 7.2rc1 on switch and router. I dunno.. but not too shaby for a quick mid day test.. didn't have much time to get testi...

kevinb361

Just saw this in the changelog for the newly release 7.2rc1:

*) queue - improved system stability when processing traffic;

kevinb361

it looks like mikrotik has lost some data.

Yep, they went down for most of the day yesterday.. it appears they restored a backup of the forum =(

kevinb361

Woohoo! I got it all working as expected! =)

Now I just need to set it up on my brothers end, and just add a cronjob to kick off the test in the middle of the night. But, I gotta run out for a few hours.. so will set that all up later.. hah I guess I will be the cron job at that point! =P

kevinb361

In an effort to take the human error out of my testing, and automation.. I am making some ansible playbooks, and then will work on making flent batch files to use these ansible playbooks to make appropriate config changes before/after each test as needed. Dunno if I am gonna have time to get it all ...

kevinb361

Oh yeah, and yes this is with the multi-queue-ethernet setting on the interface

kevinb361

Well, you shouldn't see that long term growth pattern either. This is after you tuned up the multipath tx/rx thing? What happens with bandwidth down less 20Mbit? Anyway, thx again. I'm packing up for a trip south, (not to mexico! trying to get closer to the spacex launch), and can't look at this ha...

kevinb361

Thx again for helping. Trying to decide on fleeing to mexico or not. Ok, please reload the qdisc(s), leave ecn off, and try again? Were you seeing these lumps before? lumps.png You got no throughput from dallas with ecn. nodnfromdallas.png No problem! Mexico would be nice this time of year! I have ...

kevinb361

OK, I had to redo that test because I messed up. Here is the test

rtt_fair_var with cake and ECN on and then off

https://drive.google.com/drive/folders/ ... sp=sharing

kevinb361

I'd appreciate another capture from your brothers box, of rtt_fair, blowing up, with ecn enabled. also it's easier to look at this stuff in tcptrace/xplot if you just capture those flows. tcpdump -i the_interface -s 128 -w the_capture host dallas or host sydney or host ... thx! Can do! Do you want ...

kevinb361

don't celebrate too soon. :) And you mean cake memlimit or physical memory? Is the ack-filter on on egress? Again, given my still held doubts on having the offsets right for dscp, ecn, and that, having it on may do bad things, but it's very useful on asymmetric connections if working. https://blog....

kevinb361

Another note.. not sure the interface queue change made any difference for the ring queue, BUT.. it APPEARS that it does in fact allow for better threading across the cores!

kevinb361

Another note I have noticed throughout my testing on my own DSL, and my brothers cable.. but definently more so on his.. I also see this 'crash' or whatever it is when you set the bandwidth limit TOO LOW on the egress!!

kevinb361

QUICK NOTE!! I went back and added memory to my brothers router.. and it runs cake without crashing or whatever it was doing.. WITH bandwidth on ingress AND egress! My gut feeling now is that the default memory needs to be increased.. I just didn't see it crashing on my end.. maybe because I am at 1...

kevinb361

Oh, and I do have another router.. it is a CCR1009. The one I was running at the very beginning before we started doing science. It is a 9 core 1.2Ghz TILE processor with 2GB RAM.

kevinb361

OK, I forgot to mention.. this morning I had done some testing with ECN again.. with it set to 1 on the host, the upload was a big sync'd wave.. crazy looking.. going from 2-4mb in one big wave on the RRUL test However, with it set to 2, it seemed normal. Again, atleast the RRUL test. I just set it ...

kevinb361

The behavior of multiple queues in series is kind of complex. Theorists like very much to think about things in terms of a fountain of water, but the real world is batchy in so many respects. Take packets hitting the rx ring. A batch arrives and the ring was nearly full in the first place. A whole ...

kevinb361

I imagine routerOS has no way to see or increase the rx ring? Linux uses "ethtool" to see that. OK, it does not appear from digging through the interface or through the documentation that I can change the ring buffer. Now, if I was running a CHR I could.. since it is RouterOS running on t...

kevinb361

so this last one had 200MB on ingress? Dang. I gotta point at available queue space at the provider, or a limited rx ring, (or that bug with bursty failures) to explain a failure to improve here. Yes sir, 200 on both. Well that is a bummer! I was getting excited! haha but honestly.. the internet is...

kevinb361

Also, another note.. I see the tails on the CDF plot have also made a huge improvement!

kevinb361

Moah! Moah! 8x more! You have the memory to burn. (when we developed cake, *32MB* of ram in the router was a lot) I tried to explain the "default" calculation had some overheads in it that didn't make as much sense on inbound shaping as out. I can try to explain that better.... Ahhh, so c...

kevinb361

see how the drops are sync'd on the down? Shouldn't happen. Up the memlimit? or it's the rx-ring. Or gamma radiation from Mars. BWHAH gamma radiation! Ahh, I did not notice them sync'd. Interesting! OK, so looking at the config the Memory Limit is 0 aka default. Reading the documentation, it states...

kevinb361

looks like perfection to me. rtt_fair? :P :P :P OK, work is not bad! WOO! =) This is with the same settings as last post, with cake.. running rtt_fair_var -- It seems to my untrained eyes it is pretty fair on the upload, but going hog wild on download to dallas! rtt_fair_var_-_cake_rtt_fair.png rtt...

kevinb361

fqcodel_dl.png despite this being better, it appears to my eye that you were running out of queue on the down due to the synchronized drops - which could be hitting a limit at the provider or... is there a 1000 packet limit or memory limit? Cake scales this correctly for you on the down, or should....

kevinb361

Well, I feel pretty confident with my DSL config now. As a recap, it is a 100/20 DSL which is synced at 110/22 per the DSL modem. After some playing around this morning, the best compromise to my eyes at seems like it likes to have the upstream set at the sync, 22mb. I can lower that, but only gain ...

kevinb361

Without ack filtering it is extremely difficult to achieve full download speeds at a 15x1 ratio of down to up or worse. Also rx rings need to be properly sized, as docsis is bursty. A rx ring of 256 is too small. Don't know if you can change that. i wish more folk were taking packet captures of the...

kevinb361

Been following along and don't have too much to add other than it's been very interesting and informative to read this discussion! Regarding your speed issue, I wonder if it's this issue that was mentioned by another user in a different topic? They have a 5009 and a 2.5Gb modem as well it looks lik...

kevinb361

Yeah, I noticed those interruptions or whatever is causing it as well. I am having a heck of a time with this link. I had my brother call spectrum today and verify that his modem is JUST a gateway.. else make sure it is in bridge mode. It is in fact just a gateway. It is a DOCSIS 3.1 modem with a 2...

kevinb361

thank you so much for sharing your raw flent.gz files and packet captures. So many things in this world cannot be captured by a single number, a summary plot, and while a cdf might hint at a problem, looking at a system's evolution, over time, is always helpful. The explanation for why we saw this ...

kevinb361

Well, that grouped bifurcation shouldn't be happening in that way. fq-codel suffers from the birthday problem where you get a hash collission sqrt(1024), so at 32 flows it's likely you'd see 2 flows colliding and getting different behavior from the rest. Cake uses a 8 way set associatve hash so you...

kevinb361

How much memory does this router have? And if there's a way to, say, double the packet and memory limits on the fq_codel rtt_fair test on your home machine maybe those sync'd drops would go away. I didn't see those options in the gui... a lot of people patch down the 10000 packet limit and 32MB lim...

kevinb361

turn off ecn on your brothers link? I assume you have 2 or more hardware queues on the vm? There is only one software queue on that VM, fq_codel.. but who knows what proxmox might be doing.. I know uplink from that server is 10gbit I have been running some test now that the link is quiet on their e...

kevinb361

Let me tackle the download portion of the test. :rant: *nobody* for some reason, tests up and downloads and ping simultaneously, as if people just sat there, did an upload, waited, then did a download, and then did a ping. It's a really bothersome aspect of almost all the web tests today. Real traf...

kevinb361

cake on the edgerouter: https://community.ui.com/questions/Cake-compiled-for-the-EdgeRouter-devices/fc1ff27c-f321-4344-8737-fcc755cae8a2 cake on the udm pro: https://github.com/fabianishere/udm-kernel The whole bufferbloat project is full of hackers desperate to have low latency bandwidth and willi...

kevinb361

The download component of your test looks a touch odd to me, I asked above what it was set to. Also the --te=upload_streams parameter has no function on the rtt_fair tests, they generate one stream per -H server option. Here's where fq-codel begins to pull ahead of SFQ in a couple respects. Your ba...

kevinb361

To verify - presently you have cake 100mbit on the download, and were varying the upload qdisc? And when you tested "the bare modem" both were off? cake 100 on download, and NOT varying the upload, it was set at 19M both were off when testing the bare modem Screenshot from 2021-12-14 17-2...

kevinb361

I have since setup whatever ubiquiti's default simple queue is on their USG.. I believe it is fq_codel? He is amazed, and his facetime video is super clear. He is on a 25/5 cable modem so that made an enormous improvement for them. As he said, I can stream netflix and game at the same time now! heh...

kevinb361

here is the final rtt_fair_var with fq_codel Data: https://drive.google.com/drive/folders/1sUDn9N9svEyGdefCmJlYyJebBgwwWt-U?usp=sharing rtt_fair_var_-_fqcodel_rtt_fair_totals.png rtt_fair_var_-_fqcodel_rtt_fair_cdftotal.png rtt_fair_var_-_fqcodel_rtt_fair_4.png rtt_fair_var_-_fqcodel_rtt_fair_8.png ...

kevinb361

To restore your eyeball to what the current "real world" looks like for everyone else, try that rtt_fair test with all this fancy schmancy stuff off, just the default fifo on the modem. You situation is different than that 2013 demo in that you have a vastly shorter queue than the 250+ms ...

kevinb361

since your eye is now "trained" for a fairly short rtt, try fremont.starlink.taht.net or london,singapore, or sydney .starlink.taht.net we also have tests for these competing against each other, as in the usual case we are not sending flows to a single server. SFQ will start to underperfo...

kevinb361

The FQ component of fq_codel, cake, and fq-pie has what we call the "sparse flow optimization". Request/response (DNS, syn, syn/ack) the first packet of any new flow, acks, voip, gaming, packets, usually "fly through" without observing any queuing at all. In this example we have...

kevinb361

since your eye is now "trained" for a fairly short rtt, try fremont.starlink.taht.net or london,singapore, or sydney .starlink.taht.net we also have tests for these competing against each other, as in the usual case we are not sending flows to a single server. SFQ will start to underperfo...

kevinb361

@kevinb361 the ecn result is very disturbing. But it could be mikrotik (a checksum failure or parsing the wrong bits on this encapsulation, which was a bug that I can't remember when we fixed in some release of linux and cake), the modem, the path, something at linode, where my server is. Anyway, f...

kevinb361

@kevinb361 the ecn result is very disturbing. But it could be mikrotik (a checksum failure or parsing the wrong bits on this encapsulation, which was a bug that I can't remember when we fixed in some release of linux and cake), the modem, the path, something at linode, where my server is. Anyway, f...

kevinb361

@kevinb361 I was up very late yesterday and will sleep soon. I can live with not knowing ecn works before I wake.:) thx again for going to town on this and making such "interesting" mistakes. It's all data to me, and I think the bug you had on the xanwhatever itwas kernel was rather inter...

kevinb361

ok, so if you could don a fire retardant suit, re-enable ecn, and retry cake, and if that looks substantially similar, retry fq-codel? AHAHAH roger that.. here we go! Umm.. well, the results are different.. o.O CAKE with ECN=1 Data: https://drive.google.com/drive/folders/1VErDliJTrxvJbZrj1EYb7TIk42...

kevinb361

Well, don't do that then. :O ip is big-endian.... but a good test of fq-codel with ecn disabled would comfort me, first. There should be differences in the overall distribution particularly in the 32 flows test... but throughput should stay flat, not that horrible thing that just happened.... HAH, ...

kevinb361

I just had an oh crap moment.. before this thread, I had been been toying around with other stuff, and just remembered I had put this in place.. I would imagine this has had some sort of affect in between client and router.... Firewall mangle rule in mikrotik.. ;;; Set priority for WMM chain=postrou...

kevinb361

really large string of wtf moments, there. Can you return to cake? or turn off ecn? or both? your mq - fq-codel might explain some other things, but not this. Roger that, back to cake net.ipv4.tcp_ecn = 0 Data: https://drive.google.com/drive/folders/1NJsq2LAfEgouLkrhzdbqEj3i564vn_IY?usp=sharing tcp...

kevinb361

Random thought.. I didn't answer your question earlier.. I do have qdiscs on this machine.. fq_codel in fact.. here is what is there by default.. qdisc mq 0: dev enp11s0 root qdisc fq_codel 0: dev enp11s0 parent :c limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn ...

kevinb361

And here are the FQ_Codel with ECN tests and packet capture: https://drive.google.com/drive/folders/1NZtfFnJ0vL1Vtu79yALaIn50NxftiXns?usp=sharing tcp_nup_-_fq_codel_ecn_totals.png tcp_nup_-_fq_codel_ecn_4up.png tcp_nup_-_fq_codel_ecn_8up.png tcp_nup_-_fq_codel_ecn_16up.png tcp_nup_-_fq_codel_ecn_32u...

kevinb361

OK, I am going to post SFQ and FQ_Codel separately so it doesn't get confusing.. Link to packet capture of 16 flows SFQ with ECN: https://drive.google.com/drive/folders/1FV-VK1RR6TRwjD21hp5BdSFmSooIg-pe?usp=sharing FQ test coming up in a few tcp_nup_-_sfq_ecn_total.png tcp_nup_-_sfq_ecn_4up.png tcp_...

kevinb361

OK, it's back up. ECN neg is enabled (but the bits could be getting washed out on the path, OR I'd disabled it on the previous boot). To go to your BBR vs cubic question. :lecture mode: TCP reno was the "internet standard" for a long time. It had a "sawtooth", and an initial win...

kevinb361

To summarize a few things. Yesterday we ended up in a state where a bunch of flows weren't even going through the host at the right rate, so we weren't stress testing the qdisc, and thus not seeing any difference in latency between the three different qdiscs under test. It was seeing SFQ act the sa...

kevinb361

Go ahead and do what you need to do. I need to go eat dinner myself. I will check back here before any further testing!

kevinb361

Thank you for the packet capture. You can, btw, filter out all your other traffic by specifying "host dallas.starlink.taht.net" This is the correct sort of carnage that cubic does, there's retransmits, dup acks, out of order stuff - strangely comforting after puzzling over that last captu...

kevinb361

Once again, I forgot the capture for that last round with ECN = 1 HAH! Here ya go

https://drive.google.com/drive/folders/ ... sp=sharing

kevinb361

That's MUCH more correct looking, thank you! Next, to see if ecn is working properly, you can run the exact same test series, but with: sudo sysctl -w net.ipv4.tcp_ecn=1 I use ecn primarily as an AQM debugging tool (given how rarely it's turned on in the field) and for all I know (without seeing th...

kevinb361

My brain is fried.. forgot to add the packet capture..

https://drive.google.com/drive/folders/ ... sp=sharing

kevinb361

Whew, it has been a long day. First thing this morning, I figured.. to heck with that kernel.. well I blew the box up, wouldn't boot into X.. so anyhow, decided to just blow it away and start with a clean install instead of futzing with Linux. I needed to get to work ;) Other than that, whew I havn'...

kevinb361

I hate bugs. :/ Anyway, a packet capture of the 16 flow test would be good at this point. tcpdump -i your-interface -s 128 -w 16flowscake.cap We'd never tested bonding until today... and I could imagine us having a lot of packet reordering in a variety of ways. Assuming this is a bug that isn't in ...

kevinb361

also, 8, 16, 32 with SFQ?

Here ya go!

tcp_nup_-_sfq_32up.png

tcp_nup_-_sfq_32up1.png

tcp_nup_-_sfq_32up2.png

tcp_nup_-_sfq_32up3.png

tcp_nup_-_sfq_32up4.png

tcp_nup_-_sfq_32up5.png

kevinb361

I'd wanted a tcp rtt plot for the 4up test also. You can recreate my cdf if you like comparing the sfq vs cake vs fq-codel.

Yes sir, here ya go.. The plot thickens! =P

tcp_nup_-_sfq_4up2.png

tcp_nup_-_sfq_4up3.png

kevinb361

I'm off researching kernel versions. NOT relevant to this was the wireguard patch that went into 5.7. https://github.com/dtaht/sch_cake/issues/141#issuecomment-984503893 If you have a mikrotik account (I am not a mikrotik customer), and can file a bug, I'm a bit concerned. I wouldn't mind, however,...

kevinb361

I am pretty sure you have the overhead right at this point. I'm also happy to see it not crash. In the interest of science, however, if at some point you could also repeat the 4up test with htb + fq_codel, that would be interesting. Also if you were to enable ecn for a fq_codel vs cake comparison o...

kevinb361

I'm kinda hoping this is a bug in flent!!!

Just for the heck of it.. I added some more data here.. I added 8 and 32. It looks like even with 8 it starts to drop.. and gets worse with more, however 16 and 32 are roughly the same.

kevinb361

as for the up16 anomaly, try htb + fq_codel... And at some point, when your gf is not looking, reboot and try cake again at up16? I return to my initial objective, not crashing. This is 5.6.x? cpu arch? Hmm, well after a fresh reboot.. the results are the same for the up16 and cake. This is odd. Pe...

kevinb361

no, I didn't notice. 19 makes my head hurt less for now? In general dsl tends to fluxuate in rain, over the course of a day, etc, so leaving yourself headroom is a good idea. Good point, not to mention I thought about it afterwards.. even thought the sync rate is 22, the ISP is obviously holding me...

kevinb361

by "scraping the rate" I meant rolling some sort of script to pull it off the modems sync rate, but since your isp is shaping you instead, stick to the 19. Ahh ok, gotcha! Well, that is the interesting thing not sure you noticed that last go around I had set the rate to 22 to match the ra...

kevinb361

your 16up result seems kind of anomalous.

Agreed, I just ran it again.. and got similar results as before. Almost identical, the speed is wonky, however latency is still great.

kevinb361

OK, I believe I have the framing sync'd up pretty well. To quote moeller0 above, he is spot on with the overhead of 44 and also spot on with the fact that VDSL2 likey having a lower overhead than 44. Also, to note.. I have the bandwidth limit set to 22M, which is the bonded upstream rate of the mode...

kevinb361

I think she'll be happy with your efforts so far. I have given up asking her how the internet is doing.. she is very binary. It either works or it doesn't. AHAHA!!! Only thing I could do to make her happier is move the AP on that side of the house into her room so she has a better signal from her d...

kevinb361

Is it possible to scrape that rate? cake supports dynamically changing it's config *without* reloading the qdisc, but I doubt mikrotik can do that with their api (?) tc qdisc change dev whatever cake bandwidth the_new_bandwidth. You should be able to get really close to the actual uplink rate (22xx...

kevinb361

You don't have hw flow control. Nice to know (I guess) that BBR2 still struggles with itself. Try resetting that to cubic on the up, please, and shape to 19 add ack-filter to the up I'm running cubic on that server for the down. Your baseline rtt might drop in half without bonding OR if you can dis...

kevinb361

i do dream of hardware flow control, so no shaper, bandwidth=0 for cake as a tcp_nup test. But i expect to be unlucky. Anyway, your fiddling with the frame parameters without a cake shaper active should have done nothing (I think), so that run was puzzling... cake nat besteffort the_right_dsl_optio...

kevinb361

OK. 0) Still mostly very happy it doesn't crash. 1) Your dsl device's buffer is sized in packets, not bytes. The reason we only saw a 20ms RTT before on the rrul test, vs a vs the tcp-nup test being so much larger RTT, is that the acks from the return flows on the path filled up the queue also. I l...

kevinb361

Does that VDSL device do hardware flow control? Or are you shaping via cake via htb? (I'm happy to hear the bandwidth=0 parameter seems to be working otherwise?), but the only way I can think of you getting results this good is if the vdsl modem is exerting flow control.... Anyway, your last result...

kevinb361

OK, ok, I gave in, in order to do science, could you also try a tcp_nup with upload_streams=4? and =16? The Test 1 *appears* to show an old issue raising it's head - tcp global synchronization - the amount of queue is so short that all the flows synchronize and drop simultaneously, as per panel 3 o...

kevinb361

Thx so much for testing. I have a low standard right now... "does it crash?", so far, so good. Your first result, sans cake, was really quite good, and indicates your AT&T link has only about 20ms of buffering in it, or so. Believe it or not, that's actually "underbuffered" ...

kevinb361

Good evening, and thank you Dave for your many years of work along with the rest of your team combating bufferbloat! I have been following along for many years, and still feel like I know so little. I am so glad to finally have fq_codel and cake in Mikrotik! Previously I had run an OpenBSD router at...

kevinb361

Just updated the following all working great so far!

CCR 1009, CSR 326, and two wAP AC's

Search found 84 matches