Hey hope someone could point me what i could be doing wrong.I have some basic QOS rules i created for few routers i use for my customers, now first time i installed new HEX3 and on fast optical link going up to 250mbit / 150mbit im having speed issues.
I copy pasted rules i use on 10 other rotuers which work fine and i set Queue tree limits to 300mbit for testing purposes but i get only 130-140mbit downstream on speedtest, if i disable all queue tree i get full speed of 250mbit.Upload speeds are normal entire time, just download slow!
If i disable few queues my speed goes UP, if i add more of them speed drops for every mangle + queue rules.Usually i have up to 100mbit links and i never so such problems, this is first time i have this speed internet and this kind of problem.I tried everything possible to debug it from changing queue types, i monitored resources cpu never goes above 50%, there is 200mb+ RAM free, upgrading to latest routerOS, changing how packets are marked etc..
EDIT:I just now tried disabling all mangle + queue tree rules, i created single Simple queue, i set as target my lan exit port or target IP, i set limits to 500m, again, speedtest i get 140/140mbit speed, huh ?
Huge performance drop with mangle + queue tree rules, CPU 50% max
You do not have the required permissions to view the files attached to this post.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
So not a single idea from anyone about this?No response from Mikrotik support in 4 days also..
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
On a different scenario but I have the same problem. Many simple queues on a CCR1009, CPU is fine (<50%) but throughput is really limited. Disabling queues makes it flow without issues.
I don't know what to do.
I don't know what to do.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
Yep seams same issue as me, and i just got replay from MT, here is what they replied:On a different scenario but I have the same problem. Many simple queues on a CCR1009, CPU is fine (<50%) but throughput is really limited. Disabling queues makes it flow without issues.
I don't know what to do.
"Hello,
Check /system resources cpu, most probably you will see that one CPU core have reached 100%. As one simple queue can be operated only by one CPU core, that applies limitations. You need device with more powerful CPU cores for such queue setups
Best regards,
Janis M."
Seriously MT?Do you even read properly tickets?I Specifically wrote how CPU never goes above 50%, even funnier on my screenshot you can see CPU is at only 6% during that 150mbit download speedtest, so i dont understand where does rest of my 100mbit of bandwidth go?
And are you saying your brand new dual core HEX3 router isnt capable of doing SINGLE queue up to 250mbit ?For real?
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
Can you click on system -- >settings ---> resources
Then click on CPU and run your test again. You will see this will show each of the CPU cores and you can see if one is maxing out. You can then click Tools Profile to see the name of the process that is using all your resources. I believe that only simple queues are optimised for multi core platform and not queue trees. But this may have changed in one of the recent updates. you can also post an export of your mangle and queue trees. Another thing that can help is creating a rule in your firewall raw which turns of connection tracking for lan to lan traffic.
Then click on CPU and run your test again. You will see this will show each of the CPU cores and you can see if one is maxing out. You can then click Tools Profile to see the name of the process that is using all your resources. I believe that only simple queues are optimised for multi core platform and not queue trees. But this may have changed in one of the recent updates. you can also post an export of your mangle and queue trees. Another thing that can help is creating a rule in your firewall raw which turns of connection tracking for lan to lan traffic.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
Okay im speechless now, i just loged into router to do another test and post screenshots for you, NOW IT SUDDENLY WORKS!Can you click on system -- >settings ---> resources
Then click on CPU and run your test again. You will see this will show each of the CPU cores and you can see if one is maxing out. You can then click Tools Profile to see the name of the process that is using all your resources. I believe that only simple queues are optimised for multi core platform and not queue trees. But this may have changed in one of the recent updates. you can also post an export of your mangle and queue trees. Another thing that can help is creating a rule in your firewall raw which turns of connection tracking for lan to lan traffic.
I have changed ZERO things, nobody even logged into router for few days, last thing i did is update to 6.40rc2 when was out and tested once more in hope it somehow got fixed(it didint).
Now im getting full 200+mbit speeds and cpu doesnt go over 50% cpu regardless of Simple queues or Queue tree that i have previously set when it didint work.
Now im not sure if its random bug, or was it fixed and i missed it somehow, will monitor if this problem will appear again..
You do not have the required permissions to view the files attached to this post.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
Keep us posted. But it would be good to see what the cpu usage is per core and also see the profile of which resource is using the cpu.
I am yet to get my CPU usage above 14%. The dude uses 100% of one core which pushes my total usage up to 14% acround 9 cores. Queues and queue tree have not gone above 1 % on any core as yet.
I am yet to get my CPU usage above 14%. The dude uses 100% of one core which pushes my total usage up to 14% acround 9 cores. Queues and queue tree have not gone above 1 % on any core as yet.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
@ivicask: what others are saying is that in multi-cpu boards (like your hEX or my CCR) some processes that don't use multi-threading can consume a single core to 100% but what you see in total (that one you see in winbox) is a fraction of percent. Given that you have a dual-cpu system, if one goes to 100% you should see a ~50% total cpu usage.
In my case I checked this and none of the cpus goes up to 100%, at worst I reach 80%, still I suffer of the same wrong behaviour.
In my case I checked this and none of the cpus goes up to 100%, at worst I reach 80%, still I suffer of the same wrong behaviour.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
I understand this, but my total CPU usage when it was broken was 6%, and individual cores(2+2 multithread) where never above like 50%, i understand everything, sorry if im bad at explaining problem, i was about to post the screenshots form profiles, but as i wrote above now suddenly all works..@ivicask: what others are saying is that in multi-cpu boards (like your hEX or my CCR) some processes that don't use multi-threading can consume a single core to 100% but what you see in total (that one you see in winbox) is a fraction of percent. Given that you have a dual-cpu system, if one goes to 100% you should see a ~50% total cpu usage.
In my case I checked this and none of the cpus goes up to 100%, at worst I reach 80%, still I suffer of the same wrong behaviour.
Side question, is mangle + Queue tree supposed to use multi cores or not?From what i see when running test now when it all works properly it does use all cores or im seeing wrong?
You do not have the required permissions to view the files attached to this post.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
i see firewall use about twice resources than qos. So, probably you can try to optimize your firewall rules (mangle)
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
Well i have simple mangle rules, and as long everything works and CPU isnt crossing even 50% total i dont want to touch anything..i see firewall use about twice resources than qos. So, probably you can try to optimize your firewall rules (mangle)
But as i said original problem was with ALL firewall and ALL mangle rules disabled, just single Simple queue and bandwidth was cut in half, still cant explain it why it happened, but it did, i submitted supout logs to support during when this problem happened but i guess it doesnt help them much..
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
Do test: in this simple queue, try to use queue type = sfq. (create new one or simply use wireless-default) and check again.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
I did try that! Now its pointless as i dont have problem!Do test: in this simple queue, try to use queue type = sfq. (create new one or simply use wireless-default) and check again.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
Here we go again, problem is back, nothing was touched in configuration, i tried again disabling all firewall, mangle, Queue tree rules.
I set single simple queue rule, limit it to 500/500mbit, i get top 170mbit on speedest
I disable simple rule i get 250mbit again.
Comparing to my previos screenshot above when it worked i see there is something different with CPU, when it worked properly it used CPU0 to full + CPU3 and CPU0.
Now it tops CPU3, and other CPUs are barely used.
Im 100% Sure this is some kind of bug in this Device multicore management / task distribution.
Cant we do some proper debugging?Im willing to give Team Viewer acces to my server if some Mikrotik technician wants to check whats going on.
I set single simple queue rule, limit it to 500/500mbit, i get top 170mbit on speedest
I disable simple rule i get 250mbit again.
Comparing to my previos screenshot above when it worked i see there is something different with CPU, when it worked properly it used CPU0 to full + CPU3 and CPU0.
Now it tops CPU3, and other CPUs are barely used.
Im 100% Sure this is some kind of bug in this Device multicore management / task distribution.
Cant we do some proper debugging?Im willing to give Team Viewer acces to my server if some Mikrotik technician wants to check whats going on.
You do not have the required permissions to view the files attached to this post.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
This may sound silly, but do you really need the Simple queues?
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
I'm using Queue tree, this is just example so someone doesn't tell me how mangle or Queue tree rules are bad ,or how i should change that or that, thats why i disabled all rules for test.This may sound silly, but do you really need the Simple queues?
Performance issue happens with both Queue tree or Simple queue.
And does it really matter?Its brand new powerful router, i should be able to do simple queue limits if i want, and it does work as shown im above SS, but seams there is some kind of bug which im trying to debug/prove now..
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
are you not under some sort of attack. I find it weird your router is hitting 28% on firewall as well as queues. confirm how many clients you have running off this. can you do export on your firewall.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
During tests there was like 3 clients.Nobody was active, because i would see traffic on pppoe connection..are you not under some sort of attack. I find it weird your router is hitting 28% on firewall as well as queues. confirm how many clients you have running off this. can you do export on your firewall.
Code: Select all
/ip firewall filter
add action=drop chain=forward in-interface=DISZG_GOST_BRIDGE out-interface=!pppoe-out1
add action=accept chain=input connection-state=established
add action=accept chain=input connection-state=related
add action=accept chain=input dst-address=192.168.255.2 dst-port="" in-interface=pppoe-out1 port=3389 protocol=tcp
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept established,related" connection-state=established,related
add action=drop chain=input comment="defconf: drop all from WAN" in-interface=pppoe-out1
add action=accept chain=forward comment="defconf: accept established,related" connection-state=established,related
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalidEDIT:I mean if i was under attack wouldn't i see higher firewall load during no traffic?It only goes up during speedtest..
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
Can you post your nat and mangle rules.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
/ip firewall natCan you post your nat and mangle rules.
add action=masquerade chain=srcnat comment="defconf: masquerade" out-interface=pppoe-out1
add action=masquerade chain=srcnat out-interface=WAN_IZLAZ
add action=masquerade chain=srcnat out-interface=DISZG_GOST_BRIDGE
add action=dst-nat chain=dstnat dst-port=3389 in-interface=pppoe-out1 protocol=tcp to-addresses=192.168.255.2 to-ports=3389
add action=dst-nat chain=dstnat in-interface=pppoe-out1 port=13000 protocol=tcp to-addresses=192.168.255.2 to-ports=13000
add action=dst-nat chain=dstnat in-interface=pppoe-out1 port=14000 protocol=tcp to-addresses=192.168.255.2 to-ports=14000
add action=dst-nat chain=dstnat in-interface=pppoe-out1 port=13000 protocol=udp to-addresses=192.168.255.2 to-ports=13000
add action=dst-nat chain=dstnat in-interface=pppoe-out1 port=14000 protocol=udp to-addresses=192.168.255.2 to-ports=14000
add action=dst-nat chain=dstnat in-interface=pppoe-out1 port=15000 protocol=udp to-addresses=192.168.255.2 to-ports=15000
add action=dst-nat chain=dstnat dst-port=1723 in-interface=pppoe-out1 protocol=tcp to-addresses=192.168.255.1 to-ports=1723
[admin@MikroTik] /ip firewall nat>
Mangle rules are not important, i currently deleted all to simplify things during debuging.So got nothing to export..
And again, i tried disabling all NAT rules except pppoe and WAN output interfaces, made no difference.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
what is the port 1723 dstnat rule for. Surely you dont need to dstnat to your router. I have much more in my firewall and nat than you have on a HAP lite never see CPU going above 10% total with simple queues. never see any issues. I would look out for an error here or a loop or some sort of attack
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
If I was under attack wouldn't firewall be higer during idle? I only get higher firewall cpu usage during speedtest, and same happens on 10+ other routers.what is the port 1723 dstnat rule for. Surely you dont need to dstnat to your router. I have much more in my firewall and nat than you have on a HAP lite never see CPU going above 10% total with simple queues. never see any issues. I would look out for an error here or a loop or some sort of attack
What speed you have? Also 250mbit? Because it's not fair comparison if you have much lower.
And 1723 port I need for VPN. But again, I disabled all those ports during the tests I showed on screenshot, makes no difference..
About loop, I got 2 ports, port 1 connected to provider router dialing pppoe connection, port 2 masquerade and gives Internet output to rest of network, don't see where I could have loop.
Re: Huge performance drop with mangle + queue tree rules, CPU 50% max
Same problem here. RB951G
Mangle rules + Queue (limiting at 30mbits) = 20mbits/sec on a 45mbit/sec link
No rules and no queues = 45 mbits/sec
In no case I see more than 40% CPU usage.
The worst... I work at a ISP where we have CCR1072 as core routers and 5 queue tree rules absolutely destroy our traffic (3.5gbps) .. same behaviour as my 951G but maximized...
Mangle rules + Queue (limiting at 30mbits) = 20mbits/sec on a 45mbit/sec link
No rules and no queues = 45 mbits/sec
In no case I see more than 40% CPU usage.
The worst... I work at a ISP where we have CCR1072 as core routers and 5 queue tree rules absolutely destroy our traffic (3.5gbps) .. same behaviour as my 951G but maximized...