Using RouterOS to QoS your network - 2020 Edition

pcunite · Fri May 31, 2013 9:43 pm

Title:
Using RouterOS to QoS your network - 2020 Edition

Welcome:
The following article is a high-level introduction to a QoS implementation using MikroTik RouterOS. Quality of Service is a large topic. So, this short article will not attempt to explain all edge cases, compare the many algorithms, or provide deep context on packet prioritization. However, it is possible to achieve good - even fantastic results - by creating simple classifications and actions on the most common traffic flows. The configuration presented here is suitable for small business, home, ip telephony, and gaming environments where a single device is providing QoS management.

Why prioritize traffic?
Generally speaking because there is network contention. This occurs most commonly because two or more applications are requesting enough data to exceed the interface. Maybe you want to plan ahead knowing there will be congestion. Even when individual applications and protocols are managing themselves well, they are not aware of the effect they are having on the rest of the network. QoS is then a network governor watching all packet flows and making good decision for everyone. Since network interfaces operate in a serial manner, interactive traffic will be waiting on the many packets ahead of it from big bulky traffic. Even if you could afford to add more Internet connections and more routers, it is still possible to overwhelm them. Therefore, prioritizing your network is a QoS mechanism to manage the different types of traffic flows.

Traffic Types:
You need to classify at least three: interactive, network, and then bulky. For the purposes of this article, VoIP packets are interactive traffic and considered the most important. The network traffic consists of DNS, ICMP, and ACK packets. For the bulky category we have HTTP and QUIC. We also have a catch all for everything else which gets the lowest priority. When our highly interactive traffic is occurring we will ensure it is never impeded by the other types. Indeed, all other traffic types will have secondary importance for the duration of VoIP packet flows, but only when the network is under threat of congestion. Use our model as a guide and create your own categories.

How to identify Traffic Types:
There are actually quite a few ways. Some applications have standard port numbers like DNS. Maybe you have equipment setting DSCP bits for you. You could also check ip addresses, mac addresses, or even VLAN IDs to know the importance of packets coming from those locations. It is also possible to check byte rates to identify streaming traffic. Knowing the types of applications in use and their bandwidth requirements will help you to correctly know what is important or at least what category it should go in.

TrafficTypes.png

How Interfaces and QoS work together:
It is helpful to understand a little bit about interfaces, queuing, and shaping before we jump right into the implementations. Think of interfaces as buckets and packets as different color liquids. These buckets have a drain port at the bottom to let out the liquid. Nozzles are pouring red, green, and blue liquid into our bucket. Thus we have two considerations: the speed of the drain port and the size of the bucket.

If five packets arrive every 5 minutes, it is easy to reason that our bucket can handle that just fine. But if 10,000 packets arrive every second we are going to have a problem. We could speed up the port, say with a 100GbE interface. But there are side-effects to doing that and it is not always affordable to go with faster interfaces. We could get a bigger bucket. A bucket so big that it could hold an elephant. Unfortunately we have the situation to where the last drop of red may take too long waiting for all that blue to drip out. No matter the speed of our drain port or the size of the bucket, it can be so utilized that it can not keep up with incoming data. Our bucket can overflow, throwing away some of the packets.

When we QoS packets, we do make use of port and bucket characteristics, but we also notify those pouring nozzles to release in a more responsible way for our capacity. If they don't, then we take matters into our own hands to ensure the packets we care about the most don't overflow. With QoS this is done by dropping packets. Naturally, some packets can't be dropped without effecting the application experience. We plan accordingly.

Disclaimer:
What follows is my best understanding of how to implement the stated goals. FastTrack must not be enabled. Feedback from MikroTik as well as fellow forum members is required to make this an accurate document. Please suggest changes that should be made. Let's make this issue a commonly understood one. Special thanks to bharrisau for testing and feedback.

pcunite · Fri May 31, 2013 9:43 pm

Implementing traffic prioritization (QoS) with RouterOS

To turn on the QoS capabilities of RouterOS, we implement two things: marking and then queuing.

The Marking Stage

How to Mark the Traffic Types:
RouterOS supplies the Mangle feature to mark packets. What you decide to mark is up to personal and business decisions. Here is a sample starting point. It can be appropriate to mark items, interesting to you, that will ultimately go into the same queue. This is useful for network monitoring purposes. The POP3 mark is an example of that.

Take time to get your marking correct. Test to ensure you are seeing the totals move as you expect. At this stage, we are only marking items. We will use another command to take actions on these marks.

/ip firewall mangle
# Identify DNS on the network or coming from the Router itself
add chain=prerouting  action=mark-connection connection-state=new new-connection-mark=DNS port=53 protocol=udp passthrough=yes comment="DNS"
add chain=prerouting  action=mark-packet     connection-mark=DNS  new-packet-mark=DNS passthrough=no
add chain=postrouting action=mark-connection connection-state=new new-connection-mark=DNS port=53 protocol=udp passthrough=yes
add chain=postrouting action=mark-packet     connection-mark=DNS  new-packet-mark=DNS passthrough=no

# Identify VoIP
add chain=prerouting  action=mark-connection new-connection-mark=VOIP port=5060-5062,10000-10050 protocol=udp passthrough=yes comment="VOIP"
add chain=prerouting  action=mark-packet     connection-mark=VOIP new-packet-mark=VOIP passthrough=no

# Identify HTTP/3 and Google's QUIC
add chain=prerouting  action=mark-connection connection-state=new new-connection-mark=QUIC port=80,443 protocol=udp passthrough=yes comment="QUIC"
add chain=prerouting  action=mark-packet     connection-mark=QUIC new-packet-mark=QUIC passthrough=no

# Identify UPD. Useful for further analysis. Should it be considered high priority or put in the catchall? You decide.
add chain=prerouting  action=mark-connection connection-state=new new-connection-mark=UDP protocol=udp passthrough=yes comment="UDP"
add chain=prerouting  action=mark-packet     connection-mark=UDP  new-packet-mark=UDP passthrough=no

# Identify PING on the network or coming from the Router itself
add chain=prerouting  action=mark-connection connection-state=new new-connection-mark=ICMP protocol=icmp passthrough=yes comment="ICMP"
add chain=prerouting  action=mark-packet     connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add chain=postrouting action=mark-connection connection-state=new new-connection-mark=ICMP protocol=icmp passthrough=yes
add chain=postrouting action=mark-packet     connection-mark=ICMP new-packet-mark=ICMP passthrough=no

# Identify Acknowledgment packets
add chain=postrouting action=mark-packet     new-packet-mark=ACK packet-size=0-123 protocol=tcp tcp-flags=ack passthrough=no comment="ACK"
add chain=prerouting  action=mark-packet     new-packet-mark=ACK packet-size=0-123 protocol=tcp tcp-flags=ack passthrough=no

# Identify HTTP traffic but move it to a Streaming mark if necessary.
add chain=prerouting  action=mark-connection connection-mark=no-mark  connection-state=new new-connection-mark=HTTP port=80,443 protocol=tcp passthrough=yes comment="HTTP"
add chain=prerouting  action=mark-connection connection-bytes=5M-0    connection-mark=HTTP connection-rate=2M-100M new-connection-mark=HTTP_BIG protocol=tcp passthrough=yes
add chain=prerouting  action=mark-packet     connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no
add chain=prerouting  action=mark-packet     connection-mark=HTTP     new-packet-mark=HTTP passthrough=no

# Email goes to the catchall
add chain=prerouting  action=mark-connection connection-state=new new-connection-mark=POP3 port=995,465,587 protocol=tcp passthrough=yes comment="OTHER"
add chain=prerouting  action=mark-packet     connection-mark=POP3 new-packet-mark=OTHER passthrough=no

# Unknown goes to the catchall
add chain=prerouting  action=mark-connection connection-mark=no-mark new-connection-mark=OTHER passthrough=yes
add chain=prerouting  action=mark-packet     connection-mark=OTHER   new-packet-mark=OTHER passthrough=no

The Queuing Stage

How to act on Traffic Marks:
RouterOS supplies the Queue Tree structure that enable us to act on marks. This is how we truly classify the packet flows on the network. A whole book could be written on what is occurring here. There are many options one could use to dial in a very custom Queue Tree. The purpose of this article, however, is to present a simple yet very effective implementation. A few things do need to be understood.

Max-limit:
In order for queuing to occur in our equipment, and thus give us the control on packet flows, we have to set our interfaces to operate at 10% the rate of our ISP connection. This is only a starting number and is dependent upon your CPU speed and simultaneous connections. Apply to both the upload and download links. This way buffering always occurs inside of our equipment. The max-limit parameter is required for the algorithms to function and must not be 0. In our example, we have 100M service, so we have set it to 90M.

Limit-at:
This option is not something you will use commonly and is therefore recommended to leave it at 0 (disabled). However, there is a very special situation where you must enable it. Read the Protection with Limit-at section to learn more.

bucket-size:
During congestion, this value sets the amount of tokens to accrue before the chosen queue type takes effect. This is an equation and means that after Max-limit is reached (bucket-size * max-limit) worth of bytes will be engaged by the queue type. For our purposes, we only want a small amount of time addressing packets going over the limit, enough to smooth out any protocol windowing.


/queue tree

# DOWN
add name=DOWN max-limit=90M parent=LAN bucket-size=0.01 queue=default

add name="1. VOIP"     packet-mark=VOIP     parent=DOWN priority=1 queue=default
add name="2. DNS"      packet-mark=DNS      parent=DOWN priority=2 queue=default
add name="3. ACK"      packet-mark=ACK      parent=DOWN priority=3 queue=default
add name="4. UDP"      packet-mark=UDP      parent=DOWN priority=3 queue=default
add name="5. ICMP"     packet-mark=ICMP     parent=DOWN priority=4 queue=default
add name="6. HTTP"     packet-mark=HTTP     parent=DOWN priority=5 queue=default
add name="7. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=6 queue=default
add name="8. QUIC"     packet-mark=QUIC     parent=DOWN priority=7 queue=default
add name="9. OTHER"    packet-mark=OTHER    parent=DOWN priority=8 queue=default


# UP
add name=UP max-limit=90M parent=WAN bucket-size=0.01 queue=default

add name="1. VOIP_"     packet-mark=VOIP     parent=UP priority=1 queue=default
add name="2. DNS_"      packet-mark=DNS      parent=UP priority=2 queue=default
add name="3. ACK_"      packet-mark=ACK      parent=UP priority=3 queue=default
add name="4. UDP_"      packet-mark=UDP      parent=UP priority=3 queue=default
add name="5. ICMP_"     packet-mark=ICMP     parent=UP priority=4 queue=default
add name="6. HTTP_"     packet-mark=HTTP     parent=UP priority=5 queue=default
add name="7. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=6 queue=default
add name="8. QUIC_"     packet-mark=QUIC     parent=UP priority=7 queue=default
add name="9. OTHER_"    packet-mark=OTHER    parent=UP priority=8 queue=default

pcunite · Fri May 31, 2013 9:44 pm

Additional Information:
Since QoS is such a big topic, additional information is presented here for those who want to dive a little deeper. While the example configuration is simple to understand and explain, we should go further in the real world. Also read the Bufferbloat article.

Protection with Limit-at:
In the event that traffic volume, for one particular queue, reaches at least 51% of the Max-limit value, that traffic will be subject to packet prioritization. This means packets might get dropped. This is normal and the whole point of what we are doing. But, what if that queue is your high priority queue? There is a way to carve out some protection for these special packets.

To illustrate how this is done, we'll demonstrate with VoIP packets flowing in our VOIP queue. Each VoIP call is about 86Kb, both directions. Let's say we have to protect 700 active calls to support a business contract. At 86Kb, that works out to be about 60M, which is indeed over 51% of our 90M Max-limit. Thus we set a Limit-at of 60M, on the VOIP queue only (both up & down). This will prevent those packets from ever being dropped. If you only ever expect 10 VoIP calls then Limit-at would be unnecessary.

Protection for low priority queues with Limit-at:
Why offer any protection for a low priority queue? If you have enough bandwidth to go around, it is a very good idea to carve out a minimum amount of bandwidth for busy queues. Otherwise, all higher priority queues will take nearly everything under ultra-high contention events. Thus, you could ensure at least 10M or more if you can afford it. QUIC and HTTP_BIG are queues that could use some level of protection.

Limit-at Qualifiers:

You can't protect more bandwidth than you actually have.
Protection for one queue (only when under pressure) naturally forces the other queues to battle over what's left.
Attempting to protect more than 90% is risky. Better to increase bandwidth instead.
Not everything on your network should be high priority. If so, then in reality, you only have one queue.

Implement like so:

# Set protection on VOIP queue, both directions. Also some for HTTP_BIG.
/queue tree 
add limit-at=60M max-limit=90M name="1. VOIP" packet-mark=VOIP parent=DOWN priority=1 queue=default
add limit-at=60M max-limit=90M name="1. VOIP_" packet-mark=VOIP parent=UP priority=1 queue=default
add limit-at=10M max-limit=90M name="7. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=7 queue=default
add limit-at=10M max-limit=90M name="7. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=7 queue=default

Optimizing the HTTP, HTTP_BIG, and QUIC queues:
Any queue that frequently hits the Max-limit value and has packets using global synchronization, Cubic, NewReno, or similar congestion methods, will benefit from a better queuing mechanism. Therefore, it is recommended to use RED (Random Early Detection) on these queue types. Note that WRED is an often mentioned improved version that looks at priority data in the packet. However, since we are placing packets in their own special queue anyway, we have already classified what is important.

Implement like so:

/queue type

# default queue behavior
set default kind=sfq

# queue behavior for streaming type traffic
add kind=red red-avg-packet=1514 name=redCustom

# example of how to use red, optionally set for all bulky traffic types
add name="7. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=6 queue=redCustom
add name="7. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=6 queue=redCustom

pcunite · Fri May 31, 2013 9:45 pm

How Buffering and Bufferbloat in ISP supplied equipment affects QoS and latency sensitive protocols

Understanding this phenomenon is really tricky. It helps to know how applications, network controllers, protocols, and memory all behave in isolation. As networks have grown in size and complexity, all these elements have accidentally worked together to increase Latency, the enemy of networking.

Applications:
Software applications send their data to receivers in chunks or packets about 1Kb in size. If Application 1A sends a packet destined for Application 1B, it waits for a moment, to see if it gets a reply. During this wait, other applications may try to say something. Now we have an interleaving amount of traffic coming and going. To make full use of this highway (bandwidth), applications start slowly sending data. As they get acknowledgment replies back that they are being heard, they increase the rate of packets they send. Applications try to send as much as they can.

Controllers:
Network interfaces, whether exposed as a Cat5-7, fiber ports, or wifi, are connected to a controller that is ultimately managed by low level software. This implies memory, and is what is used to hold the incoming bits that eventually make up an individual packet. As the bits come in, one by one, a whole packet is realized as it comes in off the wire. The first packet might belong to Application 1. The next packet might belong to Application 2. The next 30 packets might all belong to Application 3. A natural question might then be: how many packets should network controllers, routers, and switches store?

Protocols:
When a sending application does not receive a reply (ACK packet), within a certain time window, it begins to slow down its sending rate. It will slow down until it is heard, at which point is starts increasing again. It can enter a timeout period if it is never heard. If this happens to a graphical app, it may appear to freeze. If a file download is in progress, timeouts make the network seem "slow" or even "down". The reply window and timeouts work well when there is one type application on the network. But today, there is almost never one type of application or even one type of packet flow.

Memory:
Memory is plentiful these days. Since packets might only be 1Kb in size, it is cheap to hold a lot of them. But it stands to reason that at some point, room for holding packets is going to run out. When that happens, the network controllers start to drop packets and their buffers overflow. As applications realize they are not getting acknowledgments they begin to slow down too freeing up memory pressure. The cycle is then repeated.

PacketPath.png

Everything described so far is correct behavior, as designed, and works well under certain conditions. However, conditions and what is expected of a network have changed. Networks are not used to simply download things or view web pages any more. Increasingly, there is interactive traffic.

The interactive packet:
As packets get stored by the network, every so often a tiny little packet needs to be sent. These come from applications that want to send and reply in a loop with small amounts of data indicating state. Yet, the network is completely full causing any new packet to wait behind long lines of bulky traffic. Because of this, controllers are overflowing their buffers all along the way. Our small packets are getting dropped too. We are now experiencing Latency.

Latency:
The amount of time for a packet to get its reply. This is what makes a network feel fast or slow. There will always be a delay but how much is too much? For certain applications, that is about 150ms. If these packets are for voice or game data, the end user will think the network is unstable. Yet, big downloads, running in the background or on other system, finish and no one perceives any issues.

Buffer collapse:
Because of the way networks store packets in buffers, sending applications get bandwidth and priority that should not be available to them. When buffers along the way are seemly accepting everything being sent, applications increase their packet flow rates, never leaving room for interactive applications. At some point in time, random buffers begin to be completely maxed out. Packets get dropped, acknowledgments are not sent. Sending applications slow down. Unfortunately, they all do this at once. Then all applications ramp back up, then they all slow down again. The network is weaving back and forth inefficiently. This affects bulky traffic, and most certainly never properly allows for interactive traffic to work as needed.

Solution:
On the ISP side, buffers sized appropriately and matched to processor, controller, and bandwidth sizes and speeds. On the client side, Active Queue Management techniques as shown in this article series. The ISP may never know what you consider to be your most important traffic, so for both your up-link and down-link interfaces, you must ensure to manage your own buffers, control what overflows, and determine which packets should be dropped to signal packet rates.

pcunite · Sat Jun 01, 2013 6:02 pm

Reserved

pcunite · Sat Jun 01, 2013 6:02 pm

Reserved

Chupaka · Tue Jun 04, 2013 4:17 pm

great info, but it's better to use http://wiki.mikrotik.com for such articles - forum is more for questions, not for tutorials

pcunite · Tue Jun 04, 2013 4:22 pm

great info, but it's better to use http://wiki.mikrotik.com for such articles - forum is more for questions, not for tutorials

Thank you. My intention was to perfect this post and then have it accepted by MikroTik after all the experts had confirmed it. Then, as you say, have it posted in the Wiki.

Tue Jun 04, 2013 4:24 pm

great info, but it's better to use http://wiki.mikrotik.com for such articles - forum is more for questions, not for tutorials
Thank you. My intension was to perfect this post and then have it accepted by MikroTik after all the experts had confirmed it. Then, as you say, have it posted in the Wiki.

Good idea. Waiting for next parts.

deejayq · Mon Jul 01, 2013 1:30 pm

# HELP! I don't know how to mark what is left?

just add two rules at the bottom of the script
first mark-connection with dst-address=0.0.0.0/0 (nothing else)
second mark-packet with connection mark you set on the first rule (BULK i think you named both marks).

theprism · Sun Jul 14, 2013 3:48 am

It doesn't work.

Here's what I have:
****************************************
/ip firewall mangle

add chain=forward action=mark-connection src-address=192.168.151.7 new-connection-mark=VOIP
add chain=forward action=mark-packet passthrough=yes connection-mark=VOIP new-packet-mark=VOIP
add chain=forward action=mark-connection dst-address=192.168.151.7 new-connection-mark=VOIP
add chain=forward action=mark-packet passthrough=no connection-mark=VOIP new-packet-mark=VOIP

add chain=forward action=mark-connection dst-address-list=IPTVlist new-connection-mark=IPTV
add chain=forward action=mark-packet passthrough=yes connection-mark=IPTV new-packet-mark=IPTV
add chain=forward action=mark-connection src-address-list=IPTVlist new-connection-mark=IPTV
add chain=forward action=mark-packet passthrough=no connection-mark=IPTV new-packet-mark=IPTV

add chain=forward action=mark-connection src-address=0.0.0.0/0 new-connection-mark=BULK
add chain=forward action=mark-packet passthrough=yes connection-mark=BULK new-packet-mark=BULK
add chain=forward action=mark-connection dst-address=0.0.0.0/0 new-connection-mark=BULK
add chain=forward action=mark-packet passthrough=no connection-mark=BULK new-packet-mark=BULK

/queue tree

add name="TOTAL_U" parent=pppoe-out1 queue=default priority=8 limit-at=0 max-limit=680k
add name="TOTAL_D" parent=bridge-homelan queue=default priority=8 limit-at=0 max-limit=4300k

add name="VOIP_U" parent=TOTAL_U packet-mark=VOIP queue=default priority=1
add name="VOIP_D" parent=TOTAL_D packet-mark=VOIP queue=default priority=1

add name="IPTV_U" parent=TOTAL_U packet-mark=IPTV queue=default priority=2
add name="IPTV_D" parent=TOTAL_D packet-mark=IPTV queue=default priority=2

add name="BULK_U" parent=TOTAL_U packet-mark=BULK queue=default priority=8
add name="BULK_D" parent=TOTAL_D packet-mark=BULK queue=default priority=8

/ip firewall address-list
add address=184.84.243.193 list=IPTVlist
add address=4.27.18.126 list=IPTVlist
****************************************

In mangle I see exactly same counts for src and dst on each marking portion, which makes me think that mangles shouldn't be configured like that.
/ip firewall mangle screenshot is attached

Also prioritization doesn't work, since if I start my IPTV only, it works fine. Now, while IPTV is running, if I start a download, my IPTV is screwd. Image is distorted and become impossible to view and hear the audio.
/queue tree screenshot is attached

Any solutions?

Thanks,
T.P.

WiPoint · Thu Aug 29, 2013 3:05 pm

Hi from your Web Mangle Mark up rule MAX packet size has to to be in the range of 0-65535. Any advice on what to set it as?
ROS6.2 RB1100AHx2

pcunite · Sat Aug 31, 2013 12:56 am

We need someone from MikroTik (or popular online blogger) to help with these rules. These are what I was able to create based on documentation. Qos is a very important feature and one of the main reasons someone would want one of these advanced routers. Let's hope they chime in and we can build a good default script.

pcunite · Sat Aug 31, 2013 6:15 am

Hello everyone. I've tested and updated the script. It now works correctly on RouterOS 6.1. Note that ether1 is WAN and ether2 is LAN. Adjust those as necessary for your environment.

I would appreciate if someone could tell me how to mark big downloads over HTTP traffic. Currently, the script marks port 80 so everything HTTP gets too much priority. The idea situation would be to let short bursts of HTTP traffic get high priority and the big long downloads get less.

whiely · Sun Sep 01, 2013 11:27 pm

Hello everyone. I've tested and updated the script. It now works correctly on RouterOS 6.1. Note that ether1 is WAN and ether2 is LAN. Adjust those as necessary for your environment.

I would appreciate if someone could tell me how to mark big downloads over HTTP traffic. Currently, the script marks port 80 so everything HTTP gets too much priority. The idea situation would be to let short bursts of HTTP traffic get high priority and the big long downloads get less.

with connection byte ?

pcunite · Thu Sep 05, 2013 3:14 am

>> how to mark big downloads over HTTP traffic?

With connection byte?

Thank you, the answer is to use connection-bytes and connection-rate.

TikUser · Thu Sep 12, 2013 12:15 am

I’ve watched these 2 videos: HR13: QoS on RouterOS v6 and MUM US11: QoS workshop. In the second video (18:15min) Janis says: "Prerouting is a sum of (mangle) input and (mangle) forward. Both chains are together in here. Postrouting is a sum of (mangle) forward and (mangle) output."

Prerouting = input + forward
Postrouting = output + forward

In your Option 2 configuration You used prerouting and postrouting together. Will these two mangles come into conflict (because of 2 forwards)? If I’ve understood it correctly, in mangle postrouting we mark traffic that goes from the router (output) and through the router (forward). How will this affect the marked traffic from the prerouting and meant for forward? Will it be remarked?

pcunite · Thu Sep 12, 2013 12:55 am

Prerouting = input + forward
Postrouting = output + forward

In your Option 2 configuration You used prerouting and postrouting together. Will these two mangles come into conflict (because of 2 forwards)? If I’ve understood it correctly, in mangle postrouting we mark traffic that goes from the router (output) and through the router (forward). How will this affect the marked traffic from the prerouting and meant for forward? Will it be remarked?

I don't think the 2011 talk was referring to v6, maybe it was. But Prerouting not = input + forward. Look at the 3rd image here. The reason I use pre and post on DNS traffic is because the forward chain is not able to mark traffic sourced from the router itself and also traffic coming into the router from LAN.

Maybe there are other ways of accomplishing the same thing. Perhaps your question will reveal a better way. At the moment, this is all I know. However, when I was watching traffic from the connections menu, doing what I'm doing now was the only way to see it show up marked.

TikUser · Thu Sep 12, 2013 1:48 am

I believe the presentation (Packet Flow and QoS v6) in the 1st video (from 7:16min) could help clarify the situation. This is for ROS v6.

All traffic from the Input Interface goes to Chain Prerouting. If we choose to mark it in Prerouting, from there on, all traffic going to Input and Forward is marked. So, in a way, Prerouting is a sum of Forward and Input.
The Chain Output will remain unmarked. Maybe this is the answer on how to mark traffic sourced from the router itself. Mark it in the mangle Chain Output?

jandafields · Mon Oct 07, 2013 6:15 am

Looking at OPTION 2 above....

In Queue A, you have voip = priority 1
In Queue B, you have ack = priority 1
In Queue C, you have http = priority 1

Since the parent queues don't have a priority, what actually makes "queue A priority 1 (voip)" a higher priority than "queue C priority 1 (http)" since they are both priority 1?

Isn't there a setting that must tell which parent has a higher priority than another parent, or how will it know?

Or, if priority is the wrong term, the real question is: What keeps queue C from using up all the bandwidth and not leaving any for queue A, since they are both maxed at 900k/4M?

pcunite · Mon Oct 07, 2013 3:53 pm

Looking at OPTION 2 above....

In Queue A, you have voip = priority 1
In Queue B, you have ack = priority 1
In Queue C, you have http = priority 1
...
Or, if priority is the wrong term, the real question is: What keeps queue C from using up all the bandwidth and not leaving any for queue A, since they are both maxed at 900k/4M?

You are correct to wonder, and indeed there is something at issue here. I'm still perfecting this script. Basically what is happening currently, and why this script works for small networks is that: since Queue A is never full (never exceeds capacity) nothing is ever prioritized (dropped) within it. Queue C will always be exceeding capacity and thus traffic marked to go there will indeed be dropped (prioritized) at times. But as you've correctly noticed only at 900k/4M. This means that Queue A gets a 10% window of head room before ISP equipment starts to buffer.

What is the fix for this? I'm still researching but I think it must be a dynamic script using connection-rate and other actions. Let's review some wants.

1. Don't ever drop SIP and RTP (VoIP) packets.
2. Don't limit (drop) HTTP traffic until we absolutely have to.

To achieve the above we need to know how many SIP and HTTP (or random BULK) sessions are open (or their byte usage) at any given time. Thus a dynamic script ... when many SIP sessions are open we should increase the limits for HTTP (maybe only 100k/1M), when there are no SIP sessions, automatically allow HTTP to have 900k/4M usage again.

I've created this thread for people that know to tell us how to do this. So far, they've not see this thread and replied. But I do thank you for noticing.

jandafields · Mon Oct 07, 2013 4:21 pm

Ok, this command should tell you how many sip sessions are open. You might need to change dst-address to src-address, depending on the sip traffic:

/ip firewall connection print count-only where protocol="udp" and dst-address~":50.."

~":50.." will match any port between 5000 and 5999

We could run that every xx seconds, then use the number returned by that command and then limit the other queues accordingly as we know that every sip session is probably around 100kbps in both directions.

Craftylover · Tue Oct 15, 2013 4:27 am

Hello everyone.
It seems as though many of the QOS examples listed in the forum are individually tailored to a certain users needs and/or a particular application -this one just happens to be generic enough to make a little sense to me (completely new at this).

Might there be a way to add anything to this configuration that would help prioritize streaming video for a home network? Looks as though some of the Netflix traffic for example gets lost in the fray when there is heavy traffic marked as HTTP and the quality tends to suffer. I have seen some stuff pointing to L7 pattern matching, but I can honestly say that I do not know how to implement.
Any pointers would be appreciated.
Thank you.

ingjaab · Tue Oct 15, 2013 11:17 am

hello how would qos, wan, lan1, lan2 , LAN3, lan4, lan5, ... ..........., yes core has 12 ether

thanks

pcunite · Tue Oct 15, 2013 6:23 pm

Might there be a way to add anything to this configuration that would help prioritize streaming video for a home network? Looks as though some of the Netflix traffic for example gets lost in the fray when there is heavy traffic marked as HTTP and the quality tends to suffer. I have seen some stuff pointing to L7 pattern matching, but I can honestly say that I do not know how to implement.

Great question. You must understand that the best way to Qos a network is to have enough bandwidth to work with. People want everything ... but that is a tall order.

Want: VoIP is perfect, file downloads are max speed, video streaming is perfect ... all at the same time!

Wow! I want that too but to achieve this you must have at least 10mb down. Why? Because streaming video is 5mb down! How many users do you have? It gets worse from there.

Currently, the script does find and mark video traffic as "HTTP_BIG". To have what you want you have two options. Keep the script as is but put "HTTP_BIG" into a bigger queue and ensure internet speed is 10mb down and hope that only one user is streaming video.

Or you need to separate video streaming from big downloads. Thus you could use L7 matching or IP's from hulu, YouTube, etc. Once you've separated out your traffic types I would put "VIDEO" into PCC queues. The more users streaming the better PCC will segment them. Naturally, you'll want 5MB per user ... but that is your problem.

You might also tweak what the script marks as "HTTP_BIG". Some sites (like pinterest) have so many images on a single page it looks like streaming video for a short time.

whowantstwoknow · Wed Oct 23, 2013 2:36 am

Hi there,

In my recent thread, part of my goal is to route VOIP traffic from the netgear TA612V through the routerOS router. Came across this thread, now forgive the noob question, but in the context of the scripts in this thread and the interfaces I have (see below), which interfaces are my ether-LAN and ether-WAN?

[admin@MikroTik] /interface> print
Flags: D - dynamic, X - disabled, R - running, S - slave
# NAME TYPE MTU L2MTU MAX-L2MTU
0 R ether1-gateway ether 1500 1598 4074
1 R ether2-master-local ether 1500 1598 4074
2 R ether3-slave-local ether 1500 1598 4074
3 ether4-slave-local ether 1500 1598 4074
4 ether5-slave-local ether 1500 1598 4074
5 R wlan1 wlan 1500 2290
6 R bridge-local bridge 1500 1598

Thanks in advance
W.

2fast4youbr · Fri Nov 22, 2013 10:19 pm

Just did and when a softphone from outside company tries to connect, we can listen anything...

any tips ?

2fast4youbr · Tue Nov 26, 2013 11:37 pm

Cant wait for version 3....

good job, working nice here!

arpiska · Thu Jan 23, 2014 2:07 pm

Please help me how to change this solution, if I have 2 WAN and several LAN interface?
- wan1 60M/3M , wan2 21M/5M
- vlan1, vlan2, vlan3... as local ethernet

TikUser · Mon Feb 10, 2014 3:20 am

[Solved]

mdkberry · Fri Feb 14, 2014 7:33 am

Hi pcunite

great, post and pretty much the only thing out there on VoIP QoS at this level that I can find for Router OS.
have you progressed any yet. There seem to be some key issues still outstanding.

I am about to setup QoS for a VoIP network on an RB2011 which will be competing with other subnet LANS for a 5mbps up and 5mbps down EFM service and I really need to nail the VoIP QoS. Any further reading recommendations... other than who the best support person is to pay to do it at Mikrotik.

Rgds
Mark

Urban! · Fri Feb 14, 2014 3:40 pm

hi pcunite

im using 12M up and 12M Down

how to set the upload rates and download rates in Voip

# The secret to ensuring VoIP quality (or any UDP traffic) is to put it into
# a queue that will never be full and thus never prioritize (drop) packets.
add name="LEVEL_A_UP" parent=ether-WAN queue=default max-limit=900k
add name="LEVEL_A_DOWN" parent=ether-LAN queue=default max-limit=4M

# Next, create a queue for high priority traffic.
add name="LEVEL_B_UP" parent=ether-WAN queue=default max-limit=900k
add name="LEVEL_B_DOWN" parent=ether-LAN queue=default max-limit=4M

# Finally, create a queues for traffic that normally exceeds levels.
add name="LEVEL_C_UP" parent=ether-WAN queue=default max-limit=900k
add name="LEVEL_C_DOWN" parent=ether-LAN queue=default max-limit=4M

thanks

pcunite · Fri Feb 14, 2014 3:49 pm

Great post and pretty much the only thing out there on VoIP QoS at this level that I can find for Router OS.
Have you progressed any yet? There seem to be some key issues still outstanding.

Any further reading recommendations ... other than who the best support person is to pay to do it at Mikrotik?

There are a couple of people on the board here who you could pay: efaden, sdischer, or IPANetEngineer. They know what they're doing. I was hoping this post would bring out the VoIP experts but sadly they've not seen it yet. This is something we've all got to get nailed down.

pcunite · Fri Feb 14, 2014 4:03 pm

Hi pcunite. I'm have 12M/12M. How to set the upload rates and download rates for Voip? Thanks.

Make sure you understand how buffering works. Next, it all depends on how many active phone calls you want to support. A single VoIP call is 90kb both directions ... up and down.

With this knowledge, first we chop off 1M/1M (90%) so that the cable/adsl modem does not buffer anything. Next, let's allow for 10 active phone calls: 90kb x 10 = 900kb, which is about 1M. So, 1M for buffering, and 1M for VoIP means we set max-limit to equal 12 - 2 = 10M.

ros code

add name="LEVEL_A_UP"   parent=ether-WAN queue=default max-limit=10M
add name="LEVEL_A_DOWN" parent=ether-LAN queue=default max-limit=10M
add name="LEVEL_B_UP"   parent=ether-WAN queue=default max-limit=10M
add name="LEVEL_B_DOWN" parent=ether-LAN queue=default max-limit=10M
add name="LEVEL_C_UP"   parent=ether-WAN queue=default max-limit=10M
add name="LEVEL_C_DOWN" parent=ether-LAN queue=default max-limit=10M

Just so you know, my VoIP script is not perfect. It works, however. Until you really know how to master segmenting traffic you'll at least have VoIP calls working correctly.

mdkberry · Sun Feb 16, 2014 4:05 am

I would like to adapt pcunite's script to my scenario. and would welcome any suggestions.

my router is RB2011 routerOS 6.7

ISP service is 5mbps up and 5mbps down on EFM (4 wire symmetrical).
this is connected on Eth1 of the router.

I have a VoIP PBX which will make 25 calls max at a time.
this is on the Voice subnet x.x.20.x which is connected to Eth2

the Data LAN is on x.x.10.x connected to Eth3

another subnet is on x.x.88.x connected to Eth4 (but bridged with Eth3)

and there is a WLAN1 interface with x.x.30.x subnet on it.

what I am looking at currently with this is to QoS VoIP traffic on x.x.20.x with a set bandwidth of 2.5mbps each way up and down. to guarantee the VoIP service.and then allow the other subnets to compete for the other 2.5mps ideally using the same queues as in the example from pcunite. Though it would be nice if they could use the remainder of the 5mbps when VoIP isnt being used to the max. In the Cisco configs I have used before the way I dealt with this was to assign QoS as a percentage of bandwidth use and that worked well. I dont see a way to do that with Mikrotiks.

another problem I can't work out yet is how to associate multiple interfaces to a queue, which I will need in this example since I have seperate LANs on seperate interfaces, but they need to use the same queue. I can't bridge the interfaces and apply it there, because two of the subnets have DHCP services and that all stops working when you add bridges in.

EDIT: here are some solutions for my situation so far,

-I have been able to pretty much stick to the script provided by pcunite.
-I included the chains for my PBX using 192.168.20.2 in src-address and another one for dst-address and set ports to 5060, 5061, 10000-20000 on both of those. These are associated to Level A queue.
-all the mangle chains are on prerouting or postrouting including the above ones, instead of forward, though not sure what difference that makes I would be interested to hear comments on that, it seems to work.
-the LEVEL_DOWN queues are all set to parent 'global' resolving the NAT fine, and also multiple subnets/interfaces issue.
-the LEVEL_UP queues set to parent 'ether1-gateway' (the WAN link)
-copy and pasting the script gave me a few issues not sure why, it may have been me accidently clicking fields before saving but it didnt seem to take properly into RouterOS 6.7 and I had to double check some of the mangles to fix a few fields that got dropped randomly, as per my post later on in this thread.
- I have set the Level A queue to 4M/4M, Level B to 3M/3M and Level C to 3M/3M though I am thinking of putting a Queue above Level B and C to max-limit them collectively to 3M to allow the 2M of voip always to be available to Level A. Have to think about that as it may still cause problems for the Level B traffic which can be from my PBX. From my understanding of VoIP so far it is more important to get the ACK and SIP traffic without packet loss than the RTP media traffic which can handle it a little, though it needs to stay below <5% loss.

mdkberry · Tue Feb 18, 2014 2:38 am

Hi pcunite

in testing your script using YouTube media , I am finding that instead of port 80 traffic getting marked with HTTP or HTTP_BIG mark
it is ending up getting marked by the UDP rule in mangle.
if I disable UDP in mangle then it correctly marks the traffic with HTTP or HTTP_BIG

any thoughts on this?

pcunite · Tue Feb 18, 2014 4:22 am

In testing your script using YouTube media, I am finding that instead of port 80 traffic getting marked with HTTP or HTTP_BIG mark it is ending up getting marked by the UDP rule in mangle. If I disable UDP in mangle then it correctly marks the traffic with HTTP or HTTP_BIG.

The order of the rules is important somewhat ... but let me see what you have and I can hopefully spot the error.

mdkberry · Tue Feb 18, 2014 7:15 am

my bad...somehow when I copy-pasted it from your script into the terminal on my router, the UDP mangle didnt copy the protocol for just that mangle.
I just fixed it and everything looks to be ok.

another question, when I copy-pasted, all the mark-connection mangle rules defaulted to "passthrough=yes" should these be set to no?
I can see mark-packet should be set to no, but wasnt clear about the mark-connection default state.

TikUser · Tue Feb 18, 2014 12:52 pm

Hello pcunite! I've noticed something interesting in your rules. Have You used the examples from this Mikrotik wiki for HTTP and HTTP_BIG?
The rule structure is pretty much the same. You are using the same connection rate and connection bytes values. Everything seems fine, except your comment(explanation) for this rules. The author of the wiki talks about 4Mb (500kB), and You about 5MB. The same values, but different explanation.
4Mb (little letter b), 5MB (big letter b).
Mb= megabit
MB= megabyte
This is a huge difference.

SyCo · Tue Feb 18, 2014 2:44 pm

Hello! I've noticed something interesting in your rules. [...]

Nice find! Like you've said it's only in the comments.
I see also that the rule use the correct value of 500kB -> connection-bytes=500000-0

Then we simply need to change the comments

BTW on that same line there is also a little typo where it's written "tranfers" instead of "tranSfers".

mdkberry · Wed Feb 19, 2014 6:38 am

another router setup using the same script and I am seeing HTTP_D traffic which is supposed to be priority 1 being dropped before the other traffic in Level C which is lower priority.
any thoughts? piccy attached.

dropped_packets.tiff

TikUser · Wed Feb 19, 2014 12:52 pm

another router setup using the same script and I am seeing HTTP_D traffic which is supposed to be priority 1 being dropped before the other traffic in Level C which is lower priority.
any thoughts? piccy attached.
dropped_packets.tiff

I. Reduce the number of Queue Trees to 2. One for download & one upload. Your goal is to create a unified hierarchy.
Like this:
Queue Tree - download(the same for upload).
1. DNS - Priority 1
2. ICMP - Priority 1
3. VOIP -Priority 2
4. ACK - Priority 3
5. HTTP - Priority 4
6. HTTP_BIG - Priority 5
7. UDP - Priority 7
8. OTHER - Priority 8

II. Or rearrange the Queue Tree. Look at the picture in the attachment. The picture isn’t something, but it serves the purpose.

Merge level B & C into one queue tree. VOIP (level A) remains in its queue. At the top create the total download bandwidth queue, with VOIP (level A) and Everything else (level B+ level C) as its children.
Read this Mikrotik wiki for more information.

P.s. For the Queue Tree - download, You also need to use the HTB Interface as parent. NOT global. Global is not good for this examples. It is used for another type of QoS.

TikUser · Wed Feb 19, 2014 8:53 pm

another problem I can't work out yet is how to associate multiple interfaces to a queue, which I will need in this example since I have seperate LANs on seperate interfaces, but they need to use the same queue. I can't bridge the interfaces and apply it there, because two of the subnets have DHCP services and that all stops working when you add bridges in.

The pcunite examples won't work well in Your case. You need to create another set of mangle and Queue Tree rules.
- Replace prerouting with the forward chain for traffic going through the router.
- Use src-address-list(your network) + out. interface(WAN-Internet) for marking the connections going through the router(forward chain) to the internet and backwards.
- Create separate mangle rules for marking the download(Internet->LAN) and upload(LAN->Internet) packets from these connections.

- Replace postrouting with the output chain for traffic from the router to the internet. If needed, use out. interface option in order to make the connections in the output chain more precise.
- Create separate mangle rules for marking the download and upload packets from these connections. Which means: Packets coming from the internet to the router (Internet->router), and packets going from the router to the internet (router->Internet).

"We think of Connections as "Both Upload and Download" since most connections have packets in both directions. We think of Packets as either "Download" or "Upload"." Quoted from: NetworkPro on Quality of Service
Watch out how you order the rules!

- Now you can use parent global in Queue Tree for creating the download and upload tree. You must not use the HTB Interface (ether1, ether2,...) in Queue Tree as the parent for the download and upload queue, because You have more than 1 active LAN interfaces, which, as You said, mustn't be bridged.

This is out-of-my-head solution.

Not tested. Test it and adapt it to your needs. This is not the only possibility. For every problem, there are a couple of solutions.

And most important: You need to understand the packet flow, Queue Tree and prerouting, forward, output, postrouting chain in order to know where to apply what, and why. Until You know what You are doing, a router without prioritization is a better solution than a router with a false implemented prioritization.

Search the Mikrotik forum and You will find the answer to: “another question, when I copy-pasted, all the mark-connection mangle rules defaulted to "passthrough=yes" should these be set to no? I can see mark-packet should be set to no, but wasnt clear about the mark-connection default state.“

In most of the cases, when You mark the connections, use passthrough=yes. When You mark the packets, use passthrough=no.

pcunite · Wed Feb 19, 2014 9:47 pm

I am seeing HTTP_D traffic (which is priority 1) being dropped before other traffic in Level C, which has a lower priority.

Notes:
To QoS traffic with a MikroTik you have to be an expert and really understand their documentation. I am not an expert. Please note that my QoS script is for VoIP packet protection. I don't ever want to drop a VoIP packet ... ever ... never. You need to do extensive reading on the subject if you want to design a QoS system that differs from the one I've created.

Terms:
priority means 'drop packets' WHEN needed.
max-limit means to drop packets ONLY when exceeded.

Explanation:
If over 1M of HTTP_D traffic appears then they'll get dropped. I don't know how many PC's you have on your network, but it is real easy to exceed 1M from casual browsing. If you have over 1M of HTTP_D and HTTP_BIG_D at the exact same time then priority is calculated between the two.

TecnoNet · Sat Feb 22, 2014 10:05 am

Note: ehter1--REAL--MAIN is the internet from the ISP,
and in this QOS we Limit the HEAVY DOWNLOAD DATA and leave the rest

mdkberry · Wed Feb 26, 2014 1:07 am

thanks for all the input those that have posted that really helps. I will test it TikUser ideas out and let you know how it goes.

one thing I noticed when I tried pcunite's script tailored in the way I mentioned. once the PBX started distributing calls outbound
all the VoIP traffic ended up marked with PBX_UDP up or down as I expected, but all the actual queue traffic was in Level B under ACK and UDP. hardly any showed up in Level A VoIP despite it being marked from the PBX server fine.

anyway I had to strip out all the marking and queue info anyway as it wasnt working for my setup.

pcunite, in answer to your comment about 1M being a low limit the voice network is setup to have only phones on it, and minimal HTTP traffic should be on it. it was working ok and no prioritising was going on. that idea was really to capture anyone accidently plugging into a voice network floor port and surfing the voice network into a grinding halt.

to be continued...

RB951G2HnD · Mon Mar 24, 2014 8:11 pm

TecnoNet,

Thank you for a good QoS config script. It seems that it's working fine except P2P. In our days the most of P2P clients are using encryption and that is the reason why a simple indicated P2P option is not working in a Mikrotik ROS. I think that to make this QoS config script working with torrent's traffic we need to specify incoming ports in P2P client. For example to use incoming ports from 6881 to 6891. If it so then we need to change Mangle rule for P2P. How to relate this rule with the dedicated ports?

ros code

add action=mark-connection chain=postrouting comment="P2P UP" new-connection-mark=p2pupconn out-interface=ether1 p2p=all-p2p
add action=mark-packet chain=postrouting connection-mark=p2pupconn new-packet-mark=p2pup passthrough=no
add action=mark-connection chain=prerouting comment="P2P DOWN" in-interface=ether1 new-connection-mark=p2pdownconn p2p=all-p2p
add action=mark-packet chain=prerouting connection-mark=p2pdownconn new-packet-mark=p2pdown passthrough=no

RB951G2HnD · Wed Mar 26, 2014 10:36 am

No QoS config provided by TecnoNet is not working because all traffic marked as 'other' including http and etc. So the attached config file is wrong.

RB951G2HnD · Fri Mar 28, 2014 5:36 am

Yesterday I was able to slightly adjust QoS priorities. But somehow torrent traffic clogs channel so that for example in parallel to watch youtube is not possible. Torrent traffic is marked in the Mangle rules correctly. Please prompt me where to look and what needs to be corrected into config?

ros code

/queue tree
add max-limit=2800k name=LEVEL_A_UP parent=ether1 priority=7 queue=default
add max-limit=2800k name=LEVEL_A_DOWN parent=ether2 priority=7 queue=default
add max-limit=2800k name=LEVEL_B_UP parent=ether1 priority=7 queue=default
add max-limit=2800k name=LEVEL_B_DOWN parent=ether2 priority=7 queue=default
add max-limit=2800k name=LEVEL_C_UP parent=ether1 queue=default
add max-limit=2800k name=LEVEL_C_DOWN parent=ether2 queue=default
add name=ACK_U packet-mark=ACK parent=LEVEL_A_UP priority=1 queue=default
add name=ACK_D packet-mark=ACK parent=LEVEL_A_DOWN priority=1 queue=default
add name=DNS_U packet-mark=DNS parent=LEVEL_A_UP priority=2 queue=default
add name=DNS_D packet-mark=DNS parent=LEVEL_A_DOWN priority=2 queue=default
add name=ICMP_U packet-mark=ICMP parent=LEVEL_A_UP priority=3 queue=default
add name=ICMP_D packet-mark=ICMP parent=LEVEL_A_DOWN priority=3 queue=default
add name=UDP_U packet-mark=UDP parent=LEVEL_B_UP priority=1 queue=default
add name=UDP_D packet-mark=UDP parent=LEVEL_B_DOWN priority=1 queue=default
add name=OTHER_U packet-mark=OTHER parent=LEVEL_C_UP priority=1 queue=default
add name=OTHER_D packet-mark=OTHER parent=LEVEL_C_DOWN priority=1 queue=default
add name=HTTP_U packet-mark=HTTP parent=LEVEL_B_UP priority=3 queue=default
add name=HTTP_D packet-mark=HTTP parent=LEVEL_B_DOWN priority=3 queue=default
add name=HTTP_BIG_U packet-mark=HTTP_BIG parent=LEVEL_B_UP priority=3 queue=default
add name=HTTP_BIG_D packet-mark=HTTP_BIG parent=LEVEL_B_DOWN priority=3 queue=default
add name=E-MAIL_U packet-mark=E-MAIL parent=LEVEL_B_UP priority=2 queue=default
add name=E-MAIL_D packet-mark=E-MAIL parent=LEVEL_B_DOWN priority=2 queue=default
add name=TCP_TORRENT_U packet-mark=TCP_TORRENT parent=LEVEL_C_UP priority=2 queue=default
add name=TCP_TORRENT_D packet-mark=TCP_TORRENT parent=LEVEL_C_DOWN priority=2 queue=default
add name=UDP_TORRENT_U packet-mark=UDP_TORRENT parent=LEVEL_C_UP priority=2 queue=default
add name=UDP_TORRENT_D packet-mark=UDP_TORRENT parent=LEVEL_C_DOWN priority=2 queue=default
/ip firewall mangle
add action=mark-connection chain=prerouting comment=DNS connection-state=new new-connection-mark=DNS port=53 protocol=udp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=DNS port=53 protocol=udp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=ICMP connection-state=new new-connection-mark=ICMP protocol=icmp
add action=mark-packet chain=prerouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=ICMP protocol=icmp
add action=mark-packet chain=postrouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-packet chain=postrouting comment=ACK new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-connection chain=prerouting comment=UDP_TORRENT connection-state=new new-connection-mark=UDP_TORRENT port=6881-6891 protocol=udp
add action=mark-packet chain=prerouting connection-mark=UDP_TORRENT new-packet-mark=UDP_TORRENT passthrough=no
add action=mark-connection chain=prerouting comment=UDP connection-state=new new-connection-mark=UDP protocol=udp
add action=mark-packet chain=prerouting connection-mark=UDP new-packet-mark=UDP passthrough=no
add action=mark-connection chain=prerouting comment=E-MAIL connection-state=new new-connection-mark=E-MAIL port=110,995,143,993,25,465,587,2525 protocol=tcp
add action=mark-packet chain=prerouting connection-mark=E-MAIL new-packet-mark=E-MAIL passthrough=no
add action=mark-connection chain=prerouting comment=HTTP connection-mark=!HTTP_BIG connection-state=new new-connection-mark=HTTP port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-bytes=500000-0 connection-mark=HTTP connection-rate=200k-100M new-connection-mark=HTTP_BIG protocol=tcp
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no
add action=mark-connection chain=prerouting comment=TCP_TORRENT connection-state=new new-connection-mark=TCP_TORRENT port=6881-6891 protocol=tcp
add action=mark-packet chain=prerouting connection-mark=TCP_TORRENT new-packet-mark=TCP_TORRENT passthrough=no
add action=mark-connection chain=prerouting comment=OTHER connection-mark=no-mark new-connection-mark=OTHER
add action=mark-packet chain=prerouting connection-mark=OTHER new-packet-mark=OTHER passthrough=no

Rudios · Fri Mar 28, 2014 8:22 am

I'm a little curious why you have some rules twice

ros code

/ip firewall mangle
add chain=forward action=mark-connection protocol=udp   src-address=192.168.100.5 connection-state=new new-connection-mark="VOIP" comment="IP-PBX"
add chain=forward action=mark-packet     passthrough=no connection-mark="VOIP"    new-packet-mark="VOIP"
add chain=forward action=mark-connection protocol=udp   dst-address=192.168.100.5 connection-state=new new-connection-mark="VOIP"
add chain=forward action=mark-packet     passthrough=no connection-mark="VOIP"    new-packet-mark="VOIP"

You are marking packets twice based on the same options. In my opinion rule 2 can be removed.
Then first there are two rules marking the connection and setting connection-marks. Then use the connection-mark to mark the packets in one single rule.

mdkberry · Thu May 22, 2014 6:30 am

Ok been a while since I was on this thread but I have tried the above setup and run into a strange problem where marking the packets stops some web sites being available. I am not sure what is causing it and hoped someone could look at the below Mangle rules and tell me.

I have enabled everything except HTTP, HTTP_BIG and one I called 'THE_REST' because when I enabled these no one could access various web sites, such as http://www.adobe.com http://www.sybase.com and a host of others. It was always the same web sites and lots of them, until I disabled those three rules then it works fine, I am not sure if the other traffic like ACK or DNS is having a problem but no one has complained and it has been up a couple of days like this now.

I assume this is because those disabled rules are stopping the traffic flow passing to the dynamic rule set that appeared after I configured the PPPoE TPG fibre gateway. I have added those two dynamic rules in at the end of this post.

should just marking traffic cause it to stop going out the normal route? or have I done something wrong that I cant see here?

my mangle rules:

0   ;;; VOIP - Voip traffic FROM PBX (5060-5061, 10000-20000 udp)
     chain=forward action=mark-connection new-connection-mark=VOIP passthrough=yes connection-state=new protocol=udp
     src-address=10.10.20.5 port=5060-5061,10000-20000

 1   chain=forward action=mark-packet new-packet-mark=VOIP passthrough=no connection-mark=VOIP

 2   ;;; VOIP - Voip traffic TO PBX (5060-5061, 10000-20000 udp)
     chain=output action=mark-connection new-connection-mark=VOIP passthrough=yes connection-state=new protocol=udp
     dst-address=10.10.20.5 port=5060-5061,10000-20000

 3   chain=output action=mark-packet new-packet-mark=VOIP passthrough=no connection-mark=VOIP

 4   ;;; V_GENERAL - other traffic FROM voice network
     chain=forward action=mark-connection new-connection-mark=V_GENERAL passthrough=yes src-address-list=VoIP_Network
     connection-mark=no-mark

 5   chain=forward action=mark-packet new-packet-mark=V_GENERAL passthrough=no connection-mark=V_GENERAL

 6   ;;; V_GENERAL - other traffic TO voice network
     chain=output action=mark-connection new-connection-mark=V_GENERAL passthrough=yes dst-address-list=VoIP_Network
     connection-mark=no-mark

 7   chain=output action=mark-packet new-packet-mark=V_GENERAL passthrough=no connection-mark=V_GENERAL

 8   ;;; DNS
     chain=forward action=mark-connection new-connection-mark=DNS passthrough=yes connection-state=new protocol=udp port=53

 9   chain=forward action=mark-packet new-packet-mark=DNS passthrough=no connection-mark=DNS

10   chain=output action=mark-connection new-connection-mark=DNS passthrough=yes connection-state=new protocol=udp port=53

11   chain=output action=mark-packet new-packet-mark=DNS passthrough=no connection-mark=DNS

12   ;;; UDP
     chain=forward action=mark-connection new-connection-mark=UDP passthrough=yes connection-state=new protocol=udp

13   chain=forward action=mark-packet new-packet-mark=UDP passthrough=no connection-mark=UDP

14   ;;; ICMP
     chain=forward action=mark-connection new-connection-mark=ICMP passthrough=yes connection-state=new protocol=icmp

15   chain=forward action=mark-packet new-packet-mark=ICMP passthrough=no connection-mark=ICMP

16   chain=output action=mark-connection new-connection-mark=ICMP passthrough=yes connection-state=new protocol=icmp

17   chain=output action=mark-packet new-packet-mark=ICMP passthrough=no connection-mark=ICMP

18   ;;; ACK
     chain=output action=mark-packet new-packet-mark=ACK passthrough=no tcp-flags=ack protocol=tcp packet-size=0-123

19   chain=forward action=mark-packet new-packet-mark=ACK passthrough=no tcp-flags=ack protocol=tcp packet-size=0-123

20 X ;;; HTTP
     chain=forward action=mark-connection new-connection-mark=HTTP passthrough=yes connection-state=new protocol=tcp
     port=80,443 connection-mark=!HTTP_BIG

21 X chain=forward action=mark-connection new-connection-mark=HTTP_BIG passthrough=yes protocol=tcp connection-mark=HTTP
     connection-bytes=500000-0 connection-rate=200k-100M

22 X chain=forward action=mark-packet new-packet-mark=HTTP_BIG passthrough=no connection-mark=HTTP_BIG

23 X chain=forward action=mark-packet new-packet-mark=HTTP passthrough=no connection-mark=HTTP

24 X ;;; THE_REST
     chain=forward action=mark-connection new-connection-mark=THE_REST passthrough=yes connection-mark=no-mark

25 X chain=forward action=mark-packet new-packet-mark=THE_REST passthrough=no connection-mark=THE_REST

Below are the dynamic rules in place from the PPPoE gateway , they are the last two rules in my mangle list but dont show when doing a print from command line.

chain=forward
protocol= tcp
in interface = all ppp
TCP MSS = 1441-65535
TCP Flags = syn
action = change MSS
new TCP MSS = 1440
(no traffic through this as yet)

chain=forward
protocol = tcp
out.interface = all ppp
TCP MSS = 1441-65535
TCP Flags = syn
action = change MSS
new TCP MSS = 1440
passthrough= ticked
(quite a lot of traffic has passed through this rule)

pcunite · Sun May 25, 2014 4:25 pm

I'm a little curious why you have some rules twice.
ros code
/ip firewall mangle
add chain=forward action=mark-connection protocol=udp   src-address=192.168.100.5 connection-state=new new-connection-mark="VOIP" comment="IP-PBX"
add chain=forward action=mark-packet     passthrough=no connection-mark="VOIP"    new-packet-mark="VOIP"
add chain=forward action=mark-connection protocol=udp   dst-address=192.168.100.5 connection-state=new new-connection-mark="VOIP"
add chain=forward action=mark-packet     passthrough=no connection-mark="VOIP"    new-packet-mark="VOIP"
You are marking packets twice based on the same options. In my opinion rule 2 can be removed.
Then first there are two rules marking the connection and setting connection-marks. Then use the connection-mark to mark the packets in one single rule.

This section is from Option 1. It is marking "src" and then "dst" to the same ip address. Also, unless MikroTik documentation has changed, one must mark the connection and then the packet for performance.

TikUser · Thu May 29, 2014 6:44 pm

Hey mdkberry,
it would be good to give us a visual scheme of your network. So, we could easily see what goes where.

First, do you want to do QoS for traffic going/coming to/from the internet, or for all the traffic?
If it's the Internet, then your mangle rules need to be more precise.
Example (imprecise rule):

add action=mark-connection chain=forward connection-state=new new-connection-mark=UDP protocol=udp

This rule catches ALL the udp traffic going through your router. Not only internet udp traffic. If you have users on ether3, ether4 and ether 5 and they are communicating with each other, then this rule will also catch their udp inside the network traffic.

You said that your internet speed is 5Mb up/down. If your network is connected with Ethernet cables, then you have at least 100Mb of available bandwidth inside your network. QoS for inside traffic is NOT needed, because you have more than enough bandwidth.
If your internet connection is on ether1, then change the rule:

add action=mark-connection chain=forward connection-state=new new-connection-mark=UDP out-interface=ether1 protocol=udp
add action=mark-packet chain=forward connection-mark=UDP new-packet-mark=UDP passthrough=no

With this rule you are cathing all the udp connections going to the internet. You don't need to specify the mark-packet rule, because it uses the data from the mark-connection rule.

Why are you using output chain for the VoIP traffic to PBX and other traffic to voice network? Output chain means only traffic that is generated by the Mikrotik router.
If you have a scheme: PBX VOIP <-> Mikrotik-router <-> Internet, then the output chain will NOT catch this traffic, unless you have modified the ROS to redirect the VoIP traffic to the router itself and then to other interfaces.
Try with this rules:

add action=mark-connection chain=forward connection-state=new new-connection-mark=VOIP out-interface=ether1 port=5060-5061,10000-20000 protocol=udp src-address=10.10.20.5 
add action=mark-connection chain=forward connection-state=new dst-address=10.10.20.5 in-interface=ether1 new-connection-mark=VOIP port=5060-5061,10000-20000 protocol=udp

First mark-connection rule catches voip connections initiated from 10.10.20.5 and going to the internet(ether1). Second mark-connection rule catches connections initiated from the internet(ether1) and going to 10.10.20.5.
If on 10.10.20.5 you only have the PBX, you can remove ports and protocol from these rules. Also replace the connection-mark=no-mark with connection-state=new in rules 4 and 6.

If I were at your place, I would remove the udp rules 12 and 13 completely because, for your configuration, they don’t make any sense.
And I would integrate the ACK rules with the HTTP and HTTP_BIG rules. Like this:

add action=mark-packet chain=forward connection-mark=HTTP_BIG new-packet-mark=ACK-HTTP-BIG packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=forward connection-mark=HTTP new-packet-mark=ACK-HTTP packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack

These rules catch the ack packets from HTTP & HTTP_BIG connections. Put them AFTER the HTTP & HTTP_BIG mark-connection rules, and BEFORE the HTTP & HTTP_BIG mark-packet rules.
Example:
1. HTTP mark-connection rule
2. HTTP_BIG mark-connection rule
3. ACK-HTTP-BIG mark-packet rule
4. ACK-HTTP mark-packet rule
5. HTTP_BIG mark-packet rule
6. HTTP mark-packet rule

About the PPPOE. You will need to play with this.

I don't have a PPPOE configuration (yet), so I can't help you in this field.
- Disable all the static mangle rules.
- Check if traffic is flowing through both dynamic pppoe rules.
- Change the MSS size, if needed.
- Change the Max UDP Packet size in DNS Settings, if needed.
- Try with putting the dynamic mangle rules above all static mangle rules.
- Search this forum for more information about ppp dynamic rules and mss size.

beb · Thu Jun 26, 2014 3:58 am

How Buffering and bufferbloat in ISP supplied equipment affects Qos and latency sensitive protocols ...

Thank you for this great explanation!

Am I correct in my conclusion that without a reasonable estimate of my wan bandwidth, it's impossible to implement qos?

Specifically - I have Comcast (cable) and the download speed varies between 60Mbps and 5Mbps, depending on the time-of-day, phase of the moon, and Comcast's share price. If I understand correctly, if I want to implement qos I need to set the maximum limit to <5Mbps. This of course suffices for VOIP, but gives me much less than available bandwidth most of the time.

A bit sobering, but at least I am not wasting my time trying to fix something I cannot.

Correct?

pcunite · Fri Jun 27, 2014 9:09 pm

Am I correct in my conclusion that without a reasonable estimate of my wan bandwidth, it's impossible to implement qos? Specifically - I have Comcast (cable) and the download speed varies between 60Mbps and 5Mbps

That is quite a variance! Yes, you are correct ... you can not use "static Qos" techniques to maintain your traffic integrity. There are "dynamic Qos" ways of doing things but that is beyond the scope of this thread and my expertise.

war3boy · Sun Jun 29, 2014 4:16 pm

Hi Guys,

Need some light for the Latency QoS. I try to shape but seem it failed to do so.
I knew we can use bandwidth control to guarantee the bandwidth for gaming , but i wish can fully utilize the bandwidth when the game is not in play.
But when the game is in play, the QoS will do it work to keep the game latency as usual.

ros code

# jun/29/2014 21:38:22 by RouterOS 6.12
# software id = AF8C-ZI5T
#
/ip firewall mangle
#Skype Layer7 Marking
add action=mark-packet chain=prerouting comment=Skype in-interface=PPPoE-UniFi layer7-protocol=skypetoskype new-packet-mark=QoS_1_In passthrough=no
add action=mark-packet chain=postrouting layer7-protocol=skypetoskype new-packet-mark=QoS_1_Out out-interface=PPPoE-UniFi passthrough=no

#VPN Layer 7 Marking
add action=mark-packet chain=prerouting comment=VPN in-interface=PPPoE-UniFi new-packet-mark=QoS_2_In passthrough=no protocol=gre
add action=mark-packet chain=postrouting new-packet-mark=QoS_2_Out out-interface=PPPoE-UniFi passthrough=no protocol=gre

#QoS_4 Steaming Layer 7 Marking
add action=mark-packet chain=prerouting comment="------------QoS_4 [Streaming_Services]------------" in-interface=PPPoE-UniFi layer7-protocol=Streaming new-packet-mark=QoS_4_In passthrough=no
add action=mark-packet chain=postrouting layer7-protocol=Streaming new-packet-mark=QoS_4_Out out-interface=PPPoE-UniFi passthrough=no
add action=mark-packet chain=prerouting in-interface=PPPoE-UniFi layer7-protocol=video new-packet-mark=QoS_4_In passthrough=no
add action=mark-packet chain=postrouting layer7-protocol=video new-packet-mark=QoS_4_Out out-interface=PPPoE-UniFi passthrough=no

#QoS_8 Torrent Layer 7 Marking
add action=mark-packet chain=prerouting comment="------------QoS_8 [Torrent_Services]------------" in-interface=PPPoE-UniFi layer7-protocol=Torrent-wwws new-packet-mark=QoS_8_In passthrough=no
add action=mark-packet chain=postrouting layer7-protocol=Torrent-wwws new-packet-mark=QoS_8_Out out-interface=PPPoE-UniFi passthrough=no
add action=mark-packet chain=prerouting in-interface=PPPoE-UniFi layer7-protocol=torrentDNS new-packet-mark=QoS_8_In passthrough=no
add action=mark-packet chain=postrouting layer7-protocol=torrentDNS new-packet-mark=QoS_8_Out out-interface=PPPoE-UniFi passthrough=no
add action=mark-packet chain=prerouting in-interface=PPPoE-UniFi new-packet-mark=QoS_8_In p2p=all-p2p passthrough=no
add action=mark-packet chain=postrouting new-packet-mark=QoS_8_Out out-interface=PPPoE-UniFi p2p=all-p2p passthrough=no

#QoS_1 Steam
add action=mark-packet chain=prerouting comment="------------QoS_1 [Steam]------------" in-interface=PPPoE-UniFi new-packet-mark=QoS_1_In passthrough=no protocol=tcp src-port=27014-27050
add action=mark-packet chain=postrouting dst-port=27014-27050 new-packet-mark=QoS_1_Out out-interface=PPPoE-UniFi passthrough=no protocol=tcp
add action=mark-packet chain=prerouting in-interface=PPPoE-UniFi new-packet-mark=QoS_1_In passthrough=no protocol=udp src-port=3478,4379,4380,28960,27000-27030
add action=mark-packet chain=postrouting dst-port=3478,4379,4380,28960,27000-27030 new-packet-mark=QoS_1_Out out-interface=PPPoE-UniFi passthrough=no protocol=udp

#QoS_2 DNS
add action=mark-packet chain=prerouting comment="------------QoS_2 [DNS+NTP_PPTP_Services]------------" in-interface=PPPoE-UniFi new-packet-mark=QoS_2_In passthrough=no protocol=udp src-port=53,123,1723
add action=mark-packet chain=postrouting dst-port=53,123,1723 new-packet-mark=QoS_2_Out out-interface=PPPoE-UniFi passthrough=no protocol=udp
add action=mark-packet chain=prerouting connection-state=new in-interface=PPPoE-UniFi new-packet-mark=QoS_2_In passthrough=no protocol=tcp src-port=52,123,1723
add action=mark-packet chain=postrouting dst-port=52,123,1723 new-packet-mark=QoS_2_Out out-interface=PPPoE-UniFi passthrough=no protocol=tcp

#QoS_3 WWW Services + General_Services
add action=mark-packet chain=prerouting comment="------------QoS_3 [WWW_Services]------------" connection-bytes=0-1500000 in-interface=PPPoE-UniFi new-packet-mark=QoS_3_In passthrough=no protocol=tcp src-port=80,443
add action=mark-packet chain=postrouting connection-bytes=0-1500000 dst-port=80,443 new-packet-mark=QoS_3_Out out-interface=PPPoE-UniFi passthrough=no protocol=tcp
add action=mark-packet chain=prerouting comment="------------QoS_3 [General_Services]------------" in-interface=PPPoE-UniFi new-packet-mark=QoS_3_In passthrough=no protocol=tcp src-port=20,21,22,23,25,110,143,465,587,993,995,2525,3535
add action=mark-packet chain=postrouting dst-port=20,21,22,23,25,110,143,465,587,993,995,2525,3535 new-packet-mark=QoS_3_Out out-interface=PPPoE-UniFi passthrough=no protocol=tcp

#QoS_5 WWW_Heavy
add action=mark-packet chain=prerouting comment="------------QoS_5 [WWW_Services_Heavy]------------" connection-bytes=1500000-0 in-interface=PPPoE-UniFi new-packet-mark=QoS_5_In passthrough=no protocol=tcp src-port=80,443
add action=mark-packet chain=postrouting connection-bytes=1500000-0 dst-port=80,443 new-packet-mark=QoS_5_Out out-interface=PPPoE-UniFi passthrough=no protocol=tcp

#QoS_8 Other BULK
add action=mark-packet chain=prerouting comment="------------QoS_8 [Other_Services]------------" in-interface=PPPoE-UniFi new-packet-mark=QoS_8_In passthrough=no protocol=tcp
add action=mark-packet chain=postrouting new-packet-mark=QoS_8_Out out-interface=PPPoE-UniFi passthrough=no protocol=tcp
add action=mark-packet chain=prerouting in-interface=PPPoE-UniFi new-packet-mark=QoS_8_In passthrough=no protocol=udp
add action=mark-packet chain=postrouting new-packet-mark=QoS_8_Out out-interface=PPPoE-UniFi passthrough=no protocol=udp
add action=mark-packet chain=prerouting in-interface=PPPoE-UniFi new-packet-mark=QoS_8_In passthrough=no
add action=mark-packet chain=postrouting new-packet-mark=QoS_8_Out out-interface=PPPoE-UniFi passthrough=no

ros code

# jun/29/2014 21:27:09 by RouterOS 6.12
# software id = AF8C-ZI5T
#
/queue tree

#Upload
add max-limit=4700k name=QoS_Global_Upload parent=global queue=default
add limit-at=512k max-limit=3M name="QoS_1_Out(Steam)" packet-mark=QoS_1_Out parent=QoS_Global_Upload priority=1 queue=default
add limit-at=100k max-limit=1M name="QoS_2_Out(DNS+NTP+PPTP)" packet-mark=QoS_2_Out parent=QoS_Global_Upload priority=2 queue=default
add limit-at=1500k max-limit=4M name="QoS_3_Out(WWW)" packet-mark=QoS_3_Out parent=QoS_Global_Upload priority=3 queue=default
add burst-limit=3800k burst-time=3s max-limit=3M name="QoS_4_Out(Streaming)" packet-mark=QoS_4_Out parent=QoS_Global_Upload priority=4 queue=default
add burst-limit=3800k burst-time=3s max-limit=3M name="QoS_5_Out(WWW_Heavy)" packet-mark=QoS_5_Out parent=QoS_Global_Upload priority=5 queue=default
add burst-limit=4300k burst-time=3s max-limit=3500k name="QoS_8_Out(Default)" packet-mark=QoS_8_Out parent=QoS_Global_Upload queue=pcq-upload-default

#Download
add max-limit=4800k name=QoS_Global_Download parent=global queue=default
add limit-at=512k max-limit=3M name="QoS_1_In(Steam)" packet-mark=QoS_1_In parent=QoS_Global_Download priority=1 queue=default
add limit-at=100k max-limit=500k name="QoS_2_In(DNS+NTP+PPTP)" packet-mark=QoS_2_In parent=QoS_Global_Download priority=2 queue=default
add burst-limit=3M burst-threshold=2M burst-time=5s limit-at=1500k max-limit=2M name="QoS_3_In(WWW)" packet-mark=QoS_3_In parent=QoS_Global_Download priority=3 queue=default
add burst-limit=2500k burst-threshold=1M burst-time=5s max-limit=1M name="QoS_4_In(Streaming)" packet-mark=QoS_4_In parent=QoS_Global_Download priority=4 queue=default
add burst-limit=3800k burst-time=2s max-limit=1M name="QoS_5_In(WWW_Heavy)" packet-mark=QoS_5_In parent=QoS_Global_Download priority=5 queue=default
add burst-limit=4300k burst-threshold=4M burst-time=3s max-limit=3M name="QoS_8_In(Default+Torrent)" packet-mark=QoS_8_In parent=QoS_Global_Download queue=pcq-download-default

ceather93 · Fri Sep 12, 2014 5:56 am

Hi,

I am getting a bit confused with this. Wondering if someone could help me out.

I am trying to achieve the following in terms of QoS - in that order

1) All WAN related UDP connections
2) All Connections from and to Certain Address List
3) All Connections from and to a different address List
4) All Other Traffic
5) File Sharing/Torrents

I think I am stumbling on the packet flow part and why you need to set a limit on the LAN side of things in the queuing?
Other question is for the mangle do you need to make a separate rule for both outgoing and incoming traffic eg:

 2 X ;;; LYNC | Mark new source connections
     chain=forward action=mark-connection new-connection-mark=Lync-Connected
     passthrough=yes connection-state=new src-address-list=Lync in-interface=TPG PPPoe

 3 X ;;; LYNC | Mark Packet
     chain=forward action=mark-packet new-packet-mark=Lync_Packet passthrough=yes
     connection-mark=Lync-Connected

 4 X ;;; LYNC | Mark new dest connections
     chain=forward action=mark-connection new-connection-mark=Lync-Connected
     passthrough=yes connection-state=new dst-address-list=Lync
     out-interface=TPG PPPoe

 5 X ;;; LYNC | Mark Packet
     chain=forward action=mark-packet new-packet-mark=Lync_Packet passthrough=no
     connection-mark=Lync-Connected

Interaces:

Bridge Called 'Bridge' (Ether2 in it - connected to switch)
PPPoE Client called 'PPPoE' assigned to Ether1

vamose · Tue Mar 03, 2015 7:51 pm

Guys,

I bought the Mikrotik router and used AP and PPOE settings to enable the router and my ISP provider.

I have 10MBs service, but when I use speediest websites such as testmy.net net performance maxes out at 3.2MBps.

I need your help with streamlining my configuration and with QOS. The requirement for QOS is as per the priority listed, can any one help me.

VOIP
Streaming video content - You tube
SSH
FTP
Downloading
Torrent downloads.

toxicfusion · Tue Mar 17, 2015 8:57 pm

Hi

I've followed this guide... and have made adjustments to try and fix issues.

However, when ever I have queue tree rules created for 'no-mark'. The router is ALSO clamping the LAN network file transfer speeds... not cool.

/queue tree
add max-limit=4M name=LEVEL_A_UP parent=ether1_wan01 queue=default
add max-limit=22M name=LEVEL_A_DOWN parent=TRNK_LAN queue=default
add limit-at=3799k max-limit=3880k name=VOIP_UP packet-mark=VOIP parent=\
LEVEL_A_UP priority=1 queue=default
add limit-at=20M max-limit=22M name=VOIP_DWN packet-mark=VOIP parent=\
LEVEL_A_DOWN priority=1 queue=default
add name=DNS_UP packet-mark=DNS parent=LEVEL_A_UP priority=2 queue=default
add name=DNS_DWN packet-mark=DNS parent=LEVEL_A_DOWN priority=2 queue=default
add max-limit=60M name=LEVEL_B_DOWN parent=TRNK_LAN queue=default
add max-limit=3800k name=LEVEL_B_UP parent=ether1_wan01 queue=default
add limit-at=2600k max-limit=3M name=HTTP_UP packet-mark=HTTP parent=LEVEL_B_UP \
priority=2 queue=default
add limit-at=59M max-limit=60M name=HTTP_DWN packet-mark=HTTP parent=\
LEVEL_B_DOWN priority=2 queue=default
add limit-at=2800k max-limit=3M name=HTTP_Traff packet-mark=HTTP_Traff parent=\
LEVEL_B_UP priority=2 queue=default
add limit-at=2800k max-limit=3M name=HTTPS_up packet-mark=HTTPS parent=\
LEVEL_B_UP priority=2 queue=default
add limit-at=58M max-limit=60M name=HTTPS_Down packet-mark=HTTPS parent=\
LEVEL_B_DOWN priority=2 queue=default
add limit-at=55M max-limit=58M name=IPSEC_Down packet-mark=IPSEC parent=\
LEVEL_B_DOWN priority=2 queue=default
add limit-at=2800k max-limit=3M name=IPSEC_OUT_IN packet-mark=IPSEC parent=\
LEVEL_B_UP priority=3 queue=default
add disabled=yes limit-at=58M max-limit=60M name=misc_down packet-mark=no-mark \
parent=LEVEL_B_DOWN priority=3 queue=default

kivimart · Thu Mar 19, 2015 10:46 pm

This is my queue tree, no limits set only trying to learn how to mark packet and queue then then later set limits.

I do not use VOIP in my setup...

If you post your Mangle rules there will be easier to help you with your problem on local traffic hitting queues.
I am no expert on queues but will help you as much as i can.

My routerboard is a RB850Gx2

Gateway interface Name=GW1

/ip firewall layer7-protocol
add name=bittorrent regexp="^(\\x13bittorrent protocol|azver\\x01\$|get /scrape\\\?info_hash=ge\
t /announce\\\?info_hash=|get /client/bitcomet/|GET /data\\\?fid=)|d1:ad2:id20:|\\x08'7P\\)\
[RP]"
add name=streaming regexp=videoplayback|video|stream
add name=streaming-audio regexp=audioplayback|audio

/ip firewall mangle
add action=mark-connection chain=prerouting comment=DNS-IN new-connection-mark=DNS port=53 \
protocol=udp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=output comment=DNS-UT new-connection-mark=DNS_UP port=53 \
protocol=udp
add action=mark-packet chain=output connection-mark=DNS_UP new-packet-mark=DNS_UP passthrough=\
no
add action=mark-connection chain=prerouting comment=ICMP-IN in-interface=GW1 \
new-connection-mark=ICMP-CON protocol=icmp
add action=mark-packet chain=prerouting connection-mark=ICMP-CON new-packet-mark=ICMP \
passthrough=no
add action=mark-connection chain=postrouting comment=ICMP-UT new-connection-mark=ICMP_UP-CON \
out-interface=GW1 protocol=icmp
add action=mark-packet chain=postrouting connection-mark=ICMP_UP-CON new-packet-mark=ICMP_UP \
passthrough=no
add action=mark-connection chain=prerouting comment="WinBox, Sniffer" new-connection-mark=\
WinBox-CON port=8291,8081 protocol=tcp
add action=mark-packet chain=prerouting connection-mark=WinBox-CON new-packet-mark=WinBox \
passthrough=no
add action=mark-connection chain=postrouting comment="WinBox, Sniffer-UT" new-connection-mark=\
WinBox-CON_UP port=8291,8081 protocol=tcp
add action=mark-packet chain=postrouting connection-mark=WinBox-CON_UP new-packet-mark=\
WinBox_UP passthrough=no
add action=mark-connection chain=prerouting comment=SlingBox-IN in-interface=GW1 \
new-connection-mark=SLINGBOX-CON port=5001,5757 protocol=tcp
add action=mark-packet chain=prerouting connection-mark=SLINGBOX-CON new-packet-mark=SLINGBOX \
passthrough=no
add action=mark-connection chain=postrouting comment=SlingBox-UT new-connection-mark=\
SLINGBOX_UP out-interface=GW1 port=5001 protocol=tcp
add action=mark-packet chain=postrouting connection-mark=SLINGBOX_UP new-packet-mark=\
SLINGBOX_UP passthrough=no
add action=mark-connection chain=prerouting comment=Stream-AUDIO layer7-protocol=\
streaming-audio new-connection-mark=STREAM-CON_AUDIO
add action=mark-packet chain=prerouting connection-mark=STREAM-CON_AUDIO new-packet-mark=\
STREAM_AUDIO passthrough=no
add action=mark-connection chain=prerouting comment=Stream-VIDEO layer7-protocol=streaming \
new-connection-mark=STREAM-CON
add action=mark-packet chain=prerouting connection-mark=STREAM-CON new-packet-mark=STREAM \
passthrough=no
add action=mark-connection chain=prerouting comment=HTTP-IN connection-mark=!HTTP_BIG \
in-interface=GW1 new-connection-mark=HTTP port=80,443,8080 protocol=tcp
add action=mark-connection chain=prerouting connection-bytes=500000-0 connection-mark=HTTP \
new-connection-mark=HTTP_BIG protocol=tcp
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=ACK-HTTP-BIG \
packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=ACK-HTTP \
packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG \
passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=\
no
add action=mark-connection chain=postrouting comment=HTTP_UT connection-mark=!HTTP_BIG_UT \
new-connection-mark=HTTP_UT out-interface=GW1 port=80,443,8080 protocol=tcp
add action=mark-connection chain=postrouting connection-bytes=500000-0 connection-mark=HTTP_UT \
new-connection-mark=HTTP_BIG_UT protocol=tcp
add action=mark-packet chain=postrouting connection-mark=HTTP_BIG_UT new-packet-mark=\
ACK-HTTP-BIG_UP packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting connection-mark=HTTP_UT new-packet-mark=ACK-HTTP_UT \
packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting connection-mark=HTTP_BIG_UT new-packet-mark=\
HTTP_BIG_UT passthrough=no
add action=mark-packet chain=postrouting connection-mark=HTTP_UT new-packet-mark=HTTP_UT \
passthrough=no
add action=mark-connection chain=prerouting comment=SNMP new-connection-mark=SNMP-CON port=161 \
protocol=udp
add action=mark-packet chain=prerouting connection-mark=SNMP-CON new-packet-mark=SNMP
add action=mark-connection chain=postrouting comment=SNMP_UP new-connection-mark=SNMP-CON_UP \
port=161 protocol=udp
add action=mark-packet chain=postrouting connection-mark=SNMP-CON_UP new-packet-mark=SNMP_UP
add action=mark-connection chain=prerouting comment=BITTORRENT layer7-protocol=bittorrent \
new-connection-mark=BITTORRENT
add action=mark-packet chain=prerouting connection-mark=BITTORRENT new-packet-mark=BITTORRENT \
passthrough=no
add action=mark-connection chain=postrouting comment=BITTORRENT_UP layer7-protocol=bittorrent \
new-connection-mark=BITTORRENT_UP out-interface=GW1
add action=mark-packet chain=postrouting connection-mark=BITTORRENT_UP new-packet-mark=\
BITTORRENT_UP passthrough=no
add action=mark-connection chain=prerouting comment=THE_REST connection-mark=no-mark \
in-interface=GW1 new-connection-mark=THE_REST-CON
add action=mark-packet chain=prerouting connection-mark=THE_REST-CON new-packet-mark=THE_REST \
packet-mark=!Work passthrough=no
add action=mark-connection chain=postrouting comment=THE_REST-UP connection-mark=no-mark \
new-connection-mark=THE_REST_UP-CON out-interface=GW1
add action=mark-packet chain=postrouting connection-mark=THE_REST_UP-CON new-packet-mark=\
THE_REST_UP passthrough=no

/queue tree
add name=Down parent=global queue=default
add name=Up parent=global queue=default
add name=THE_REST packet-mark=THE_REST parent=Down priority=7 queue=default
add name=THE_REST_UP packet-mark=THE_REST_UP parent=Up priority=7 queue=default
add name=HTTP packet-mark=HTTP parent=Down priority=3 queue=default
add name=HTTP_UP packet-mark=HTTP_UT parent=Up priority=3 queue=default
add name=HTTP_BIG_UP packet-mark=HTTP_BIG_UT parent=Up priority=6 queue=default
add name=HTTP_BIG packet-mark=HTTP_BIG parent=Down priority=7 queue=default
add name=ACK packet-mark=ACK-HTTP parent=Down priority=1 queue=default
add name=ACK_UP packet-mark=ACK-HTTP_UT parent=Up priority=1 queue=default
add name=DNS packet-mark=DNS parent=Down priority=2 queue=default
add name=DNS_UP packet-mark=DNS_UP parent=Up priority=2 queue=default
add name=ICMP packet-mark=ICMP parent=Down priority=2 queue=default
add name=ICMP_UP packet-mark=ICMP_UP parent=Up priority=2 queue=default
add name=SLINGBOX packet-mark=SLINGBOX parent=Down priority=6 queue=default
add max-limit=128k name=SLINGBOX_UP packet-mark=SLINGBOX_UP parent=Up queue=default
add name=BITTORRENT packet-mark=BITTORRENT parent=Down queue=default
add max-limit=64k name=BITTORRENT_UP packet-mark=BITTORRENT_UP parent=Up queue=default
add name=Stream packet-mark=STREAM parent=Down priority=5 queue=pcq-download-default
add name=STREAM_UP packet-mark=STREAM_UP parent=Up priority=4 queue=default
add name=STREAM-AUDIO packet-mark=STREAM_AUDIO parent=Down priority=4 queue=default
add name=WinBox packet-mark=WinBox parent=Down priority=2 queue=default
add name=WinBox_UP packet-mark=WinBox_UP parent=Up priority=2 queue=default
add name=ACK-HTTP-BIG packet-mark=ACK-HTTP-BIG parent=Down priority=1 queue=default
add name=ACK-HTTP-BIG_UP packet-mark=ACK-HTTP-BIG_UP parent=Up priority=1 queue=default
add name=SNMP packet-mark=SNMP parent=Down priority=2 queue=default
add name=SNMP_UP packet-mark=SNMP_UP parent=Up priority=2 queue=default

--------------------

toxicfusion · Fri Mar 27, 2015 8:04 pm

Below are my mangle rules

I use similar / same QoS for customers and they experience same issues on local UPLOAD traffic being clamped. meaning, if I do a file transfer over the network from one computer to another -- that transfer gets capped.

Mangle rules are essentially same for myself and i use for clients

/ip firewall mangle
add action=mark-connection chain=prerouting comment=VOIP_TRAFFIC \
new-connection-mark=VOIP protocol=udp src-port=5060-5099
add action=mark-packet chain=prerouting connection-mark=VOIP new-packet-mark=\
VOIP passthrough=no protocol=udp src-port=5060-5099
add action=mark-connection chain=prerouting comment=DNS dst-port=53 \
new-connection-mark=DNS protocol=udp
add action=mark-packet chain=prerouting connection-mark=DNS dst-port=53 \
new-packet-mark=DNS passthrough=no protocol=udp
add action=mark-connection chain=postrouting connection-mark=DNS \
connection-state=new dst-port=53 new-connection-mark=DNS protocol=udp
add action=mark-packet chain=postrouting connection-mark=DNS dst-port=53 \
new-packet-mark=DNS packet-mark=DNS passthrough=no protocol=udp
add action=mark-connection chain=prerouting comment="HTTP TRAFFIC" dst-port=80 \
new-connection-mark=HTTP protocol=tcp
add action=mark-packet chain=prerouting connection-mark=HTTP dst-port=80 \
new-packet-mark=HTTP passthrough=no protocol=tcp
add action=mark-connection chain=prerouting comment=HTTPS_Traffic dst-port=443 \
new-connection-mark=HTTPS protocol=tcp
add action=mark-packet chain=prerouting connection-mark=HTTPS dst-port=443 \
new-packet-mark=HTTPS passthrough=no protocol=tcp
add action=mark-connection chain=postrouting comment="IPSEC TRAFFIC" \
new-connection-mark=IPSEC protocol=gre
add action=mark-packet chain=postrouting connection-mark=IPSEC new-packet-mark=\
IPSEC passthrough=no protocol=gre
add action=mark-connection chain=postrouting new-connection-mark=IPSEC \
protocol=ipsec-esp
add action=mark-packet chain=postrouting connection-mark=IPSEC new-packet-mark=\
IPSEC passthrough=no protocol=ipsec-esp
add action=mark-connection chain=postrouting dst-port=500 new-connection-mark=\
IPSEC protocol=udp
add action=mark-packet chain=postrouting connection-mark=IPSEC dst-port=500 \
new-packet-mark=IPSEC protocol=udp
add action=mark-connection chain=prerouting comment=OTHER in-interface=\
ether1_wan01 new-connection-mark=no-mark
add action=mark-packet chain=prerouting connection-mark=no-mark in-interface=\
ether1_wan01 new-packet-mark=no-mark passthrough=no
add action=mark-connection chain=postrouting comment=TORRENTS \
new-connection-mark=p2p_conns p2p=all-p2p
add action=mark-packet chain=postrouting connection-mark=p2p_conns \
new-packet-mark=p2p_traff p2p=all-p2p passthrough=no

kivimart · Fri Mar 27, 2015 8:17 pm

I put my master interface in a bridge and the problem off cap local traffic went away.

toxicfusion · Wed Apr 01, 2015 5:17 pm

Thanks to know!

I have RB2011 routers out in field with similar configuration, except VLAN's and they're using BRIDGES.

So they might be fine.

It might just be an issue with this RB1100AH as I'm using 'router on stick' configuration with VLAN's assigned to ether-trunks

I'm worried that If I put the trunks into a bridge, it will have a huge negative impact on performance?

EngineerAustin · Sat Jun 20, 2015 7:41 pm

I am new to VOIP prioritization and I need a bit of advice and it looks like I'm at the correct forum.

I have a multiple site network with multiple subnet vlans:

Main Location Connections:
1) 50/50Mb fiber internet, protected with sonicwall 2600 firewall (not my choice of equipment, but what I have to use)
2) 20/20Mb fiber ethernet MPLS provided by ISP to circuit to remote locations

I have a mikrotik router that I have configured at main to do NAT & Routing with vlans.
the way I have configured the MikroTik is this:
1 subnet for VOIP VLAN network 51.0/24 DHCP on MikroTik VLAN Bridge, 1 access port for local network switch. PBX onsite.
1 subnet for local computer DATA VLAN network 50.0/24 DHCP on MikroTik VLAN Bridge, 1 access port, for Local Network Switch computers, printers, and all other
3 x subnet for remote sites DATA VLAN network 40.0/24, 30.0/24, 20.0/24 DHCP on MikroTik VLAN for computers, printers, and all other at remote sties that need to communicate with each other site
1 MPLS (ether5) Trunk Port with 4 DATA Vlans & 1 VOIP VLAN51 (20/20Mb to MPLS ether switch)

The remote sites are all identical and look like this:
1) access port for VOIP VLAN 51 (ether1)
2) access port for MAIN Data VLAN 51 (ether2)
3) access port for Local Network Vlan, 40, 30, 20 (ether3)
4) ether4 spare
5) MPLS (ether5) Trunk Port with 4 DATA Vlans & 1 VOIP VLAN51 (5/5Mb to MPLS ether switch)

I need QoS for VLAN 51 prioritized over all other traffic except primary default network traffic for routing.

Am I at the right spot? Please contact me if you can help or check over my setup and help with QoS.

Thank you for your time in this. This forum seemed to be pointing me in the correct direction with QoS, but I'm a little confused how to prioritize VLAN 51, at remote sites and then at the main location with different limits?

Also, I was wondering if I could just prioritize all VLAN 51 VOIP subnet, or if I needed to do port prioritization too.

Any guidance would be greatly appreciated.

Sincerely,

Lynn

alaskanjackal · Tue Sep 29, 2015 1:57 pm

I spent a gazillion hours tonight Googling and working through a bunch of different threads on the topic of QOS. (I'm new to Mikrotik and was going to wade in slowly, but I encountered some choppy voice on a test call today and thought I'd try to clean that up.)

I started with the script posted upthread by pcunite, but after reading some of the documentation, I made some changes and thought I'd post them here to share (in case it helps anyone, or in case someone would like to fix any massive errors I've created).

One of the issues with pcunite's script is that you have to back off from the rated line speed by 10% or so in order to give Queue A (the one with the VoIP markings) a chance to breathe, since there doesn't appear to be any logic that actually prioritizes Queue A over Queue C--it just basically relies on the fact that Queue C can't quite use the entire connection to prevent Queue A's packets from queuing at the ISP edge (in my case, the cable modem).

I think I've encountered a way to get a bit closer to the actual maximum throughput and let HTTP and other bulk downloads use the full connection without compromising VoIP (and other priority) packets. This is done by making all of the traffic classes child queues of a single parent queue and then ordering these child queues in priority--my understanding in the Mikrotik wiki documentation is that this should work.

I did some testing and it seems to work when I forced the link to saturation with HTTP traffic (there was no audible VoIP packet loss, and HTTP traffic showed a small amount of packets in the queue while the VoIP queue showed traffic and no queuing, indicating they were passed through without delay).

In my case, I'm on a 50m down, 3m up cable connection, and my ISP does ever so slightly overprovision the connection (~51mbps down and ~3.2mbps up, and they do a pretty good job of giving me my rated speed 24/7), so I wrote the script to push me right to the edge of 50M/3M, and it seems to work OK and gives me just barely shy of those figures to my workstation all while keeping the VoIP connection clean and ICMP pings to nearby servers in the single digits. That said, this is really just a test and a draft and a quick hack of the script (plus an attempt to remove a few of the seemingly redundant lines), so for those just stumbling upon this thread from Google, don't put this into production without it being vetted by the more experienced community members here.

I also threw in a few extra things--some lines (with limit-at=) to guarantee bandwidth for at least a couple of concurrent calls--this may be unnecessary since it's first in the priority list, but I figured it couldn't hurt. I also capped the bandwidth of VoIP and other priority traffic to a shade less than my actual connection, just in case for some reason something happens and a UDP flood attempts to monopolize my upstream bandwidth--there will be a little left over for HTTP and other traffic.

One thing--both the original script by pcunite and this one seem to fail to properly mark the HTTP_BIG traffic. I don't know if there have been any changes in RouterOS since pcunite last updated his script that may affect that, but I spent a good two hours working on trying to get that to work to no avail. Ultimately, I decided that as long as VoIP traffic is properly classified and prioritized and ACK packets are given priority on the uplink (which helps with the browser feeling responsive even on a saturated uplink), the need to separately classify bulk uploads is mitigated. Perhaps someone can troubleshoot that further.

Oh, one other thing--I did change the VoIP mangle rules to go by DSCP markings rather than ports, since my VoIP infrastructure does mark packets, but if yours doesn't, you'll want to uncomment the port-based matching lines and delete the ones that reference DSCP.

Anyway, it seems to be working here, but there's a very good chance I just got lucky in my testing and it really isn't working at all or something, so hopefully someone will come along to stop me from spreading misinformation if that's the case...but I hope this helps in some way.

# Date: September 29, 2015
# Version: 1.3b1, based on work by pcunite
# Tested with RouterOS 6.32.1
# Rename ether1-gateway and bridge-local to match your environment

###############################################################################
# Mangle
#
# Using prerouting/postrouting since we don't have dst or src checks.
#
/ip firewall mangle
###############################################################################

# DNS requests. Mark in two places because DNS is sent out by the router (itself) too.
add chain=prerouting action=mark-connection protocol=udp port=53 connection-state=new new-connection-mark="DNS" comment="DNS"
add chain=postrouting action=mark-connection protocol=udp port=53 connection-state=new new-connection-mark="DNS"
add chain=postrouting action=mark-packet passthrough=no connection-mark="DNS" new-packet-mark="DNS"

# Mark all VoIP TCP traffic based on DSCP marking 46 and 26.
add chain=prerouting action=mark-connection protocol=tcp dscp=46 new-connection-mark="VOIP" comment="VOIP"
add chain=prerouting action=mark-connection protocol=udp dscp=46 new-connection-mark="VOIP"
add chain=prerouting action=mark-connection protocol=tcp dscp=26 new-connection-mark="VOIP"
add chain=prerouting action=mark-connection protocol=udp dscp=26 new-connection-mark="VOIP"
add chain=prerouting action=mark-packet passthrough=no connection-mark="VOIP" new-packet-mark="VOIP"

# Mark all VoIP traffic. We've set all our equiptment to use SIP 5060,5061 and RTP 10000-20000.
# add chain=prerouting action=mark-connection protocol=udp port=5060,5061,10000-20000 new-connection-mark="VOIP" comment="VOIP"
# add chain=prerouting action=mark-packet passthrough=no connection-mark="VOIP" new-packet-mark="VOIP"

# Mark all UDP traffic. Mark different UDP streams if you want more granularity.
add chain=prerouting action=mark-connection protocol=udp connection-state=new new-connection-mark="UDP" comment="UDP"
add chain=prerouting action=mark-packet passthrough=no connection-mark="UDP" new-packet-mark="UDP"

# Ping replies. Mark in two places because ICMP is sent out by the router (itself) too.
add chain=prerouting action=mark-connection protocol=icmp connection-state=new new-connection-mark="ICMP" comment="ICMP"
add chain=postrouting action=mark-connection protocol=icmp connection-state=new new-connection-mark="ICMP"
add chain=postrouting action=mark-packet passthrough=no connection-mark="ICMP" new-packet-mark="ICMP"

# ACK traffic. Based on viewtopic.php?f=2&t=67965
add chain=postrouting action=mark-packet passthrough=no protocol=tcp tcp-flags=ack packet-size=0-123 new-packet-mark="ACK" comment="ACK"
add chain=prerouting action=mark-packet passthrough=no protocol=tcp tcp-flags=ack packet-size=0-123 new-packet-mark="ACK"

# Mark all new HTTP(s) connections with "HTTP" if they have not previously been marked as "HTTP_BIG".
# If the current mark of "HTTP" tranfers more than 5MB and at a rate of 200k+ then mark it as "HTTP_BIG" for the duration of the TCP session.
add chain=prerouting action=mark-connection protocol=tcp new-connection-mark="HTTP" connection-state=new port=80,443 comment="HTTP"
add chain=prerouting action=mark-connection protocol=tcp connection-mark="HTTP" new-connection-mark="HTTP_BIG" connection-bytes=500000-0 connection-rate=200k-1000M
add chain=prerouting action=mark-packet passthrough=no connection-mark="HTTP_BIG" new-packet-mark="HTTP_BIG"
add chain=prerouting action=mark-packet passthrough=no connection-mark="HTTP" new-packet-mark="HTTP"

# Mark everything else that has no mark applied.
add chain=prerouting action=mark-connection connection-mark=no-mark new-connection-mark="OTHER" comment="OTHER"
add chain=prerouting action=mark-packet passthrough=no connection-mark="OTHER" new-packet-mark="OTHER"

###############################################################################
# HTB Queue Tree a unidirectional queue
#
# Based on 50M down / 3M up connection (measured at 50400k/50.4M down, 3008k/3.0M up).
# Replace with your own values as needed and adjust downward if connection is unreliable.
#
# Notes:
# priority means 'drop packets' WHEN needed.
# When limit-at=0 priority starts when max-limit is reached.
# When limit-at=123 priority starts when limit-at is reached.
#
# The priority option applies to children not parents. Parent is for setting
# overall limits. Therefore use limit-at and max-limit on the children if
# you want more granularity.
#
# max-limit must always be set or priority will not happen.
#
# Tips for TCP (not VoIP) SOHO network:
# limit-at = Total bandwidth / max hosts
# max-limit = Total bandwidth / min hosts
#
/queue tree
###############################################################################

# The secret to ensuring VoIP quality (or any UDP traffic) is to put it into
# a queue that will never be full and thus never prioritize (drop) packets.
add name="MASTER_UP" parent=ether1-gateway queue=default max-limit=3000k
add name="MASTER_DOWN" parent=bridge-local queue=default max-limit=50000k

add name="VOIP_U" parent="MASTER_UP" packet-mark="VOIP" queue=default limit-at=200k max-limit=2850k priority=1
add name="VOIP_D" parent="MASTER_DOWN" packet-mark="VOIP" queue=default limit-at=200k max-limit=45000k priority=1
add name="ACK_U" parent="MASTER_UP" packet-mark="ACK" queue=default limit-at=200k max-limit=2850k priority=2
add name="ACK_D" parent="MASTER_DOWN" packet-mark="ACK" queue=default limit-at=2000k max-limit=45000k priority=2
add name="DNS_U" parent="MASTER_UP" packet-mark="DNS" queue=default limit-at=200k max-limit=2850k priority=3
add name="DNS_D" parent="MASTER_DOWN" packet-mark="DNS" queue=default limit-at=2000k max-limit=45000k priority=3
add name="UDP_U" parent="MASTER_UP" packet-mark="UDP" queue=default limit-at=200k max-limit=2850k priority=4
add name="UDP_D" parent="MASTER_DOWN" packet-mark="UDP" queue=default limit-at=2000k max-limit=45000k priority=4
add name="ICMP_U" parent="MASTER_UP" packet-mark="ICMP" queue=default limit-at=200k max-limit=2850k priority=5
add name="ICMP_D" parent="MASTER_DOWN" packet-mark="ICMP" queue=default limit-at=2000k max-limit=45000k priority=5
add name="HTTP_U" parent="MASTER_UP" packet-mark="HTTP" queue=default limit-at=200k max-limit=3000k priority=6
add name="HTTP_D" parent="MASTER_DOWN" packet-mark="HTTP" queue=default limit-at=2000k max-limit=50000k priority=6
add name="HTTP_BIG_U" parent="MASTER_UP" packet-mark="HTTP_BIG" queue=default limit-at=200k max-limit=3000k priority=7
add name="HTTP_BIG_D" parent="MASTER_DOWN" packet-mark="HTTP_BIG" queue=default limit-at=2000k max-limit=50000k priority=7
add name="OTHER_U" parent="MASTER_UP" packet-mark="OTHER" queue=default limit-at=200k max-limit=3000k priority=8
add name="OTHER_D" parent="MASTER_DOWN" packet-mark="OTHER" queue=default limit-at=2000k max-limit=50000k priority=8

freemannnn · Tue Sep 29, 2015 7:56 pm

dear "alaskanjackal" giving higher priority to udp packets is not good when u download torrents, right?
i think i will place them under http_big

alaskanjackal · Tue Sep 29, 2015 8:35 pm

dear "alaskanjackal" giving higher priority to udp packets is not good when u download torrents, right?
i think i will place them under http_big

Perhaps not. I don't (often) torrent, so that isn't a concern. But that was in pcunite's original script, and while prioritizing UDP traffic like that wouldn't necessarily have been something I would have thought to do if I were writing a script myself, I saw no reason to change it.

There are other example scripts on the forums here that do a better job of classifying and deprioritizing torrent traffic, though none of the scripts I came across are perfect--most are just suggestions/ideas that require some heavy editing to get working. Despite its limited scope, the ones brought up in this thread are the most functional, at least from what I saw.

alaskanjackal · Wed Sep 30, 2015 2:00 am

OK, so I just did a 1-hour long test (playing music through the line) while completely saturating my connection with multiple concurrent speed tests, uploads, downloads, and several very large Linux torrents going at once, and aside from one 20-second episode during the whole hour of some lightly choppy upstream audio, the VoIP call performed flawlessly.

I realized that shaping UDP downstream is pointless because it won't cause any slowdown on the transmission, so there is no point in throwing away UDP packets that have managed to make it as far as my edge router. Perhaps it's better just to not mark downstream UDP traffic or to put it in a separate (unlimited) queue or something, so I'll mess with that later.

Also, there's still some weirdness going on with HTTP and HTTP_BIG--stuff that isn't big is going into BIG and stuff that is BIG is not, and then other stuff that should go into BIG gets left over and goes into OTHER. I'm not super worried about it, as the setup will likely work fine for daily use (and still prioritize VoIP, which is the main goal here), but it causes some unpredictable weirdness in choking/throttling larger uploads for no reason (occasionally, suddenly all upstream HTTP traffic will zero out for several seconds or even longer until it eases back to link speed). I probably need a better understanding of mangling and queue priorities in Mikrotik to diagnose that much more.

freemannnn · Wed Sep 30, 2015 8:39 am

connection-rate is broken from firmware 6.28 and after. i dont understand why mikrotik dont fix it.
http://forum.mikrotik.com/viewtopic.php ... te#p501644
downgrade to 6.28 and you will see that it works!
thank you for the nice qos script

alaskanjackal · Fri Oct 02, 2015 10:03 pm

connection-rate is broken from firmware 6.28 and after. i dont understand why mikrotik dont fix it.
http://forum.mikrotik.com/viewtopic.php ... te#p501644
downgrade to 6.28 and you will see that it works!
thank you for the nice qos script

Thanks. I dropped the connection-rate parameter from Mangle and it works now and properly classifies large transfers in HTTP_BIG once 5MB of data has been transferred. This may have negative consequences for some long-lived HTTP TCP connections that don't use up much speed but may, over time, use more than 5MB of data, but I'm thinking it all probably makes very little practical difference in my network.

I also made a couple of additional changes just now: I separated out VOIP into VOIP-TCP and VOIP-UDP. I then put VOIP-UDP_D into a separate queue with a parent of my inbound interface (rather than MASTER_DOWN) without a limit. I also moved the other UDP_D outside of MASTER_DOWN with no limit. I think this is probably good practice, since, as I mentioned above, there's no point in dropping inbound UDP packets if they've already made it this far. I left VOIP-TCP_D in the MASTER_DOWN queue with a limit as well as both VOIP-UDP_U and VOIP-TCP_U under MASTER_UP with limits (very unlikely that I'll ever have more than a few dozen Kbps in VoIP traffic, but you never know what might happen, so why not).

Seems to be working, but I'll play with it a bit more before posting a revised script.

haffari · Mon Dec 28, 2015 11:38 pm

Implementing VoIP traffic prioritization (Qos) with RouterOS v6
Option 3 ... coming soon

Hi
Waiting for next parts.......................................

freemannnn · Wed Feb 24, 2016 11:55 am

is there a newer version of this qos script? (1.3b1 is the latest i found)
can we use simple queues (pcq or sfq) instead of queue tree?

MikroTikOkay · Mon Mar 07, 2016 5:34 pm

connection-rate is broken from firmware 6.28 and after. i dont understand why mikrotik dont fix it.
http://forum.mikrotik.com/viewtopic.php ... te#p501644
downgrade to 6.28 and you will see that it works!
thank you for the nice qos script
Thanks. I dropped the connection-rate parameter from Mangle and it works now and properly classifies large transfers in HTTP_BIG once 5MB of data has been transferred. This may have negative consequences for some long-lived HTTP TCP connections that don't use up much speed but may, over time, use more than 5MB of data, but I'm thinking it all probably makes very little practical difference in my network.

I also made a couple of additional changes just now: I separated out VOIP into VOIP-TCP and VOIP-UDP. I then put VOIP-UDP_D into a separate queue with a parent of my inbound interface (rather than MASTER_DOWN) without a limit. I also moved the other UDP_D outside of MASTER_DOWN with no limit. I think this is probably good practice, since, as I mentioned above, there's no point in dropping inbound UDP packets if they've already made it this far. I left VOIP-TCP_D in the MASTER_DOWN queue with a limit as well as both VOIP-UDP_U and VOIP-TCP_U under MASTER_UP with limits (very unlikely that I'll ever have more than a few dozen Kbps in VoIP traffic, but you never know what might happen, so why not).

Seems to be working, but I'll play with it a bit more before posting a revised script.

alaskanjackal,

Would you mind sharing your revised script? Thanks!

porch · Sat Mar 12, 2016 10:24 am

Had anyone got Option 2 to do anything? I have a 951Ui-2HnD 6.33.3 running in a small office with 5 VOIP phones connected to an internet VIOP server. Bandwidth is 8Mbs/1Mbs.
I have Option 2 installed set for 6Mbs/600Kbs. It's tagging the packets right and the queues look good, but nothing happens. Speedtest.net reports the speed being 8Mbs/1Mbs, and if I run Speedtest.net while on a call, it gets all choppy.
So it's not really doing anything.
I have spent the last 4 hours trying to get this to work, and bricked one 951 to the point that netinstall won't see it trying to get this to work.
HELP!!!!!

MosquitoCR · Mon Aug 01, 2016 4:49 am

alaskanjackal,

Please send me the updated MK script .

IntrusDave · Mon Aug 01, 2016 7:54 am

Try using pure DSCP
http://forum.mikrotik.com/viewtopic.php?f=9&t=110717

nichky · Mon Sep 05, 2016 12:08 am

Does anyone know what is happen with option 3? Does it come out?
appreciate you

Thanks

mrjurek · Mon Dec 19, 2016 12:01 am

alaskanjackal,

Would you mind sharing your new script?
(My mail: mrjurek@poczta.onet.pl)

Thanks.

mltobing · Wed Dec 21, 2016 4:33 am

Hello,

I am still using RouterOS 6.27 because connection-rate bug. Is it no fixed in new version ?
If already fixed, please tell me in which version.

Thanks

freemannnn · Wed Dec 21, 2016 9:31 am

is already fix after 6.31 or 6.33. i cant remember exactly
u should install and test bugfix version 6.36.4

49er · Wed Dec 21, 2016 12:39 pm

Hi,
This is a very interesting topic.
I want to use it on a transparent mikrotik device.
So no router.
Is that also possible?
And than make difference in up and download

satskiy · Thu Dec 22, 2016 9:40 am

if i hawe spf as WAN and all lan+ wifi are in bridge
then
Ether-WAN= SPF
Ether-LAN=Bridge right ?

mltobing · Mon Dec 26, 2016 4:22 am

is already fix after 6.31 or 6.33. i cant remember exactly
u should install and test bugfix version 6.36.4

RouterOS v6.36.4 working properly, then upgrade to 6.37.3. Bug fix confirmed

Thanks freemannnn

alaskanjackal · Mon Mar 20, 2017 1:12 am

alaskanjackal,

Would you mind sharing your new script?
(My mail: mrjurek@poczta.onet.pl)

Thanks.

Sorry for my long absence here.

Here's an updated version of the script (from another device I have deployed, so I think the interface names might be different than my first script and the connection speed is also different--adjust as necessary) that's been running for >1y without any issues.

I do see the report that connection-rate may be fixed in a newer release, but I have not re-incorporated it into my scripts. Perhaps I will someday. For now, here is my latest operating script:

# Date: March 19, 2017
# Version: 1.4, based on work by pcunite
# Tested with RouterOS 6.38.5
# Rename ether1-gateway and bridge-local to match your environment

###############################################################################
# Mangle
#
# Using prerouting/postrouting since we don't have dst or src checks.
#
/ip firewall mangle
###############################################################################

# DNS requests. Mark in two places because DNS is sent out by the router (itself) too.
add action=mark-connection chain=prerouting comment=DNS connection-state=new new-connection-mark=DNS passthrough=yes port=53 protocol=udp
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=DNS passthrough=yes port=53 protocol=udp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no

# Mark all VoIP traffic based on DSCP marking 46 and 26.
add action=mark-connection chain=prerouting comment=VOIP dscp=46 new-connection-mark=VOIP-TCP passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting dscp=46 new-connection-mark=VOIP-UDP passthrough=yes protocol=udp
add action=mark-connection chain=prerouting new-connection-mark=VOIP-UDP passthrough=yes port=4500 protocol=udp
add action=mark-connection chain=prerouting dscp=26 new-connection-mark=VOIP-TCP passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting dscp=26 new-connection-mark=VOIP-UDP passthrough=yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=VOIP-TCP new-packet-mark=VOIP-TCP passthrough=no
add action=mark-packet chain=prerouting connection-mark=VOIP-UDP new-packet-mark=VOIP-UDP passthrough=no

# Mark all UDP traffic. Mark different UDP streams if you want more granularity.
add action=mark-connection chain=prerouting comment=UDP connection-state=new new-connection-mark=UDP passthrough=yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=UDP new-packet-mark=UDP passthrough=no

# Ping replies. Mark in two places because ICMP is sent out by the router (itself) too.
add action=mark-connection chain=prerouting comment=ICMP connection-state=new new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=postrouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no

# ACK traffic. Based on viewtopic.php?f=2&t=67965
add action=mark-packet chain=postrouting comment=ACK new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack

# Mark all new HTTP(s) connections with "HTTP" if they have not previously been marked as "HTTP_BIG".
# If the current mark of "HTTP" tranfers more than 5MB and at a rate of 200k+ then mark it as "HTTP_BIG" for the duration of the TCP session.
add action=mark-connection chain=prerouting comment=HTTP connection-state=new new-connection-mark=HTTP passthrough=yes port=80,443,8080 protocol=tcp
add action=mark-connection chain=prerouting connection-bytes=500000-0 connection-mark=HTTP new-connection-mark=HTTP_BIG passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no

# Mark everything else that has no mark applied.
add action=mark-connection chain=prerouting comment=OTHER connection-mark=no-mark new-connection-mark=OTHER passthrough=yes
add action=mark-packet chain=prerouting connection-mark=OTHER new-packet-mark=OTHER passthrough=no

###############################################################################
# HTB Queue Tree a unidirectional queue
#
# Based on 50M down / 5M up connection (measured at 54000k/54.0M down, 5600k/5.6M up).
# Replace with your own values as needed and adjust downward if connection is unreliable.
#
# Notes:
# priority means 'drop packets' WHEN needed.
# When limit-at=0 priority starts when max-limit is reached.
# When limit-at=123 priority starts when limit-at is reached.
#
# The priority option applies to children not parents. Parent is for setting
# overall limits. Therefore use limit-at and max-limit on the children if
# you want more granularity.
#
# max-limit must always be set or priority will not happen.
#
# Tips for TCP (not VoIP) SOHO network:
# limit-at = Total bandwidth / max hosts
# max-limit = Total bandwidth / min hosts
#
/queue tree
###############################################################################

# The secret to ensuring VoIP quality (or any UDP traffic) is to put it into
# a queue that will never be full and thus never prioritize (drop) packets.

add max-limit=5500k name=MASTER_UP parent=ether1-gateway queue=default
add max-limit=53M name=MASTER_DOWN parent=bridge-local queue=default
add limit-at=200k max-limit=5M name=VOIP-TCP_U packet-mark=VOIP-TCP parent=MASTER_UP priority=1 queue=default
add limit-at=200k max-limit=45M name=VOIP-TCP_D packet-mark=VOIP-TCP parent=MASTER_DOWN priority=1 queue=default
add limit-at=200k max-limit=5M name=ACK_U packet-mark=ACK parent=MASTER_UP priority=2 queue=default
add limit-at=4M max-limit=45M name=ACK_D packet-mark=ACK parent=MASTER_DOWN priority=2 queue=default
add limit-at=200k max-limit=5M name=DNS_U packet-mark=DNS parent=MASTER_UP priority=3 queue=default
add limit-at=4M max-limit=45M name=DNS_D packet-mark=DNS parent=MASTER_DOWN priority=3 queue=default
add limit-at=200k max-limit=5500k name=UDP_U packet-mark=UDP parent=MASTER_UP priority=4 queue=default
add name=UDP_D packet-mark=UDP parent=bridge-local priority=4 queue=default
add limit-at=200k max-limit=5M name=ICMP_U packet-mark=ICMP parent=MASTER_UP priority=5 queue=default
add limit-at=4M max-limit=45M name=ICMP_D packet-mark=ICMP parent=MASTER_DOWN priority=5 queue=default
add limit-at=200k max-limit=5500k name=HTTP_U packet-mark=HTTP parent=MASTER_UP priority=6 queue=default
add limit-at=4M max-limit=53M name=HTTP_D packet-mark=HTTP parent=MASTER_DOWN priority=6 queue=default
add limit-at=200k max-limit=5500k name=HTTP_BIG_U packet-mark=HTTP_BIG parent=MASTER_UP priority=7 queue=default
add limit-at=4M max-limit=53M name=HTTP_BIG_D packet-mark=HTTP_BIG parent=MASTER_DOWN priority=7 queue=default
add limit-at=200k max-limit=5500k name=OTHER_U packet-mark=OTHER parent=MASTER_UP queue=default
add limit-at=4M max-limit=53M name=OTHER_D packet-mark=OTHER parent=MASTER_DOWN queue=default
add name=VOIP-UDP_D packet-mark=VOIP-UDP parent=bridge-local priority=1 queue=default
add limit-at=200k max-limit=5M name=VOIP-UDP packet-mark=VOIP-UDP parent=MASTER_UP priority=1 queue=default

Note that the limit-at amounts could probably use some playing around with, and (as I mentioned above) connection-rate could probably be re-implemented. But this ruleset seems to be working pretty bulletproof for me--allowing me access to very, very near my max possible throughput while preventing any sort of connection issues on VoIP calls.

nichky · Wed Apr 19, 2017 12:11 am

Can i implement PCQ on that script? i mean..for e.g.

add limit-at=200k max-limit=5M name=ACK_U packet-mark=ACK parent=MASTER_UP priority=2 queue=PCQ_U

And also i would like to specified any IP_Subnet for e.g. (address list) can i do this?

if it is posible will be exselent

)

Thanks a lot

nichky · Thu Apr 20, 2017 4:47 am

just I'm wondering it' possible to do, on second part of QoS on this post, i wold like to do with Address List.Can anyone help me?

Thanks

korcom · Mon Aug 07, 2017 10:48 am

@alaskanjackal your scripts are incredible.

How would I implement it with 2 WANs both with different speeds?

Thank you

marwooj · Wed Mar 07, 2018 7:23 pm

Implementing VoIP traffic prioritization (Qos) with RouterOS v6 – Option 2

If I have s2s ipsec VPN will it go into "OTHER"?

marwooj · Wed Mar 07, 2018 7:36 pm

Implementing VoIP traffic prioritization (Qos) with RouterOS v6 – Option 2

I also do not understand how LEVEL_A_ gets priority over LEVEL_B_ and then over LEVEL_C_.
Would somebody be so kind explain this?

sindy · Sun Mar 11, 2018 10:55 pm

Implementing VoIP traffic prioritization (Qos) with RouterOS v6 – Option 2

If I have s2s ipsec VPN will it go into "OTHER"?

In my opinion, the only way to understand how the traffic prioritization works is to try to implement a simple one on your own and debug it, not to try to understand someone else's complex example like the one above. Once you understand how it works using your simple case, reading the complex example becomes much easier.

So as I wrote in the other thread, you have to prioritize VoIP traffic among everything you send through the uplink. To do that, you have to set a DSCP mark (better to use the standard one, i.e. 46, for VoIP media and maybe also signalling) as DSCP is the only attribute of the packet which remains unchanged after IPsec encryption and encapsulation.

To answer your question - when talking about the classifier rules in the example above, the IPSec transport packets would definitely fall into the OTHER group, while the packets before encryption and encapsulation could fall into any of the groups depending on the criteria. So if you would use that script without modification of the classifier rules, you would prioritize VoIP traffic among everything that is going to be sent encrypted, but that would be useless because you would give the lowest (OTHER) priority to all the encrypted traffic.

So the whole implementation would have to include

for the upload direction:
- mangle rules setting DSCP value for anything which is going to be sent via the IPsec tunnel (it is possible that the phones already do that, but you also need to eventually remove priority marking from other than VoIP packets should it eventually be set on them)
- mangle rules marking connections and packets going to be sent via the uplink, including the IPsec transport packets, based on DSCP and other criteria
- queue setup prioritizing the packets going to be sent via the uplink based on the marks assigned in the step above
for the download direction:
- mangle rules marking connections and packets received based on protocol type and, possibly, DSCP marking (if it survives transit through internet as stated earlier)
- queue setup throttling the low priority packets in such a way that the necessary amount of download bandwidth is kept free for the priority packets.

The download margin you have to reserve for the priority packets will differ significantly depending on whether the DSCP markings survive the transit or not. If they do, you may keep the margin only as big as to accommodate the incoming VoIP traffic; if they don't, you have to limit the IPsec transport bandwidth as a whole on the remote site upload and reserve enough space for all of it at local download because in such case, there is no way to freely distribute the "non-VoIP" download bandwidth between the tunnelled and non-tunnelled traffic.

marwooj · Mon Mar 12, 2018 1:47 pm

I think I will forget about putting VoIP into VPN, this definitely sounds to complicated. And I will try to tune the example I have already implemented. Just after I understood how LEVEL_A_ gets priority over LEVEL_B_ and then over LEVEL_C_

sindy · Mon Mar 12, 2018 2:55 pm

I think I will forget about putting VoIP into VPN, this definitely sounds to complicated. And I will try to tune the example I have already implemented. Just after I understood how LEVEL_A_ gets priority over LEVEL_B_ and then over LEVEL_C_

Funny. To me the idea of setting the DSCP field doesn't sound complicated at all as it is basically a single mangle rule. What may be complicated is to distinguish VoIP packets from non-VoIP ones but if you reserve an address range or subnet on each site for your VoIP devices, or if you can ask the VoIP devices to set the DSCP themselves, it is also not very complex. Softphones are a problem unless they can set DSCP themselves because they cannot be identified by IP address and the PCs they run at generate both kinds of traffic.

I mean, once you pushstart the prioritization as such, adding VoIP needs just a tiny bit compared to the overall effort.

What I still have to try is to practically deploy the queues, I didn't have the need so far.

But I've had a look at the setup with LEVEL_X and it seems to me that LEVEL_A and LEVEL_B have no priority over each other, they simply each have their dedicated share of the total bandwidth as they have a common parent queue which doesn't have any bandwidth limits set. So both LEVEL_A and LEVEL_B may reach their limits simultaneously, so the summary traffic may be 7800k up and 82M down, but none of them may "borrow" the other one's bandwidth even if it is completely unused. So GROUP_A and GROUP_B would be more appropriate names.

marwooj · Mon Mar 12, 2018 3:10 pm

Looks like my PBX boxes are adding DSCP = 40 already, I just afraid that by putting this traffic into VPN will add unnecessary overheat and I will end up with having bigger disaster that I have now.

So to make LEVEL_B and LEVEL_C always lower priority than LEVEL_A I will change parent of B and C to A?

sindy · Mon Mar 12, 2018 3:39 pm

Looks like my PBX boxes are adding DSCP = 40 already, I just afraid that by putting this traffic into VPN will add unnecessary overheat and I will end up with having bigger disaster that I have now.

Do you use VoIP phones or analog/ISDN phones and only the PBXes would talk VoIP to each other? If VoIP phones, do the PBXes tell the phones to send media to each other directly or do they force themselves into the media path? All that plays a role - if VoIP phones send media directly to each other, they must set DSCP or you must do it on their behalf when forwarding the packets to the remote site.

So to make LEVEL_B and LEVEL_C always lower priority than LEVEL_A I will change parent of B and C to A?

Yes but not only, you must also give the LEVEL_B queue the lowest priority among all the other child queues of LEVEL_A. And LEVEL_C must have the lowest priority among all child queues of LEVEL_B.

The queues play two roles - one is speed limitation and another one is prioritization. Speed limit doesn't allow more data to be sent even if bandwidth is available. Priority determines the order of searching through the queues at each level for the next packet to be sent once the "line" gets free after the previous packet has been sent.

On the upload, it is basically enough for QoS to do the prioritization, speed limitation is only necessary if you want to guarantee some minimum bandwidth to individual users.

On the download, you cannot actually affect the priority with which the ISP is sending the packets down your connection, so you must make use of the TCP's flow control - if you limit the speed with which received TCP packets are delivered from WAN to LAN, the remote senders limit their sending speed accordingly as they wait for ACKs from the recipients before sending another burst of packets. There is no point in throttling incoming UDP as it has no effect on the sender. So the trick is to limit the bandwidth for TCP in download direction and leave the rest for the peak amount of UDP you can expect (which is some 100k per audio RTP stream, but you have to inspect your traffic to see what else on top of RTP, DNS responses, NTP responses arrives to you as UDP). You'll likely also see some icmp there.

marwooj · Mon Mar 12, 2018 3:59 pm

I also have ipsec s2s not for the purpose of VoIP, to completely different site. It is mostly RDP.
Since IPsec s2s is also UDP (I think), how I can put it "to not disturb" my VoIP but not to go into LEVEL_C / OTHER?

sindy · Mon Mar 12, 2018 4:43 pm

I also have ipsec s2s not for the purpose of VoIP, to completely different site. It is moatly RDP.
Since IPsec s2s is also UDP (I think), how I can put it "to not disturb" my VoIP but not to go into LEVEL_C / OTHER?

That's the role of DSCP as mentioned in the other (s2s) thread. The "plaintext" traffic on the WAN can be classified as "TCP" and "other protocols" directly; the IPsec transport packets will either be ESP ones or UDP ones (depending on existence of NAT in the path) with a particular source and destination; whether "TCP" or "other" is encrypted inside them can only be determined if the sending side sets the DSCP field and the transit through internet doesn't destroy it completely.

On upload, you have to send the packets to the IPsec machinery without any throttling, only set their DSCP. The outgoing prioritization will take place on the plaintext and IPsec transport packets together, where the DSCP shows to which group or level the IPsec transport packets belong.

On download, there is a special situation - the IPsec transport packet and its decapsulated an decrypted payload are treated as two separate packets which came in via the same interface. As you do not want to queue the same actual packet twice, you can

either inspect the remainders of DSCP (if any) on the IPsec transport packets and choose a queue for them on that basis, but in such case you have to exclude from the queueing the decapsulated an decrypted payload
or exclude from queueing the IPsec transport packets, but in such case you have to inspect the protocol type and other attributes of the decapsulated and decrypted packets and choose the queue for them on that basis

I am afraid that "exclude from queueing" actually means to put them into a topmost queue with highest priority and no speed limits.

Classification of IPsec transport packets by DSCP in download saves CPU because the classification rules are simpler as the class has already been determined by the sending Mikrotik.

marwooj · Tue Mar 13, 2018 11:19 am

So to make LEVEL_B and LEVEL_C always lower priority than LEVEL_A I will change parent of B and C to A?
Yes but not only, you must also give the LEVEL_B queue the lowest priority among all the other child queues of LEVEL_A. And LEVEL_C must have the lowest priority among all child queues of LEVEL_B.

Should LEVEL_A as a parent of everything have

priority=1

?
Does priority is valid only among members of the same branch. So all children in sub branch will always step down to traffic if they parent priority is smaller than priority of an other branches on the same level. Does not mutter they have 1 but there is another purrent that have 7 ant they parrent has 8?

Regarding bandwidth, how to make C not to stal everything from B, and then B not to eat everything of A.
Do I need to change theirs max-limit to be smaller from each other?

and the example of proper "do not disturb my VoIP" should look like:

add name="LEVEL_A_UP"   parent=ether-WAN  queue=default max-limit=900k [b]priority=1
[/b]add name="LEVEL_A_DOWN" parent=bridge-LAN queue=default max-limit=4M [b]priority=1
[/b]add name="VOIP_U"       parent="LEVEL_A_UP"    packet-mark="VOIP"     queue=default priority=1
add name="VOIP_D"       parent="LEVEL_A_DOWN"  packet-mark="VOIP"     queue=default priority=1

add name="LEVEL_B_UP"   parent=LEVEL_A_UP  queue=default max-limit=[b]something smaler than parent A[/b] [b]priority=2
[/b]add name="LEVEL_B_DOWN" parent=LEVEL_A_DOWN queue=default max-limit=something smaler than parent A [b]priority=2
[/b]
add name="ACK_U"        parent="LEVEL_B_UP"    packet-mark="ACK"      queue=default priority=1
add name="ACK_D"        parent="LEVEL_B_DOWN"  packet-mark="ACK"      queue=default priority=1
add name="SOMETHING"        parent="LEVEL_B_UP"    packet-mark="SOMETHING"      queue=default [b]priority=7
[/b]add name="SOMETHING"        parent="LEVEL_B_DOWN"  packet-mark="SOMETHING"      queue=default [b]priority=7
[/b]
add name="LEVEL_C_UP"   parent=[b]LEVEL_B_UP[/b]  queue=default max-limit=[b]something smaler than parent B[/b] [b]priority=8
[/b]add name="LEVEL_C_DOWN" parent=[b]LEVEL_B_UP[/b] queue=default max-limit=[b]something smaler than parent B[/b] [b]priority=8
[/b]
add name="HTTP_U"       parent="LEVEL_C_UP"    packet-mark="HTTP"     queue=default [b]priority=1
[/b]add name="HTTP_D"       parent="LEVEL_C_DOWN"  packet-mark="HTTP"     queue=default [b]priority=1
[/b]
add name="OTHER_U"      parent="LEVEL_C_UP"    packet-mark="OTHER"    queue=default priority=8
add name="OTHER_D"      parent="LEVEL_C_DOWN"  packet-mark="OTHER"    queue=default priority=8

sindy · Tue Mar 13, 2018 12:01 pm

So to make LEVEL_B and LEVEL_C always lower priority than LEVEL_A I will change parent of B and C to A?
Yes but not only, you must also give the LEVEL_B queue the lowest priority among all the other child queues of LEVEL_A. And LEVEL_C must have the lowest priority among all child queues of LEVEL_B.
Should LEVEL_A as a parent of everything have
Code: Select all
priority=1
?
Does priority is valid only among members of the same branch. So all children in sub branch will always step down to traffic if they parent priority is smaller than priority of an other branches on the same level. Does not mutter they have 1 but there is another purrent that have 7 ant they parrent has 8?

Regarding bandwidth, how to make C not to stal everything from B, and then B not to eat everything of A.
Do I need to change theirs max-limit to be smaller from each other?

and the example of proper "do not disturb my VoIP" should look like:
Code: Select all
add name="LEVEL_A_UP"   parent=ether-WAN  queue=default max-limit=900k [b]priority=1
[/b]add name="LEVEL_A_DOWN" parent=bridge-LAN queue=default max-limit=4M [b]priority=1
[/b]add name="VOIP_U"       parent="LEVEL_A_UP"    packet-mark="VOIP"     queue=default priority=1
add name="VOIP_D"       parent="LEVEL_A_DOWN"  packet-mark="VOIP"     queue=default priority=1

add name="LEVEL_B_UP"   parent=LEVEL_A_UP  queue=default max-limit=[b]something smaler than parent A[/b] [b]priority=2
[/b]add name="LEVEL_B_DOWN" parent=LEVEL_A_DOWN queue=default max-limit=something smaler than parent A [b]priority=2
[/b]
add name="ACK_U"        parent="LEVEL_B_UP"    packet-mark="ACK"      queue=default priority=1
add name="ACK_D"        parent="LEVEL_B_DOWN"  packet-mark="ACK"      queue=default priority=1
add name="SOMETHING"        parent="LEVEL_B_UP"    packet-mark="SOMETHING"      queue=default [b]priority=7
[/b]add name="SOMETHING"        parent="LEVEL_B_DOWN"  packet-mark="SOMETHING"      queue=default [b]priority=7
[/b]
add name="LEVEL_C_UP"   parent=[b]LEVEL_B_UP[/b]  queue=default max-limit=[b]something smaler than parent B[/b] [b]priority=8
[/b]add name="LEVEL_C_DOWN" parent=[b]LEVEL_B_UP[/b] queue=default max-limit=[b]something smaler than parent B[/b] [b]priority=8
[/b]
add name="HTTP_U"       parent="LEVEL_C_UP"    packet-mark="HTTP"     queue=default [b]priority=1
[/b]add name="HTTP_D"       parent="LEVEL_C_DOWN"  packet-mark="HTTP"     queue=default [b]priority=1
[/b]
add name="OTHER_U"      parent="LEVEL_C_UP"    packet-mark="OTHER"    queue=default priority=8
add name="OTHER_D"      parent="LEVEL_C_DOWN"  packet-mark="OTHER"    queue=default priority=8

I don't like to provide theoretical-only answers and as I've stated several times, I didn't have a strong enough reason to test this practically so far. So already my previous response was wrong, as the manual says the following:
priority (1..8 ) : Prioritize one child queue over other child queue. Does not work on parent queues (if queue has at least one child). One is the highest, eight is the lowest priority. Child queue with higher priority will have chance to reach its max-limit before child with lower priority. Priority have nothing to do with bursts.

I have assumed that the priority works among all children of the same queue and recursively, but it seems not to be the case. So either you have to split the total available banwidth into subbands which do not affect each other, or you have to manage everything within the 8 priority levels of the common band. I just remind that in download direction there is basically no other way than throttling the traffic which can be throttled to leave a "fast lane" always free for the real-time traffic.

SamWCL · Tue Mar 27, 2018 10:09 am

Did anyone figure out MPLS/VPLS QoS using DSCP/EXP bits?

mixig · Tue Nov 06, 2018 9:29 pm

I'm a little curious why you have some rules twice
ros code
/ip firewall mangle
add chain=forward action=mark-connection protocol=udp   src-address=192.168.100.5 connection-state=new new-connection-mark="VOIP" comment="IP-PBX"
add chain=forward action=mark-packet     passthrough=no connection-mark="VOIP"    new-packet-mark="VOIP"
add chain=forward action=mark-connection protocol=udp   dst-address=192.168.100.5 connection-state=new new-connection-mark="VOIP"
add chain=forward action=mark-packet     passthrough=no connection-mark="VOIP"    new-packet-mark="VOIP"
 
You are marking packets twice based on the same options. In my opinion rule 2 can be removed.
Then first there are two rules marking the connection and setting connection-marks. Then use the connection-mark to mark the packets in one single rule.

This two rules are not the same... src/dst address are in use

Chupaka · Tue Jan 08, 2019 10:31 am

Should this topic be moved to Useful user articles forum?

pcunite · Wed Jan 09, 2019 12:18 am

Should this topic be moved to Useful user articles forum?

I would like to rewrite it, make it a little easier to follow.

pegasus123 · Fri Jan 25, 2019 12:27 pm

I'm a little curious why you have some rules twice
ros code
/ip firewall mangle
add chain=forward action=mark-connection protocol=udp   src-address=192.168.100.5 connection-state=new new-connection-mark="VOIP" comment="IP-PBX"
add chain=forward action=mark-packet     passthrough=no connection-mark="VOIP"    new-packet-mark="VOIP"
add chain=forward action=mark-connection protocol=udp   dst-address=192.168.100.5 connection-state=new new-connection-mark="VOIP"
add chain=forward action=mark-packet     passthrough=no connection-mark="VOIP"    new-packet-mark="VOIP"
 
You are marking packets twice based on the same options. In my opinion rule 2 can be removed.
Then first there are two rules marking the connection and setting connection-marks. Then use the connection-mark to mark the packets in one single rule.
This two rules are not the same... src/dst address are in use

isn't it enough to just mark it once? once a packet is read, it will be marked with connection mark, connection mark works both ways upload and download in which case you can mark the packet and be used in queue tree.

is my understanding correct?

sindy · Fri Jan 25, 2019 4:33 pm

is my understanding correct?

Your understanding is correct in terms that the rule translating connection-mark into packet-mark may be there only once (as the last one after the two assigning the connection-mark). Regarding the need for two rules assigning the connection-mark, it is a more complex question.

One approach would be to have only one such rule (for src- or dst-address) and say that if a single packet in the "wrong" direction doesn't get connection-marked, it causes no harm as the first subsequent packet in the opposite direction will fix this. The other approach, however, is to save CPU by having the connection-mark->packet-mark translation rule before the connection-marking rules so that these rules would only handle packets belonging to not yet marked connections. And this is usually combined with allowing these rules to handle only the initial packet of each connection (connection-state=new), so then you need to use both rules because you don't know in advance which RTP packet will be the first one in a given call.

And of course, the translation rules have to be doubled in the latter case - the first one handles packets belonging to already marked connections early in the chain, and the other one has to be there to handle the translation for the initial packets after the connection-mark has been just assigned.

pegasus123 · Sat Jan 26, 2019 5:49 am

is my understanding correct?
Your understanding is correct in terms that the rule translating connection-mark into packet-mark may be there only once (as the last one after the two assigning the connection-mark). Regarding the need for two rules assigning the connection-mark, it is a more complex question.

One approach would be to have only one such rule (for src- or dst-address) and say that if a single packet in the "wrong" direction doesn't get connection-marked, it causes no harm as the first subsequent packet in the opposite direction will fix this. The other approach, however, is to save CPU by having the connection-mark->packet-mark translation rule before the connection-marking rules so that these rules would only handle packets belonging to not yet marked connections. And this is usually combined with allowing these rules to handle only the initial packet of each connection (connection-state=new), so then you need to use both rules because you don't know in advance which RTP packet will be the first one in a given call.

And of course, the translation rules have to be doubled in the latter case - the first one handles packets belonging to already marked connections early in the chain, and the other one has to be there to handle the translation for the initial packets after the connection-mark has been just assigned.

Thanks for the detailed explanation!

denisun · Fri Feb 08, 2019 8:56 am

I am also writing here because I have not found a solution.
Has anyone managed to give priority to VoIP, and to work with no problem, with full load on the line?
My problem is this.

pcunite · Fri Feb 08, 2019 3:13 pm

I am also writing here because I have not found a solution. Has anyone managed to give priority to VoIP, and to work with no problem, with full load on the line? My problem is this.

Someday, I hope to do another write up on this subject, when I get time. I had hoped that one would not need to be an expert to get this correct, but as of 2019, it still does. However, I think a better article would help.

From the moment that audio transmits from your equipment, until it gets back, there must not be greater than a 150ms interruption (or thereabout) otherwise you will notice the delay. Now, think about everything that an audio packet has to go through to make that happen. When you understand that, you'll have solved your VoIP issue.

The permanent fix, a separate Internet service line. Can't do that? Then fast hardware performing QoS where VoIP packets get the best treatment when bad things happen (and they will).

denisun · Fri Feb 08, 2019 4:13 pm

I am also writing here because I have not found a solution. Has anyone managed to give priority to VoIP, and to work with no problem, with full load on the line? My problem is this.

Someday, I hope to do another write up on this subject, when I get time. I had hoped that one would not need to be an expert to get this correct, but as of 2019, it still does. However, I think a better article would help.

From the moment that audio transmits from your equipment, until it gets back, there must not be greater than a 150ms interruption (or thereabout) otherwise you will notice the delay. Now, think about everything that an audio packet has to go through to make that happen. When you understand that, you'll have solved your VoIP issue.

The permanent fix, a separate Internet service line. Can't do that? Then fast hardware performing QoS where VoIP packets get the best treatment when bad things happen (and they will).

I have a CRS109-8G.
I can not believe that my provider's simple modem/router equipment work perfect with VoIP packets and full load
and the MT have problem.
My question is...
- Have i somewhere wrong in mangles, tree or queue types?
- Is MT hw low specifications and can't process the load?
- Is ROS bug?
With ubiquity the result is much perfect.

pcunite · Fri Feb 08, 2019 4:50 pm

I have a CRS109-8G. With ubiquity the result is much better.

I would not use the CRS109 for QoS tasks. Too under powered in my opinion. However, there are many variables.

denisun · Fri Feb 08, 2019 7:26 pm

I have a CRS109-8G. With ubiquity the result is much better.

I would not use the CRS109 for QoS tasks. Too under powered in my opinion. However, there are many variables.

I don't believe that the problem is the hw specifications of CRC.
My ISPs equipment cost ~20-50e and work perfect with traffic (VoIP and dw).

bolean · Sat Feb 23, 2019 3:08 pm

version 3, running smooth

pcunite · Sat Feb 23, 2019 6:43 pm

Version 3, running smooth

@bolean,

How does this test compared to what you're doing?

/queue tree

# DOWN
add name=DOWN max-limit=90M parent=BR_LAN queue=default

add name="1. VOIP"     packet-mark=VOIP     parent=DOWN priority=1 queue=default
add name="2. DNS"      packet-mark=DNS      parent=DOWN priority=2 queue=default
add name="3. ACK"      packet-mark=ACK      parent=DOWN priority=3 queue=default
add name="4. UDP"      packet-mark=UDP      parent=DOWN priority=3 queue=default
add name="5. ICMP"     packet-mark=ICMP     parent=DOWN priority=4 queue=default
add name="6. HTTP"     packet-mark=HTTP     parent=DOWN priority=5 queue=default
add name="7. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=6 queue=default
add name="8. QUIC"     packet-mark=QUIC     parent=DOWN priority=7 queue=default
add name="9. OTHER"    packet-mark=OTHER    parent=DOWN priority=8 queue=default


# UP
add name=UP max-limit=90M parent=ether1 queue=default

add name="1. VOIP_"     packet-mark=VOIP     parent=UP priority=1 queue=default
add name="2. DNS_"      packet-mark=DNS      parent=UP priority=2 queue=default
add name="3. ACK_"      packet-mark=ACK      parent=UP priority=3 queue=default
add name="4. UDP_"      packet-mark=UDP      parent=UP priority=3 queue=default
add name="5. ICMP_"     packet-mark=ICMP     parent=UP priority=4 queue=default
add name="6. HTTP_"     packet-mark=HTTP     parent=UP priority=5 queue=default
add name="7. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=6 queue=default
add name="8. QUIC_"     packet-mark=QUIC     parent=UP priority=7 queue=default
add name="9. OTHER_"    packet-mark=OTHER    parent=UP priority=8 queue=default

pcunite · Tue Dec 31, 2019 2:28 am

I have a question to ask the admins (how do we private message you and talk about the forums)? I am planning on rewriting this article. What is the best course of action to maintain the link (which is pinned and also maybe linked elsewhere)? I would like for all posts to be deleted except for the first six ones (I'm going to go back and edit my own). Is there a better recommendation?

anav · Sun Jan 05, 2020 9:24 pm

I would hope the admins give you another thread to start that only you can edit.
Folks could use your original thread to comment and keep the conversation going leaving your new thread as the more polished pre-wiki draft to tweak.

digin4 · Tue Jan 07, 2020 11:27 pm

How to add some TCP ports and prioritize them as VOIP? can I simply do this and it will be enough?:

/ip firewall mangle
add chain=prerouting  action=mark-connection new-connection-mark=VOIP port=4244,5222,5223,5228,5242,50318,59234,1119,3724,6113 protocol=tcp passthrough=yes comment="VOIP"

Thank you!

pcunite · Thu Jan 09, 2020 1:26 am

How to add some TCP ports and prioritize them as VOIP? can I simply do this and it will be enough?

Please study the example more closely. VoIP packets on your network will not be the same as others. Using Mangle, you will choose what you need to mark. It could be via IP Address, standard SIP or RTP ports, or other metrics unique to your environment.

gcser · Thu Jan 09, 2020 2:48 pm

Hi Everybody,

I have the following scenario (only for testing the limiter, this is not a production environment):
PC->CCR eth3 (vlan id: 2205 untagged)
CCR eth2->RB600eth3 (2205, 2206, 2207 vlans configured, all traffic are tagged)
RB600 eth2->Server (vlan id: 2205 untagged).

On th CCR I have a bridge containing 2 ports, with VLAN filtering enabled. I have 3 VLANs on it (2205, 2206, 2207), all traffic are tagged on eth2 interface, on the eth3 interface the 2205 vlan is untagged, pvid set to 2205. I connected my test pc to eth3.
On the RB600 I the the same situation (I would like to simulate a trunk between the 2 MTs). I connected a server pc to the RB600.
I can ping the server, I can download files from it, the network works as expected.

I would like to make a traffice shaping/priorization based on VLAN-s.
On the CCR, at the bridge configuration I checked the "use IP firewall" and the "user IP firewall for VLAN" checkboxes.
I created a bridge filter rule: chain: forward, MAC-protocol-num=8100 (vlan). On Advanced tab I filled the VLAN ID, on the action tab: mark packet with mark "office-traffic"
After that, I created a queue (in queue tree), set the packet mark to "office-traffic", and made a bandwith limit to 10mbit/sec.

My problem is, that if I make a download from the server, the bandwidth limit is not applied. If I remove the vlan ID from the rule, and set the MAC-protocol back to "ip", the limiter works as expected.
If I set the VLAN tag, in the queue I can see only a ~700kbit/sec traffic. If I remove the vlan filter from the rule, the queue shows 10mbit/sec traffic.

What doing I wrong? It seems to me, that the MAC-protocol =vlan not capturing all the packets... Is it possible?

Tabalugach · Sun Jan 19, 2020 7:39 pm

Hi Everybody,

I setup queue tree like this :

/queue tree

# DOWN
add name=DOWN max-limit=9M parent=LAN queue=default

add name="1. Gaming"     packet-mark=Gaming parent=DOWN priority=1 queue=default
add name="2. DNS"      packet-mark=DNS      parent=DOWN priority=2 queue=default
add name="3. ACK"      packet-mark=ACK      parent=DOWN priority=3 queue=default
add name="4. UDP"      packet-mark=UDP      parent=DOWN priority=3 queue=default
add name="5. ICMP"     packet-mark=ICMP     parent=DOWN priority=4 queue=default
add name="6. HTTP"     packet-mark=HTTP     parent=DOWN priority=5 queue=default
add name="7. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=6 queue=default
add name="8. QUIC"     packet-mark=QUIC     parent=DOWN priority=7 queue=default
add name="9. OTHER"    packet-mark=OTHER    parent=DOWN priority=8 queue=default


# UP
add name=UP max-limit=540k parent=WAN queue=default

add name="1. Gaming_"     packet-mark=Gaming parent=UP priority=1 queue=default
add name="2. DNS_"      packet-mark=DNS      parent=UP priority=2 queue=default
add name="3. ACK_"      packet-mark=ACK      parent=UP priority=3 queue=default
add name="4. UDP_"      packet-mark=UDP      parent=UP priority=3 queue=default
add name="5. ICMP_"     packet-mark=ICMP     parent=UP priority=4 queue=default
add name="6. HTTP_"     packet-mark=HTTP     parent=UP priority=5 queue=default
add name="7. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=6 queue=default
add name="8. QUIC_"     packet-mark=QUIC     parent=UP priority=7 queue=default
add name="9. OTHER_"    packet-mark=OTHER    parent=UP priority=8 queue=default

I only mark two things, one is Apex Legends and 2nd is everything else.
Apex Legends is correctly marked and it's go to gaming queue, everything else go to OTHER queue.

But still when i hit speed test (https://www.speedtest.net/), when upload start, apex will lost connection.

How is this possible that priority=8 overcomes priority=1?

Another thing which I don't understand is if I setup :

#DOWN
add name=DOWN max-limit=1M parent=LAN queue=default
# UP
add name=UP max-limit=100k parent=WAN queue=default

I will still get :

Download Mbps 9.68
Upload Mbps 0.56

How is this possible?

Please help!

StubArea51 · Mon Jan 20, 2020 6:48 pm

You need to use 'limit-at' to guarantee bandwidth instead of 'max-limit' try that and see if you get different results.

pcunite · Mon Jan 20, 2020 10:00 pm

#DOWN
add name=DOWN max-limit=1M parent=LAN queue=default
# UP
add name=UP max-limit=100k parent=WAN queue=default

I will still get : Download Mbps 9.68, Upload Mbps 0.56

How is this possible?

You can not use Fast Track and Queues Tree together.

FRANKNunoz · Mon Jan 20, 2020 10:39 pm

version 3, running smooth

friend how is your connection dial and packet marking in mangrove?

Tabalugach · Tue Jan 21, 2020 2:11 am

You can not use Fast Track and Queues Tree together.

I found it today! Thank you!
Btw, do you prefer to put highest priority 1 (in my situation game : Apex) to fast track?

pcunite · Tue Jan 21, 2020 4:57 am

do you prefer to put highest priority 1 (in my situation game : Apex) to fast track?

Fast Track is CPU usage mitigation technique. Queuing is a bandwidth utilization technique. Different goals. If the CPU can handle it, you need to use Queue technique only.

theprism · Fri Jan 24, 2020 2:04 pm

Hello everyone. I've tested and updated the script. It now works correctly on RouterOS 6.1. Note that ether1 is WAN and ether2 is LAN. Adjust those as necessary for your environment.

I would appreciate if someone could tell me how to mark big downloads over HTTP traffic. Currently, the script marks port 80 so everything HTTP gets too much priority. The idea situation would be to let short bursts of HTTP traffic get high priority and the big long downloads get less.
with connection byte ?

OK for WAN, but what about LAN in cases when more than 1 interface is used?
For example, I use ether2+wlan1+wlan2 in homebridge, but I also have ether3+wlan1-guest in guestbridge. How I identify them both together as LAN in the QoS rules and don't let one of them to take the whole bandwidth if both are traffic-hungry?

Thanks,
T.P.

pcunite · Sat Jan 25, 2020 10:00 pm

what about when there are several LAN interfaces?

There are several ways, in RouterOS, to combine several things into one. Maybe VLAN, maybe interface lists, maybe address lists. Its up to you. Then you simply mangle them and send them to the queue.

louisren · Thu Jan 30, 2020 4:41 am

If anyone can add a tutorial on how to conduct QoS on CRS328-24P-4S+RM that would be great...........

Thu Jan 30, 2020 6:49 am

If anyone can add a tutorial on how to conduct QoS on CRS328-24P-4S+RM that would be great...........

maybe when MikroTik enable DSCP and TOS remarking

maybe when MikroTik enable 4 or 8 hardware queues per port

all of this in hardware switch ASIC to do it at wire-speed without almost CPU usage

then will be nice to do QoS on MikroTik switch's, but today is not

helper · Tue Feb 04, 2020 1:45 pm

Thanks alot .....Trying

Chupaka · Tue Feb 04, 2020 2:00 pm

Shouldn't the topic be moved to viewforum.php?f=23?

bharrisau · Wed Feb 05, 2020 9:02 am

Did a pile of testing today with Flent measuring bufferbloat and performance. The final system that gave the best results (no increase in latency when limited to 90% of line bandwidth) is below.

Set the two parent queues (UP and DOWN) to have a bucket-size of 0.005. This means that up to 5ms of token can accumulate.

Create a bulkUp queue of kind PCQ, set the pcq-limit to 11*[upload rate in Mbps] (100ms of upload bandwidth) and the pcq-total-limit to 10 times that. Select all 4 classifier options.

Create a bulkDown queue of kind sqf. (Using RED gave similar performance, but multiple downloads were less fair)

Change the default queue to sfq. (This will only effect something if the interactive traffic is saturating the link, which is a bad state to get into)

Set the HTTP/HTTP_BIG/QUIC/OTHER queues to be bulkUp in the UP direction, and bulkDown in the DOWN direction.

Rationale for the 0.005 size is to copy CoDel as much as possible. We set bulkUp to 100ms of data to smooth out throughput, 100ms of latency for bulk traffic isn't expected to be noticeable in practice. In the down direction we have a much smaller buffering capacity to ensure we are dropping early - the down direction isn't controlled as well so early drops are the only way to avoid bloat at the ISP end. RED and SFQ work well here - I had more fairness during testing using SFQ.

An example of 100ms of data setting for my 40 Mbps upload is pcq-limit = 440. Because pcq-limit is a value in KiB, multiplying the Mbps by 11 gives us approximately 100ms of data transfer at 90% of the line speed.

Attached image is my final Flent test. I'm testing from Australia to Europe so the baseline latency is pretty high, upload and download limits in router are 90% of the line speeds (90/36). At the start of this test I also did a simultaneous speedtest.com test to see how fair things are.

pcunite · Wed Feb 05, 2020 3:12 pm

Shouldn't the topic be moved to viewforum.php?f=23?

I'm okay with that. Please keep the original url and redirect it to whatever the new one will be. Also, remove many of the old posts that don't advance the topic, since the new 2020 edition for example.

pcunite · Wed Feb 05, 2020 3:16 pm

Set the two parent queues (UP and DOWN) to have a bucket-size of 0.005.

Create a bulkUp queue of kind PCQ, set the pcq-limit to 11*[upload rate in Mbps] (100ms of upload bandwidth) and the pcq-total-limit to 10 times that. Select all 4 classifier options.

Create a bulkDown queue of kind sqf.

Change the default queue to sfq.

Set the HTTP/HTTP_BIG/QUIC/OTHER queues to be bulkUp in the UP direction, and bulkDown in the DOWN direction.

Wow, thank you for doing this. To remove any ambiguity, would you please post your queue config (between code tags) so I can study it? How well did my reference implementation do for you? I want to make sure that your config is not too specific to your needs as we do need something general that people can start and learn from.

Chupaka · Thu Feb 06, 2020 10:14 pm

I'm okay with that. Please keep the original url and redirect it to whatever the new one will be.

Moved. Topic ID is the same, so all links are still valid.

hknet · Fri Feb 07, 2020 3:52 am

as this is trending and the initial target if I understand it correctly was VoIP priority, here is a small example on how we do it.

the assumption is like this: download-link can be oversaturated, though this is not desireable we usually can't do much about it if incoming packets are thrown at the CPE, therefore the provider-end is assumed to also hold it's end of the bargain and do their VoIP prio right. in our situation we are the provider end ourselves and yes our BRAS do this quite well.

so the real job of the mikrotik at the customer-site is handle the upload well.
also we assume VoIP packets are well marked using DSCP-markings (usually all VoIP equipment does this quite well).

therefore we first build a DSCP-marking to packet-marking within MT setup and we slide in a marking for remote-winbox-connects also:

/ip firewall mangle add action=mark-packet chain=postrouting comment="mgmt prio winbox" \
dst-address=<dudeserver-mgmt-IP> new-packet-mark=dscp.48 passthrough=no protocol=tcp src-port=8291
/ip firewall mangle add action=mark-packet chain=postrouting comment=dscp.0_no-marking disabled=no \
dscp=0 new-packet-mark=dscp.0 passthrough=no
/ip firewall mangle add action=mark-packet chain=postrouting comment=dscp.46_VoIP-marking disabled=no \
dscp=46 new-packet-mark=dscp.46 passthrough=no
/ip firewall mangle add action=mark-packet chain=postrouting comment=dscp.48_network-mark disabled=no \
dscp=48 new-packet-mark=dscp.48 passthrough=no
:for x from 1 to 45 do={/ip firewall mangle add action=mark-packet chain=postrouting \
comment=dscp.1-45 disabled=no dscp=$x new-packet-mark=dscp.other passthrough=no}
/ip firewall mangle add action=mark-packet chain=postrouting comment=dscp.47 disabled=no \
dscp=47 new-packet-mark=dscp.other passthrough=no
:for x from 49 to 63 do={/ip firewall mangle add action=mark-packet chain=postrouting \
comment=dscp.49-63 disabled=no dscp=$x new-packet-mark=dscp.other passthrough=no}

and as you sure do provide IPv6 connectivity to your customers we do this very same thing for IPv6 packets:

/ipv6 firewall mangle add action=mark-packet chain=postrouting comment="mgmt prio winbox" \
dst-address=<dudeserver-mgmt-IP> new-packet-mark=dscp.48 passthrough=no protocol=tcp src-port=8291
/ipv6 firewall mangle add action=mark-packet chain=postrouting comment=dscp.0_no-marking disabled=no \
dscp=0 new-packet-mark=dscp.0 passthrough=no
/ipv6 firewall mangle add action=mark-packet chain=postrouting comment=dscp.46_VoIP-marking disabled=no \
dscp=46 new-packet-mark=dscp.46 passthrough=no
/ipv6 firewall mangle add action=mark-packet chain=postrouting comment=dscp.48_network-mark disabled=no \
dscp=48 new-packet-mark=dscp.48 passthrough=no
:for x from 1 to 45 do={/ipv6 firewall mangle add action=mark-packet chain=postrouting \
comment=dscp.1-45 disabled=no dscp=$x new-packet-mark=dscp.other passthrough=no}
/ipv6 firewall mangle add action=mark-packet chain=postrouting comment=dscp.47 disabled=no \
dscp=47 new-packet-mark=dscp.other passthrough=no
:for x from 49 to 63 do={/ipv6 firewall mangle add action=mark-packet chain=postrouting \
comment=dscp.49-63 disabled=no dscp=$x new-packet-mark=dscp.other passthrough=no}

now we have packets marked for further use in our queues.
the basic queue max-limit depends on the upload speed, usually we automate this and take for example 90% of the theoretically available bandwidth (as we have IP/PPP/.. overhead to accomodate):

/queue tree
add max-limit=666k name=line01 parent=ppp-out-line1 queue=default

then we add priority based on the previously created dscp-markings:

add name="line01_dscp-48-prio=2" packet-mark=dscp.48 parent=line01 priority=2
add name="line01_dscp-46-prio=3" packet-mark=dscp.46 parent=line01 priority=3
add name="line01_dscp-other-prio=7" packet-mark=dscp.other parent=line01 priority=7
add name="line01_dscp-none_besteffort" packet-mark=dscp.0 parent=line01

if you have a non-overhead-connection (eg. ethernet-connection) to your provider, you could do this (assuming the uplink is on ether2):

/queue tree
add max-limit=1000M name=ether2-queue parent=ether2 queue=default
add name="ether2_dscp-48-prio=2" packet-mark=dscp.48 parent=ether2-queue priority=2
add name="ether2_dscp-46-prio=3" packet-mark=dscp.46 parent=ether2-queue priority=3
add name="ether2_dscp-other-prio=7" packet-mark=dscp.other parent=ether2-queue priority=7
add name="ether2_dscp-none_besteffort" packet-mark=dscp.0 parent=ether2-queue

now in case you're wondering how the provider-end holds up to their promise - we do this on ciscos like this (simplified but also working if you want to try it):

class-map match-any MEDIA
 match dscp ef
class-map match-all VOICE
  description express forward = VoIP
 match ip dscp ef
class-map match-any SIGNALING
  description VoIP Signalling
 match ip dscp cs3
 match dscp cs3
 match dscp af31

now we have kind of what is a marking in mikrotiks for packets matching those classes in dscp.
we then need a policy what to do:

policy-map VOIP1M
 description VoIP Prio for 1Mbit
 class SIGNALING
  bandwidth 100
 class MEDIA
  bandwidth 1000
 class class-default
  fair-queue

we simpy define the bandwidth we want to be guaranteed downstream to the CPE and then
we also need to create a policy for the interface towards the user limiting the bandwidth to their expected bandwidth:

policy-map 10MVOIP
 class class-default
  shape average 10000000
   service-policy VOIP1M

this shapes the connection to 10Mbps and hands over the calculation to the before defined 1Mbps VoIP prio.
in your radius-server there then you add a pair like this to the user-account: cisco-avpair += ip:sub-qos-policy-out=10MVOIP

the result on the provider-end looks like this (here for a 30Mbps access-service):

sh policy-map interface Vi2.17

 Virtual-Access2.17

 SSS session identifier 447 -

  Service-policy output: 30MVOIP

    Class-map: class-default (match-any)
      7255033 packets, 949096170 bytes
      30 second offered rate 0000 bps, drop rate 0000 bps
      Match: any
      Queueing
      queue limit 64 packets
      (queue depth/total drops/no-buffer drops) 0/1401/0
      (pkts output/bytes output) 5906077/903171025
      shape (average) cir 30000000, bc 120000, be 120000
      target shape rate 30000000

      Service-policy : VOIP1M

        Class-map: SIGNALING (match-any)
          614466 packets, 396421487 bytes
          30 second offered rate 0000 bps, drop rate 0000 bps
          Match: ip dscp cs3 (24)
          Match:  dscp cs3 (24)
          Match:  dscp af31 (26)
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 614468/396422741
          bandwidth 100 kbps

        Class-map: MEDIA (match-any)
          453497 packets, 102353156 bytes
          30 second offered rate 0000 bps, drop rate 0000 bps
          Match:  dscp ef (46)
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 453497/102353156
          bandwidth 1000 kbps

        Class-map: class-default (match-any)
          6187064 packets, 450322568 bytes
          30 second offered rate 0000 bps, drop rate 0000 bps
          Match: any
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops/flowdrops) 0/1401/0/1401
          (pkts output/bytes output) 4838112/404395128
          Fair-queue: per-flow queue limit 16 packets

the drops if bandwidth is exceeded happen in the default-class (as expected).

at first DSCP might seem uncontrollable (and for some users who choose their own dscp-markings at their devices it is), but if your users also want to have the QoS for their VoIP calls and they follow the guidelines to simply use DSCP and leave the rest alone this usually works out very well and everybody gets what they want - good service.

maybe this helps someone.

bharrisau · Fri Feb 07, 2020 7:29 am

Wow, thank you for doing this. To remove any ambiguity, would you please post your queue config (between code tags) so I can study it? How well did my reference implementation do for you? I want to make sure that your config is not too specific to your needs as we do need something general that people can start and learn from.

Need text here for the code block to work.

/queue type
set 0 kind=sfq
add kind=pcq name=bulkUp pcq-classifier=src-address,dst-address,src-port,dst-port pcq-limit=450KiB pcq-total-limit=4500KiB
add kind=sfq name=bulkDown

/queue tree
add bucket-size=0.005 max-limit=90M name=DOWN parent=BR1 queue=default
add name="1. VOIP" packet-mark=VOIP parent=DOWN priority=1 queue=default
add name="2. DNS" packet-mark=DNS parent=DOWN priority=2 queue=default
add name="3. ACK" packet-mark=ACK parent=DOWN priority=3 queue=default
add name="4. UDP" packet-mark=UDP parent=DOWN priority=3 queue=default
add name="5. ICMP" packet-mark=ICMP parent=DOWN priority=4 queue=default
add name="6. HTTP" packet-mark=HTTP parent=DOWN priority=5 queue=bulkDown
add name="7. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=6 queue=bulkDown
add name="8. QUIC" packet-mark=QUIC parent=DOWN priority=7 queue=bulkDown
add name="9. OTHER" packet-mark=OTHER parent=DOWN priority=8 queue=bulkDown
add bucket-size=0.005 max-limit=36M name=UP parent=WAN1 queue=default
add name="1. VOIP_" packet-mark=VOIP parent=UP priority=1 queue=default
add name="2. DNS_" packet-mark=DNS parent=UP priority=2 queue=default
add name="3. ACK_" packet-mark=ACK parent=UP priority=3 queue=default
add name="4. UDP_" packet-mark=UDP parent=UP priority=3 queue=default
add name="5. ICMP_" packet-mark=ICMP parent=UP priority=4 queue=default
add name="6. HTTP_" packet-mark=HTTP parent=UP priority=5 queue=bulkUp
add name="7. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=6 queue=bulkUp
add name="8. QUIC_" packet-mark=QUIC parent=UP priority=7 queue=bulkUp
add name="9. OTHER_" packet-mark=OTHER parent=UP priority=8 queue=bulkUp

I'll re-run the tests next week with a baseline and the reference implementation to compare. Just trying to get a test server set up closer so there is less natural latency variability.

I'm also keen to test out the "Layer Cake" style system where instead of 9 categories you just have 3 - 'priority', 'sparse' and 'bulk'. VoIP becomes priority (maybe DNS and ACK too), and the rest of the connections are classified as either sparse or bulk depending on their connection-rate/packet-size/not-sure-yet. Cake is making them 'bulk' when the queue for that connection is not emptying, they are 'sparse' when packet/bit rate is low enough to not queue any packets (at least that is what I'm seeing from a quick scan through the code). I don't have access to the PCQ data at the mangle stage, so I need to experiment with a different heuristics - similar to your HTTP vs HTTP_BIG heuristic. Thinking something along the lines of this - with tweaking based on experiments next week. Mainly using connection-rate, and including some dst-limit to work around the fact that rate is calculated every second.

/ip firewall mangle

add action=mark-connection chain=prerouting comment="SPARSE by default" connection-mark=no-mark new-connection-mark=SPARSE passthrough=yes
add action=mark-connection chain=postrouting connection-mark=no-mark new-connection-mark=SPARSE passthrough=yes
add action=mark-connection chain=prerouting comment="SPARSE low rate" new-connection-mark=SPARSE connection-rate=0-1000 passthrough=yes
add action=mark-connection chain=postrouting new-connection-mark=SPARSE connection-rate=0-1000 passthrough=yes

add action=mark-connection chain=prerouting comment="SPARSE first packets when idle or new" new-connection-mark=SPARSE_FORCE connection-rate=0-1 dst-limit=1,1,addresses-and-dst-port/1m40s passthrough=yes
add action=mark-connection chain=postrouting new-connection-mark=SPARSE_FORCE connection-rate=0-1 dst-limit=1,1,addresses-and-dst-port/1m40s passthrough=yes
add action=mark-connection chain=prerouting comment="BULK large packets in idle or new" connection-mark=SPARSE new-connection-mark=BULK packet-size=!0-500 connection-rate=0-1 passthrough=yes
add action=mark-connection chain=postrouting connection-mark=SPARSE new-connection-mark=BULK packet-size=!0-500 connection-rate=0-1 passthrough=yes
add action=mark-connection chain=prerouting comment="BULK high rate" new-connection-mark=BULK connection-rate=!0-10000 passthrough=yes
add action=mark-connection chain=postrouting new-connection-mark=BULK connection-rate=!0-10000 passthrough=yes

add action=mark-connection chain=prerouting connection-mark=SPARSE_FORCE new-connection-mark=SPARSE passthrough=yes
add action=mark-connection chain=postrouting connection-mark=SPARSE_FORCE new-connection-mark=SPARSE passthrough=yes
add action=mark-packet chain=prerouting connection-mark=BULK new-packet-mark=BULK passthrough=no
add action=mark-packet chain=postrouting connection-mark=BULK new-packet-mark=BULK passthrough=no
add action=mark-packet chain=prerouting connection-mark=SPARSE new-packet-mark=SPARSE passthrough=no
add action=mark-packet chain=postrouting connection-mark=SPARSE new-packet-mark=SPARSE passthrough=no

Sun Feb 09, 2020 3:37 am

nice job guys, Thanks a lot

StubArea51 · Sun Feb 09, 2020 4:38 pm

Agreed, this is great work.

Gombeen666 · Sun Feb 09, 2020 5:57 pm

I get error "input does not match any value of parent"

Anyone got a complete file to test please?

bharrisau · Mon Feb 10, 2020 7:55 am

I've done some testing with a closer server. Four cases considered: baseline, QOS2020, QOS2020-Optimised (my recommended changes), mikroCAKE (pseudo-CAKE implementation).

Graphs of the bandwidth vs latency, and a latency distribution plot. All tests were Perth to Sydney, 30 seconds of baseline either side of a 120 second test. Local connection is 100/40 Mbps fibre, remote connection is 1000/250 Mbps OVH dedicated server. I've used 90% of the rated line speed locally for the settings.

Configuration exports below

Baseline

/ip firewall mangle
/queue type
/queue tree
add max-limit=36M name=UP packet-mark=no-mark parent=WAN1 queue=default
add max-limit=90M name=DOWN packet-mark=no-mark parent=BR1 queue=default

QOS2020

/ip firewall mangle
add action=mark-connection chain=prerouting comment=DNS connection-state=new new-connection-mark=DNS passthrough=yes port=53 protocol=udp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=DNS passthrough=yes port=53 protocol=udp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=VOIP new-connection-mark=VOIP passthrough=yes port=5060-5062,10000-20000 protocol=udp
add action=mark-packet chain=prerouting connection-mark=VOIP new-packet-mark=VOIP passthrough=no
add action=mark-connection chain=prerouting comment=QUIC connection-state=new new-connection-mark=QUIC passthrough=yes port=80,443 \
    protocol=udp
add action=mark-packet chain=prerouting connection-mark=QUIC new-packet-mark=QUIC passthrough=no
add action=mark-connection chain=prerouting comment=UDP connection-state=new new-connection-mark=UDP passthrough=yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=UDP new-packet-mark=UDP passthrough=no
add action=mark-connection chain=prerouting comment=ICMP connection-state=new new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=prerouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=postrouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-packet chain=postrouting comment=ACK new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-connection chain=prerouting comment=HTTP connection-mark=no-mark connection-state=new new-connection-mark=HTTP passthrough=\
    yes port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-bytes=5000000-0 connection-mark=HTTP connection-rate=2M-100M new-connection-mark=\
    HTTP_BIG passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no
add action=mark-connection chain=prerouting comment=OTHER connection-state=new new-connection-mark=POP3 passthrough=yes port=995,465,587 \
    protocol=tcp
add action=mark-packet chain=prerouting connection-mark=POP3 new-packet-mark=OTHER passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=OTHER passthrough=yes
add action=mark-packet chain=prerouting connection-mark=OTHER new-packet-mark=OTHER passthrough=no

/queue type
add kind=red name=customRED red-avg-packet=1514

/queue tree
add max-limit=36M name=UP parent=WAN1 queue=default
add max-limit=90M name=DOWN parent=BR1 queue=default
add name="1. VOIP" packet-mark=VOIP parent=DOWN priority=1 queue=default
add name="2. DNS" packet-mark=DNS parent=DOWN priority=2 queue=default
add name="3. ACK" packet-mark=ACK parent=DOWN priority=3 queue=default
add name="4. UDP" packet-mark=UDP parent=DOWN priority=3 queue=default
add name="5. ICMP" packet-mark=ICMP parent=DOWN priority=4 queue=default
add name="6. HTTP" packet-mark=HTTP parent=DOWN priority=5 queue=customRED
add name="7. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=6 queue=customRED
add name="8. QUIC" packet-mark=QUIC parent=DOWN priority=7 queue=customRED
add name="9. OTHER" packet-mark=OTHER parent=DOWN queue=default
add name="1. VOIP_" packet-mark=VOIP parent=UP priority=1 queue=default
add name="2. DNS_" packet-mark=DNS parent=UP priority=2 queue=default
add name="3. ACK_" packet-mark=ACK parent=UP priority=3 queue=default
add name="4. UDP_" packet-mark=UDP parent=UP priority=3 queue=default
add name="5. ICMP_" packet-mark=ICMP parent=UP priority=4 queue=default
add name="6. HTTP_" packet-mark=HTTP parent=UP priority=5 queue=customRED
add name="7. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=6 queue=customRED
add name="8. QUIC_" packet-mark=QUIC parent=UP priority=7 queue=customRED
add name="9. OTHER_" packet-mark=OTHER parent=UP queue=default

QOS2020 - Optimised

# Mangle is as above
/queue type
set 0 kind=sfq
add kind=pcq name=bulkUp pcq-classifier=src-address,dst-address,src-port,dst-port pcq-limit=450KiB pcq-total-limit=4500KiB

/queue tree
add bucket-size=0.005 max-limit=90M name=DOWN parent=BR1 queue=default
add name="1. VOIP" packet-mark=VOIP parent=DOWN priority=1 queue=default
add name="2. DNS" packet-mark=DNS parent=DOWN priority=2 queue=default
add name="3. ACK" packet-mark=ACK parent=DOWN priority=3 queue=default
add name="4. UDP" packet-mark=UDP parent=DOWN priority=3 queue=default
add name="5. ICMP" packet-mark=ICMP parent=DOWN priority=4 queue=default
add name="6. HTTP" packet-mark=HTTP parent=DOWN priority=5 queue=default
add name="7. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=6 queue=default
add name="8. QUIC" packet-mark=QUIC parent=DOWN priority=7 queue=default
add name="9. OTHER" packet-mark=OTHER parent=DOWN priority=8 queue=default
add bucket-size=0.005 max-limit=36M name=UP parent=WAN1 queue=default
add name="1. VOIP_" packet-mark=VOIP parent=UP priority=1 queue=default
add name="2. DNS_" packet-mark=DNS parent=UP priority=2 queue=default
add name="3. ACK_" packet-mark=ACK parent=UP priority=3 queue=default
add name="4. UDP_" packet-mark=UDP parent=UP priority=3 queue=default
add name="5. ICMP_" packet-mark=ICMP parent=UP priority=4 queue=default
add name="6. HTTP_" packet-mark=HTTP parent=UP priority=5 queue=bulkUp
add name="7. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=6 queue=bulkUp
add name="8. QUIC_" packet-mark=QUIC parent=UP priority=7 queue=bulkUp
add name="9. OTHER_" packet-mark=OTHER parent=UP priority=8 queue=bulkUp

mikroCAKE

/ip firewall mangle
add action=mark-connection chain=prerouting comment=DNS connection-state=new new-connection-mark=PRIO passthrough=yes port=53 protocol=udp
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=PRIO passthrough=yes port=53 protocol=udp
add action=mark-connection chain=prerouting comment=VOIP new-connection-mark=PRIO passthrough=yes port=5060-5062,10000-20000 protocol=udp
add action=mark-packet chain=postrouting connection-mark=PRIO new-packet-mark=PRIO passthrough=no
add action=mark-packet chain=prerouting connection-mark=PRIO new-packet-mark=PRIO passthrough=no
add action=mark-packet chain=postrouting comment=ACK new-packet-mark=PRIO packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting new-packet-mark=PRIO packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting comment=Sparse connection-rate=0-1 dst-limit=5,2,addresses-and-dst-port/1m40s new-packet-mark=\
    SPARSE passthrough=no
add action=mark-packet chain=postrouting connection-rate=1-10k new-packet-mark=SPARSE passthrough=no
add action=mark-packet chain=postrouting comment=Bulk new-packet-mark=BULK passthrough=no

/queue type
set 0 kind=sfq
add kind=pcq name=bulkUp pcq-classifier=src-address,dst-address,src-port,dst-port pcq-limit=450KiB pcq-total-limit=4500KiB

/queue tree
add bucket-size=0.005 max-limit=90M name=DOWN parent=BR1 queue=default
add name="1. Priority" packet-mark=PRIO parent=DOWN priority=1 queue=default
add name="2. Sparse" packet-mark=SPARSE parent=DOWN priority=2 queue=default
add name="3. Bulk" packet-mark=BULK parent=DOWN priority=3 queue=default
add bucket-size=0.005 max-limit=36M name=UP parent=WAN1 queue=default
add name="1. Priority_" packet-mark=PRIO parent=UP priority=1 queue=default
add name="2. Sparse_" packet-mark=SPARSE parent=UP priority=2 queue=default
add name="3. Bulk_" packet-mark=BULK parent=UP priority=3 queue=bulkUp

erhtun · Tue Feb 25, 2020 5:28 pm

I've done some testing with a closer server. Four cases considered: baseline, QOS2020, QOS2020-Optimised (my recommended changes), mikroCAKE (pseudo-CAKE implementation).

Graphs of the bandwidth vs latency, and a latency distribution plot. All tests were Perth to Sydney, 30 seconds of baseline either side of a 120 second test. Local connection is 100/40 Mbps fibre, remote connection is 1000/250 Mbps OVH dedicated server. I've used 90% of the rated line speed locally for the settings.

Configuration exports below

Baseline

/ip firewall mangle
/queue type
/queue tree
add max-limit=36M name=UP packet-mark=no-mark parent=WAN1 queue=default
add max-limit=90M name=DOWN packet-mark=no-mark parent=BR1 queue=default

QOS2020

/ip firewall mangle
add action=mark-connection chain=prerouting comment=DNS connection-state=new new-connection-mark=DNS passthrough=yes port=53 protocol=udp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=DNS passthrough=yes port=53 protocol=udp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=VOIP new-connection-mark=VOIP passthrough=yes port=5060-5062,10000-20000 protocol=udp
add action=mark-packet chain=prerouting connection-mark=VOIP new-packet-mark=VOIP passthrough=no
add action=mark-connection chain=prerouting comment=QUIC connection-state=new new-connection-mark=QUIC passthrough=yes port=80,443 \
    protocol=udp
add action=mark-packet chain=prerouting connection-mark=QUIC new-packet-mark=QUIC passthrough=no
add action=mark-connection chain=prerouting comment=UDP connection-state=new new-connection-mark=UDP passthrough=yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=UDP new-packet-mark=UDP passthrough=no
add action=mark-connection chain=prerouting comment=ICMP connection-state=new new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=prerouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=postrouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-packet chain=postrouting comment=ACK new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-connection chain=prerouting comment=HTTP connection-mark=no-mark connection-state=new new-connection-mark=HTTP passthrough=\
    yes port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-bytes=5000000-0 connection-mark=HTTP connection-rate=2M-100M new-connection-mark=\
    HTTP_BIG passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no
add action=mark-connection chain=prerouting comment=OTHER connection-state=new new-connection-mark=POP3 passthrough=yes port=995,465,587 \
    protocol=tcp
add action=mark-packet chain=prerouting connection-mark=POP3 new-packet-mark=OTHER passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=OTHER passthrough=yes
add action=mark-packet chain=prerouting connection-mark=OTHER new-packet-mark=OTHER passthrough=no

/queue type
add kind=red name=customRED red-avg-packet=1514

/queue tree
add max-limit=36M name=UP parent=WAN1 queue=default
add max-limit=90M name=DOWN parent=BR1 queue=default
add name="1. VOIP" packet-mark=VOIP parent=DOWN priority=1 queue=default
add name="2. DNS" packet-mark=DNS parent=DOWN priority=2 queue=default
add name="3. ACK" packet-mark=ACK parent=DOWN priority=3 queue=default
add name="4. UDP" packet-mark=UDP parent=DOWN priority=3 queue=default
add name="5. ICMP" packet-mark=ICMP parent=DOWN priority=4 queue=default
add name="6. HTTP" packet-mark=HTTP parent=DOWN priority=5 queue=customRED
add name="7. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=6 queue=customRED
add name="8. QUIC" packet-mark=QUIC parent=DOWN priority=7 queue=customRED
add name="9. OTHER" packet-mark=OTHER parent=DOWN queue=default
add name="1. VOIP_" packet-mark=VOIP parent=UP priority=1 queue=default
add name="2. DNS_" packet-mark=DNS parent=UP priority=2 queue=default
add name="3. ACK_" packet-mark=ACK parent=UP priority=3 queue=default
add name="4. UDP_" packet-mark=UDP parent=UP priority=3 queue=default
add name="5. ICMP_" packet-mark=ICMP parent=UP priority=4 queue=default
add name="6. HTTP_" packet-mark=HTTP parent=UP priority=5 queue=customRED
add name="7. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=6 queue=customRED
add name="8. QUIC_" packet-mark=QUIC parent=UP priority=7 queue=customRED
add name="9. OTHER_" packet-mark=OTHER parent=UP queue=default

QOS2020 - Optimised

# Mangle is as above
/queue type
set 0 kind=sfq
add kind=pcq name=bulkUp pcq-classifier=src-address,dst-address,src-port,dst-port pcq-limit=450KiB pcq-total-limit=4500KiB

/queue tree
add bucket-size=0.005 max-limit=90M name=DOWN parent=BR1 queue=default
add name="1. VOIP" packet-mark=VOIP parent=DOWN priority=1 queue=default
add name="2. DNS" packet-mark=DNS parent=DOWN priority=2 queue=default
add name="3. ACK" packet-mark=ACK parent=DOWN priority=3 queue=default
add name="4. UDP" packet-mark=UDP parent=DOWN priority=3 queue=default
add name="5. ICMP" packet-mark=ICMP parent=DOWN priority=4 queue=default
add name="6. HTTP" packet-mark=HTTP parent=DOWN priority=5 queue=default
add name="7. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=6 queue=default
add name="8. QUIC" packet-mark=QUIC parent=DOWN priority=7 queue=default
add name="9. OTHER" packet-mark=OTHER parent=DOWN priority=8 queue=default
add bucket-size=0.005 max-limit=36M name=UP parent=WAN1 queue=default
add name="1. VOIP_" packet-mark=VOIP parent=UP priority=1 queue=default
add name="2. DNS_" packet-mark=DNS parent=UP priority=2 queue=default
add name="3. ACK_" packet-mark=ACK parent=UP priority=3 queue=default
add name="4. UDP_" packet-mark=UDP parent=UP priority=3 queue=default
add name="5. ICMP_" packet-mark=ICMP parent=UP priority=4 queue=default
add name="6. HTTP_" packet-mark=HTTP parent=UP priority=5 queue=bulkUp
add name="7. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=6 queue=bulkUp
add name="8. QUIC_" packet-mark=QUIC parent=UP priority=7 queue=bulkUp
add name="9. OTHER_" packet-mark=OTHER parent=UP priority=8 queue=bulkUp

mikroCAKE

/ip firewall mangle
add action=mark-connection chain=prerouting comment=DNS connection-state=new new-connection-mark=PRIO passthrough=yes port=53 protocol=udp
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=PRIO passthrough=yes port=53 protocol=udp
add action=mark-connection chain=prerouting comment=VOIP new-connection-mark=PRIO passthrough=yes port=5060-5062,10000-20000 protocol=udp
add action=mark-packet chain=postrouting connection-mark=PRIO new-packet-mark=PRIO passthrough=no
add action=mark-packet chain=prerouting connection-mark=PRIO new-packet-mark=PRIO passthrough=no
add action=mark-packet chain=postrouting comment=ACK new-packet-mark=PRIO packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting new-packet-mark=PRIO packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting comment=Sparse connection-rate=0-1 dst-limit=5,2,addresses-and-dst-port/1m40s new-packet-mark=\
    SPARSE passthrough=no
add action=mark-packet chain=postrouting connection-rate=1-10k new-packet-mark=SPARSE passthrough=no
add action=mark-packet chain=postrouting comment=Bulk new-packet-mark=BULK passthrough=no

/queue type
set 0 kind=sfq
add kind=pcq name=bulkUp pcq-classifier=src-address,dst-address,src-port,dst-port pcq-limit=450KiB pcq-total-limit=4500KiB

/queue tree
add bucket-size=0.005 max-limit=90M name=DOWN parent=BR1 queue=default
add name="1. Priority" packet-mark=PRIO parent=DOWN priority=1 queue=default
add name="2. Sparse" packet-mark=SPARSE parent=DOWN priority=2 queue=default
add name="3. Bulk" packet-mark=BULK parent=DOWN priority=3 queue=default
add bucket-size=0.005 max-limit=36M name=UP parent=WAN1 queue=default
add name="1. Priority_" packet-mark=PRIO parent=UP priority=1 queue=default
add name="2. Sparse_" packet-mark=SPARSE parent=UP priority=2 queue=default
add name="3. Bulk_" packet-mark=BULK parent=UP priority=3 queue=bulkUp

Hi, I am trying your configuration. Until now there are too much drops ACK upload packets than HTTP or HTTP_BIG packets. Download ACK packets are not dropped. Is this normal?

Also I want to ask a question aboud bulkUp pcq queue size and total queue size. I have set upload max-limit to 4M. What I should set queue size and total queue size in bulkUp queue type for my connection?

zlsd · Sat Feb 29, 2020 11:56 am

Why don't you use "PCQ" instead of the default queue?
Could you create a rule that would reduce the priority for a large number of connections for applications like windows update?
Why is "SFQ" better for download than "PCQ"?

KingKeA · Sun Mar 15, 2020 4:37 pm

Thank you so much for this!

I have 940mbps/35mbps cable using a RB4011 router. When applying the optimized QOS, I see better results at dslreport's bufferbloat test, but my overall throughput is much lower, around 500mbps. Is there a way to have QOS and still have close to line speeds with this model?

Thank you again!

purplecloud · Sun Mar 29, 2020 1:04 pm

Thank you so much for this!

I have 940mbps/35mbps cable using a RB4011 router. When applying the optimized QOS, I see better results at dslreport's bufferbloat test, but my overall throughput is much lower, around 500mbps. Is there a way to have QOS and still have close to line speeds with this model?

Thank you again!

I second this question. I have similar problems. Would it be similar with QOS?

Jotne · Thu Apr 09, 2020 4:10 pm

How to Mark the Traffic Types:

Nice thread.

I have not read all in detail yet, but there are on thing I do miss or not understand.
Do you need to mark the connection before mark the packet? Yes/No/Why?
Please add some words about this in your second post where you talk about marking.

Jotne · Thu Apr 09, 2020 4:29 pm

I have a question to ask the admins (how do we private message you and talk about the forums)? I am planning on rewriting this article. What is the best course of action to maintain the link (which is pinned and also maybe linked elsewhere)? I would like for all posts to be deleted except for the first six ones (I'm going to go back and edit my own). Is there a better recommendation?

phpBB that this forum is based on do support private message user/user user/admin etc, but for some reason MikroTik has turned this function off. It would be nice to have it one, since there are times you like to exchange private information that are not for all to read.

sindy · Thu Apr 09, 2020 5:01 pm

Do you need to mark the connection before mark the packet? Yes/No/Why?

Before @pcunite addresses that, a quick answer:
Connection marking is a way to give all packets of the same "connection" (a TCP session, or a bi-directional UDP stream) the same treatment without need to analyse all the selection criteria for each individual packet. So usually the processing of the initial packet of a new connection results in choosing a treatment for the whole connection, and all connections whose packets should be treated that way are labeled with the same connection mark. Whenever a packet arrives to the firewall, the connection tracking module matches it to the list of existing connections, and if it finds a match, the connection mark is retrieved from the connection context (along with e.g. the eventual NAT treatment which is, however, applied directly by the connection tracking module) and attached as a metafield to the packet, so that further layers of the firewall (and queueing) can refer to it instead of re-evaluating all the other conditions again. Routing cannot refer to connection-mark directly but needs a routing-mark, so a rule must be used to translate the former into the latter for each packet; queues need a packet-mark so a translation rule is needed too. NAT rules can refer directly to connection-mark because they're an organic part of the firewall; routing and queueing are not. A connection mark may be changed during the lifetime of the connection, but each connection may only bear a single connection mark at any given time, so policy routing along with queueing is a bit complex, yet doable. The change of connection mark in context of QoS is useful to de-prioritize large downloads to allow quick response for small-volume exchanges.

pcunite · Thu Apr 09, 2020 8:23 pm

Do you need to mark the connection before mark the packet? Yes/No/Why? Please add some words about this in your second post where you talk about marking.

How to use RouterOS to accomplish a task vs. how RouterOS itself works are two different concepts. I only focus on the former. However, everyone is welcome to discuss. The examples show what you need to do in order to facilitate QoS.

Jotne · Thu Apr 09, 2020 8:56 pm

I am trying to learning, so bear with me.

I could mark the HTTP packets directly by setting TCP and port 80 and select mark packet, without marking connection?

sindy · Thu Apr 09, 2020 9:01 pm

I could mark the HTTP packets directly by setting TCP and port 80 and select mark packet, without marking connection?

Yes, that would work too. The only drawback is that matching two items (protocol & port) takes more CPU than matching just one (connection mark), and if you start thinking about discrimination between several categories of http connections, so you start matching e.g. address lists, the amount of CPU power spent on handling every single packet in the firewall grows even higher.

Jotne · Thu Apr 09, 2020 11:00 pm

Thanks for the explanation, some clearer now

pcunite · Thu Apr 16, 2020 10:46 pm

Set parent queues to have a bucket-size of 0.005. Changed the default queue to sfq. (Using red gave similar performance, but multiple downloads seemed less fair). Rationale for the 0.005 size is to copy CoDel as much as possible.

@bharrisau,

I've have tested your feedback and have made changes to the original post to include bucket-size recommendations, which indeed was helpful. However, I do find SFQ to be situational dependent. Yes, it drops very quickly and responds well to speed tests, but it does not handle global synchronization well, which is a real problem in the wild. Thus, I still show the recommendation for using RED, howbeit, as you did for all queues, which is a cleaner config example. There is still tweaking here to do. More testing is needed under real workloads.

bharrisau · Sat Apr 18, 2020 9:55 am

@pcunite
No worries. I'll see if I can rerun the test graphs with the updated baseline when office reopens.

Dude2048 · Sat Apr 18, 2020 10:16 am

@pcunite
Thanks for the hard work.

pcunite · Sun Apr 19, 2020 8:33 pm

@pcunite
No worries. I'll see if I can rerun the test graphs with the updated baseline when office reopens.

I've seen some weird packet issues when using RED for the defaults. So, I've gone back to SFQ for default but use RED for the bulky flows. The behavior of RED as default causes the VoIP queue to drop a single packet here and there when it does not appear to be under pressure. Perhaps there is something else going on, but I never want a packet to drop there, so SFQ (or probably any other type) prevents that behavior.

zx128k · Fri Apr 24, 2020 11:52 am

Guys, can anyone share QOS configuration which prioritizes gaming, browsing and some other important packets over downloads/uploads and video traffics, please?

digin4 · Sat Apr 25, 2020 2:20 pm

Does it make sense to use RED for VPN queue running on UDP only? the congestion methods mentioned all seem to apply to TCP exclusively, but in the guide it's mentioned that it can be applied to QUIC which uses UDP so I gues this means it's fine to use it on UDP VPN?

bharrisau · Sun Apr 26, 2020 3:21 am

the congestion methods mentioned all seem to apply to TCP exclusively

TCP and QUIC are stream protocols - as in you have a large amount of data and you will send multiple packets to transfer it. What the connection methods do is to drop some packets as a signal that the stream should slow down. So we are using RED for example to say 'speed up' or 'slow down'. TCP and QUIC have built in mechanisms to resend lost data so it doesn't cause much issue to drop data.

UDP is just a packet format. Depending on the application it will behave differently. Some applications don't have and contention management so dropping packets doesn't cause it to slow down (it actually speeds up because it has to send everything twice). For example VoIP and gaming is usually a fixed rate, and if you drop a packet it causes stuttering or lag. BitTorrent is also UDP, but uses dropped packets to slow the stream down.

So the simple answer is - the router doesn't care if it is TCP or UDP. All TCP connections can be slowed by dropping packets. Some UDP can be slowed, some UDP you don't want to drop packets. It is based on the application though, and not anything to do with UDP.

jaguar666 · Mon May 04, 2020 7:30 pm

@bharrisau, @pcunite and to all experts out there

Is it recommended to use 2 QOS servers, 1 is for the 2020 QOS-Optimized server (as main bandwidth management) and the other one is for the microCAKE and this will be running inside PPPoE/ Hotspot server?

santyx32 · Mon May 04, 2020 8:10 pm

@bharrisau I'm using your QOS2020 Optimized script and it's working awesome but on my home network I have two bridges, one for the main network and the other for a guest wifi, how should I proceed?

aletrevizoli · Mon May 04, 2020 11:40 pm

I have an BIG doubt.....My network is distributed, I have dozens of MK routers with OSPF on network and on each site I have an OLT and one Router that is PPPoE server.
I have too two places that arrive Internet(with two or more Carrier Link), on each place I have an MK to control this links.
So, where I should use the QOS? in all routers, In PPPoE servers or in Last router (link control).
Thanks a lot!
Very good work!

kenyloveg · Sat May 09, 2020 4:48 pm

My network only have two devices connect to one HAP AC2 through 2 RJ45 cable.(one is an openwrt router, and other is windows 7 computer)
What is the most effecient way to make windows 7 computer the higher priority than the other?
Maybe something like mac/ether port based QoS can be easily set up? (for ex, ether2 in bridge1 is higher priority than ether3 in bridge1)
Thanks.

jaguar666 · Sat May 09, 2020 9:32 pm

My network only have two devices connect to one HAP AC2 through 2 RJ45 cable.(one is an openwrt router, and other is windows 7 computer)
What is the most effecient way to make windows 7 computer the higher priority than the other?
Maybe something like mac/ether port based QoS can be easily set up? (for ex, ether2 in bridge1 is higher priority than ether3 in bridge1)
Thanks.

You may create a simple queues entry specifying the target IP or the entire interface you want to prioritize.

santyx32 · Sat May 09, 2020 10:26 pm

You may create a simple queues entry specifying the target IP or the entire interface you want to prioritize.

Yeah, that would get the job done for OP but beware that doing that on a wireless network will also limit LAN-LAN speed. I say this since I've seen lots of tutorials suggesting people do this.

santyx32 · Sun May 10, 2020 12:13 am

This is my config but I can't get it to work

/ip firewall mangle
add action=mark-connection chain=forward comment=MAIN-DOWNLOAD connection-state=new in-interface=bridge new-connection-mark=main-download passthrough=yes
add action=mark-packet chain=forward connection-mark=main-download new-packet-mark=main-download passthrough=yes
add action=mark-connection chain=forward comment=GUEST-DOWNLOAD connection-state=new in-interface=bridge-invitados new-connection-mark=guest-download passthrough=yes
add action=mark-packet chain=forward connection-mark=guest-download new-packet-mark=guest-download passthrough=yes
add action=mark-connection chain=forward comment=MAIN-UPLOAD connection-state=new new-connection-mark=main-upload out-interface=bridge passthrough=yes
add action=mark-packet chain=forward connection-mark=main-upload new-packet-mark=main-upload passthrough=yes
add action=mark-connection chain=forward comment=GUEST-UPLOAD connection-state=new new-connection-mark=guest-upload out-interface=bridge-invitados passthrough=yes
add action=mark-packet chain=forward connection-mark=guest-upload new-packet-mark=guest-upload passthrough=yes
add action=mark-connection chain=prerouting comment=DNS connection-state=new new-connection-mark=DNS passthrough=yes port=53 protocol=udp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=DNS passthrough=yes port=53 protocol=udp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=VOIP new-connection-mark=VOIP passthrough=yes port=5060-5062,10000-20000 protocol=udp
add action=mark-packet chain=prerouting connection-mark=VOIP new-packet-mark=VOIP passthrough=no
add action=mark-connection chain=prerouting comment=QUIC connection-state=new new-connection-mark=QUIC passthrough=yes port=80,443 protocol=udp
add action=mark-packet chain=prerouting connection-mark=QUIC new-packet-mark=QUIC passthrough=no
add action=mark-connection chain=prerouting comment=UDP connection-state=new new-connection-mark=UDP passthrough=yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=UDP new-packet-mark=UDP passthrough=no
add action=mark-connection chain=prerouting comment=ICMP connection-state=new new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=prerouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=postrouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-packet chain=postrouting comment=ACK new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-connection chain=prerouting comment=HTTP connection-mark=no-mark connection-state=new new-connection-mark=HTTP passthrough=yes port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-bytes=5000000-0 connection-mark=HTTP connection-rate=2M-100M new-connection-mark=HTTP_BIG passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no
add action=mark-connection chain=prerouting comment=OTHER connection-state=new new-connection-mark=POP3 passthrough=yes port=995,465,587 protocol=tcp
add action=mark-packet chain=prerouting connection-mark=POP3 new-packet-mark=OTHER passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=OTHER passthrough=yes
add action=mark-packet chain=prerouting connection-mark=OTHER new-packet-mark=OTHER passthrough=no


/queue tree
add max-limit=68M name=GLOBAL-DOWN parent=global queue=default
add bucket-size=0.005 max-limit=68M name=MAIN-DOWN packet-mark=main-download parent=GLOBAL-DOWN priority=1 queue=default
add name="1. VOIP" packet-mark=VOIP parent=MAIN-DOWN priority=1 queue=default
add name="2. DNS" packet-mark=DNS parent=MAIN-DOWN priority=2 queue=default
add name="3. ACK" packet-mark=ACK parent=MAIN-DOWN priority=3 queue=default
add name="4. UDP" packet-mark=UDP parent=MAIN-DOWN priority=3 queue=default
add name="5. ICMP" packet-mark=ICMP parent=MAIN-DOWN priority=4 queue=default
add name="6. HTTP" packet-mark=HTTP parent=MAIN-DOWN priority=5 queue=default
add name="7. HTTP_BIG" packet-mark=HTTP_BIG parent=MAIN-DOWN priority=6 queue=default
add name="8. QUIC" packet-mark=QUIC parent=MAIN-DOWN priority=7 queue=default
add name="9. OTHER" packet-mark=OTHER parent=MAIN-DOWN queue=default
add bucket-size=0.005 max-limit=20M name=GUEST-DOWN packet-mark=guest-download parent=GLOBAL-DOWN priority=2 queue=default
add name="1. VOIP_GD" packet-mark=VOIP parent=GUEST-DOWN priority=1 queue=default
add name="2. DNS_GD" packet-mark=DNS parent=GUEST-DOWN priority=2 queue=default
add name="3. ACK_GD" packet-mark=ACK parent=GUEST-DOWN priority=3 queue=default
add name="4. UDP_GD" packet-mark=UDP parent=GUEST-DOWN priority=3 queue=default
add name="5. ICMP_GD" packet-mark=ICMP parent=GUEST-DOWN priority=4 queue=default
add name="6. HTTP_GD" packet-mark=HTTP parent=GUEST-DOWN priority=5 queue=default
add name="7. HTTP_BIG_GD" packet-mark=HTTP_BIG parent=GUEST-DOWN priority=6 queue=default
add name="8. QUIC_GD" packet-mark=QUIC parent=GUEST-DOWN priority=7 queue=default
add max-limit=70M name=GLOBAL-UP parent=global queue=default
add bucket-size=0.005 max-limit=70M name=MAIN-UP packet-mark=main-upload parent=GLOBAL-UP priority=1 queue=default
add name="1. VOIP_" packet-mark=VOIP parent=MAIN-UP priority=1 queue=default
add name="2. DNS_" packet-mark=DNS parent=MAIN-UP priority=2 queue=default
add name="3. ACK_" packet-mark=ACK parent=MAIN-UP priority=3 queue=default
add name="4. UDP_" packet-mark=UDP parent=MAIN-UP priority=3 queue=default
add name="5. ICMP_" packet-mark=ICMP parent=MAIN-UP priority=4 queue=default
add name="6. HTTP_" packet-mark=HTTP parent=MAIN-UP priority=5 queue=bulkUp-Main
add name="7. HTTP_BIG_" packet-mark=HTTP_BIG parent=MAIN-UP priority=6 queue=bulkUp-Main
add name="8. QUIC_" packet-mark=QUIC parent=MAIN-UP priority=7 queue=bulkUp-Main
add name="9. OTHER_" packet-mark=OTHER parent=MAIN-UP queue=bulkUp-Main
add bucket-size=0.005 max-limit=20M name=GUEST-UP packet-mark=guest-upload parent=GLOBAL-UP priority=2 queue=default
add name="1. VOIP_GU" packet-mark=VOIP parent=GUEST-UP priority=1 queue=default
add name="2. DNS_GU" packet-mark=DNS parent=GUEST-UP priority=2 queue=default
add name="3. ACK_GU" packet-mark=ACK parent=GUEST-UP priority=3 queue=default
add name="4. UDP_GU" packet-mark=UDP parent=GUEST-UP priority=3 queue=default
add name="5. ICMP_GU" packet-mark=ICMP parent=GUEST-UP priority=4 queue=default
add name="6. HTTP_GU" packet-mark=HTTP parent=GUEST-UP priority=5 queue=bulkUp-Guest
add name="7. HTTP_BIG_GU" packet-mark=HTTP_BIG parent=GUEST-UP priority=6 queue=bulkUp-Guest
add name="8. QUIC_GU" packet-mark=QUIC parent=GUEST-UP priority=7 queue=bulkUp-Guest
add name="9. OTHER_GU" packet-mark=OTHER parent=GUEST-UP queue=bulkUp-Guest

kenyloveg · Sun May 10, 2020 6:19 pm

My network only have two devices connect to one HAP AC2 through 2 RJ45 cable.(one is an openwrt router, and other is windows 7 computer)
What is the most effecient way to make windows 7 computer the higher priority than the other?
Maybe something like mac/ether port based QoS can be easily set up? (for ex, ether2 in bridge1 is higher priority than ether3 in bridge1)
Thanks.
You may create a simple queues entry specifying the target IP or the entire interface you want to prioritize.

Thanks for your replay.
Can you discribe this step more detailed?
I have pppoe-out1 for wan, bridge1 (includ port ether3 and ether4) for internal lan, and deive A connected to ether3, device B to ether4.
I want deivce A get higher priority than device B, just play games regardless the bandwidth.

Zacharias · Fri May 15, 2020 8:56 pm

I ve used Queue Trees mostly with Global Parent.
My question is, why do we use as Parent the Lan Interface for the Download Traffic and the WAN for the Upload Traffic ?

dmuzikero · Wed May 20, 2020 11:24 am

version 3, running smooth

Hi can u share script? thanks man

mikegleasonjr · Sat May 23, 2020 2:48 am

Hi, thanks for the ver. 2020 of the guide. I learned a ton while implementing it and playing with it.

But for me there was a few issues. Mainly regarding the lack of flexibility and problems about incoming packets not being accounted into the queues (even if properly marked by the mangle rules). I'm no expert, only a software engineer so feel free to comment on my findings.

1. lack of flexibility

Having the queue tree main parents (in and out) using interfaces as parents (LAN & WAN) assumes the router is an "internet gateway" only router and that all the traffic would only be internet traffic. What if I want to track (or not track) forwarded traffic between the lan interfaces, etc ? What if I want non-internet related queues at the same time ? I wanted a common root for the Internet traffic, not 2 nodes at the root (in and out).

2. packets not going through the queues

When I was checking a ping and a pong packets going out of the router itself (not forwarded), they would be properly marked by the mangle rules but only the outgoing ones would be accounted in the queues. Since the incoming queue subtree has the LAN as parent, the incoming pong packet would not be accounted (since it never reaches the LAN). You can try and reproduce by yourself by ticking the log checkbox in the mangle rules for icmp and by doing a ping from the router to the outside. Ping might not be a big deal but my DNS traffic were doing the same, because the router makes the queries (in my case). Unless I missed something, that's what I observed.

My solution

I came up which seems the be one of the 2 main approach I saw on the Internet. Mark packets irrespective of their direction (this 2020 guide) or mark separately the ins from outs packets (my final solution). I was able to fix the 2 problems I mentioned earlier with this:

Mangle rules (wlan-2G-client is my wan)

/ip firewall mangle
    add action=mark-connection chain=postrouting comment=icmp connection-state=new new-connection-mark=icmp-conn out-interface=wlan-2G-client passthrough=yes protocol=icmp
    add action=mark-packet chain=postrouting connection-mark=icmp-conn new-packet-mark=icmp-out-pk out-interface=wlan-2G-client passthrough=no
    add action=mark-packet chain=prerouting connection-mark=icmp-conn in-interface=wlan-2G-client new-packet-mark=icmp-in-pk passthrough=no
    add action=mark-connection chain=postrouting comment=dns connection-state=new dst-port=53 new-connection-mark=dns-conn out-interface=wlan-2G-client passthrough=yes protocol=udp
    add action=mark-connection chain=postrouting connection-state=new dst-port=53 new-connection-mark=dns-conn out-interface=wlan-2G-client passthrough=yes protocol=tcp
    add action=mark-packet chain=postrouting connection-mark=dns-conn new-packet-mark=dns-out-pk out-interface=wlan-2G-client passthrough=no
    add action=mark-packet chain=prerouting connection-mark=dns-conn in-interface=wlan-2G-client new-packet-mark=dns-in-pk passthrough=no
    add action=mark-connection chain=postrouting comment=voip connection-state=new dst-port=5060,5061 new-connection-mark=voip-conn out-interface=wlan-2G-client passthrough=yes protocol=udp
    add action=mark-connection chain=postrouting connection-state=new dst-port=5060,5061 new-connection-mark=voip-conn out-interface=wlan-2G-client passthrough=yes protocol=tcp
    add action=mark-packet chain=postrouting connection-mark=voip-conn new-packet-mark=voip-out-pk out-interface=wlan-2G-client passthrough=no
    add action=mark-packet chain=prerouting connection-mark=voip-conn in-interface=wlan-2G-client new-packet-mark=voip-in-pk passthrough=no
    add action=mark-connection chain=postrouting comment=zoom connection-state=new dst-port=3478,3479,8801-8810,5090 new-connection-mark=zoom-conn out-interface=wlan-2G-client passthrough=yes protocol=udp
    add action=mark-connection chain=postrouting connection-state=new dst-port=8801,8802,5090,5091 new-connection-mark=zoom-conn out-interface=wlan-2G-client passthrough=yes protocol=tcp
    add action=mark-packet chain=postrouting connection-mark=zoom-conn new-packet-mark=zoom-out-pk out-interface=wlan-2G-client passthrough=no
    add action=mark-packet chain=prerouting connection-mark=zoom-conn in-interface=wlan-2G-client new-packet-mark=zoom-in-pk passthrough=no
    add action=mark-connection chain=postrouting comment=rocket-league connection-state=new dst-port=7000-8800,8811-8887,8889-9000 new-connection-mark=rocketleague-conn out-interface=wlan-2G-client passthrough=yes protocol=udp
    add action=mark-packet chain=postrouting connection-mark=rocketleague-conn new-packet-mark=rocketleague-out-pk out-interface=wlan-2G-client passthrough=no
    add action=mark-packet chain=prerouting connection-mark=rocketleague-conn in-interface=wlan-2G-client new-packet-mark=rocketleague-in-pk passthrough=no
    add action=mark-connection chain=postrouting comment=other connection-state=new new-connection-mark=other-conn out-interface=wlan-2G-client passthrough=yes
    add action=mark-packet chain=postrouting connection-mark=other-conn new-packet-mark=other-out-pk out-interface=wlan-2G-client passthrough=no
    add action=mark-packet chain=prerouting connection-mark=other-conn in-interface=wlan-2G-client new-packet-mark=other-in-pk passthrough=no

Queues

/queue tree
    add max-limit=22M name=internet parent=global queue=ethernet-default
    add max-limit=10M name=internet-in parent=internet queue=ethernet-default
    add max-limit=12M name=internet-out parent=internet queue=ethernet-default
    add name=zoom-in packet-mark=zoom-in-pk parent=internet-in priority=4 queue=ethernet-default
    add name=zoom-out packet-mark=zoom-out-pk parent=internet-out priority=4 queue=ethernet-default
    add name=icmp-out packet-mark=icmp-out-pk parent=internet-out priority=2 queue=ethernet-default
    add name=icmp-in packet-mark=icmp-in-pk parent=internet-in priority=2 queue=ethernet-default
    add name=dns-out packet-mark=dns-out-pk parent=internet-out priority=2 queue=ethernet-default
    add name=dns-in packet-mark=dns-in-pk parent=internet-in priority=2 queue=ethernet-default
    add name=voip-in packet-mark=voip-in-pk parent=internet-in priority=2 queue=ethernet-default
    add name=voip-out packet-mark=voip-out-pk parent=internet-out priority=2 queue=ethernet-default
    add name=rocketleague-in packet-mark=rocketleague-in-pk parent=internet-in priority=4 queue=ethernet-default
    add name=rocketleague-out packet-mark=rocketleague-out-pk parent=internet-out priority=4 queue=ethernet-default
    add bucket-size=0.01 max-limit=10M name=other-out packet-mark=other-out-pk parent=internet-out queue=pcq-upload-default
    add max-limit=8M name=other-in packet-mark=other-in-pk parent=internet-in queue=pcq-download-default

A picture is worth a thousand words

winbox.png

With this setup, I am able to track any traffic that goes out of my wan port, whether or not it originates from the router itself or not. No more, no less.

Hope this helps someone.

jamesbutterfill · Fri Jun 05, 2020 5:18 pm

Hi All. Firstly thanks for all this hard work. I've been having a bit of an issue though. I have the LHG Cat6 LTE Mikrotik router but my speed on a day by day basis varies significantly with downloads varying from 20-80mbits depending on environmental/network conditions. My upload can vary from 2-12mbits too. So what speeds should I specify without inhibiting performance on the days when the UL/DL rates are much higher?

WayneF · Thu Jun 11, 2020 2:26 am

How should I go about choosing sensible max limits for the queues.

I have a X Mpbs down and a Y Mbps up; well so the ISP claims. When I actually do tests I hardly ever see those results when using speedtest.net,
using various test servers, it seems to vary as most test servers fail to deliver the speeds.

Should I be setting the limit's to more practical values, values I most typically see?

Also should one be considering the Maximum WiFi Speeds and use different queues for the WiFi and have different Max values?
(to me this makes sense as I cannot get the maximum values over WiFi; my crude testing I could see icmp response time increases when I
sent packets over the WiFi while also doing a speedtest over the WiFi).

Also how does one establish the WiFi max Speed using Mac to Mikrotik?

Here is my crude testing strategy:
1) I start sending (icmp packets) pings continuously to googles dns server 8.8.8.8 - using the Mikrotik Ping tool
2) then I do a speedtest over WiFi to the fastest server I can find.
3) record the icmp response times and dropped packets in the queues.

[Download test]
No increase in response times.
No dropped packets. (Which is expect as my WiFi cannot get close to the Max Download of X Mpbs - 10%.)

[Upload test]
No increase in response times.
And some dropped packets for the "OTHER" entry.

When I ping over WiFi the icmp response times are affected as I am maximising the WiFi throughput I assume.

kswong · Thu Jun 18, 2020 7:54 am

TCP uses a three-way handshake to establish a reliable connection, why prioritize the ack traffic only??

erhtun · Thu Jun 25, 2020 12:43 pm

@pcunite, @kswong what is your queue > interface types for Wan interface(ether1) and wifi interfaces?

sbernesto · Thu Jun 25, 2020 10:00 pm

Hi,
Why is it necessary to deactivate the filter fasttrack connection?
or only in this particular configuration is it required or does it apply to all config shared?

TCP uses a three-way handshake to establish a reliable connection, why prioritize the ack traffic only??

My setup able to track the TCP 3-way & 4-way handshake traffic ...

Mangle rules

/ip firewall mangle
add action=mark-packet chain=prerouting comment="FIN / SYN / ACK / RST in" in-interface-list=WAN new-packet-mark=FIN-I packet-size=0-64 passthrough= no protocol=tcp tcp-flags=fin
add action=mark-packet chain=prerouting in-interface-list=WAN new-packet-mark=SYN-I packet-size=0-64 passthrough=no protocol=tcp tcp-flags=syn
add action=mark-packet chain=prerouting in-interface-list=WAN new-packet-mark=RST-I packet-size=0-64 passthrough=no protocol=tcp tcp-flags=rst
add action=mark-packet chain=prerouting in-interface-list=WAN new-packet-mark=ACK-I packet-size=0-64 passthrough=no protocol=tcp tcp-flags=ack,!psh
add action=mark-packet chain=postrouting comment="FIN / SYN / ACK / RST out" new-packet-mark=FIN-O out-interface-list=WAN packet-size=0-64 passthrough=no protocol=tcp tcp-flags=fin
add action=mark-packet chain=postrouting new-packet-mark=SYN-O out-interface-list=WAN packet-size=0-64 passthrough=no protocol=tcp tcp-flags=syn
add action=mark-packet chain=postrouting new-packet-mark=RST-O out-interface-list=WAN packet-size=0-64 passthrough=no protocol=tcp tcp-flags=rst
add action=mark-packet chain=postrouting new-packet-mark=ACK-O out-interface-list=WAN packet-size=0-64 passthrough=no protocol=tcp tcp-flags=ack,!psh
add action=mark-connection chain=prerouting comment="ICMP in" connection-mark=no-mark new-connection-mark=CM-ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=prerouting connection-mark=CM-ICMP in-interface-list=WAN new-packet-mark=ICMP-I passthrough=no
add action=mark-connection chain=postrouting comment="ICMP out" connection-mark=no-mark new-connection-mark=CM-ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=postrouting connection-mark=CM-ICMP new-packet-mark=ICMP-O out-interface-list=WAN passthrough=no
add action=mark-connection chain=prerouting comment="DNS in" connection-mark= no-mark dst-port=53,853 new-connection-mark=CM-DNS passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-port= 53,853 new-connection-mark=CM-DNS passthrough=yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=CM-DNS in-interface-list=WAN new-packet-mark=DNS-I passthrough=no
add action=mark-connection chain=postrouting comment="DNS out" connection-mark=no-mark dst-port=53,853 new-connection-mark=CM-DNS passthrough=yes protocol=tcp
add action=mark-connection chain=postrouting connection-mark=no-mark dst-port=53,853 new-connection-mark=CM-DNS passthrough=yes protocol=udp
add action=mark-packet chain=postrouting connection-mark=CM-DNS new-packet-mark=DNS-O out-interface-list=WAN passthrough=no
add action=mark-connection chain=prerouting comment="VPN in" connection-mark= no-mark dst-port=1723 new-connection-mark=CM-VPN passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-port= 500,1194,1701,4500 new-connection-mark=CM-VPN passthrough=yes protocol= udp
add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=CM-VPN passthrough=yes protocol=gre
add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=CM-VPN passthrough=yes protocol=ipsec-esp
add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=CM-VPN passthrough=yes protocol=ipsec-ah
add action=mark-packet chain=prerouting connection-mark=CM-VPN in-interface-list=WAN new-packet-mark=VPN-I passthrough=no
add action=mark-connection chain=postrouting comment="VPN out" connection-mark=no-mark dst-port=1723 new-connection-mark=CM-VPN passthrough=yes protocol=tcp
add action=mark-connection chain=postrouting connection-mark=no-mark dst-port=500,1194,1701,4500 new-connection-mark=CM-VPN passthrough=yes protocol=udp
add action=mark-connection chain=postrouting connection-mark=no-mark new-connection-mark=CM-VPN passthrough=yes protocol=gre
add action=mark-connection chain=postrouting connection-mark=no-mark new-connection-mark=CM-VPN passthrough=yes protocol=ipsec-esp
add action=mark-connection chain=postrouting connection-mark=no-mark new-connection-mark=CM-VPN passthrough=yes protocol=ipsec-ah
add action=mark-packet chain=postrouting connection-mark=CM-VPN new-packet-mark=VPN-O out-interface-list=WAN passthrough=no
add action=mark-connection chain=prerouting comment="VOIP in" connection-mark=no-mark dscp=26 new-connection-mark=CM-VOIP passthrough= yes
add action=mark-connection chain=prerouting connection-mark=no-mark dscp=46 new-connection-mark=CM-VOIP passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-port= 3478-3479,5060-5061,8801-8802 new-connection-mark=CM-VOIP passthrough= yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=CM-VOIP in-interface-list=WAN new-packet-mark=VOIP-I passthrough=no
add action=mark-connection chain=postrouting comment="VOIP out" connection-mark=no-mark dscp=26 new-connection-mark=CM-VOIP passthrough= yes
add action=mark-connection chain=postrouting connection-mark=no-mark dscp=46 new-connection-mark=CM-VOIP passthrough=yes
add action=mark-connection chain=postrouting connection-mark=no-mark dst-port=3478-3479,5060-5061,8801-8802 new-connection-mark=CM-VOIP passthrough=yes protocol=udp
add action=mark-packet chain=postrouting connection-mark=CM-VOIP new-packet-mark=VOIP-O out-interface-list=WAN passthrough=no
add action=mark-connection chain=prerouting comment="EMAIL in" connection-mark=no-mark dst-port=25,110,143,465,587,993,995 new-connection-mark=CM-EMAIL passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting connection-mark=CM-EMAIL in-interface-list=WAN new-packet-mark=EMAIL-I passthrough=no
add action=mark-connection chain=postrouting comment="EMAIL out" connection-mark=no-mark dst-port=25,110,143,465,587,993,995 new-connection-mark=CM-EMAIL passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting connection-mark=CM-EMAIL new-packet-mark=EMAIL-O out-interface-list=WAN passthrough=no
add action=mark-connection chain=prerouting comment="QUIC in" connection-mark=no-mark dst-port=80,443 new-connection-mark=CM-QUIC passthrough=yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=CM-QUIC in-interface-list=WAN new-packet-mark=QUIC-I passthrough=no
add action=mark-connection chain=postrouting comment="QUIC out" connection-mark=no-mark dst-port=80,443 new-connection-mark=CM-QUIC passthrough=yes protocol=udp
add action=mark-packet chain=postrouting connection-mark=CM-QUIC new-packet-mark=QUIC-O out-interface-list=WAN passthrough=no
add action=mark-connection chain=prerouting comment="HTTP in" connection-mark=no-mark dst-port=80,443,8080,8443 new-connection-mark= CM-HTTP passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting connection-bytes=1000000-0 connection-mark=CM-HTTP connection-rate=!0-8k new-connection-mark= CM-HTTP-L passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting connection-mark=CM-HTTP-L in-interface-list=WAN new-packet-mark=HTTP-B-I passthrough=no
add action=mark-packet chain=prerouting connection-mark=CM-HTTP in-interface-list=WAN new-packet-mark=HTTP-I passthrough=no
add action=mark-connection chain=postrouting comment="HTTP out" connection-mark=no-mark dst-port=80,443,8080,8443 new-connection-mark= CM-HTTP passthrough=yes protocol=tcp
add action=mark-connection chain=postrouting connection-bytes=1000000-0 connection-mark=CM-HTTP connection-rate=!0-8k new-connection-mark= CM-HTTP-L passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting connection-mark=CM-HTTP-L new-packet-mark=HTTP-B-O out-interface-list=WAN passthrough=no
add action=mark-packet chain=postrouting connection-mark=CM-HTTP new-packet-mark=HTTP-O out-interface-list=WAN passthrough=no
add action=mark-connection chain=prerouting comment="OTHER in" connection-mark=no-mark new-connection-mark=CM-OTHER passthrough=yes
add action=mark-connection chain=prerouting connection-mark=CM-OTHER new-connection-mark=CM-OTHER-L packet-size=!0-400 passthrough=yes
add action=mark-packet chain=prerouting connection-mark=CM-OTHER-L in-interface-list=WAN new-packet-mark=OTHER-B-I passthrough=no
add action=mark-packet chain=prerouting connection-mark=CM-OTHER in-interface-list=WAN new-packet-mark=OTHER-I passthrough=no
add action=mark-connection chain=postrouting comment="OTHER out" connection-mark=no-mark new-connection-mark=CM-OTHER passthrough=yes
add action=mark-connection chain=postrouting connection-mark=CM-OTHER new-connection-mark=CM-OTHER-L packet-size=!0-400 passthrough=yes
add action=mark-packet chain=postrouting connection-mark=CM-OTHER-L new-packet-mark=OTHER-B-O out-interface-list=WAN passthrough=no
add action=mark-packet chain=postrouting connection-mark=CM-OTHER new-packet-mark=OTHER-O out-interface-list=WAN passthrough=no

Queue tree

/queue tree
add bucket-size=0.01 max-limit=210M name=INTERNET parent=global queue=default
add bucket-size=0.01 max-limit=200M name=IN parent=INTERNET queue=default
add bucket-size=0.01 max-limit=10M name=OUT parent=INTERNET queue=default
add name=DNS_IN packet-mark=DNS-I parent=IN priority=1 queue=default
add name=NETWORK_IN packet-mark= FIN-I,SYN-I,RST-I,ACK-I,ICMP-I parent=IN priority=3 queue=default
add name=VOIP_IN packet-mark=VOIP-I parent=IN priority=2 queue=default
add name=OTHER_IN packet-mark=OTHER-I,VPN-I parent=IN priority=4 queue=default
add name=HTTP_IN packet-mark=HTTP-I,EMAIL-I parent=IN priority=5 queue=default
add name=HTTP_BIG_IN packet-mark=HTTP-B-I parent= IN priority=6 queue=default
add name=QUIC_IN packet-mark=QUIC-I parent=IN priority=7 queue=default
add name=OTHER_BIG_IN packet-mark=OTHER-B-I parent=IN queue=default
add name=DNS_OUT packet-mark=DNS-O parent=OUT priority=1 queue=default
add name=NETWORK_OUT packet-mark= FIN-O,SYN-O,RST-O,ACK-O,ICMP-O parent=OUT priority=3 queue=default
add name=VOIP_OUT packet-mark=VOIP-O parent=OUT priority=2 queue=default
add name=OTHER_OUT packet-mark=OTHER-O,VPN-O parent=OUT priority=4 queue=default
add name=HTTP_OUT packet-mark=HTTP-O,EMAIL-O parent=OUT priority=5 queue=default
add name=HTTP_BIG_OUT packet-mark=HTTP-B-O parent=OUT priority=6 queue=default
add name=QUIC_OUT packet-mark=QUIC-O parent=OUT priority=7 queue=default
add name=OTHER_BIG_OUT packet-mark=OTHER-B-O parent=OUT queue=default

Make sure that the fasttrack connection is turned off

bharrisau · Fri Jun 26, 2020 3:37 pm

TCP uses a three-way handshake to establish a reliable connection, why prioritize the ack traffic only??

We aren't trying to prioritize the handshake. The handshake is only 3 small packets to establish the TCP connection.

Once you establish the connection and then start downloading at full speed, your computer need to tell the remote end that is successfully received the data. It does this by sending ACK packets. By prioritizing these ACK packets we can ensure that uploads on a connection do not impact downloads. If we didn't prioritize the ACK then the downloads would be limited because the remote computer would be waiting on the ACK until is sends more data.

Hope that clarifies somewhat.

erhtun · Sat Jun 27, 2020 12:21 am

TCP uses a three-way handshake to establish a reliable connection, why prioritize the ack traffic only??
We aren't trying to prioritize the handshake. The handshake is only 3 small packets to establish the TCP connection.

Once you establish the connection and then start downloading at full speed, your computer need to tell the remote end that is successfully received the data. It does this by sending ACK packets. By prioritizing these ACK packets we can ensure that uploads on a connection do not impact downloads. If we didn't prioritize the ACK then the downloads would be limited because the remote computer would be waiting on the ACK until is sends more data.

Hope that clarifies somewhat.

I have 100M Down / 5M Up cable connection. I don't have voip calls. I rarely upload files. So under this circumstances should I prioritize ACK packets?

Another question I want to ask is about "parent" settings in queue tree. When to use parent=global or parent=interface(ether1, bridge etc.) for optimum performance? Why?

jaguar666 · Sat Jun 27, 2020 7:09 am

Why is it necessary to deactivate the filter fasttrack connection?

The mangle rules and queue tree will not act accordingly if you enable fasttrack connections in /ip firewall filter (default config).

DarkNate · Sun Jun 28, 2020 4:00 pm

# Identify VoIP
add chain=prerouting  action=mark-connection new-connection-mark=VOIP port=5060-5062,10000-20000 protocol=udp passthrough=yes comment="VOIP"
add chain=prerouting  action=mark-packet     connection-mark=VOIP new-packet-mark=VOIP passthrough=no

Does this account for all VoIP like VoWiFi, Google Duo, FaceTime etc?

Is this correct for ICMPv6 marking? Of course, I used the same marking name "ICMP" from IPv4 rules.

# Identify PING on the network or coming from the Router itself
add chain=prerouting  action=mark-connection connection-state=new new-connection-mark=ICMP protocol=icmpv6 passthrough=yes comment="ICMP"
add chain=prerouting  action=mark-packet     connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add chain=postrouting action=mark-connection connection-state=new new-connection-mark=ICMP protocol=icmpv6 passthrough=yes
add chain=postrouting action=mark-packet     connection-mark=ICMP new-packet-mark=ICMP passthrough=no

dikrek · Thu Jul 02, 2020 9:52 pm

How would one use this approach (say, MikroCake) with LTE? The speeds are variable, but I'd argue it would still benefit from ACK prioritization etc.

Anyone care to show a sample?

digin4 · Sun Jul 12, 2020 5:58 pm

Is it possible to prioritize DoH (DNS over HTTPS) traffic such as 1.1.1.1?
I created a src-list with all the IP address of DoH providers, then added this mangle rule:

add action=mark-packet chain=forward new-packet-mark=DoH passthrough=no src-address-list=DoH comment="DoH"

But the packets aren't getting marked at all, what am I doing wrong?

sindy · Mon Jul 13, 2020 12:13 am

Do you really want to mark the responses from the DoH servers? If you actually want to mark the queries, the rule must match on dst-address-list, not the src one. Regardless that - if a rule's statistics doesn't count at all although you know that matching packets do exist, that rule is likely shadowed by some other one before (above) it, which matches the same packets and has passthrough=no (implicitly or explicitly).

DarkNate · Thu Jul 16, 2020 2:19 am

Do you really want to mark the responses from the DoH servers? If you actually want to mark the queries, the rule must match on dst-address-list, not the src one. Regardless that - if a rule's statistics doesn't count at all although you know that matching packets do exist, that rule is likely shadowed by some other one before (above) it, which matches the same packets and has passthrough=no (implicitly or explicitly).

I tried this, but it is not working.

add action=mark-connection chain=prerouting comment=HTTP connection-mark=no-mark connection-state=new dst-address-list=!RFC1918 new-connection-mark=HTTP passthrough=yes port=80,443 protocol=\
    tcp
add action=mark-connection chain=prerouting connection-bytes=5000000-0 connection-mark=HTTP connection-rate=3M-100M dst-address-list=!RFC1918 new-connection-mark=HTTP_BIG passthrough=yes \
    protocol=tcp
add action=mark-packet chain=forward comment=DoH connection-mark=HTTP dst-address-list=DoH log=yes log-prefix=DoHLogging new-packet-mark=DNS passthrough=no tcp-flags=""
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no

digin4 · Thu Jul 16, 2020 10:22 am

This is what I'm using and it's working:

# Identify DoH
add chain=prerouting  action=mark-connection connection-state=new new-connection-mark=DoH passthrough=yes dst-address-list=DoH comment="DoH"
add chain=prerouting  action=mark-packet     connection-mark=DoH  new-packet-mark=DoH passthrough=no
add chain=postrouting action=mark-connection connection-state=new new-connection-mark=DoH passthrough=yes dst-address-list=DoH
add chain=postrouting action=mark-packet     connection-mark=DoH  new-packet-mark=DoH passthrough=no

Do you really want to mark the responses from the DoH servers? If you actually want to mark the queries, the rule must match on dst-address-list, not the src one. Regardless that - if a rule's statistics doesn't count at all although you know that matching packets do exist, that rule is likely shadowed by some other one before (above) it, which matches the same packets and has passthrough=no (implicitly or explicitly).
I tried this, but it is not working.
Code: Select all
add action=mark-connection chain=prerouting comment=HTTP connection-mark=no-mark connection-state=new dst-address-list=!RFC1918 new-connection-mark=HTTP passthrough=yes port=80,443 protocol=\
    tcp
add action=mark-connection chain=prerouting connection-bytes=5000000-0 connection-mark=HTTP connection-rate=3M-100M dst-address-list=!RFC1918 new-connection-mark=HTTP_BIG passthrough=yes \
    protocol=tcp
add action=mark-packet chain=forward comment=DoH connection-mark=HTTP dst-address-list=DoH log=yes log-prefix=DoHLogging new-packet-mark=DNS passthrough=no tcp-flags=""
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no

DarkNate · Thu Jul 16, 2020 2:11 pm

This is what I'm using and it's working:

# Identify DoH
add chain=prerouting  action=mark-connection connection-state=new new-connection-mark=DoH passthrough=yes dst-address-list=DoH comment="DoH"
add chain=prerouting  action=mark-packet     connection-mark=DoH  new-packet-mark=DoH passthrough=no
add chain=postrouting action=mark-connection connection-state=new new-connection-mark=DoH passthrough=yes dst-address-list=DoH
add chain=postrouting action=mark-packet     connection-mark=DoH  new-packet-mark=DoH passthrough=no

Do you really want to mark the responses from the DoH servers? If you actually want to mark the queries, the rule must match on dst-address-list, not the src one. Regardless that - if a rule's statistics doesn't count at all although you know that matching packets do exist, that rule is likely shadowed by some other one before (above) it, which matches the same packets and has passthrough=no (implicitly or explicitly).
I tried this, but it is not working.
Code: Select all
add action=mark-connection chain=prerouting comment=HTTP connection-mark=no-mark connection-state=new dst-address-list=!RFC1918 new-connection-mark=HTTP passthrough=yes port=80,443 protocol=\
    tcp
add action=mark-connection chain=prerouting connection-bytes=5000000-0 connection-mark=HTTP connection-rate=3M-100M dst-address-list=!RFC1918 new-connection-mark=HTTP_BIG passthrough=yes \
    protocol=tcp
add action=mark-packet chain=forward comment=DoH connection-mark=HTTP dst-address-list=DoH log=yes log-prefix=DoHLogging new-packet-mark=DNS passthrough=no tcp-flags=""
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no

Thanks that seems to be working!

I've added TCP protocol and port 53, 443 and 853 to the mangle rule to ensure it only deals with specific connectivity to DoH address list. A more cleaner approach.

Why port 53, 853? It is to help catch DoH traffic from my client devices who may run independently for DoH queries.
Source: https://en.wikipedia.org/wiki/DNS_over_ ... _scenarios

DarkNate · Fri Jul 17, 2020 4:00 am

@pcunite

Does "connection-state" matter? If so why is VoIP lacking connection state?

Should we only mark the "new" connections? What about established and related?

When I look through connection tracking I can see a lot of what I'd assume to be "related" connections that are not being marked by a mangle.

smileymattj · Fri Jul 17, 2020 6:10 am

First off, nice work pcunite. That's a lot of effort you put in and very well documented.

Here is something I was thinking of. If you trust the DSCP markings applied by the devices on your network. Typically you can unless your devices are compromised. You could first check to see if the device is already prioritizing it's packets with a DSCP and use that first.

Also for the OTHER, I set it to no-mark and took out the mangle for those. Should make it slightly more efficient.

/ip setting set allow-fast-path=no


# Place at very top

/ip firewall mangle

add action=mark-connection chain=prerouting dscp=0 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=prerouting dscp=1 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=prerouting dscp=2 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=prerouting dscp=3 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=prerouting dscp=4 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=prerouting dscp=5 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=prerouting dscp=6 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=prerouting dscp=7 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=prerouting dscp=8 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=prerouting dscp=9 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=prerouting dscp=10 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=prerouting dscp=11 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=prerouting dscp=12 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=prerouting dscp=13 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=prerouting dscp=14 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=prerouting dscp=15 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=prerouting dscp=16 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=prerouting dscp=17 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=prerouting dscp=18 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=prerouting dscp=19 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=prerouting dscp=20 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=prerouting dscp=21 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=prerouting dscp=22 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=prerouting dscp=23 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=prerouting dscp=24 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=prerouting dscp=25 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=prerouting dscp=26 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=prerouting dscp=27 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=prerouting dscp=28 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=prerouting dscp=29 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=prerouting dscp=30 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=prerouting dscp=31 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=prerouting dscp=32 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=prerouting dscp=33 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=prerouting dscp=34 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=prerouting dscp=35 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=prerouting dscp=36 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=prerouting dscp=37 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=prerouting dscp=38 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=prerouting dscp=39 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=prerouting dscp=40 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=prerouting dscp=41 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=prerouting dscp=42 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=prerouting dscp=43 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=prerouting dscp=44 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=prerouting dscp=45 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=prerouting dscp=46 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=prerouting dscp=47 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=prerouting dscp=48 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=prerouting dscp=49 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=prerouting dscp=50 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=prerouting dscp=51 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=prerouting dscp=52 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=prerouting dscp=53 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=prerouting dscp=54 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=prerouting dscp=55 comment=TOS_6 new-connection-mark=QUIC
# add action=mark-connection chain=prerouting dscp=56 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=prerouting dscp=57 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=prerouting dscp=58 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=prerouting dscp=59 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=prerouting dscp=60 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=prerouting dscp=61 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=prerouting dscp=62 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=prerouting dscp=63 comment=TOS_7 new-connection-mark=OTHER

add action=mark-connection chain=postrouting dscp=0 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=postrouting dscp=1 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=postrouting dscp=2 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=postrouting dscp=3 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=postrouting dscp=4 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=postrouting dscp=5 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=postrouting dscp=6 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=postrouting dscp=7 comment=TOS_0 new-connection-mark=VOIP
add action=mark-connection chain=postrouting dscp=8 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=postrouting dscp=9 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=postrouting dscp=10 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=postrouting dscp=11 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=postrouting dscp=12 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=postrouting dscp=13 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=postrouting dscp=14 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=postrouting dscp=15 comment=TOS_1 new-connection-mark=DNS
add action=mark-connection chain=postrouting dscp=16 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=postrouting dscp=17 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=postrouting dscp=18 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=postrouting dscp=19 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=postrouting dscp=20 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=postrouting dscp=21 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=postrouting dscp=22 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=postrouting dscp=23 comment=TOS_2 new-connection-mark=ACK
add action=mark-connection chain=postrouting dscp=24 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=postrouting dscp=25 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=postrouting dscp=26 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=postrouting dscp=27 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=postrouting dscp=28 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=postrouting dscp=29 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=postrouting dscp=30 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=postrouting dscp=31 comment=TOS_3 new-connection-mark=ICMP
add action=mark-connection chain=postrouting dscp=32 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=postrouting dscp=33 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=postrouting dscp=34 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=postrouting dscp=35 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=postrouting dscp=36 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=postrouting dscp=37 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=postrouting dscp=38 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=postrouting dscp=39 comment=TOS_4 new-connection-mark=HTTP
add action=mark-connection chain=postrouting dscp=40 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=postrouting dscp=41 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=postrouting dscp=42 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=postrouting dscp=43 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=postrouting dscp=44 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=postrouting dscp=45 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=postrouting dscp=46 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=postrouting dscp=47 comment=TOS_5 new-connection-mark=HTTP_BIG
add action=mark-connection chain=postrouting dscp=48 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=postrouting dscp=49 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=postrouting dscp=50 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=postrouting dscp=51 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=postrouting dscp=52 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=postrouting dscp=53 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=postrouting dscp=54 comment=TOS_6 new-connection-mark=QUIC
add action=mark-connection chain=postrouting dscp=55 comment=TOS_6 new-connection-mark=QUIC
# add action=mark-connection chain=postrouting dscp=56 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=postrouting dscp=57 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=postrouting dscp=58 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=postrouting dscp=59 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=postrouting dscp=60 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=postrouting dscp=61 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=postrouting dscp=62 comment=TOS_7 new-connection-mark=OTHER
# add action=mark-connection chain=postrouting dscp=63 comment=TOS_7 new-connection-mark=OTHER


# Insert other/custom capture rules here




# pcunite's mangle

# Identify DNS on the network or coming from the Router itself
add chain=prerouting  action=mark-connection connection-state=new new-connection-mark=DNS port=53 protocol=udp passthrough=yes comment="DNS"
add chain=postrouting action=mark-connection connection-state=new new-connection-mark=DNS port=53 protocol=udp passthrough=yes comment="DNS"

# Identify VoIP
add chain=prerouting  action=mark-connection new-connection-mark=VOIP port=5060-5062,10000-20000 protocol=udp passthrough=yes comment="VOIP"

# Identify HTTP/3 and Google's QUIC
add chain=prerouting  action=mark-connection connection-state=new new-connection-mark=QUIC port=80,443 protocol=udp passthrough=yes comment="QUIC"

# Identify UPD. Useful for further analysis. Should it be considered high priority or put in the catchall? You decide.
add chain=prerouting  action=mark-connection connection-state=new new-connection-mark=UDP protocol=udp passthrough=yes comment="UDP"

# Identify PING on the network or coming from the Router itself
add chain=prerouting  action=mark-connection connection-state=new new-connection-mark=ICMP protocol=icmp passthrough=yes comment="ICMP"
add chain=postrouting action=mark-connection connection-state=new new-connection-mark=ICMP protocol=icmp passthrough=yes comment="ICMP"

# Identify Acknowledgment packets
add chain=postrouting action=mark-packet new-packet-mark=ACK packet-size=0-123 protocol=tcp tcp-flags=ack passthrough=no comment="ACK"
add chain=prerouting  action=mark-packet new-packet-mark=ACK packet-size=0-123 protocol=tcp tcp-flags=ack passthrough=no comment="ACK"

# Identify HTTP traffic but move it to a Streaming mark if necessary.
add chain=prerouting  action=mark-connection connection-mark=no-mark connection-state=new new-connection-mark=HTTP port=80,443 protocol=tcp passthrough=yes comment="HTTP"
add chain=prerouting  action=mark-connection connection-bytes=5M-0 connection-mark=HTTP connection-rate=2M-100M new-connection-mark=HTTP_BIG protocol=tcp passthrough=yes comment="HTTP_BIG"


# Mark Packets
add action=mark-packet chain=prerouting connection-mark=VOIP new-packet-mark=VOIP passthrough=no comment=VOIP
add action=mark-packet chain=postrouting connection-mark=VOIP new-packet-mark=VOIP passthrough=no comment=VOIP

add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no comment=DNS
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no comment=DNS

add action=mark-packet chain=prerouting connection-mark=ACK new-packet-mark=ACK passthrough=no comment=ACK
add action=mark-packet chain=postrouting connection-mark=ACK new-packet-mark=ACK passthrough=no comment=ACK

add action=mark-packet chain=prerouting connection-mark=UDP new-packet-mark=UDP passthrough=no comment=UDP
add action=mark-packet chain=postrouting connection-mark=UDP new-packet-mark=UDP passthrough=no comment=UDP

add action=mark-packet chain=prerouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no comment=ICMP
add action=mark-packet chain=postrouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no comment=ICMP

add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no comment=HTTP
add action=mark-packet chain=postrouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no comment=HTTP

add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no comment=HTTP_BIG
add action=mark-packet chain=postrouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no comment=HTTP_BIG

add action=mark-packet chain=prerouting connection-mark=QUIC new-packet-mark=QUIC passthrough=no comment=QUIC
add action=mark-packet chain=postrouting connection-mark=QUIC new-packet-mark=QUIC passthrough=no comment=QUIC


/queue tree

# DOWN
add name=DOWN max-limit=900M parent=LAN bucket-size=0.01 queue=default

add name="1. VOIP"     packet-mark=VOIP     parent=DOWN priority=1 queue=default
add name="2. DNS"      packet-mark=DNS      parent=DOWN priority=2 queue=default
add name="3. ACK"      packet-mark=ACK      parent=DOWN priority=3 queue=default
add name="4. UDP"      packet-mark=UDP      parent=DOWN priority=3 queue=default
add name="5. ICMP"     packet-mark=ICMP     parent=DOWN priority=4 queue=default
add name="6. HTTP"     packet-mark=HTTP     parent=DOWN priority=5 queue=default
add name="7. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=6 queue=default
add name="8. QUIC"     packet-mark=QUIC     parent=DOWN priority=7 queue=default
add name="9. OTHER"    packet-mark=no-mark  parent=DOWN priority=8 queue=default


# UP
add name=UP max-limit=900M parent=WAN bucket-size=0.01 queue=default

add name="1. VOIP_"     packet-mark=VOIP     parent=UP priority=1 queue=default
add name="2. DNS_"      packet-mark=DNS      parent=UP priority=2 queue=default
add name="3. ACK_"      packet-mark=ACK      parent=UP priority=3 queue=default
add name="4. UDP_"      packet-mark=UDP      parent=UP priority=3 queue=default
add name="5. ICMP_"     packet-mark=ICMP     parent=UP priority=4 queue=default
add name="6. HTTP_"     packet-mark=HTTP     parent=UP priority=5 queue=default
add name="7. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=6 queue=default
add name="8. QUIC_"     packet-mark=QUIC     parent=UP priority=7 queue=default
add name="9. OTHER_"    packet-mark=no-mark  parent=UP priority=8 queue=default

tawh · Sun Jul 19, 2020 12:49 pm

Backgound:
I have built a MPLS network over full mesh EOIP tunnels on all edge routers. I also configured VPLS tunnels in the MPLS network. all network function are worked as expected.

Problem:
There are both MPLS and non-MPLS traffics running over the EOIP tunnels, sometimes the non-MPLS traffic are so huge that some of the MPLS packets get dropped.

I want the priority all VPLS traffic (or generalized as all MPLS packets) to take higher precedence over non-MPLS traffics over the EOIP tunnels. However, I cannot figure out a mangle rule to identify the MPLS/VPLS packets.

Question:
Is there anything I missed or is there any alternative way to achieve this goal?

sindy · Sun Jul 19, 2020 4:19 pm

MPLS packets are invisible to the ip firewall rules (unless you set use-ip-firewall=yes in /interface bridge settings, which affects a lot of things so don't do that unless you absolutely have to). So this topic suggests two approaches - to assign some packet-mark to all IP traffic, and let the queue for MPLS traffic be chosen by an absence of any packet-mark, or to connect the EoIP interfaces to dedicated bridges, and use /interface bridge filter rules rather than /ip firewall mangle ones to assign packet marks to the MPLS packets.

tawh · Sun Jul 19, 2020 10:48 pm

MPLS packets are invisible to the ip firewall rules (unless you set use-ip-firewall=yes in /interface bridge settings, which affects a lot of things so don't do that unless you absolutely have to). So this topic suggests two approaches - to assign some packet-mark to all IP traffic, and let the queue for MPLS traffic be chosen by an absence of any packet-mark, or to connect the EoIP interfaces to dedicated bridges, and use /interface bridge filter rules rather than /ip firewall mangle ones to assign packet marks to the MPLS packets.

Thanks for your reply.
I tried the solution proposed by your reference link, I found I cannot work out the expected result. I suspect the main problem is that in my case the ingress MPLS router and the edge router are at the same router, while there were two separate routers in the reference link.
I further investigated the packet flow diagram from Mikrotik, In my case, the packets reaches my router with VLAN tag, then it bridges to my VPLS interface. since my VPLS circuit is further encapsulated in EOIP, as a result, the following flow shall be seen (This diagram is derived from 4.3 and 4.4 of https://wiki.mikrotik.com/wiki/Manual:Packet_Flow):

I believe the packet marking at the early stage (step 8 to step 9) of this flow didn't inherit to the VPLS (step 13) and the final EOIP packets (step 21).

Observation:
(i) I created bridge rule to add packet marks "abc" to all VLAN1 packets
(ii) I created a queue tree which selects packet mark "abc" and "eoip" interface as parent (step 18). NO packets were queued
(iii) I changed the parent interface to the "vpls" interface (step 10), packets were queued.

Can anyone review whether the packet flow is correct and any solution to this scenario? any help is greatly appreciated.

sindy · Mon Jul 20, 2020 1:10 pm

I found I cannot work out the expected result

Can you please create a dedicated topic and place a link to it here? I think your case is quite specific and should be treated separately.

ivicask · Tue Jul 21, 2020 12:58 pm

Are you sure about marking DNS / ICMP or ACK connections than their packet marks, because what im seeing that when i run speedtest some or sometimes entire bandwidth goes thru this QUEUE(DNS/ICMP/ACK) and as they have top prio actually choke my net.

I changed it to only mark packets directly without prior connection marking and now works fine, only this packets pass thru.(rest of mangle rules still uses connection +packet marking)

I tried absolutely destroying my net with few torrents, http downloads, windows updates, some youtube, radio stream(doesnt break), dropbox upload, and ping is still stable without drops, and web pages opens super fast.

This is how my QOS looks now that im using for some time now.

DarkNate · Wed Jul 22, 2020 6:33 pm

Are you sure about marking DNS / ICMP or ACK connections than their packet marks, because what im seeing that when i run speedtest some or sometimes entire bandwidth goes thru this QUEUE(DNS/ICMP/ACK) and as they have top prio actually choke my net.

I changed it to only mark packets directly without prior connection marking and now works fine, only this packets pass thru.(rest of mangle rules still uses connection +packet marking)

I tried absolutely destroying my net with few torrents, http downloads, windows updates, some youtube, radio stream(doesnt break), dropbox upload, and ping is still stable without drops, and web pages opens super fast.

This is how my QOS looks now that im using for some time now.

The reason why he marked connections instead of packets is to reduce load on the CPU.

I've encountered these anamolies you described as well. I've made a lot of modifications to the mangle rules since though.

One thing I'm not sure about is the priority of the mangle rules, like should ACK be marked before UDP/ICMP/QUIC or what? Should mark-packet http_big come first before mark-parket HTTP?
Maybe you know something about the mangle rules priority.

I also added established/related connection state when I discovered some connections would escape the mangel rules via connections tab. DoH traffic is also marked via address-list, DNS over UDP 443 is marked especially for Google Chrome as it uses that port for DNS queries sometimes, something I caught from the connections tab.

Here's mine:

add action=mark-connection chain=prerouting comment=DNS connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=DNS passthrough=yes port=53 protocol=udp \
    tcp-flags=""
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=DNS passthrough=yes port=53 protocol=udp \
    src-address-list=""
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=DoH connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=53,443,853 protocol=tcp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=53,443,853 protocol=tcp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment="DoH (UDP 443)" connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=443 protocol=udp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=443 protocol=udp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=VOIP connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=VOIP passthrough=yes port=\
    5060-5062,10000-15251,15253-20000,3478-3497,16384-16387,16393-16402,19302-19309 protocol=udp
add action=mark-packet chain=prerouting connection-mark=VOIP new-packet-mark=VOIP passthrough=no tcp-flags=""
add action=mark-connection chain=prerouting comment=QUIC connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=QUIC passthrough=yes port=80,443 \
    protocol=udp
add action=mark-packet chain=prerouting connection-mark=QUIC new-packet-mark=QUIC passthrough=no
add action=mark-connection chain=prerouting comment=UDP connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=UDP passthrough=yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=UDP new-packet-mark=UDP passthrough=no
add action=mark-connection chain=prerouting comment=ICMP connection-state=new dst-address-list=!not_in_internet new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=prerouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-connection chain=postrouting connection-state=new dst-address-list=!not_in_internet new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=postrouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-packet chain=postrouting comment=ACK new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-connection chain=prerouting comment=HTTP connection-mark=no-mark connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=HTTP \
    passthrough=yes port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-bytes=5000000-0 connection-mark=HTTP connection-rate=2M-300M connection-state=established,related,new dst-address-list=!not_in_internet \
    new-connection-mark=HTTP_BIG passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no
add action=mark-connection chain=prerouting comment=OTHER connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=POP3 passthrough=yes port=995,465,587 \
    protocol=tcp
add action=mark-packet chain=prerouting connection-mark=POP3 new-packet-mark=OTHER passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=OTHER packet-mark=no-mark \
    passthrough=yes
add action=mark-packet chain=prerouting connection-mark=OTHER new-packet-mark=OTHER packet-mark=no-mark passthrough=no

I've taken advantage of the 5 second bandwidth burst from my ISP that maxed out at 300M that I've seen so far. So I added burst for everything. I capped individual traffic types to 130M to prevent bufferbloat. That extra 300M for parent queue gives it that extra headroom.

The queue tree:

/queue tree
add bucket-size=0.01 burst-limit=300M burst-threshold=300M burst-time=1s max-limit=99M name=DOWN parent=bridge queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="1. DNS" packet-mark=DNS parent=DOWN priority=1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=2M max-limit=99M name="2. VOIP" packet-mark=VOIP parent=DOWN priority=2 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="3. ACK" packet-mark=ACK parent=DOWN priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=18M max-limit=99M name="4. UDP" packet-mark=UDP parent=DOWN priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="5. ICMP" packet-mark=ICMP parent=DOWN priority=4 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="6. HTTP" packet-mark=HTTP parent=DOWN priority=5 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="7. QUIC" packet-mark=QUIC parent=DOWN priority=6 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="8. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=7 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s name="9. OTHER" packet-mark=OTHER parent=DOWN queue=redCustom
add bucket-size=0.01 burst-limit=300M burst-threshold=300M burst-time=1s max-limit=99M name=UP parent=pppoe-out1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="1. DNS_" packet-mark=DNS parent=UP priority=1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=2M max-limit=99M name="2. VOIP_" packet-mark=VOIP parent=UP priority=2 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="3. ACK_" packet-mark=ACK parent=UP priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=18M max-limit=99M name="4. UDP_" packet-mark=UDP parent=UP priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="5. ICMP_" packet-mark=ICMP parent=UP priority=4 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="6. HTTP_" packet-mark=HTTP parent=UP priority=5 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="7. QUIC_" packet-mark=QUIC parent=UP priority=6 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="8. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=7 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s name="9. OTHER_" packet-mark=OTHER parent=UP queue=redCustom

ivicask · Wed Jul 22, 2020 7:18 pm

Are you sure about marking DNS / ICMP or ACK connections than their packet marks, because what im seeing that when i run speedtest some or sometimes entire bandwidth goes thru this QUEUE(DNS/ICMP/ACK) and as they have top prio actually choke my net.

I changed it to only mark packets directly without prior connection marking and now works fine, only this packets pass thru.(rest of mangle rules still uses connection +packet marking)

I tried absolutely destroying my net with few torrents, http downloads, windows updates, some youtube, radio stream(doesnt break), dropbox upload, and ping is still stable without drops, and web pages opens super fast.

This is how my QOS looks now that im using for some time now.

The reason why he marked connections instead of packets is to reduce load on the CPU.

I've encountered these anamolies you described as well. I've made a lot of modifications to the mangle rules since though.

One thing I'm not sure about is the priority of the mangle rules, like should ACK be marked before UDP/ICMP/QUIC or what? Should mark-packet http_big come first before mark-parket HTTP?
Maybe you know something about the mangle rules priority.

I also added established/related connection state when I discovered some connections would escape the mangel rules via connections tab. DoH traffic is also marked via address-list, DNS over UDP 443 is marked especially for Google Chrome as it uses that port for DNS queries sometimes, something I caught from the connections tab.

Here's mine:

add action=mark-connection chain=prerouting comment=DNS connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=DNS passthrough=yes port=53 protocol=udp \
    tcp-flags=""
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=DNS passthrough=yes port=53 protocol=udp \
    src-address-list=""
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=DoH connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=53,443,853 protocol=tcp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=53,443,853 protocol=tcp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment="DoH (UDP 443)" connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=443 protocol=udp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=443 protocol=udp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=VOIP connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=VOIP passthrough=yes port=\
    5060-5062,10000-15251,15253-20000,3478-3497,16384-16387,16393-16402,19302-19309 protocol=udp
add action=mark-packet chain=prerouting connection-mark=VOIP new-packet-mark=VOIP passthrough=no tcp-flags=""
add action=mark-connection chain=prerouting comment=QUIC connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=QUIC passthrough=yes port=80,443 \
    protocol=udp
add action=mark-packet chain=prerouting connection-mark=QUIC new-packet-mark=QUIC passthrough=no
add action=mark-connection chain=prerouting comment=UDP connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=UDP passthrough=yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=UDP new-packet-mark=UDP passthrough=no
add action=mark-connection chain=prerouting comment=ICMP connection-state=new dst-address-list=!not_in_internet new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=prerouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-connection chain=postrouting connection-state=new dst-address-list=!not_in_internet new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=postrouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-packet chain=postrouting comment=ACK new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-connection chain=prerouting comment=HTTP connection-mark=no-mark connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=HTTP \
    passthrough=yes port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-bytes=5000000-0 connection-mark=HTTP connection-rate=2M-300M connection-state=established,related,new dst-address-list=!not_in_internet \
    new-connection-mark=HTTP_BIG passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no
add action=mark-connection chain=prerouting comment=OTHER connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=POP3 passthrough=yes port=995,465,587 \
    protocol=tcp
add action=mark-packet chain=prerouting connection-mark=POP3 new-packet-mark=OTHER passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=OTHER packet-mark=no-mark \
    passthrough=yes
add action=mark-packet chain=prerouting connection-mark=OTHER new-packet-mark=OTHER packet-mark=no-mark passthrough=no

I've taken advantage of the 5 second bandwidth burst from my ISP that maxed out at 300M that I've seen so far. So I added burst for everything. I capped individual traffic types to 130M to prevent bufferbloat. That extra 300M for parent queue gives it that extra headroom.

The queue tree:

/queue tree
add bucket-size=0.01 burst-limit=300M burst-threshold=300M burst-time=1s max-limit=99M name=DOWN parent=bridge queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="1. DNS" packet-mark=DNS parent=DOWN priority=1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=2M max-limit=99M name="2. VOIP" packet-mark=VOIP parent=DOWN priority=2 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="3. ACK" packet-mark=ACK parent=DOWN priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=18M max-limit=99M name="4. UDP" packet-mark=UDP parent=DOWN priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="5. ICMP" packet-mark=ICMP parent=DOWN priority=4 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="6. HTTP" packet-mark=HTTP parent=DOWN priority=5 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="7. QUIC" packet-mark=QUIC parent=DOWN priority=6 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="8. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=7 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s name="9. OTHER" packet-mark=OTHER parent=DOWN queue=redCustom
add bucket-size=0.01 burst-limit=300M burst-threshold=300M burst-time=1s max-limit=99M name=UP parent=pppoe-out1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="1. DNS_" packet-mark=DNS parent=UP priority=1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=2M max-limit=99M name="2. VOIP_" packet-mark=VOIP parent=UP priority=2 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="3. ACK_" packet-mark=ACK parent=UP priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=18M max-limit=99M name="4. UDP_" packet-mark=UDP parent=UP priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="5. ICMP_" packet-mark=ICMP parent=UP priority=4 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="6. HTTP_" packet-mark=HTTP parent=UP priority=5 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="7. QUIC_" packet-mark=QUIC parent=UP priority=6 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="8. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=7 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s name="9. OTHER_" packet-mark=OTHER parent=UP queue=redCustom

I know reason is CPU load, but as i said this causes the entire connection (speedtest bandwidth for example) getting captured by ACK mangle rule and its queue tree and never reaching other rules which where supposed to actually capture it.

Im constantly changing and adapting my rules as i notice problems so not completely sure, but I prio ICMP and DNS above all, than ACK packets than rest..

About http packets, i mark big ones first than small ones, and have separate rule for SSL (443) and HTTP(80+8080) traffic (both small + big) while giving one step bigger priority to SSL traffic over HTTP in queue.

And yeah, if your ISP doesnt support big bursting same as i did in my above screenshot, you need to sacrifice some bandwidth, best is to limit high traffic consumers in one group and drop their total speed around 10-15% below total internet speed so they can never saturate it, while leaving other full bandwidth to other "bursty" queues so they have time to drop this big traffic consumer queues via their "limit-at" limits down if you get what i mean.

BTW there is one more trick im testing for some times, what do you think of some basic QOS in NAT it self, i use this on locations where there is very variable speed LTE links and i cant make proper QUEUE tree.

/ip firewall nat
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN protocol=icmp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN port=53 protocol=udp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN port=443 protocol=tcp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN port=80 protocol=tcp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN protocol=tcp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN protocol=udp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN

As nat is processed in order, this gives sort of packet priority which packets will be nated first in chain.
For example with this rules u wont notice such big slowdowns if someone is downloading torrents, or some other bandwidth hungry protocols, pages will still open snappy without any other mangle/queue rules.

Opinions?

DarkNate · Wed Jul 22, 2020 7:35 pm

Are you sure about marking DNS / ICMP or ACK connections than their packet marks, because what im seeing that when i run speedtest some or sometimes entire bandwidth goes thru this QUEUE(DNS/ICMP/ACK) and as they have top prio actually choke my net.

I changed it to only mark packets directly without prior connection marking and now works fine, only this packets pass thru.(rest of mangle rules still uses connection +packet marking)

I tried absolutely destroying my net with few torrents, http downloads, windows updates, some youtube, radio stream(doesnt break), dropbox upload, and ping is still stable without drops, and web pages opens super fast.

This is how my QOS looks now that im using for some time now.

The reason why he marked connections instead of packets is to reduce load on the CPU.

I've encountered these anamolies you described as well. I've made a lot of modifications to the mangle rules since though.

One thing I'm not sure about is the priority of the mangle rules, like should ACK be marked before UDP/ICMP/QUIC or what? Should mark-packet http_big come first before mark-parket HTTP?
Maybe you know something about the mangle rules priority.

I also added established/related connection state when I discovered some connections would escape the mangel rules via connections tab. DoH traffic is also marked via address-list, DNS over UDP 443 is marked especially for Google Chrome as it uses that port for DNS queries sometimes, something I caught from the connections tab.

Here's mine:

add action=mark-connection chain=prerouting comment=DNS connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=DNS passthrough=yes port=53 protocol=udp \
    tcp-flags=""
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=DNS passthrough=yes port=53 protocol=udp \
    src-address-list=""
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=DoH connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=53,443,853 protocol=tcp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=53,443,853 protocol=tcp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment="DoH (UDP 443)" connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=443 protocol=udp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=443 protocol=udp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=VOIP connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=VOIP passthrough=yes port=\
    5060-5062,10000-15251,15253-20000,3478-3497,16384-16387,16393-16402,19302-19309 protocol=udp
add action=mark-packet chain=prerouting connection-mark=VOIP new-packet-mark=VOIP passthrough=no tcp-flags=""
add action=mark-connection chain=prerouting comment=QUIC connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=QUIC passthrough=yes port=80,443 \
    protocol=udp
add action=mark-packet chain=prerouting connection-mark=QUIC new-packet-mark=QUIC passthrough=no
add action=mark-connection chain=prerouting comment=UDP connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=UDP passthrough=yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=UDP new-packet-mark=UDP passthrough=no
add action=mark-connection chain=prerouting comment=ICMP connection-state=new dst-address-list=!not_in_internet new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=prerouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-connection chain=postrouting connection-state=new dst-address-list=!not_in_internet new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=postrouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-packet chain=postrouting comment=ACK new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-connection chain=prerouting comment=HTTP connection-mark=no-mark connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=HTTP \
    passthrough=yes port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-bytes=5000000-0 connection-mark=HTTP connection-rate=2M-300M connection-state=established,related,new dst-address-list=!not_in_internet \
    new-connection-mark=HTTP_BIG passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no
add action=mark-connection chain=prerouting comment=OTHER connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=POP3 passthrough=yes port=995,465,587 \
    protocol=tcp
add action=mark-packet chain=prerouting connection-mark=POP3 new-packet-mark=OTHER passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=OTHER packet-mark=no-mark \
    passthrough=yes
add action=mark-packet chain=prerouting connection-mark=OTHER new-packet-mark=OTHER packet-mark=no-mark passthrough=no

I've taken advantage of the 5 second bandwidth burst from my ISP that maxed out at 300M that I've seen so far. So I added burst for everything. I capped individual traffic types to 130M to prevent bufferbloat. That extra 300M for parent queue gives it that extra headroom.

The queue tree:

/queue tree
add bucket-size=0.01 burst-limit=300M burst-threshold=300M burst-time=1s max-limit=99M name=DOWN parent=bridge queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="1. DNS" packet-mark=DNS parent=DOWN priority=1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=2M max-limit=99M name="2. VOIP" packet-mark=VOIP parent=DOWN priority=2 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="3. ACK" packet-mark=ACK parent=DOWN priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=18M max-limit=99M name="4. UDP" packet-mark=UDP parent=DOWN priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="5. ICMP" packet-mark=ICMP parent=DOWN priority=4 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="6. HTTP" packet-mark=HTTP parent=DOWN priority=5 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="7. QUIC" packet-mark=QUIC parent=DOWN priority=6 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="8. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=7 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s name="9. OTHER" packet-mark=OTHER parent=DOWN queue=redCustom
add bucket-size=0.01 burst-limit=300M burst-threshold=300M burst-time=1s max-limit=99M name=UP parent=pppoe-out1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="1. DNS_" packet-mark=DNS parent=UP priority=1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=2M max-limit=99M name="2. VOIP_" packet-mark=VOIP parent=UP priority=2 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="3. ACK_" packet-mark=ACK parent=UP priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=18M max-limit=99M name="4. UDP_" packet-mark=UDP parent=UP priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="5. ICMP_" packet-mark=ICMP parent=UP priority=4 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="6. HTTP_" packet-mark=HTTP parent=UP priority=5 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="7. QUIC_" packet-mark=QUIC parent=UP priority=6 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="8. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=7 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s name="9. OTHER_" packet-mark=OTHER parent=UP queue=redCustom

I know reason is CPU load, but as i said this causes the entire connection (speedtest bandwidth for example) getting captured by ACK mangle rule and its queue tree and never reaching other rules which where supposed to actually capture it.

Im constantly changing and adapting my rules as i notice problems so not completely sure, but I prio ICMP and DNS above all, than ACK packets than rest..

About http packets, i mark big ones first than small ones, and have separate rule for SSL (443) and HTTP(80+8080) traffic (both small + big) while giving one step bigger priority to SSL traffic over HTTP in queue.

And yeah, if your ISP doesnt support big bursting same as i did in my above screenshot, you need to sacrifice some bandwidth, best is to limit high traffic consumers in one group and drop their total speed around 10-15% below total internet speed so they can never saturate it, while leaving other full bandwidth to other "bursty" queues so they have time to drop this big consumer queues via their limit-at limits down if you get what i mean.

BTW there is one more trick im testing for some times, what do you think of some basic QOS in NAT it self, i use this on locations where there is very variable speed LTE links and i cant make proper QUEUE tree.

/ip firewall nat
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN protocol=icmp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN port=53 protocol=udp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN port=443 protocol=tcp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN port=80 protocol=tcp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN protocol=tcp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN protocol=udp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN

As nat is processed in order, this gives sort of packet priority which packets will be nated first in chain.
For example with this rules u wont notice such big slowdowns if someone is downloading torrents, or some other bandwidth hungry protocols, pages will still open snappy without any other mangle/queue rules.

Opinions?

Yeah, I am still confused about the order of the mangle rules.

I think pcunite needs to look into out input on the issues from over the past few months and give us his opinion on this things.

NAT QoS, is something I haven't thought of before, but it could be problematic if the ISP is using dynamic IPs.

Also I have IPv6 connectivity from my ISP, NAT QoS would limit it only for IPv4 traffic and therefore make it inefficient. In my opinion I suggest you stick with mangles/other mechanisms that would work for both IPv4/IPv6 smoothly.

QoS is such a pain...

ivicask · Wed Jul 22, 2020 7:45 pm

Are you sure about marking DNS / ICMP or ACK connections than their packet marks, because what im seeing that when i run speedtest some or sometimes entire bandwidth goes thru this QUEUE(DNS/ICMP/ACK) and as they have top prio actually choke my net.

I changed it to only mark packets directly without prior connection marking and now works fine, only this packets pass thru.(rest of mangle rules still uses connection +packet marking)

I tried absolutely destroying my net with few torrents, http downloads, windows updates, some youtube, radio stream(doesnt break), dropbox upload, and ping is still stable without drops, and web pages opens super fast.

This is how my QOS looks now that im using for some time now.
The reason why he marked connections instead of packets is to reduce load on the CPU.

I've encountered these anamolies you described as well. I've made a lot of modifications to the mangle rules since though.

One thing I'm not sure about is the priority of the mangle rules, like should ACK be marked before UDP/ICMP/QUIC or what? Should mark-packet http_big come first before mark-parket HTTP?
Maybe you know something about the mangle rules priority.

I also added established/related connection state when I discovered some connections would escape the mangel rules via connections tab. DoH traffic is also marked via address-list, DNS over UDP 443 is marked especially for Google Chrome as it uses that port for DNS queries sometimes, something I caught from the connections tab.

Here's mine:
Code: Select all
add action=mark-connection chain=prerouting comment=DNS connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=DNS passthrough=yes port=53 protocol=udp \
    tcp-flags=""
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=DNS passthrough=yes port=53 protocol=udp \
    src-address-list=""
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=DoH connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=53,443,853 protocol=tcp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=53,443,853 protocol=tcp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment="DoH (UDP 443)" connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=443 protocol=udp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=443 protocol=udp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=VOIP connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=VOIP passthrough=yes port=\
    5060-5062,10000-15251,15253-20000,3478-3497,16384-16387,16393-16402,19302-19309 protocol=udp
add action=mark-packet chain=prerouting connection-mark=VOIP new-packet-mark=VOIP passthrough=no tcp-flags=""
add action=mark-connection chain=prerouting comment=QUIC connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=QUIC passthrough=yes port=80,443 \
    protocol=udp
add action=mark-packet chain=prerouting connection-mark=QUIC new-packet-mark=QUIC passthrough=no
add action=mark-connection chain=prerouting comment=UDP connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=UDP passthrough=yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=UDP new-packet-mark=UDP passthrough=no
add action=mark-connection chain=prerouting comment=ICMP connection-state=new dst-address-list=!not_in_internet new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=prerouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-connection chain=postrouting connection-state=new dst-address-list=!not_in_internet new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=postrouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-packet chain=postrouting comment=ACK new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-connection chain=prerouting comment=HTTP connection-mark=no-mark connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=HTTP \
    passthrough=yes port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-bytes=5000000-0 connection-mark=HTTP connection-rate=2M-300M connection-state=established,related,new dst-address-list=!not_in_internet \
    new-connection-mark=HTTP_BIG passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no
add action=mark-connection chain=prerouting comment=OTHER connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=POP3 passthrough=yes port=995,465,587 \
    protocol=tcp
add action=mark-packet chain=prerouting connection-mark=POP3 new-packet-mark=OTHER passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=OTHER packet-mark=no-mark \
    passthrough=yes
add action=mark-packet chain=prerouting connection-mark=OTHER new-packet-mark=OTHER packet-mark=no-mark passthrough=no
I've taken advantage of the 5 second bandwidth burst from my ISP that maxed out at 300M that I've seen so far. So I added burst for everything. I capped individual traffic types to 130M to prevent bufferbloat. That extra 300M for parent queue gives it that extra headroom.

The queue tree:
Code: Select all
/queue tree
add bucket-size=0.01 burst-limit=300M burst-threshold=300M burst-time=1s max-limit=99M name=DOWN parent=bridge queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="1. DNS" packet-mark=DNS parent=DOWN priority=1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=2M max-limit=99M name="2. VOIP" packet-mark=VOIP parent=DOWN priority=2 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="3. ACK" packet-mark=ACK parent=DOWN priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=18M max-limit=99M name="4. UDP" packet-mark=UDP parent=DOWN priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="5. ICMP" packet-mark=ICMP parent=DOWN priority=4 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="6. HTTP" packet-mark=HTTP parent=DOWN priority=5 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="7. QUIC" packet-mark=QUIC parent=DOWN priority=6 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="8. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=7 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s name="9. OTHER" packet-mark=OTHER parent=DOWN queue=redCustom
add bucket-size=0.01 burst-limit=300M burst-threshold=300M burst-time=1s max-limit=99M name=UP parent=pppoe-out1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="1. DNS_" packet-mark=DNS parent=UP priority=1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=2M max-limit=99M name="2. VOIP_" packet-mark=VOIP parent=UP priority=2 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="3. ACK_" packet-mark=ACK parent=UP priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=18M max-limit=99M name="4. UDP_" packet-mark=UDP parent=UP priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="5. ICMP_" packet-mark=ICMP parent=UP priority=4 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="6. HTTP_" packet-mark=HTTP parent=UP priority=5 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="7. QUIC_" packet-mark=QUIC parent=UP priority=6 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="8. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=7 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s name="9. OTHER_" packet-mark=OTHER parent=UP queue=redCustom
I know reason is CPU load, but as i said this causes the entire connection (speedtest bandwidth for example) getting captured by ACK mangle rule and its queue tree and never reaching other rules which where supposed to actually capture it.

Im constantly changing and adapting my rules as i notice problems so not completely sure, but I prio ICMP and DNS above all, than ACK packets than rest..

About http packets, i mark big ones first than small ones, and have separate rule for SSL (443) and HTTP(80+8080) traffic (both small + big) while giving one step bigger priority to SSL traffic over HTTP in queue.

And yeah, if your ISP doesnt support big bursting same as i did in my above screenshot, you need to sacrifice some bandwidth, best is to limit high traffic consumers in one group and drop their total speed around 10-15% below total internet speed so they can never saturate it, while leaving other full bandwidth to other "bursty" queues so they have time to drop this big consumer queues via their limit-at limits down if you get what i mean.

BTW there is one more trick im testing for some times, what do you think of some basic QOS in NAT it self, i use this on locations where there is very variable speed LTE links and i cant make proper QUEUE tree.
Code: Select all
/ip firewall nat
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN protocol=icmp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN port=53 protocol=udp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN port=443 protocol=tcp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN port=80 protocol=tcp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN protocol=tcp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN protocol=udp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN
As nat is processed in order, this gives sort of packet priority which packets will be nated first in chain.
For example with this rules u wont notice such big slowdowns if someone is downloading torrents, or some other bandwidth hungry protocols, pages will still open snappy without any other mangle/queue rules.

Opinions?
Yeah, I am still confused about the order of the mangle rules.

I think pcunite needs to look into out input on the issues from over the past few months and give us his opinion on this things.

NAT QoS, is something I haven't thought of before, but it could be problematic if the ISP is using dynamic IPs.

Also I have IPv6 connectivity from my ISP, NAT QoS would limit it only for IPv4 traffic and therefore make it inefficient. In my opinion I suggest you stick with mangles/other mechanisms that would work for both IPv4/IPv6 smoothly.

QoS is such a pain...

No provider supports IPV6 in Croatia

You are using IPV6 in local network also ? There is no nat involved in router?If yes, than my rules would still work regardless as long your router is NAT-ing something at some point.

DarkNate · Thu Jul 23, 2020 12:20 am

Are you sure about marking DNS / ICMP or ACK connections than their packet marks, because what im seeing that when i run speedtest some or sometimes entire bandwidth goes thru this QUEUE(DNS/ICMP/ACK) and as they have top prio actually choke my net.

I changed it to only mark packets directly without prior connection marking and now works fine, only this packets pass thru.(rest of mangle rules still uses connection +packet marking)

I tried absolutely destroying my net with few torrents, http downloads, windows updates, some youtube, radio stream(doesnt break), dropbox upload, and ping is still stable without drops, and web pages opens super fast.

This is how my QOS looks now that im using for some time now.
The reason why he marked connections instead of packets is to reduce load on the CPU.

I've encountered these anamolies you described as well. I've made a lot of modifications to the mangle rules since though.

One thing I'm not sure about is the priority of the mangle rules, like should ACK be marked before UDP/ICMP/QUIC or what? Should mark-packet http_big come first before mark-parket HTTP?
Maybe you know something about the mangle rules priority.

I also added established/related connection state when I discovered some connections would escape the mangel rules via connections tab. DoH traffic is also marked via address-list, DNS over UDP 443 is marked especially for Google Chrome as it uses that port for DNS queries sometimes, something I caught from the connections tab.

Here's mine:
Code: Select all
add action=mark-connection chain=prerouting comment=DNS connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=DNS passthrough=yes port=53 protocol=udp \
    tcp-flags=""
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=DNS passthrough=yes port=53 protocol=udp \
    src-address-list=""
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=DoH connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=53,443,853 protocol=tcp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=53,443,853 protocol=tcp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment="DoH (UDP 443)" connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=443 protocol=udp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=established,related,new dst-address-list=DoH new-connection-mark=DNS passthrough=yes port=443 protocol=udp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=VOIP connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=VOIP passthrough=yes port=\
    5060-5062,10000-15251,15253-20000,3478-3497,16384-16387,16393-16402,19302-19309 protocol=udp
add action=mark-packet chain=prerouting connection-mark=VOIP new-packet-mark=VOIP passthrough=no tcp-flags=""
add action=mark-connection chain=prerouting comment=QUIC connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=QUIC passthrough=yes port=80,443 \
    protocol=udp
add action=mark-packet chain=prerouting connection-mark=QUIC new-packet-mark=QUIC passthrough=no
add action=mark-connection chain=prerouting comment=UDP connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=UDP passthrough=yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=UDP new-packet-mark=UDP passthrough=no
add action=mark-connection chain=prerouting comment=ICMP connection-state=new dst-address-list=!not_in_internet new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=prerouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-connection chain=postrouting connection-state=new dst-address-list=!not_in_internet new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=postrouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-packet chain=postrouting comment=ACK new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-connection chain=prerouting comment=HTTP connection-mark=no-mark connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=HTTP \
    passthrough=yes port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-bytes=5000000-0 connection-mark=HTTP connection-rate=2M-300M connection-state=established,related,new dst-address-list=!not_in_internet \
    new-connection-mark=HTTP_BIG passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting connection-mark=HTTP_BIG new-packet-mark=HTTP_BIG passthrough=no
add action=mark-packet chain=prerouting connection-mark=HTTP new-packet-mark=HTTP passthrough=no
add action=mark-connection chain=prerouting comment=OTHER connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=POP3 passthrough=yes port=995,465,587 \
    protocol=tcp
add action=mark-packet chain=prerouting connection-mark=POP3 new-packet-mark=OTHER passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=established,related,new dst-address-list=!not_in_internet new-connection-mark=OTHER packet-mark=no-mark \
    passthrough=yes
add action=mark-packet chain=prerouting connection-mark=OTHER new-packet-mark=OTHER packet-mark=no-mark passthrough=no
I've taken advantage of the 5 second bandwidth burst from my ISP that maxed out at 300M that I've seen so far. So I added burst for everything. I capped individual traffic types to 130M to prevent bufferbloat. That extra 300M for parent queue gives it that extra headroom.

The queue tree:
Code: Select all
/queue tree
add bucket-size=0.01 burst-limit=300M burst-threshold=300M burst-time=1s max-limit=99M name=DOWN parent=bridge queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="1. DNS" packet-mark=DNS parent=DOWN priority=1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=2M max-limit=99M name="2. VOIP" packet-mark=VOIP parent=DOWN priority=2 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="3. ACK" packet-mark=ACK parent=DOWN priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=18M max-limit=99M name="4. UDP" packet-mark=UDP parent=DOWN priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="5. ICMP" packet-mark=ICMP parent=DOWN priority=4 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="6. HTTP" packet-mark=HTTP parent=DOWN priority=5 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="7. QUIC" packet-mark=QUIC parent=DOWN priority=6 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="8. HTTP_BIG" packet-mark=HTTP_BIG parent=DOWN priority=7 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s name="9. OTHER" packet-mark=OTHER parent=DOWN queue=redCustom
add bucket-size=0.01 burst-limit=300M burst-threshold=300M burst-time=1s max-limit=99M name=UP parent=pppoe-out1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="1. DNS_" packet-mark=DNS parent=UP priority=1 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=2M max-limit=99M name="2. VOIP_" packet-mark=VOIP parent=UP priority=2 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="3. ACK_" packet-mark=ACK parent=UP priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=18M max-limit=99M name="4. UDP_" packet-mark=UDP parent=UP priority=3 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s name="5. ICMP_" packet-mark=ICMP parent=UP priority=4 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="6. HTTP_" packet-mark=HTTP parent=UP priority=5 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="7. QUIC_" packet-mark=QUIC parent=UP priority=6 queue=default
add burst-limit=130M burst-threshold=130M burst-time=1s limit-at=10M max-limit=99M name="8. HTTP_BIG_" packet-mark=HTTP_BIG parent=UP priority=7 queue=redCustom
add burst-limit=130M burst-threshold=130M burst-time=1s name="9. OTHER_" packet-mark=OTHER parent=UP queue=redCustom
I know reason is CPU load, but as i said this causes the entire connection (speedtest bandwidth for example) getting captured by ACK mangle rule and its queue tree and never reaching other rules which where supposed to actually capture it.

Im constantly changing and adapting my rules as i notice problems so not completely sure, but I prio ICMP and DNS above all, than ACK packets than rest..

About http packets, i mark big ones first than small ones, and have separate rule for SSL (443) and HTTP(80+8080) traffic (both small + big) while giving one step bigger priority to SSL traffic over HTTP in queue.

And yeah, if your ISP doesnt support big bursting same as i did in my above screenshot, you need to sacrifice some bandwidth, best is to limit high traffic consumers in one group and drop their total speed around 10-15% below total internet speed so they can never saturate it, while leaving other full bandwidth to other "bursty" queues so they have time to drop this big consumer queues via their limit-at limits down if you get what i mean.

BTW there is one more trick im testing for some times, what do you think of some basic QOS in NAT it self, i use this on locations where there is very variable speed LTE links and i cant make proper QUEUE tree.
Code: Select all
/ip firewall nat
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN protocol=icmp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN port=53 protocol=udp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN port=443 protocol=tcp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN port=80 protocol=tcp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN protocol=tcp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN protocol=udp
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN
As nat is processed in order, this gives sort of packet priority which packets will be nated first in chain.
For example with this rules u wont notice such big slowdowns if someone is downloading torrents, or some other bandwidth hungry protocols, pages will still open snappy without any other mangle/queue rules.

Opinions?
Yeah, I am still confused about the order of the mangle rules.

I think pcunite needs to look into out input on the issues from over the past few months and give us his opinion on this things.

NAT QoS, is something I haven't thought of before, but it could be problematic if the ISP is using dynamic IPs.

Also I have IPv6 connectivity from my ISP, NAT QoS would limit it only for IPv4 traffic and therefore make it inefficient. In my opinion I suggest you stick with mangles/other mechanisms that would work for both IPv4/IPv6 smoothly.

QoS is such a pain...
No provider supports IPV6 in Croatia

You are using IPV6 in local network also ? There is no nat involved in router?If yes, than my rules would still work regardless as long your router is NAT-ing something at some point.

Your rules are only limited to IPv4 traffic. NAT is for IPv4. There's no NAT for IPv6 except something like NAT64 which is something I wished didn't exist to begin with. I'm anti-NAT anyway. The whole point of IPv6 was to bring back end-to-end principle.

I have native IPv6 connectivity from my ISP. My client devices each gets two publicly routable IPv6. I had to create separate mangle rules for v4 and v6.

srcrosset · Fri Jul 24, 2020 10:50 pm

@pcunite,
I followed your VLAN tutorial and set up router-on-a-stick which has been working great.
Now I am looking through your QoS tutorial and would like to know how this applied to the VLAN setup.
Would the LAN interface be the bridge?
Would this QoS setup only apply to traffic going to and from the WAN (ether1) port and not affect internal inter-LAN traffic

haj3s29a · Sun Jul 26, 2020 12:27 am

/ip firewall mangle add action=mark-connection chain=prerouting connection-bytes=5000000-0 connection-mark=HTTP connection-rate=2M-300M connection-state=established,related,new dst-address-list=!not_in_internet \
    new-connection-mark=HTTP_BIG passthrough=yes protocol=tcp

connection-bytes=5000000-0 doesn't work in RouterOS v7.1beta1 - it is possible to setup only 0-xxxxx.

Can someone confirm whether it works in stable version or not, please?

Sadly, documentation is outdated and I didn't find any further info.

kswong · Sun Jul 26, 2020 6:49 pm

Code: Select all
/ip firewall mangle add action=mark-connection chain=prerouting connection-bytes=5000000-0 connection-mark=HTTP connection-rate=2M-300M connection-state=established,related,new dst-address-list=!not_in_internet \
    new-connection-mark=HTTP_BIG passthrough=yes protocol=tcp
connection-bytes=5000000-0 doesn't work in RouterOS v7.1beta1 - it is possible to setup only 0-xxxxx.

Can someone confirm whether it works in stable version or not, please?

Sadly, documentation is outdated and I didn't find any further info.

try this....connection-bytes=!0-5000000

haj3s29a · Sun Jul 26, 2020 7:25 pm

try this....connection-bytes=!0-5000000

not possible to use "!" in connection-bytes. I have tried connection-bytes=5M-100M and it seems to work

UPDATE:

5M-0 seems to work over CLI but WebUI considers it as an error.

osc86 · Mon Jul 27, 2020 9:14 pm

Any idea how a mangle setup with policy routing (mark routing) and connection / packet marking for qos could look like?
Both need "stop processing" in prerouting chain.
I experimented a bit with my existing rules, but I didn't get both working at the same time. Maybe I have to use a new chain and some jump & return rules.