Hello. I'm a technician operating a small WISP, and I'm currently trying to make some improvements to our infrastructure. The network we have currently has 40-50 clients. At present, our system is using simple queues for each user to set maximum upload/download speeds for each user. The limits for each client are not the same.

I would like to move away from this model, and have the following requirements for the new implementation:
- Distribution of available bandwidth through PCQ (or another mechanism)
- Different clients have different maximum download/upload rates (for example 256/128, 512/256, and 1024/512)
- Each client's traffic can still be viewed on the graphs page (as is the case with the simple queues we use presently)
- Queue limitations should only affect WAN traffic, and not limit traffic between clients

Caveat: our WAN connection is a VSAT satellite Internet connection. It is a 2Mbps/512kbps connection on a 10:1 contention ratio, which means that the available throughput at any given time varies (generally 400kbps to 2Mbps download).
I have read extensively the wiki and forum posts about queue trees, PCQ, connection/packet marks, etc. However, I am having problems understanding how to implement the proper queue tree. It is my understanding that I need one parent PCQ queue for upload, and another for download. Though, I'm not sure which settings to use in that queue since we don't have consistent bandwidth availability. I want the parent queue to evenly distribute whatever is available, and the individual queues to control maximum limits. If this is possible, how might the queue tree look?

I'm not sure I fully understand connection/packet tagging. Most sources say to tag the new WAN connection first, then use the connection tag as a criteria to tag packets. I understand that this will save resources over evaluating every single packet. However, I still have these questions:
What is the best criteria to use for connection marks? I imagine in-interface = Ether1 for would work for download, but maybe there's a better option.
What is the best way to mark WAN upload traffic for a masqueraded network?
Once a connection coming in on Ether1 is tagged with the download mark, will the upload component of that connection (ACK packets, for example) also be tagged with the download mark, therefor counting against the download queue limit (and vice versa)?

Thanks for your assistance!

which means that the available throughput at any given time varies (generally 400kbps to 2Mbps download).

It's quite impossible to implement QOS when you don't have a guaranteed bandwidth available. The router itself does not know what available
bandwidth is there.

. It is my understanding that I need one parent PCQ queue for upload, and another for download. Though, I'm not sure which settings to use in that queue since we don't have consistent bandwidth availability. I want the parent queue to evenly distribute whatever is available, and the individual queues to control maximum limits.

Yes, you need one parent for download and another for upload, but it is not the parent queue which distributes "equally" the bandwidth. The parent queue basically establishes the max-limit for that queue and its own child queues. The child queues, deal then with the distribution, priorities, minimum bandwidth guaranteed.

What is the best criteria to use for connection marks? I imagine in-interface = Ether1 for would work for download, but maybe there's a better option.

When ether1 is your ISP connection, as it looks in your case, yes. But you need to use the forward chain, so that your are marking only connections that pass through the router. And the same is true for upload, only in reverse.

Once a connection coming in on Ether1 is tagged with the download mark, will the upload component of that connection (ACK packets, for example) also be tagged with the download mark, therefor counting against the download queue limit (and vice versa)?

No, that is another connection.

Thanks for the helpful clarification Caci99 (and sorry for my rediculously late response). Since PCQ is not going to work due to fluctuating throughput, would there be another mechanism that would help allow equal access to the available bandwidth? After reading a bit more I was thinking that maybe a SFQ queue would be able to do this, but correct me if I'm wrong. I need to ensure that no one user hogs all the bandwidth when other users are contending for it, and I like the graphs for individual clients available with simple queues.

I am pleased I could have been of little help.
As for your problem, I am not familiar with SFQ queue, but again I don't see how the router can distribute evenly the bandwidth when the total is unknown.
The graphs are a great feature of RouterOS, but if you want sth reliable which stores the graphs even if you reboot the router, or you decide to change that one, you definitely need a Cacti for example which captures the traffic based on snmp protocol. What's more, simple queues do generate graphs, while queue tree do not. But queue tree is far better for some good QOS, simple queues are for a simpler QOS and configuration.

Sorry if this sounds crazy, but have you tried a Windows PC to be the VSAT gateway and to have cFosSpeed turned On on that PC? After this PC you can have a RouterBOARD for max-limit per user and additional control.

Not to mention you need a web proxy.

P.S. and three more VSAT accounts to loadbalance to increase usability of the whole thing.

Does the bandwidth have a schedule ? i.e During daytime its closer to 400kb/s but at nighttime its 2mb/s or something similar ? You could add 2 seperate sets of queue's and enable/disable them via script, I believe there was something like that in the script repository on the wiki.

It's quite impossible to implement QOS when you don't have a guaranteed bandwidth available. The router itself does not know what available
bandwidth is there.

As I understand it, Queue trees provide the bare minimum for each queue (assuming they're all equal priority) before providing any spare bandwidth available for the queues to draw on, which I imagine is what OwenITGuy wants. So he simply has to ensure that the bare minimum of all his users is sufficiently low to avoid taxing the connection beyond its bandwidth capacity during rush hours. Then when the line gains more bandwidth its shared to the rest of the users as part of their maximum cap. As I undertand most sat providers offer a guaranteed minimum bandwidth as well similar to what you will be providing your users. You can simply use that number in your initial calculations.


On a side note, I have a setup quite similar to yours using simple queues and such, except its a wired network and I was looking into expanding to WiFi, what kind of hardware do you have and whats your overall WiFi range ?

Queue trees provide the bare minimum for each queue (assuming they're all equal priority) before providing any spare bandwidth

Queue tree as well as simple queue provide the minimum bandwidth (CIR), and it is not affected by priority of the queue itself. And it will provide the minimum specified whether or not the queue has a high priority. Priority enters in play when minimum of all queues is satisfied, that's the idea of minimum guaranteed, no matter what you always provide the minimum.
It could be an approach this as well, implementing minimums, but ....

but if the bandwidth was insufficient for all those minimums it would address the ones with the highest priority no ?

Thanks everyone for the suggestions. I think I should elaborate a little more, because my situation probably doesn't fit into most of your paradigms for a typical setup. I'm with a non-profit in eastern Congo. Our client base is pretty much NGOs, missionaies, and a few churches, and our total number of clients (and monthly revenue) won't likely grow significantly any time soon.

Power is also an issue here. Our system runs on an inverter and batteries that are charged primarilly by solar power, and supplemented with a generator. We do have a server, but we actually shut it down overnight because of the power drain. For this reason, solutions like Cacti or other proxy servers are not a viable option, because it would depend on the server being online. The graphing available with simple queues on the router is sufficient, again I just want to make sure we equalize access to the bandwidth a little more.

Not to mention you need a web proxy...and three more VSAT accounts to loadbalance to increase usability of the whole thing.

Does the bandwidth have a schedule ? i.e During daytime its closer to 400kb/s but at nighttime its 2mb/s or something similar ?

OK, let me address these two comments, and give everyone a quick primer on VSAT communication in case you're not familiar. When we buy bandwidth from the satellite company we buy it at a certain bandwidth and contention ratio. For the station I operate, we have a 2 Mbps download channel. We have a 10:1 contention ratio, which means that 9 other stations share this particular channel on the satellite transponder. The stations use time division multiple access (TDMA) to alternate who gets to use the channel. TDMA grants x miliseconds to each station per cycle. If only one station is passing traffic, then x is larger, and the throughput will be near 100%. If all 10 stations are passing traffic, then x is smaller, and the throughput will be near 10% (in the case of a 10:1 contention ratio). For this reason the throughput varies constantly. If you watch the traffic in real time, there are times during the day where our station gets speeds from 1-1.5 Mbps, but throughout the day we see speeds of 400-700 kbps of throughput when averaged over time. This is why I can't really set large CIRs for each queue, nor can I use parent queues that specify a certain level of available bandwidth. It is possible to change our bandwidth package to a lower contention ratio to improve speeds, but we're not likely to do that at this time because of the price difference (VSAT bandwidth is quite expensive). You would also still have throughput fluctuations on, for example, a 5:1 contention ratio connection.

As for your problem, I am not familiar with SFQ queue, but again I don't see how the router can distribute evenly the bandwidth when the total is unknown.

From what I understand from looking at the manual (http://wiki.mikrotik.com/wiki/Manual:Queue#SFQ), SFQ schedules all streams and gives them equal opportunities to pass traffic, regardless of the overall bandwidth. However, other than that single paragraph, I haven't found much other useful information on the topic. One site (http://wiki.ispadmin.eu/index.php/Docum ... otik_guide) provided some clarification that this is a prioritization of individual streams, rather than individual users/queues. Still, it sounds like it will keep one single stream (like a large file download) from eating up all the bandwidth. I just wish I could find some more examples, or people who have experience with this. Maybe it's time to get my MTCTCE certification!

What I've decided to try at the moment is a parent queue (SFQ) with no limits specified. Then each client has a child simple queue (pfifo) with maximum upload and download limits, as well as a small limit-at CIR specified. If I'm correct, this will use SFQ to keep things like large file loads from monopolizing the bandwidth, and still give me access to the simple queue graphs. Any other suggestions or experiences would be great.

Have you tried to put all users through a Windows PC with cFosSpeed and no other queues and limitations after that ? This PC can be Intel Atom 14W.

You can have a slow and useful proxy in another 14W consumption.

You can let your buddies test these on 3g ~ same variable bandwidth situation and send you the stuff to Congo once you are sure that it's worth it.

P.S. all individual clients having cFosSpeed will also help if they can afford the license / get discount.

P.S.2 you should be making plans and establish partnerships with near/far WISPs to bring more connectivity by nstreme/NV2 links.

P.S. all individual clients having cFosSpeed will also help if they can afford the license / get discount.

P.S.2 you should be making plans and establish partnerships with near/far WISPs to bring more connectivity by nstreme/NV2 links.

NetworkPro, I can't help but think you may be a salesman for cFosSpeed. Thanks for your input, but I'm really not interested in switching to another platform for our gateway.

As to the comment that I "should" be making partnerships with other WISPs, we're not in an environment where there are other WISPs. Besides, I don't really see how that contributes to answering my initial question. Thanks again for your input though.

Yes. You do not see how. I do because I have more information. I am giving you the relevant. The problem that you have can not be solved with RouterOS. You would be needing a WAN optimisation solution. The piece of software I recommended - I have tested. Pirate it for all I care.

P.S.2 you should be making plans and establish partnerships with near/far WISPs to bring more connectivity by nstreme/NV2 links.

This is what crossed my mind as well. If you can not guarantee the available bandwidth there is little RouterOS can do.
I met a guy form Nigeria 5 years ago, and he was in situation like yours, with only 2Mbps. It was by the time when MikroTik was releasing ROS rc3.x. This guy named Sundy, had come up with a great solution, installing squid proxy and making sure that traffic from proxy was not limited by the queues. He said he could save nearly 40% of the bandwidth. I think his tutorial is on the wiki. Of course, proxy will not improve skype or youtube or things like this experience, but webpages and small downloaded files will be a lot faster.
I understand that you have a problem with power, but in the current situation of yours, there is little you can do, at least I can not think of anything else.

Well, I've been using a parent SFQ simple queue (type wireless-default) for a few days now. I've also set some small limit-at values for each queue. I really can't say conclusively if it's helping significantly, but in theory it's working. If we get to a point where we can power the server 24-7, then maybe I'll look at a Squid install. I will at least start using MikroTik's stripped down implementation of Squid within the router, as that may help. I already have a queue to allow local traffic at the maximum speed possible, so this may help a bit. For now, I think this is the best we can manage given all that is Congo...cest la vie! Thank you everyone for your input!

On a side note, I have a setup quite similar to yours using simple queues and such, except its a wired network and I was looking into expanding to WiFi, what kind of hardware do you have and whats your overall WiFi range ?

The infrastucture that was here when I arrived a few months ago was a mixture between Ubiquiti (mostly NanoStation M2, some NanoStation 2, and some PowerStations), and Giganet antennas. Giganet isn't even in business anymore. I can understand why. When the antennas work, they have pretty descent range. However, we regularaly have problems where they just stop transmitting a wireless signal, and won't let you log into them. There is no reset button, so you have to short out some of the circuitry on the board...even that doesn't work all the time. They're about 4-5 years old though, Over the next few months I'm replacing all the old Giganet stuff with Ubiquiti hardware. I'm using NanoStation M2 antennas for the CPEs, 120-degree sector antennas/Rocket M2 access points, and a couple of NanoStation M2s as APs where 120-degree sectors are not quite as useful. Our farthest client is only 3.8 km (2.4 miles). I'd love to use more MikroTik, but until MIMO implementations follow an established standard, I think moving to a single vendor for the wireless hardware is the best option.

Hi,

I am based myself in Nairobi and are working a lot with all sorts of W-ISPs in central Africa.
E.g. Datco in Buvaku and Goma and several more.
I might have a solution for you...

Just let me know if I can help.

Heiko

Reviving this post hahahaha...
I have a similar situation from you my friend.

VSAT at 20 Mbp,s unlimited traffic, about 10 customer, with maximum 3 Mbps each one.

But, sometimes, the congestion give me about 2 or 3 Mbps..

How did you handle your request?

Thank you

I know this is old, but perhaps I can add to some understanding here. It is important to know how queues work.

The concept of 'fair queing' needs to be clarified to the way it is carried out, because how it is carried out is all you get out of it.

The real problem is not that one download will swamp other traffic, the problem is that one PERSON will swamp everyone else.

This is therefore not a problem in connections or connection types, so we don't really care how bandwidth is shared between downloads, or streaming or simple web page hits, unless you do, but thats a whole other story of prioritization.

What we care about is that each person gets the same over all banwidth at the exact moment that he needs it that everyone else does, regardless of what he is doing.

Thus if you have 3 megs, and 3 customers, and all hit the enter key at the same time, each one gets 1 meg for the time that they need it, and when one doesn't need it, it goes to someone else so they get more.

What we DO care about is the person who opens 8 connections to do his download, and the other two open 1 each. That's
10 total connections contending for bandwidth, but 8 belong to one person. The last thing we want is for that one person to get 80 percent of the 3 megs and everyone else to get only 10 percent. We want everyone to get 33.3 percent of the 3 megs regardless of how many connections they have open or what they are doing.

This is implemented, by PCQ automatically by opening a separate queue for each IP that has a connection open. As packets of any kind come into the router destined for an end customer they are placed into their own queue per dst IP, regardless of maximum or minimums. Then one packet is dequeued from each queue in round robin fashion. Thus if each queue has packets in it, each queue will be dequeued at the same rate as each other queue, one packet per dequeue cycle, thus everyone gets the same bandwidith! Very simple and works perfectly.

If packets are coming into the router slow enough that they can be dequeued immediately, there is no need to queue them at all. But if they are coming into the router faster than the router can deliver them to the end customers, then they get queued as they come in, and then each queue is dequeued one packet at a time, giving each end customer exactly the same bandwidth in packets per second. They can be dequeued in some systems in bytes rather than packets, but it has to be in whole packets one way or the other.

And there-in lies the rub, if you have a slow sat connection for your upstream, and a fast wireless to your customers, then the router CAN send packets faster than they are coming in, and they will never get queued. Thus one person can open up 8 connections and thus the incoming stream is 80 percent his stuff and only 20 percent the other two's stuff. Thus by setting your maxes on each customer to the sum of the incoming bandwidth, or a little less, you can make sure that your delivery speed is SLOWER than your incoming feed, so your packets will get queued by PCQ and then dequeued with fair share precision. This will cause the 80 percent customer to lose packets, and so
his stream will slow down until its equal to the other customers streams which will speed up until they are equal to each other.

Also be aware that in a highly daisy chained wireless system, where wireless is used to get from A to B and B to C and C to D, the customers
at D will be suffering way more total packet loss than those at B, and so TCP will slow down not from congestion but from failed packets and acks, and they will need more bandwidth pushed to them to make up for it.

Homer