I've never been very happy with my plain 802.11 qos. I am considering at least providing an option to my customers of a more expensive install and setting aside a channel for nstreme to provide higher quality.

I don't want to sell my customers a bill of goods though. I'm wondering if Nstreme really delivers in the real world. For all the talk about it, I've never heard a single wisp with typical heavily loaded APs (>40 customers) say they have done it and got good results.


Has anyone out there actually tried this?

The big problem with NStreme on heavily loaded AP's is CPU speed. The average RB230/RB532/RB112 doesn't have enough power to deliver 30Mbps of NStreme packets to multiple clients. A board that uses Pentium M processors or similar is necessary to really make it work. Like I am testing now a Biostar board w/ NX1750 AMD Geode processor (1.4ghz, only uses a small 1" fan on the CPU) that ends up being <$125 for the board, CPU & 128MB memory & 2 PCI slots).. If it can withstand the heat without major cooling concerns (i'll probably upgrade the little fan to a big heatsink instead of a fan at all), then it'll have plenty of CPU power to run lots of boards (I have a 1.8 celeron link that runs 4 links, 2 of which can BTEST on the same unit at 65Mbps each).

Right now I don't get anywhere near 30mbits using 802.11. If I could deliver 8Mbits without all the dropped packets and high latency storms that would be fine. Then when we all figure out how to do a faster AP that won't burn up I could switch them out.

Does anyone have this experience?

The RB532s start showing large latency variations (high jitter) if you use WDS. Although, we haven't tested without WDS to verify this. The latency issue starts showing up after about 3 associated clients.

Well, I have a 3 clients on one unit, a WRAP board, on 5ghz-10mhz, connected to an RB112 at 2.5 miles away with WDS that gets about 9.5Mbps TCP, 17Mbps UDP. That's using the RB112 as the reciever, and a secondary board here as the BTest source. No NStreme. If I turn on NStreme, the CPU maxes out much sooner. Double that to get nearly 19Mbps TCP, 34Mbps UDP and it starts to make sense where you can get the higher rates. The true "max" for 802.11a is about 27Mbps, Nstreme's optimizations and packet aggregation helps a lot to bring it up to around 37Mbps. TCP's killer is latency more than anything, so if you can reduce packet latency it helps bring the "usable" speed up.

We have 8 links coming off one radio using nstream, clients vary from 1km to 3km away and achieve around 11mbit TCP each, using RB112 as receiver and athlon 64 as the base Low latency 1-10ms depending on load. So i think nstream does work

All this feedback is good, and I appreciate it but I REALLY hope to hear from someone who has >40 customers on an Nstreme AP. Ulesss it can reach at least 40 customers I don't consider it worth doing. Better to cave in and get Canopy gear.

that becomes your loss

The big problem with NStreme on heavily loaded AP's is CPU speed. The average RB230/RB532/RB112 doesn't have enough power to deliver 30Mbps of NStreme packets to multiple clients. A board that uses Pentium M processors or similar is necessary to really make it work. Like I am testing now a Biostar board w/ NX1750 AMD Geode processor (1.4ghz, only uses a small 1" fan on the CPU) that ends up being <$125 for the board, CPU & 128MB memory & 2 PCI slots).. If it can withstand the heat without major cooling concerns (i'll probably upgrade the little fan to a big heatsink instead of a fan at all), then it'll have plenty of CPU power to run lots of boards (I have a 1.8 celeron link that runs 4 links, 2 of which can BTEST on the same unit at 65Mbps each).

I like the idea of PC based APs and I have quit a few VIA boards running now and I havent had any problems so far.
BTY, New Egg has those boards for $16. (OEM)
http://www.newegg.com/Product/Product.a ... 813138019R
Man, thats cheap!
This link shows a big heat sink on NX CPU.
http://www.logicpd.com/downloads/562/1003388_Rev_B.pdf

The RB532s start showing large latency variations (high jitter) if you use WDS. Although, we haven't tested without WDS to verify this. The latency issue starts showing up after about 3 associated clients.

I can can't directly confirm this... we use 532's/SR5's for APs, and 112's/CM9's for CPEs...

it seems that WDS may add about 5-10ms, of latency, presumbably due to CPU, but that's not the real issue

While NStream makes things much more consistant for latency issues (large spikes and contention/hidden node issues are eliminated), it has a downside... because the CPE's only transmit when it's their turn, if there is any interference that causes the CSMA/CA subsystem to wait before transmitting, the radio can actually miss it's turn to transmit. (MikroTik seems to have commited to trying to turn this off in a new version of NStream for v3.x, but they've not clearly and specifically said so yet, but they have said they are looking into it...).

It is our experience that with NStream and polling, you can put 25 customers on a AP, more of less regardless of the load from any individual customer, and all customers will have a reasonably consistant and reliable service with accecptable ping times.

when using polling, average ping times seem from AP to CPE seem to be roughly related to the number of radios associated to the AP, so if you have an AP with 7 users, average ping times would be between 5ms and 10ms. when we put 25 CPE's on one AP, our average ping times to the CPE are usually between 20ms and 25ms.

I think this is because the more radio's associated, the longer between polling intervals, and therefore the more pronounced the wait is if the CPE misses one or two time slots for transmitting.

Since WDS increases the number of un-necessary packets transmitted, with it being a full bridge, this adds to the problem as it will sometimes be waiting to transmit the traffic that would have otherwise been blocked if you didn't have WDS, and so your real traffic sits in queue and waits it's turn... This part of the problem would go away if you didn't have WDS and had off Default Forward, then only CPE to AP communicatins would ever be transmitted, and never anything CPE to CPE...

Well, that is hopeful. Sounds like it is headed for 40ms ping times for 40 customers which is not great, but bearable. Thanks for the feedback.

It is really strange though that no-one with a fully loaded AP ever responded. I hate being a pioneer.

I've talked with a few other WISP's about how many people to put on a sector (using nstream and the hated wds), we've all decided to keep our sectors to the 20-30 range per sector.... I don't want to go any higher, because then I'll have to drop customers because we're overloaded, to maintain performance for the rest. and I don't want to do that....

I run many APs with over 40 subscribers. It seems that Nstreme and polling are the path to overcome the 802.11 deficiency of the tail wagging the dog, i.e. the client has too much control because its heritage is to serve clients on wireless LANs. By polling, logically it would seem that the congestion and retries gemerated by wireless cards that had savvy engineers make them get more than their 'fair' (?) share of resources can be overcome and support significant subscribers per AP. Thats why some vendors that figured this out have equipment that works better. I am coming into this forum with much experience that seems to be from the other direction. I am beginning to deploy MT and build radios because of the flexibility, scalability of the platform, and cost sure seems to be an issue. However....

If you want to reliably serve 40 customers and keep them, it would seem to me that identifying the ROI, the potential to keep customers, and the overall reliability of the delivered service should be your consideration. If you know you can charge a certain rate and keep those 40 customers, which seem to be your breakeven threshold, then put in equipment that is designed to do that. Therefore, the notion that Canopy is too expensive is not true because it supports at least twice or more subscribers before you have to spend more money. It just works, and doesnt take high end installers to deploy. With the right situation, you cant go wrong with Canopy. However again, I am finding certain scenarios that warrant alternate technologies due to environmental considerations. It seems that NLOS capability and other new capabilities will be introduced into the pre-Wimax equipment space. This is where MT based technology will deliver custom application specific functionality when the requirements get out of the mainstream of serving 40 customers on an AP...
With good success at loading Canopy to ridiculous levels, I am leary about using anything else to the customer, but I am having great success using MT and Atheros based radios to deliver backbone connectivity without breaking the bank. Thats where the ROI really comes in. Seems like all WISPs will argue this ad-nauseum, but I just dont like to have to hear from unhappy subscribers. I would rather expand the network with more and more bandwidth and deliver bandwidth to places that otherwise couldnt be served.
Enough of my soapbox, do what works for your business plan. If the customer pays for the CPE, all the more better...

My experience is that I'm getting better results (more bandwith) on CPE devices when using nstreme on not heavily loaded AP (15-20 CPE antennas). BUT when trying to use it on APs with 50 customers or more I can get better download speeds without nstreme.

I'm also trying to find out how nstreme could be of any help in those heavily loaded places.

TCP's killer is latency more than anything

i disagree. in my opinion, packet loss, even as little as 1%, is worse.
latency is overcome by window scaling.

packet loss can occur because of congestion. there is simply no bandwidth left, so the packet it not being sent. in this case it is not correct to call this `loss`

packet loss can occur because of congestion. there is simply no bandwidth left, so the packet it not being sent. in this case it is not correct to call this `loss`

I disagree, it is still loss since the packet is dropped.
The word "loss" might have a strictly negative meaning to some but it really shouldn't. Loss is used to fake congestion to throttle/shape bandwidth usage.

My experience is that I'm getting better results (more bandwith) on CPE devices when using nstreme on not heavily loaded AP (15-20 CPE antennas). BUT when trying to use it on APs with 50 customers or more I can get better download speeds without nstreme.

I'm also trying to find out how nstreme could be of any help in those heavily loaded places.

I have to correct myself. Most of the problems we had when trying to migrate to nstreme were caused by CPE antennas without nstreme enabled. They were causing errors on APs to nstreme enabled.

This is the same problem I am trying to solve. Because of the cost of tower leases, a tower can not be profitable (at a per subscriber price that is bearable to the market) without 50-70 subscribers per antenna.

It seems like 802.11 just can't handle that kind of density, any way you slice it

I have been trying to determine if, by using nstream 40+ clients per antenna could be supported, and no one seems to be using it as such. I'm sure if they were, they would be an active forum participant.

Currently our maximum density is 34 clients, and I will say that it "usually" works ok - but we are not using nstream!

Just one question.
I have 20 CPE(DLINK 2000AP+ client mode) on a RB112.
But on the registration table , obviosly, I see 40 MAC address (the CPE + customer pc).
When you wrote 40+ customers do you consider 40 MAC address on the registration table or 40 customer with 80 MAC address on the registration table?-

Thanks
Massimo

Just one question.
I have 20 CPE(DLINK 2000AP+ client mode) on a RB112.
But on the registration table , obviosly, I see 40 MAC address (the CPE + customer pc).
When you wrote 40+ customers do you consider 40 MAC address on the registration table or 40 customer with 80 MAC address on the registration table?-

As far as I know, you should only see one MAC in the wireless registration table - that of the radio associated to the AP. By saying 40+ customers, I mean exactly that. More than 40 stations associated to the AP.

Following this thread it seems to me that ROS is not engineered to do
large amount of customers per sector? If it would, we would have
heard some nstream test numbers from MT in this thread.

Anyway what are the parameters to optimize the service for
as much customers per sector as possible:

- default forward off to reduce useless traffic
- nstream with polling
- reduce amount of supported rates to reduce retries
- disable wds and use routing to avoid useless traffic
- use only RB with Atheros as AP and Client to avoid misbehavior

May be it can help a lot to eliminate or upgrade customers antenna
with bad signal. Reduces retries and keeps medium free.

If I understand enabling nstream is changing from csma/cd to
token based network. Token based works better on full medium
than csma/cd, but csma/cd has lower overhead and performs
better if medium is not used heavly.

Stefan

Following this thread it seems to me that ROS is not engineered to do large amount of customers per sector? If it would, we would have
heard some nstream test numbers from MT in this thread.

I think the problem is with 802.11, not ROS. I have had the same issues with older Orinoco AP-1000 units, as well as newer Prism 2 based "hardware" AP units.

If I understand enabling nstream is changing from csma/cd to
token based network. Token based works better on full medium
than csma/cd, but csma/cd has lower overhead and performs
better if medium is not used heavly.

In 2.9.x CSMA/CD is still enabled, even when nstream is in use. This is being changed for v3, this is the only handy thread in the beta forum I found (I didn't look deeply, please browse the beta forum for more information) http://forum.mikrotik.com/viewtopic.php?t=12217

Following this thread it seems to me that ROS is not engineered to do large amount of customers per sector? If it would, we would have
heard some nstream test numbers from MT in this thread.

I think the problem is with 802.11, not ROS. I have had the same issues with older Orinoco AP-1000 units, as well as newer Prism 2 based "hardware" AP units.

Clear. But I think it would be possible to tune this. Especially in
homogen systems with only ROS. Nstream is such a tuning.
Proxim call it Worp, I think it's nearly the same.

If I understand enabling nstream is changing from csma/cd to
token based network. Token based works better on full medium
than csma/cd, but csma/cd has lower overhead and performs
better if medium is not used heavly.

In 2.9.x CSMA/CD is still enabled, even when nstream is in use. This is being changed for v3, this is the only handy thread in the beta forum I found (I didn't look deeply, please browse the beta forum for more information) http://forum.mikrotik.com/viewtopic.php?t=12217

I've followed this thread. Disabling CSMA/CD is only necessary in
a frequency fighting situation. With nstream the master controls
access, so the clients should not even try to send without order so
there is no collision. Disabling CSMA/CD without nstream will kill
your own sendings.

Stefan

ForePoint

What is your CPU load on your 532 and what type of service level are you providing? I'm wondering if some of the latency is due to processor speed or is it inherint in the polling mechinism, maybe MT can put in a word on how the polling is done.

We are use to this kind of latecy as it is typical on WaveRider equipment, while faster is better it is livable. Most of the brandname equipment is faster in this regards but I don't have any info from a loaded moto or trango.

Any chance we can get you to put up a P4 system next week and add another 40-50 customers so these questions can be answered.

I came across an old thread, and the person was using a Lippert mini ITX board (P4m) and was running 100+ customers, but I don't know the config details....I try and find it again when I have time and see if I can get some details.