Request for change in Hotspot settings

dfwair · Wed Nov 18, 2009 2:37 am

In our deployment of hotspots, I have found something that is really awkward. We have a PC-based MT box running 4.2 (Pentium 4/3GHz, single-core but with hyperthreading, Intel PRO/100 and Via ethernet) and we noticed that when the hotspot started getting 200-300 customers on it, we could run a ping to localhost and we would get timeouts from time to time. This setup is backed by RADIUS and is mac-address authenticated. We noticed when a bunch of users would log in at the same time (for example, adding another tower to the hotspot server) these timeouts would be frequent, and would eventually stablize, but there is an impact on VoIP services due to some dropped packets.

So I started looking and while this isn't a perfect solution, I have found one thing that has been a huge help. In the hotspot profiles, the setting to limit the hotspot's total bandwidth -- this setting still adds an unlimited queue. When the queues are added for newly logged-in users, the queues have to be rearranged. We halved the packet loss by going into the queues after they have been enabled and removed that unlimited queue. This did allow for a huge improvement, and the users are still authenticated.

There are situations where it will make sense for us to limit the throughput of a hotspot in our network down to a particular rate down the road, and we understand other people will have that same need. We're not asking to have that option removed, but what we would like to be seen is that when the speed for the hotspot isn't filled in, we would like that queue to not be installed.

Obviously removing the queue sounds like a good idea to us, however it doesn't stay when the hotspots are worked on.

Is there anything that can be done about this? Also are there any tips you could give on how to have a higher number of queues (today we're hitting about 1300) and eliminating the packet loss issue when a large number of queues are being managed?

dssmiktik · Wed Nov 18, 2009 2:59 am

I think you're on the right track with too many queues. As each client authenticates, a new simple queue is created. Mikrotik says it's better to create a queue-tree queues instead of simple ones, as it's more efficient with multiple queues (I'll have to search where I found this).

To disable queues being created when a Hotspot user authenticates, clear rate-limit setting:

/ip hotspot user profile set <user profile> rate-limit=""

If you set rate-limit to 0/0 it will still create a queue.

I haven't found a way to disable queues being create for the Hotspot interface itself though.

You could also run a script at user login to clear certain (or all) queues as well.

fewi · Wed Nov 18, 2009 3:05 am

This interests me. I'm still in the process of testing Mikrotik as a Hotspot platform and haven't exceeded a few hundred users.

Does anyone have any numbers on an RB1000 and how it performs with very large numbers of users taking out the simple queues as described above and replacing them with a queue tree that either uses PCQ to rate limit per user, or just assigns a large bandwidth pool to the entire network? How many users could one expect to support that way?

Our Hotspots usually only run a few hundred users but occasionally spike to several thousand.

dfwair · Wed Nov 18, 2009 5:41 pm

I think you're on the right track with too many queues. As each client authenticates, a new simple queue is created. Mikrotik says it's better to create a queue-tree queues instead of simple ones, as it's more efficient with multiple queues (I'll have to search where I found this).

If there was a way to do queue trees with RADIUS, that would be awesome, but unfortunately the RADIUS Mikrotik-Rate-Limit parameter sets the queue parameters per IP address that is authenticated.

To disable queues being created when a Hotspot user authenticates, clear rate-limit setting:
Code: Select all
/ip hotspot user profile set <user profile> rate-limit=""
If you set rate-limit to 0/0 it will still create a queue.

This once again is true without RADIUS running. We can delete the queues from the user profiles and get around this issue, but we're trying to centralize the bandwidth management in this network. That's the problem with after-thoughts. Unfortunately we ran into issues with hardware locking up with decent sized hotspot loads (as in RB450G that simply died in the field, 75 degree Fahrenheit day in a fiberglass NEMA enclosure -- very little heat, but the RB450G itself was very hot. RB450s with the same problem -- dying completely where even a power cycle doesn't bring them back up. Centralizing the bandwidth management and authentication at the head end of the network has proven itself because we have more control over the environmental variables, and MPLS/VPLS reduces the load of the router in the field and provides higher stability.

fewi · Wed Nov 18, 2009 5:45 pm

Are you able to use Mikrotik-Group as a RADIUS attribute instead? That should work around that particular issue as long as the value is a User Profile set up in the way dssmiktik describes, and no simple queues should be created.

dfwair · Wed Nov 18, 2009 6:06 pm

This interests me. I'm still in the process of testing Mikrotik as a Hotspot platform and haven't exceeded a few hundred users.

Does anyone have any numbers on an RB1000 and how it performs with very large numbers of users taking out the simple queues as described above and replacing them with a queue tree that either uses PCQ to rate limit per user, or just assigns a large bandwidth pool to the entire network? How many users could one expect to support that way?

Our Hotspots usually only run a few hundred users but occasionally spike to several thousand.

The RB1000, with per-user queueing (as is done with RADIUS), and seeing a decent number of logins (as does happen when the keepalive timeout is set to 2 minutes, we were able to get about 130 sessions online without major issues (occasional packet drop when a login was processed by the system and a queue was built). That was on a tower with 180 customers on it. The problem is that it did lock up, and we could never figure out why. With the hotspot services turned off we had no problems with the boards locking up or with packet loss. I have not pushed these systems to a higher number of customers after the lockups and packet loss issues. I have turned off Hotspot on several of them because they couldn't handle the loads, and it had to be pushed upstream to one box. We have also had to implement session timeouts and turn off idle timeouts and keepalive timeouts to avoid the occasional packet drops that were occurring when a queue would be built. The packet loss was bad enough to affect VoIP badly when we had idle timeouts and session timeouts enabled.

dfwair · Wed Nov 18, 2009 6:08 pm

Are you able to use Mikrotik-Group as a RADIUS attribute instead? That should work around that particular issue as long as the value is a User Profile set up in the way dssmiktik describes, and no simple queues should be created.

Per the Mikrotik RADIUS Client wiki...

Mikrotik-Group - Router local user group name (defines in /user group) for local users. HotSpot default profile for HotSpot users.

That would only specify local login parameters. See read/write/full groups for local login to the terminal or winbox.

fewi · Wed Nov 18, 2009 6:28 pm

For Hotspot RADIUS logins Mikrotik-Group selects a Hotspot User Profile and applies it to the user. I do use that and it works fine, the only caveat is that the login fails if you specify a profile that doesn't exist on the router. I do not send rate limit attributes, and the rate limit of the profile gets applied to the user. If the rate limit were to be empty, no queue should be installed and the queue tree should take. I'll try it in a lab later.

dfwair · Wed Nov 18, 2009 7:34 pm

For Hotspot RADIUS logins Mikrotik-Group selects a Hotspot User Profile and applies it to the user. I do use that and it works fine, the only caveat is that the login fails if you specify a profile that doesn't exist on the router. I do not send rate limit attributes, and the rate limit of the profile gets applied to the user. If the rate limit were to be empty, no queue should be installed and the queue tree should take. I'll try it in a lab later.

That's interesting since it's not in the documentation that it works that way. It only states local user login being set by the group.

Also there isn't a parameter in the user profiles to set a queue tree instead of the Rate-Limit. I can see how that could be beneficial though.

fewi · Wed Nov 18, 2009 8:29 pm

Yes, this would get a little convoluted, but the following is working in a quick test:

Created a User Profile 'profile1' with an empty rate limit.

[admin@MikroTik] > /ip hot use pro pri where name="profile1"    
Flags: * - default 
 1   name="profile1" idle-timeout=none keepalive-timeout=15m status-autorefresh=1m shared-users=unlimited transparent-proxy=no

Created mangle rules to mark traffic up and down:

[admin@MikroTik] > /ip fir man pri
Flags: X - disabled, I - invalid, D - dynamic 
 0   chain=prerouting action=mark-packet new-packet-mark=hotspot1-up passthrough=no in-interface=hotspot1 

 1   chain=postrouting action=mark-packet new-packet-mark=hotspot1-down passthrough=no out-interface=hotspot1

Then created PCQ queue types for 1meg down, 512k up:

[admin@MikroTik] > /que ty pr where name~"^hotspot1"
 0 name="hotspot1-up" kind=pcq pcq-rate=512000 pcq-limit=50 pcq-classifier=src-address pcq-total-limit=2000 

 1 name="hotspot1-down" kind=pcq pcq-rate=1024000 pcq-limit=50 pcq-classifier=dst-address pcq-total-limit=2000

And then some queue trees that use them:

[admin@MikroTik] > /que tr pri
Flags: X - disabled, I - invalid 
 0   name="hotspot1-up" parent=global-in packet-mark=hotspot1-up limit-at=0 queue=hotspot1-up priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s 

 1   name="hotspot1-down" parent=global-out packet-mark=hotspot1-down limit-at=0 queue=hotspot1-down priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s

Then logged into the Hotspot via RADIUS with a Mikrotik-Group attribute set to 'profile1'. No simple queues were created for the user, then ran a test against speedtest.net, the rate limits were applied correctly and only the queue trees were counting packets - the simple queue created for the Hotspot itself was not being used.

I'll try and test with more users at some point in time. I don't know how much less resource intensive this is than the normal Hotspot setup, but according to all documentation queue trees and PCQ are much more effective. It's a little more work to have to track mangle rules every time you add a profile and having to edit queues instead of profiles for rate limits, and it's not great to not be able to specify rate limits dynamically via RADIUS - but I guess in my situation very large Hotspots are OK with those limitations and the smaller ones can be run with default settings.

I also briefly tried using incoming-packet-mark/outgoing-packet-mark and the address-list properties of Hotspot User Profiles to create the marking to be able to have more granular control over how packets are marked so that more than one profile can be used per Hotspot but that didn't work out too well - I'll try again later.

dssmiktik · Wed Nov 18, 2009 9:23 pm

Just an addition also:

If you use Hotspot in v4.1, when a user logs in, you can add their client IP to an address-list. You can then control traffic to/from against that address list using mangle without necessarily specifying the interface.

What's new in 4.1:

*) hotspot - added support for dynamic address-list entries;

I'm not sure if this is quite what you're looking for though.

fewi · Wed Nov 18, 2009 9:29 pm

I can't seem to get that to work. I specify the address-list argument and log in, and no address-list entry is created for the host. No difference between using an address-list that doesn't vs that does exist.

I can't get the incoming-packet-mark and outgoing-packet-mark arguments to work, either. That would be ideal because then you wouldn't have to manually mangle at all, the queue would just pick things up automatically. I don't see any dynamic mangle rules created (which is how I'd expect it to work), and mangle rules configured in the prerouting chain with an action of log targeting those packet marks don't log anything. Is there anything special that has to be done to get those arguments to work?

dssmiktik · Wed Nov 18, 2009 9:42 pm

I can't seem to get that to work. I specify the address-list argument and log in, and no address-list entry is created for the host. No difference between using an address-list that doesn't vs that does exist.

I can't get the incoming-packet-mark and outgoing-packet-mark arguments to work, either. That would be ideal because then you wouldn't have to manually mangle at all, the queue would just pick things up automatically. I don't see any dynamic mangle rules created (which is how I'd expect it to work), and mangle rules configured in the prerouting chain with an action of log targeting those packet marks don't log anything. Is there anything special that has to be done to get those arguments to work?

Hmm... on my v4.2 address-list works good. If the list exists, it appends the client IP. If the list doesn't exist, it creates it.

[doug@MainMKT1] /ip hotspot user> print detail
Flags: X - disabled, D - dynamic
 0   name="doug" password="<hidden>" profile=Trusted-Users email="<hidden>"
     uptime=9w6d12h2m27s bytes-in=18285027636 bytes-out=111736274800 packets-in=88641593
     packets-out=113149062

[doug@MainMKT1] /ip hotspot user profile> print
Flags: * - default
 0 * name="default" idle-timeout=12h status-autorefresh=1m shared-users=2 transparent-proxy=no
 1   name="Trusted-Users" idle-timeout=5m status-autorefresh=1m shared-users=unlimited
     address-list="Trusted_Hosts" transparent-proxy=no

[doug@MainMKT1] /ip hotspot active> print
Flags: R - radius, B - blocked
 #    USER                          ADDRESS         UPTIME       SESSION-TIME-LEFT IDLE-TIMEOUT
 0    doug                          192.168.101.170 15s                            5m

[doug@MainMKT1] /ip firewall address-list> print where list=Trusted_Hosts
Flags: X - disabled, D - dynamic
 #   LIST                                                       ADDRESS
239 D Trusted_Hosts                                              192.168.101.170

fewi · Wed Nov 18, 2009 9:57 pm

Ah - the moment I use a local user account instead of RADIUS authentication it works fine for both of them (dynamic mangle rules get created for incoming-packet-mark and outgoing-packet-mark, and the address-list entry is created), so this seems to not be supported by RADIUS.

Thinking about it, in RADIUS the Mikrotik-Mark-Id attribute could probably be used for this, though. I'll play with that.

Edit: that creates two rules per user and would probably suffer the same performance penalties as with lots of users the ruleset would grow very large. I'll ask support if they can either add a RADIUS attribute that specifies a dynamic address-list to add to, or to enable address-list inheritance from the profile if a profile is set via RADIUS.

dfwair · Wed Nov 18, 2009 10:47 pm

I am going to play with this and see what I can pull out of this information. The problem that I am running in to is that we have different service levels based on different prices, and the package set in the billing system sets those parameters. Where it gets even more confusing is that we have grandfathered plans from 10 different providers with an average of 5 plans that each provider was using. Yes, some of them were very similar, but then again there are a bunch that aren't the same.

So I can see that based on this information you have given me, I should be able to do one limit for an entire hotspot, but I will play with it some more and see what can be done.

fewi · Thu Nov 19, 2009 6:32 pm

From support: There is a new RADIUS attribute that isn't documented anywhere yet, 'Mikrotik-Address-List' with vendor 'Mikrotik', id 19 and type 'string'.

Tested and it works.

dssmiktik · Fri Nov 20, 2009 10:59 pm

From support: There is a new RADIUS attribute that isn't documented anywhere yet, 'Mikrotik-Address-List' with vendor 'Mikrotik', id 19 and type 'string'.

Tested and it works.

Nice! This is what I like to hear

Thank you, and thank you Mikrotik.

fewi · Fri Nov 20, 2009 11:26 pm

OK, got done testing, using RADIUS attributes for address-lists to use PCQ instead of simple queues seems to work. Unfortunately I can't test it in a larger environment for a while to see whether or not this really performs better under larger user loads. Maybe someone else can verify or check this in the meantime.

Here the setup:

Simple default Hotspot on an interface.

[admin@MikroTik] > /ip hotspot profile print where name=hotspot
Flags: * - default
 1   name="hotspot" dns-name="hotspot.example.com" html-directory=hotspot rate-limit="" login-by=https ssl-certificate=wildcard use-radius=yes radius-accounting=yes
[admin@MikroTik] > /ip hotspot print detail
Flags: X - disabled, I - invalid, S - HTTPS
 0 S name="hotspot" interface=hotspot1 profile=hotspot idle-timeout=none keepalive-timeout=none ip-of-dns-name=10.1.0.1 proxy-status="running"

Two user profiles without much information on them, I guess they don't really have to exist for this but we often have cases where the shared-users property is set to something other than unlimited.

[admin@MikroTik] > /ip hotspot user profile print where name~"^pro"
Flags: * - default
 1   name="profile1" idle-timeout=none keepalive-timeout=2m status-autorefresh=1m shared-users=unlimited transparent-proxy=no
 2   name="profile2" idle-timeout=none keepalive-timeout=2m status-autorefresh=1m shared-users=unlimited transparent-proxy=no

Here the RADIUS debug from the router side:

13:15:52 radius,debug,packet sending Access-Request with id 31 to 1.1.1.2:1812
13:15:52 radius,debug,packet     Signature = 0x08edbdab79838cb24353d0cd0b03e0c6
13:15:52 radius,debug,packet     NAS-Port-Type = 19
13:15:52 radius,debug,packet     Calling-Station-Id = "00:1E:52:87:F4:4A"
13:15:52 radius,debug,packet     Called-Station-Id = "hotspot"
13:15:52 radius,debug,packet     NAS-Port-Id = "hotspot1"
13:15:52 radius,debug,packet     User-Name = "banana"
13:15:52 radius,debug,packet     NAS-Port = 2149580801
13:15:52 radius,debug,packet     Acct-Session-Id = "80200001"
13:15:52 radius,debug,packet     Framed-IP-Address = 10.2.0.254
13:15:52 radius,debug,packet     MT-Host-IP = 10.2.0.254
13:15:52 radius,debug,packet     User-Password = 0x7065656c
13:15:52 radius,debug,packet     Service-Type = 1
13:15:52 radius,debug,packet     WISPr-Logoff-URL = "http://10.1.0.1/logout"
13:15:52 radius,debug,packet     NAS-Identifier = "MikroTik"
13:15:52 radius,debug,packet     NAS-IP-Address = 1.1.1.3
13:15:52 radius,debug,packet received Access-Accept with id 31 from 1.1.1.2:1812
13:15:52 radius,debug,packet     Signature = 0x74253fd3958d99e3025a917a6f0ec25e
13:15:52 radius,debug,packet     Idle-Timeout = 600
13:15:52 radius,debug,packet     Service-Type = 512
13:15:52 radius,debug,packet     Connect-Info = "Unknown"
13:15:52 radius,debug,packet     Calling-Station-Id = "profile1"
13:15:52 radius,debug,packet     MT-Group = "profile1"
13:15:52 radius,debug,packet     MT-Address-List = "profile1"
13:15:52 radius,debug received reply for 3f:27
13:15:52 hotspot,debug banana (10.2.0.254): Access-Accept from RADIUS
13:15:52 hotspot,debug banana (10.2.0.254): user profile <profile1> from RADIUS
13:15:52 hotspot,debug banana (10.2.0.254): using profile <profile1>
13:15:52 hotspot,debug banana (10.2.0.254): address list <profile1> from RADIUS
13:15:52 hotspot,debug banana (10.2.0.254): idle timeout <600> from RADIUS
13:15:52 hotspot,debug banana (10.2.0.254): adding ip->user binding
13:15:52 hotspot,debug banana (10.2.0.254): adding address-list rule <profile1>
13:15:52 hotspot,account,info,debug banana (10.2.0.254): logged in

And a view of the address-list that the IP was added to:

[admin@MikroTik] > /ip firewall address-list print detail where list~"^pro"
Flags: X - disabled, D - dynamic
 0 D list=profile1 address=10.2.0.254

Only two mangle rules are needed for each profile, one for downstream, one for upstream:

[admin@MikroTik] > /ip firewall mangle print
Flags: X - disabled, I - invalid, D - dynamic
 0   chain=prerouting action=mark-packet new-packet-mark=profile1-up passthrough=yes src-address-list=profile1 hotspot=from-client
 1   chain=postrouting action=mark-packet new-packet-mark=profile1-down passthrough=yes dst-address-list=profile1 hotspot=to-client
 2   chain=prerouting action=mark-packet new-packet-mark=profile2-up passthrough=yes src-address-list=profile2 hotspot=from-client
 3   chain=postrouting action=mark-packet new-packet-mark=profile2-down passthrough=yes dst-address-list=profile2 hotspot=to-client

Two queue types are needed for each profile, though I guess queue types could be shared if the rate limit per stream is the same. On a system with many users pcq-total-limit would have to be raised significantly:

[admin@MikroTik] > /queue type print where name~"^pro"
 0 name="profile1-up" kind=pcq pcq-rate=512000 pcq-limit=50 pcq-classifier=src-address pcq-total-limit=2000
 1 name="profile1-down" kind=pcq pcq-rate=1024000 pcq-limit=50 pcq-classifier=dst-address pcq-total-limit=2000
 2 name="profile2-up" kind=pcq pcq-rate=256000 pcq-limit=50 pcq-classifier=src-address pcq-total-limit=2000
 3 name="profile2-down" kind=pcq pcq-rate=512000 pcq-limit=50 pcq-classifier=dst-address pcq-total-limit=2000

And the queues themselves:

[admin@MikroTik] > /queue tree print
Flags: X - disabled, I - invalid
 0   name="profile1-up" parent=global-in packet-mark=profile1-up limit-at=0 queue=profile1-up priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s
 1   name="profile1-down" parent=global-out packet-mark=profile1-down limit-at=0 queue=profile1-down priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s
 2   name="profile2-up" parent=global-in packet-mark=profile2-up limit-at=0 queue=profile2-up priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s
 3   name="profile2-down" parent=global-out packet-mark=profile2-down limit-at=0 queue=profile2-down priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s

The PCQ rate limits are applied correctly and the simple queues that still exist for the Hotspot traffic before the user is authenticated do not register traffic after authentication.

2fast4youbr · Fri May 17, 2013 7:38 pm

Fewi,

Very nice post!!!

I need to do the same... but this post is from 2009, today would be in the same way ?

cheers

Request for change in Hotspot settings

Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Re: Request for change in Hotspot settings

Who is online