MikroTik

Hello there, we recently installed an RB333 running RouterOS V4 3.0RC14 + XR2 card for PtMP operation. We have about 30 clients, mostly Tranzeo CPQ clients. We use WPA/AES encryption. All clients have the newest firmware installed.

Everything was working great with the new system for almost 5 days without a hiccup, then suddenly, all but one or two clients disassociated and are now showing the following errors while they try to re-associate:

echo: wireless,debug WFCB-EAST: 00:60:B3:5D:5B:FD not in local ACL, by default accept
echo: wireless,debug WFCB-EAST: 00:60:B3:28:7C:F5 attempts to associate
echo: wireless,debug WFCB-EAST: 00:60:B3:28:7C:F5 not in local ACL, by default accept
echo: wireless,debug WFCB-EAST: 00:60:B3:39:30:41 attempts to associate
echo: wireless,debug WFCB-EAST: 00:60:B3:39:30:41 not in local ACL, by default accept
echo: wireless,debug WFCB-EAST: 00:60:B3:28:6C:D1 attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:28:6C:D1, banned (last failure - group key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:3D:97:3F attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:3D:97:3F, banned (last failure - unicast key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:30:9F:DC attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:30:9F:DC, banned (last failure - received deauth: 4-way handshake timeout (15))
echo: wireless,debug WFCB-EAST: 00:60:B3:45:32:DC attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:45:32:DC, banned (last failure - group key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:01:F9:53 attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:01:F9:53, banned (last failure - group key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:28:6C:D1 attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:28:6C:D1, banned (last failure - group key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:3D:97:3F attempts to associate
echo: wireless,debug WFCB-EAST: 00:60:B3:3D:97:3F not in local ACL, by default accept
echo: wireless,debug WFCB-EAST: 00:60:B3:30:9F:DC attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:30:9F:DC, banned (last failure - received deauth: 4-way handshake timeout (15))
echo: wireless,debug WFCB-EAST: 00:60:B3:5D:5B:FD attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:5D:5B:FD, banned (last failure - group key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:28:7C:F5 attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:28:7C:F5, banned (last failure - group key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:39:30:41 attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:39:30:41, banned (last failure - group key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:45:32:DC attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:45:32:DC, banned (last failure - group key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:01:F9:53 attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:01:F9:53, banned (last failure - group key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:28:6C:D1 attempts to associate
echo: wireless,debug WFCB-EAST: 00:60:B3:28:6C:D1 not in local ACL, by default accept
echo: wireless,debug WFCB-EAST: 00:60:B3:30:9F:DC attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:30:9F:DC, banned (last failure - received deauth: 4-way handshake timeout (15))
echo: wireless,debug WFCB-EAST: 00:60:B3:5D:5B:FD attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:5D:5B:FD, banned (last failure - group key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:28:7C:F5 attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:28:7C:F5, banned (last failure - group key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:39:30:41 attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:39:30:41, banned (last failure - group key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:45:32:DC attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:45:32:DC, banned (last failure - group key exchange timeout)
echo: wireless,debug WFCB-EAST: 00:60:B3:3D:97:3F attempts to associate
echo: wireless,debug WFCB-EAST: reject 00:60:B3:3D:97:3F, banned (last failure - group key exchange timeout)

The list goes on and on with the majority of clients attempting to associate, then being dropped due to the following errors:
exchange timeout or received deauth:
4-way handshake timeout (15) error.
group key exchange timeout
unicast key exchange timeout

We've tried a number of things like adding some of the clients that won't associate to an ACL (we have not been using one), changing preamble type, allow both tkip and/or AES cypers with WPA, TKIP only, turning WMM off; none of these changes worked in getting any more people associated. We ended up changing back to all of our original settings.

We've also tried changing the channels with limited results. Originally we were seeing a -95 noise floor on CH5, we then changed to CH9 (-92 noise). After changing to CH9 we saw a few more people associate, we then changed back to CH5 and even more people were able to associate. Then again within about 10 minutes of changing back to CH5, almost everyone had disassociated again. We noticed that right after everyone disassociates, the noise floor goes to -89. Also, during these "storms", the TX CCQ on the AP goes down below 20%; during normal operation it typically is around 70-80%.

We've also checked the list of MAC's trying to associate and there are no anomalies there. All MAC's trying to associate are known, working good clients. No new clients have been installed, or changed. Before the Mikrotik we had a Tranzeo AP at this site for over a year with none of these types of issues.

Any ideas? Thanks in advance!

Change your hardware retries to 4 and upgrade to 3.0 full version.

Thanks for the reply. The hardware retries are currently set to 8. I'm not sure what you mean by upgrade to 3.0 full version? I believe I have the latest and greatest installed @ 3.0rc14. Any other ideas?

You are using rc14. The full release of 3.0 not beta is now available for download.

Downloaded and updated Mikrotik to 3.1. Clients still are showing the same association issues in the log file after rebooting the AP. We discovered that if the clients having issues power cycle their systems they seem to associate correctly. I hope the recent updates fix whatever caused this mystery issue. My guess is that there's some sort of WPA-PSK synchronization issue between the AP and the clients.

Set your hardware retries to 4.

Set your hardware retries to 4.

Can you explain why?

-----
Hai frens
i am so sorry, my suggestion is seven, in javanese Seven=PITU same PITULUNGAN or an english is "Will make help you"
all my roamimg machine set it to seven, and unitl now great.... and nice maybe since 8 months ago.
again, thanks Mikrotik and Team we stay on v2....49.
also, a week ago i have success to made link by wds to next 4th AP will supply my custome far far ways....

regaard
Hasbullah.com
-----

Set your hardware retries to 4.

4 is Mikrotik's suggestion for disconnect issues...

----
yes,
cause mikrotik in Latvia, and my suggestion in JAVANESE.
oh....my god. thanks for your help me to make nice sleep until now....

regards
Hasbullah.com
----

4 is Mikrotik's suggestion for disconnect issues...

The original issues went away after we updated ROS, and had the clients power cycle. Now we're having these issues again with a different, newly installed base station. Exact same configuration as in my first post, except now we're running ROS 3.6. I've tried upgrading to 3.7, and tried tweaking settings with no fix yet. The difference is that roughly 50% of the clients are disassociating after approx. 24 hours, and do not reconnect until they're power cycled. The same errors listed in my first post are being displayed in the log for the clients that cannot re-associate unless they power cycle. Could this be some sort of Tranzeo CPQ + WPA1/AES + Mikrotik/XR2 issue?

Yes, change your hardware retries to 4 and you should be fine.

They are, and always have been set to 4. I double checked just to make sure

So change back to 15 :-p

Not sure what you mean by 'change back to 15'. I tried upping the hardware retries to 5-10 without any positive results. It seems as though additional hardware retries do not fix the issue, and the few clients that are able to associate do so much slower compared to the recommended number of 4. I still really would like to know what the errors that I mentioned mean, and what they pertain to. It seems as though the Mikrotik knows that the clients in question are trying to associate, but rejecting them for some reason. The log messages are not verbose enough, and there is not enough information online to decipher what they mean or how to resolve them. This seems to be a very rare issue.

At this point I'm leaning towards the possibility that it may be a bad XR2 card, as we've deployed many more of these units now, without issue. In fact, the same Mikrotik box has a second XR2 card serving another sector with exactly the same settings, and client equipment associating without any of these issues. Going to try swapping this out soon and see if it does any good. Please let me know if you have any further suggestions. Thanks!

We are seeing the same issue. We took down an AP and moved it to a new pole and now all of the Tranzeo CPE simultaneously disconnect at random periods. The Mikrotik CPE stay associated.

Hardware retries=4. Firmware = 3.7. TxPower backed off to 19 (12) card rates.

Anyone?

It appears as though more and more people are having this issue. Can you set your wireless logging to debug mode to make it more verbose and check to see if you're getting any of the same errors that I'm getting? Look for stuff like:

exchange timeout or received deauth:
4-way handshake timeout (15) error.
group key exchange timeout
unicast key exchange timeout

we either need access to one of those APs with the disconnections, or we need one such tranzeo device to repeat the problem here in testing. So far support hasn't received any valuable information on this. Please email support so we can start working on it.

Sent in all information to Mikrotik support, including supout.rif. Waiting for response.

i am seeing same issue and with mostly mikrotik clients and we now have a few nano stations in the mix
i will aslo send a autosupout

Same issue here with RB333 and R52H.
Different client cards, but same result: "unicast key exchange timeout" all day long...

TL

I've submitted this issue to Mikrotik, Tranzeo, and Ubiquiti. Their responses: Mikrotik basically said it was a Tranzeo issue, and they couldn't do anything about it. They claim it has something to do with how fast the AP/ROS can process registration/authentication requests. The Tranzeo is requesting too quickly and the AP/ROS can't keep up. Noise also could be a factor. Tranzeo responded with bewildered confusion and some strange questions that was completely unrelated (which I answered in detail) and then I never heard from them again. Ubiquiti never responded.

I ended up moving the AP to a different channel, which has resolved the random disconnects that were occurring every few hours. I rarely get the "unicast key exchange timeout" error anymore; and if the clients do disassociate, they immediately come right back without a power-cycle on the client end.

Strange thing is... we originally had a Tranzeo TR-6500 on that sector and it never had any of these types of issues. Only after we swapped to Mikrotik/ROS with RB333+XR2 did we start having these issues. The noise floor is a little bad, hovering around -94 to -98, but we have other sectors with worse noise that run this same Mikrotik config that don't have any of these issues. This sector now resides on a channel with worse noise then the one it was having disconnect issues on. I'm wondering if there's some unknown, intermittent noise 'knocking' the clients off.

There's a Motorola 2.4GHz Canopy system on a near by butte... but again, we're running Mikrotik/ROS on other sectors that face this same site with no issues. So strange...

At this point it appears all three vendors have no real will to help figure out and resolve the issue(s).

Ok, we are seeing this same issue with all Mikrotik Gear.

We've recently been putting up installations using an assortment of MT Access Points (RB 532 and RB 333) and an assortment of radio cards (8602S, SR5, CM9). The clients are also mainly RB 411s and some RB 133s.

We've been trying to run exclusively ROS 3.10 with the latest firmware. The issue we've been seeing is that the clients will connect and run for a while (anywhere from 2 - 24 hrs), but will then disconnect and go into some kind of mode where they just bounce on and off and will not remain associated. After a client power cycle, the client will reconnect and remain on for a short period of time before going into the disconnect / reconnect mode.

We've seen this on several different routerboards and the common denominator seems to be ROS 3.10. We will try downgrading one of the affected clients (RB 133/SR5) to ROS 2.9.51 tomorrow morning and see if this resolves the issue.

We run hundreds of APs and clients throughout our network and are seeing this on some known good APs that have had solid clients associated with them for some time. It's just the new clients that we've put up recently and we're still trying to nail down the common denominator.

Is anyone else still seeing this on ROS 3.10? We thought it could be related to the 8602S cards that are new to us or the RB411 boards that we haven't used a whole lot of before, but now after seeing it on an RB133/SR5 combo, I'm guessing it may be ROS 3.10 related.

Have you gotten any help on this? Where's the support from Mikrotik? There are several threads about this issue with no real answers. One solution was to change the channel - but what if that's not possible and what if there is no other noise on the channel I'm currently using... Another solution was to downgrade to 2.9.51 - that tells me it is a problem in the 3.n versions where's the acknowledgement and fix... Another solution was to turn off security - is that really a solution... Another was recommended to email the support team and then later posted a response from Mikrotik that didn't seem to fix the problem... Several have recommended to change the group update rate - no one has found this to help...

I'm running an AP and a WDS slave with about 6 clients connected to the primary AP and about 4 clients connected to the WDS slave... The clients are primarily NS2s and a few RB133s w/ R52H or crossroads... The WDS slave is the one on my network that is getting the unicast key exchange timeout from the primary AP... when I check the WDS slave, it thinks it has remained associated the entire time... Disabling the wireless interface on the WDS slave and re-enabling it temporarily fixes the problem (couple hrs to a whole day), but the problem always comes back. I've tested the network having all the mikrotik nodes at 3.9, all at 3.10 and now all at 3.11 with the same error. I am now planning to downgrade them all to 2.9.51 to see if that fixes the problem...

We were hesitant to downgrade to 2.9.51 as we are hoping to expand this access point and bring on some of the new routerboards, so we decided to switch our security over to WEP and have stayed up for 1.5 days so far. I don't consider this a solution, just a workaround until someone from Mikrotik responds with some answers.

Hello all,

I am having the same isue as posted but with some different layout.

AP: Mikrotik RouterBoard RB433AH with ROS 3.20 level 5 (mode ap-bridge with dynamic wds, wpa2 enabled, mac authentication using radius)

Client WDS: Ubiquiti NS5 v3.3.1

When I made tests using this both I put security wpa2 and it worked just fine. When I remove "Default Authenticate" from RB433AH them it send mac to radius that returns ok to that but, in the mean time, MK returns to NS5 that the key was timeout.

I checked all configuration including NTP/Clock settings that are all ok.

I actualy don't know what can I do to solve this problem. Any ideas?

Also seen this issue, tried digging around and people were saying it was a signal quality issue. I have seen it with RB/600's + r52h or XR5's in 5.5-5.8ghz range.

Signal's from -68db to -79db (at least 5 different connections) all of them had so-so LOS (perhaps trees in the way).

Changing Antenna, cards / cables didn't make a difference.
Also tried changing ACK, Signal rates, radio power, etc...
Nothing worked. Had to break up links with shorter hops.

I have the same problem, with clients associated with my "ap bridge". If the clients having issues power cycle their systems they seem to associate correctly.

RB433 + ROS 3.27 + R52 and R52H cards. Hardware retries set to 4.

Anyone help me?
tks

again, enable wireless debug logs and post them here
http://wiki.mikrotik.com/wiki/Wireless_Debug_Logs

------
hello frens,

thanks your any suggestion for, but for me nice to sleep. and until now all my unwires system and Centralized AAA still in v2.9.49 as AP-Bridge to Station and AP-Bridge to AP-Bridge and parameter setting for HW-RETRIES=7

again, thanks to Mikrotik & Teams
regards
Hasbullah.com
-------

I have the same problem, with clients associated with my "ap bridge". If the clients having issues power cycle their systems they seem to associate correctly.

RB433 + ROS 3.27 + R52 and R52H cards. Hardware retries set to 4.

Anyone help me?
tks

I have the same trouble



RB 433UAH - ROS 4.2 - R52N <--> Asus EEE PC 1005AH(Atheros AR9285) with Windows XP Home.

hmm, but my Dell XPS 1330 with Intel 4965AGN with Vista Ultimate - work fine

sorry - my problem was in Windows XP.
I was disable IPSec Service and Wireless Zero Configuration

I've seen several reports of this deauth issue, and I believe what they all have in common are Tranzeos and UBNT XR2 cards. I have experienced it myself at two towers, both times with this combination (see this thread).

I experienced it again last night, and theorized that the problem was that the XR2 was going weak, because the nearest Tranzeos stayed online, the mid-distance units kept associating and then de-authing, and the most distant units never even attempted to register.

I replaced the radio card this morning and everything is rainbows and unicorns again.

I thought I would document this possibility for all you others who have experienced the same problem.