Community discussions

MikroTik App
 
MyThoughts
Member Candidate
Member Candidate
Topic Author
Posts: 218
Joined: Sat Sep 17, 2005 9:07 pm

RB600 Reboot Issue - Revisit

Wed Dec 02, 2009 7:40 am

The past month I have been noticed 5 of my RB600s in deployment were rebooting at seemingly random times. The longest uptime I noticed was just over 2 days, with most units rebooting 2-3 times per day. This is in stark contrast to many of the other RB600s which have up-times well over 200 days.

We tried all the recommended tips, newest firmware, latest version of RouterOS, disable watchdog. None of this helped.

After attempting to alter some wireless setting on one of the units, it hard locked.

We decided that while said router was down we would pull all of the problematic units from the cell site.
After getting the units taken apart we discovered a very unnerving issue. All the problem units had the same 5 capacitors leaking electrolyte.

I would recommend to anyone that is still having RB600 reboot issue to physically inspect their board where it is feasible.

If this is a known issue (all my boards had serial numbers quite close to eachother) I have yet to hear anything about it. If their is a known problem with a specific range of serial numbers it would be great to get that info.

Cheers
 
aaa
Member
Member
Posts: 458
Joined: Mon Apr 11, 2005 11:54 am
Location: Latvia, Baltezers

Re: RB600 Reboot Issue - Revisit

Wed Dec 02, 2009 12:06 pm

Can you make leaking capacitors photo and fault RB s/n list? Else in what configuration you use this RB ( case,POE/Jack, daughterboard, wireless cards )?
 
SurferTim
Forum Guru
Forum Guru
Posts: 4636
Joined: Mon Jan 07, 2008 10:31 pm
Location: Miramar Beach, Florida

Re: RB600 Reboot Issue - Revisit

Wed Dec 02, 2009 1:36 pm

Take a look at the caps. See if any have the manufacturer/rating showing. They should have a working voltage (WV) of at least 35vdc for this application (by my standards anyway). Normally leaking electrolyte is overvoltage or, on polarized caps, reverse voltage. Maybe wrong caps got loaded into the machine for that batch?

The boards were all on different power supplies, correct?

ADD: Does anybody have experience at connecting any of the RB routers reverse polarity? Any RP protection onboard? Or does it launch the caps into orbit?

WARNING! Don't try it just to check, unless you are wearing safety glasses and full body protection. They will blow the sticky-as-glue lava-hot electrolyte as far as 6 feet (2m).
 
aaa
Member
Member
Posts: 458
Joined: Mon Apr 11, 2005 11:54 am
Location: Latvia, Baltezers

Re: RB600 Reboot Issue - Revisit

Thu Dec 03, 2009 12:42 am

All RB have reverse polarity protection. Include POE connection.
 
MyThoughts
Member Candidate
Member Candidate
Topic Author
Posts: 218
Joined: Sat Sep 17, 2005 9:07 pm

Re: RB600 Reboot Issue - Revisit

Thu Dec 03, 2009 7:13 am

I'll take some pictures and post them when I get a chance.

The boards in question were all configured identically.

1 x XR5
1 x XR2
Using Routerboard indoor aluminum case for RB600
48V 1A PoE injector (each board had its own PoE injector)
The each RB indoor case was grounded

The capacitors in question are identical in rating and they seem to be the identical capacitors on the new RB600s we replaced these with. They were not installed backwards.

I have seen leaking electrolyte quite a bit. Back in the 2001-2002 Panasonic capacitor fiasco, we got tens of system mainboards with similar issue.

The biggest problem with bad electrolyte is it does not manifest right away, it can take months, sometimes years to leak or bow to the point that it becomes an issue. The RB600s in question are just over 1 year old (installed in Aug 2008 if I remember correctly). If this becomes widespread it is a major problem as technically the boards were received with the unknown manufacturers defect but by the time it causes issues the warranty period is over.

The capacitors experiencing the leak seem to all be related to the MPCI slots, without any cards installed the RB600s are much more stable, but as cards are added the problem gets much worse, with 2 high power cards installed (XR2 and XR5) when you attempt to transmit using both simultaneously it triggers an immediate reboot and/or lockup.
 
MyThoughts
Member Candidate
Member Candidate
Topic Author
Posts: 218
Joined: Sat Sep 17, 2005 9:07 pm

Re: RB600 Reboot Issue - Revisit

Tue Dec 15, 2009 10:45 pm

Finally got around to taking a picture of one of the boards.

Image

I was able to get 3 of the serial numbers before the boards went out for repair.

189801FD8323/827
189801FD83E6/827
189801FD8332/827

Cheers
 
aaa
Member
Member
Posts: 458
Joined: Mon Apr 11, 2005 11:54 am
Location: Latvia, Baltezers

Re: RB600 Reboot Issue - Revisit

Wed Dec 16, 2009 10:49 am

It seems it's capacitor quality problem same like PC mainboard. Simple replace capacitors with same capacitance an that's all.
Thanks!
 
MyThoughts
Member Candidate
Member Candidate
Topic Author
Posts: 218
Joined: Sat Sep 17, 2005 9:07 pm

Re: RB600 Reboot Issue - Revisit

Fri Dec 18, 2009 10:59 pm

Easy to fix, hard problem to find.

When boards are in operation mounted in outdoor enclourses, its a serious pain the ass to find the issue and/or check to see if it is a problem. The other issue is it takes time for capacitors to visibility show the problem, the capacitors don't go bad they are manufactured bad. So technically the boards were DOA.

With the number of reports of RB600 reboot issue, and the fact I had this many boards with this problem I would wager that there are alot more RB600s that have the same defect.

Anyone in the electronics industry will remember the 2000-2005 capacitor issue that plagued all kinds of devices. Some PC mainboard companies nearly went backrupt from the problem, that is why so many mainboard manufacturers switched to solid state capacitors.

I would gladly pay the small 10-20% premium and have all future RouterBoards designed with the more reliable solid state capacitors.

Cheers
 
ayger
just joined
Posts: 12
Joined: Sun Nov 20, 2011 7:44 pm

Re: RB600 Reboot Issue - Revisit

Sun Nov 20, 2011 8:21 pm

Hi all,

After beeing visitor in the forums for a long time (thank you! community for the tons of information!) i've decided to describe in detail my "adventure" with RB 600A's.


Please bear with me to describe the current status:
My network consists of several 433's, 411's and two RB 600A which are the main routers for my network, with 4 wireless interfaces each. Main traffic is internet sharing (2x2mpbs adsl), with strict profiles,destination ports, and services. Mainly web browsing and VoIP. No shares between clients are allowed.

The RB 600A's were rebooting with no apparent reason in random times and random bandwidth usage, after almost 19 months of painless operation. My peak bandiwth usage is 4Mbps (the total bw that my provider can provide me).Apparently, 600's are capable to handle more traffic than this .

Firmware and software revision were 2.17 and 3.25 (not sure for the first number).My philosophy : If ain't broken, don't fix it.

Taking a closer look after after the point were the end users complaining about dropped connectivity, in my syslog (centos) server no clues were stored. After logging in the rb 600, i could only see "system error critical : router was rebooted without proper shutdown (cause 1)".

Apparently this is a generic message.

Actions taken :

-Replaced my UPSed power supplies (48V, 1A) with brand new ones. No change. Reboots were taking placing constantly in random times, cpu loads and network traffic.

- Upgraded firmware and software versions, as per tech support of mikrotik suggested. No luck.

- Clearing configuration and setting up the router again. No progress. Reboots were taking place again.

Since the routers are located to remote sites, and they are sealed in IP67's cases, i've decided to climb to the tower and check them closely.

I had in mind the capacitor problem that i've read in the forums and i thought "no way it could be happened to me".

Please check the picture. At least 3 (one not showing in the pic) capacitors are "brown" on the top.Should i consider them as failed?


For the moment, i clocked down both routers at 266Mhz and running the wireless interfaces with the minimum tx power needed for my links (below 8db). Situation got better, but i have still random reboots, mainly in the 1-3 hour time window. It's better than rebooting in 1-20 minutes frequency. (...)

Unfortunatelly, i don't have spare boards to replace them.

I have to take them down (and therefore my network will be down, since these are my backbone routers) in order to repair them...

2/2 600A's failed for me. Murphy's law i suppose.

best regards,
A.


Image
 
MyThoughts
Member Candidate
Member Candidate
Topic Author
Posts: 218
Joined: Sat Sep 17, 2005 9:07 pm

Re: RB600 Reboot Issue - Revisit

Mon Nov 21, 2011 3:35 am

Yes those are bad electrolytic capacitors. The bad electrolyte will cause the capacitors function at a reduced capacitance, which leads to the instability of the RB600s.

In my testing dropping the Tx power of the radio cards does help, however the problem will only get worse as the capacitors degrade and continue to lose capacitance. Despite the pain it is to change them on a tower, and the obvious outage it would cause your best bet is to get them replaced as soon as possible. I would suggest replacing them with newer boards. I've fixed many RB600s by replacing the caps with new ones but unfortunately they continued to be unreliable by my standards.

In another sad note, I replaced my 1st RB411 the other day that had bad caps. I am sincerly hoping it was a one off, as I will be looking to a new company to do business with if the thousand or so I've deployed all start to fail.

Cheers
 
ayger
just joined
Posts: 12
Joined: Sun Nov 20, 2011 7:44 pm

Re: RB600 Reboot Issue - Revisit

Mon Nov 28, 2011 12:45 pm

Hi to all,

Does somebody has a full list with the capacitor specifications (voltage rating and capacity) ?
My boards are still on the tower, have no spare, and i will try to repair them on site. (200 km away from here)

I'll buy in advance the capacitors, so i'd be *GREATFUL* if this piece of information is available.


Thanks.
 
ayger
just joined
Posts: 12
Joined: Sun Nov 20, 2011 7:44 pm

Re: RB600 Reboot Issue - Revisit

Thu Dec 01, 2011 11:50 am

For future reference, if somebody needs the info.

The spec's of RB600 capacitors are :

four 680uF 6.3V
three 100uF 63V