RB450G + openwrt Metarouter strange problem

webasdf · Fri Oct 16, 2009 5:24 pm

I compiled my own version of openWRT for Metarouter according to the instructions found in the Metarouter wiki article. I used this metarouter image with no problems at all on a 433AH (mikrotik firmware 3.30). When I tried the same image on the 450G (both 3.30 and 4.0 firmware), I get the following strange behavior:

Metarouter starts fine on router boot
After anywhere between 5 and 45 minutes, the router reboots itself
I turned on logging and saw "system error - router was rebooted without proper shutdown by watchdog timer" (Kernel panic from RouterOS??)
Sometimes the meta router also locks up on the 450G and I have to manually disable and re-enable the metarouter (effectively rebooting it) to get it working again.

Has anyone else seen these problems in this configuration? I tried Mikrotik 3.30 and new 4.0 firmware and still same strange issues. Any ideas on how to correct this?

janisk · Mon Oct 19, 2009 10:12 am

can you get more information about your router? i am trying to reproduce it on my test RB450G with 2.23 boot loader 4.1 RouterOS and it does not hand or reboot.

edit: also, what is CPU frequency of that board?

webasdf · Mon Oct 19, 2009 4:22 pm

CPU frequency is 680MHz.
I'm not sure on the boot loader version. Is this only visible on the serial interface? I'm not local to the unit as I work off-site, but I can get someone to serial into the 450G to find this out if you think it is important.

I've noticed that problems seems to happen when I'm doing network I/O on the Metarouter. If no network I/O on the metarouter happens, it is very stable. But, once I SSH into the metarouter and start doing a top or other command line tasks, it only lasts a little while before lock up or reboot. One thing I did differently just a few minutes ago was to re-upload the image by scp. I previously uploaded by FTP and I'm not sure if it was in binary mode. Obviously, this would cause big problems. I'm trying the same tests on the new metarouter now and will post results.

I also haven't tried the default WRT build that Normis posted in another thread (I think Normis posted it). I am still using the build that I built.

webasdf · Mon Oct 19, 2009 9:47 pm

Tried the pre-built image posted by Normis. Made sure I used scp to ensure binary transfer. Same issue. We were testing some VLAN settings on the same board and it ended up in a perpetual boot state.

We attempted to reload firmware via netinstall, but no success. Stuck on the message "sending offer". We tried to flash it with the 4.0 net installer (per other forum posts), but same thing. After taking power off for a while and powering it back up, it was able to boot again. We did a memory scan from bios and it didn't appear there were any memory problems. Then, we reset all configurations. I also found a way to upgrade the boot loader from the forums. So, I did that. Previous version was 2.20 and I upgraded to 2.23. Then, re-configured router from scratch.

After all this, same issue. Random reboots when doing I/O on the metarouter on 450G. Just to check, I did the same thing to a 433AH and no issue there.

Should we keep trying to netinstall or does this point to a hardware problem?

webasdf · Mon Oct 19, 2009 10:46 pm

BTW, exactly what I'm doing when this happens:
In winbox, I have console open and running top

In an ssh session remotely, I am executing a shell script:

#!/bin/sh
while true
do
clear
date
df -h
uptime
sleep 3
done

If I let these conditions go for 40-60 minutes, router will reboot at some point in there. I've also noticed that router will reboot during these conditions if some other random events happen on the router.

The meta router is set up on a bridge called "wrt". IP address on wrt bridge is 10.20.1.1/30. IP address on meta router is 10.20.1.2/30. Gateway and DNS are set up on meta router. Dstnat is used to get to meta router ssh remotely via tcp port 3333.

I was also able to get the 450G to reboot when simply letting top run in that same image (while leaving winbox open to monitor it) and disabling the metarouter interfaces and metarouter bridge altogether. It happened in about the same amount of time....last time, it took about an hour.

Tried a different 450G board w/ the posted image (not my compiled one) and same result. I've tried everything I know to try and still get this problem.

janisk · Tue Oct 20, 2009 9:17 am

thank you for reporting in detail. We are looking into the problem.

Bootloader version can be seen under /system routerboard print

[admin@MikroTik] > system routerboard print

       routerboard: yes
             model: "600"
     serial-number: "ASDFASDFASD"
  current-firmware: "2.20"
  upgrade-firmware: "2.20"

under that menu there are many interesting things that usually you can change at boot time while hanging in bootlaoders menus.

webasdf · Tue Oct 20, 2009 3:47 pm

Were you able to reproduce the problem?

janisk · Tue Oct 20, 2009 3:53 pm

i got one problem, but not sure that is one you got on your router.

can you do the following:

1) disable watchdog timer

system watchdog set watchdog-timer=no

2) check if your router reboots. or something freezes when you think that it should have been hanged. for example do /ip proxy print

soho · Tue Oct 20, 2009 5:24 pm

i got one problem, but not sure that is one you got on your router.

can you do the following:

1) disable watchdog timer
Code: Select all
system watchdog set watchdog-timer=no
2) check if your router reboots. or something freezes when you think that it should have been hanged. for example do /ip proxy print

I have this same problem on RB450G.

I did update for 4.1, bootloader 2.23, watchdog disable and this problem persist.
I did downgrade for 3.30, bootloader 2.20, watchdog disable and this problem persist.
I did downgrade for 3.28, bootloader 2.20, watchdog disable and this problem persist.

I have a RB493AUH with 3.28 version and 2.20 bootloader and this problem not occurs (yet).

webasdf · Tue Oct 20, 2009 6:03 pm

OK.

Disabled watchdog timer.
Started top from command line on console inside my wrt metarouter (previously posted image by Normis).
The metarouter had no network communications enabled, so it was not doing network i/o
About an hour later, it locked up and didn't reboot...had to kill power and restart manually

I also tried to import the pre-built wrt metarouter image onto a regular RB450 and got an immediate reboot. After reboot happened, the metarouter listing was not there.

Thanks for the input Soho. At least I know I'm not doing something wrong with the Metarouter.

soho · Tue Oct 20, 2009 10:28 pm

OK.

Disabled watchdog timer.

Started top from command line on console inside my wrt metarouter (previously posted image by Normis).

The metarouter had no network communications enabled, so it was not doing network i/o

About an hour later, it locked up and didn't reboot...had to kill power and restart manually
I also tried to import the pre-built wrt metarouter image onto a regular RB450 and got an immediate reboot. After reboot happened, the metarouter listing was not there.

Thanks for the input Soho. At least I know I'm not doing something wrong with the Metarouter.

Interesting is the problem, as you mentioned ... The system (Mikrotik) freezes and back without me doing anything. You lose access to Mikrotik, it seems that the interfaces simply stop responding. When the system returns, there is no log, as if nothing had happened.

In one of these frozen I turned off the RB and when I turned on, the image of OpenWRT simply disappeared (deleted).

webasdf · Wed Oct 21, 2009 4:10 pm

Can we hope for a fix from Mikrotik in the next stable software release?

janisk · Thu Oct 22, 2009 12:58 pm

what else you have configured on your board?

i am running RB450G with normis posted OpenWRT image and its uptime is several hours already and nothing has hanged. i am using that router as gateway.

soho · Thu Oct 22, 2009 3:04 pm

what else you have configured on your board?

i am running RB450G with normis posted OpenWRT image and its uptime is several hours already and nothing has hanged. i am using that router as gateway.

[admin@MetaRouter] > metarouter export 
# oct/22/2009 10:02:39 by RouterOS 3.28
# software id = 8SUH-T8YM
#
/metarouter
add comment="" disabled=no disk-size=unlimited memory-size=128MiB name=mr2
/metarouter interface
add comment="" disabled=no dynamic-bridge=none dynamic-mac-address=02:5D:55:6F:7F:98 type=dynamic virtual-machine=mr2 \
    vm-mac-address=02:11:54:6C:27:89

[admin@MetaRouter] > interface export 
# oct/22/2009 10:03:27 by RouterOS 3.28
# software id = 8SUH-T8YM
#
/interface ethernet
set 0 arp=enabled auto-negotiation=yes comment="" disabled=no full-duplex=yes l2mtu=1526 mac-address=00:0C:42:54:05:ED \
    mtu=1500 name=ether1 speed=100Mbps
set 1 arp=enabled auto-negotiation=yes bandwidth=unlimited/unlimited comment="" disabled=no full-duplex=yes l2mtu=1524 \
    mac-address=00:0C:42:54:05:EE master-port=none mtu=1500 name=ether2 speed=100Mbps
set 2 arp=enabled auto-negotiation=yes bandwidth=unlimited/unlimited comment="" disabled=no full-duplex=yes l2mtu=1524 \
    mac-address=00:0C:42:54:05:EF master-port=none mtu=1500 name=ether3 speed=100Mbps
set 3 arp=enabled auto-negotiation=yes bandwidth=unlimited/unlimited comment="" disabled=no full-duplex=yes l2mtu=1524 \
    mac-address=00:0C:42:54:05:F0 master-port=none mtu=1500 name=ether4 speed=100Mbps
set 4 arp=enabled auto-negotiation=yes bandwidth=unlimited/unlimited comment="" disabled=yes full-duplex=yes \
    mac-address=00:0C:42:54:05:F1 master-port=none mtu=1500 name=ether5 speed=100Mbps
/interface bridge settings
set use-ip-firewall=yes use-ip-firewall-for-pppoe=yes use-ip-firewall-for-vlan=yes
/interface ethernet mirror
set mirror-port=none source-port=none
/interface l2tp-server server
set authentication=pap,chap,mschap1,mschap2 default-profile=default-encryption enabled=no max-mru=1460 max-mtu=1460 \
    mrru=disabled
/interface ovpn-server server
set auth=sha1,md5 certificate=none cipher=blowfish128,aes128 default-profile=default enabled=no keepalive-timeout=60 \
    mac-address=FE:D0:41:7D:84:65 max-mtu=1500 mode=ip netmask=24 port=1194 require-client-certificate=no
/interface pptp-server server
set authentication=mschap1,mschap2 default-profile=default-encryption enabled=no keepalive-timeout=30 max-mru=1460 \
    max-mtu=1460 mrru=disabled

[admin@MetaRouter] > ip address print 
Flags: X - disabled, I - invalid, D - dynamic 
 #   ADDRESS            NETWORK         BROADCAST       INTERFACE                                                        
 0   192.168.1.254/24   192.168.1.0     192.168.1.255   vif1                                                             
 1 D 10.10.200.2/28     10.10.200.0     10.10.200.15    ether1                                                           
[admin@MetaRouter] > ip route print 
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY-STATE GATEWAY                        DISTANCE INTERFACE               
 0 ADS  0.0.0.0/0                          reachable     10.10.200.1                    0        ether1                  
 1 ADC  10.10.200.0/28     10.10.200.2                                                  0        ether1                  
 2 ADC  192.168.1.0/24     192.168.1.254                                                0        vif1                    
[admin@MetaRouter] >

[admin@MetaRouter] > system watchdog export 
# oct/22/2009 11:29:29 by RouterOS 3.28
# software id = 8SUH-T8YM
#
/system watchdog
set auto-send-supout=no automatic-supout=no no-ping-delay=5m watch-address=none watchdog-timer=no

No more configuration in RB450G.

webasdf · Thu Oct 22, 2009 4:48 pm

what else you have configured on your board?

i am running RB450G with normis posted OpenWRT image and its uptime is several hours already and nothing has hanged. i am using that router as gateway.

OpenWRT image runs fine until you start doing things with it. For example, we run top command from OpenWRT, leave console open and router will reboot itself (kernel panic) within an hour. If we aren't really doing anything with OpenWRT, it is fine and doesn't reboot. I also just got the box to reboot itself in about 5 minutes (console in winbox was closed) running top on the WRT image in an SSH session from a remote client...give it a try...

The system is currently not in production anywhere...is a development system (we are developing some custom html pages inside the metarouter...at least we hope to if this problem can be solved.)

General specs

The board itself is a standard RB450G
256MB memory
CPU: MIPS 24K V7.4 @ 680MHz
520MB flash HDD space
No other devices are attached (I don't think there are any ports to do this anyway)
Running RouterOS version 3.30 (tried 4.0 and 4.1....all do the same thing)
Boot loader version 2.20 (although on another board we tried, 2.23 was installed)

Installed packages
We are primarily using these as hotspot gateways (basic NAT only), so if there are any suggested packages that we can remove or disable that may fix the problem, please suggest

calea
routeros-mipsbe
advanced-tools
dhcp
hotspot
ipv6 (disabled)
ppp
routerboard
routing
routing-test (disabled)
security
system
wireless
user-manager

Metarouter specs

openWRT image (posted by Normis)
memory 16 MB
disk size 128000kiB

Do you need any other information? If you need, I can send a support.rif as well.

Thanks.

enderx · Fri Oct 23, 2009 5:46 pm

Ditto.

Things I've tried that don't fix the problem:
Underclocking doesn't help.
Cannot be forced to produce.. (doing a bunch of stressful tasks in the openwrt does not reboot it.)
never writing to the disk subsystem has no effect.
Openwrt never shows any kernel errors.

I'm kinda concerned about the metarouter feature.. just doesn't seem to be very important.

webasdf · Fri Oct 23, 2009 8:23 pm

Ditto.

Things I've tried that don't fix the problem:
Underclocking doesn't help.
Cannot be forced to produce.. (doing a bunch of stressful tasks in the openwrt does not reboot it.)
never writing to the disk subsystem has no effect.
Openwrt never shows any kernel errors.

I'm kinda concerned about the metarouter feature.. just doesn't seem to be very important.

Hopefully Mikrotik can reproduce our problems and fix in the next RouterOS version, but I haven't heard that they will do so at this time. IMHO - Almost seems like a memory leak somewhere in metarouter causing kernel panic and router reboot...

QpoX · Sat Oct 24, 2009 6:09 am

Did also have problems with Metarouter on my RB750 (i have 4) and RB411U when i add a metarouter (just RouterOS) and both with and without network on it.
It starts up (can see it using metarouter console), then i get disconnected from Winbox and i can't get in again...
Just gets random reboots after i got kick'd out from the box.
Even using telnet or ssh i can't login, and typing username takes some time before i see what was type'd and same on serial on RB411U.
Did also try with OpenWRT with 8 and 16MB of Ram same result.

So did need to netinstall both before i could them running again!

I was using 4.1 and 3.30 running the latest firmware.
And making a Supout was a no go since i could not login after this!

janisk · Mon Oct 26, 2009 8:38 am

metarouter by default assigns minimum amount of RAM to the system - 16MB that might cause problems with it on boards that have 32MB of ram..

About Metarouter on RB450G, that have plentiful storage and RAM to run several metarouter instances - Normis posted image hanged once, last week. after that - metarouter has ran bash script posted in one of previous posts (http://forum.mikrotik.com/viewtopic.php ... 06#p176206) non-stop every second.

From last post about configuration - i do not have calea package installed. Maybe you could send e-mail to support with details - how we could get connected to that board, since my RB450G at this moment seem to be resilient to this problem.

webasdf · Mon Oct 26, 2009 6:16 pm

From last post about configuration - i do not have calea package installed.

I disabled calea ran same tests, and still same result. Rebooted itself in only 5 minutes this time.

Maybe you could send e-mail to support with details - how we could get connected to that board, since my RB450G at this moment seem to be resilient to this problem.

I have contacted you at support with router IP details.

enderx · Wed Oct 28, 2009 6:22 pm

Conclusion.. Bad power supplies..

badpowersupply.jpg

I've moved to using POE and different supply.. works good so far.. 18 hours. on a 12v 1A supply.

enderx · Wed Oct 28, 2009 9:34 pm

Maybe not bad power supply.. maybe more toward something wrong with using the barrel jacks on the 450g.
That would be pretty wild if the flaw in it would only be exposed when you used metarouter features with openwrt images... please someone tell me I'm not crazy.

Edited:
Never mind.. finally crashed using poe with stock power supply. Definitely a power supply problem. regardless of POE or barrels.

webasdf · Thu Oct 29, 2009 4:00 pm

We are using the same power supply. We will try a different power supply with more watts to see if that helps.

enderx · Thu Oct 29, 2009 4:26 pm

Seems like the power regulating circuitry is more robust on the POE instead of the barrels... I've ran 19 hours using the POE and Original power supply pictured above. One reboot in 23 hours. My other box is still running on the 12v 1A via POE, 23 hours no reboots.
For a while there ASDF I thought I caught your virus.. you mentioned instability and Boom! instability appeared.
Sure glad we figured this out.. I was getting worried I was going to have to come up with a different more expensive and less elegant solution. We planned on buying 250 boxes.. I was having nightmares of what that was going to be like.
Sam.

PS wish I could get a level4+ x86 license

webasdf · Thu Oct 29, 2009 11:26 pm

Seems like the power regulating circuitry is more robust on the POE instead of the barrels... I've ran 19 hours using the POE and Original power supply pictured above. One reboot in 23 hours. My other box is still running on the 12v 1A via POE, 23 hours no reboots.

Well, that seems to have worked! We found a 12v 1.5A power supply in the lab and gave it a shot...uptime is currently 23 hours and it seems to be very stable. We are planning on testing the higher wattage power supply available from roc-noc or other Tik dealers (http://www.roc-noc.com/product.php?prod ... t=3&page=1). It's 24VDC 1.6 amp instead of the 24VDC 800 mA on the other power supply that seems to have caused this issue.

I will post our results. Strange thing is, I have a different 433AH board that I power with 12VDC 300mA supply and it is very stable (no issues w/ metarouter at all on that board).

Any input from Mikrotik as to why a lower wattage power supply would cause this problem? Does the board use more power when Metarouter is enabled?

janisk · Fri Oct 30, 2009 11:37 am

RB433AH is different board, different hardware, that is why there is a difference.

soho · Sat Oct 31, 2009 10:47 pm

hummm ... interesting, I'll change the power supply and test.

janisk · Tue Nov 03, 2009 11:24 am

my PSU change failed, so do not have high hopes about that, it might do something, but not solve the problem completely.
we are still working on the problem. Thank you for supout.rif files sent.

webasdf · Thu Nov 05, 2009 5:10 pm

my PSU change failed, so do not have high hopes about that, it might do something, but not solve the problem completely.
we are still working on the problem. Thank you for supout.rif files sent.

An update. I am developing some custom html screens on a 433AH. Running with a 12V 300mA PS. I was running 4.1 firmware and it was relatively stable (would reboot once every 2 or 3 days). I upgraded to 4.2 firmware this morning and it rebooted itself 2 times in half an hour. I downgraded to 4.0 (because I lost my 4.1 install packages

). It is more stable now.

On the 450G board. Our power supply upgrade to 12V 1.5 amp made that board really stable. It still reboots itself, though...but only once every 4 to 5 days instead of once per hour.

Hope this helps any further diagnosis of the issue.

UPDATE:
On the 450G board we tried the higher voltage 24V power supply (24V 1.6amp). It rebooted by itself within 27 minutes. We switched back to 12V 1.5 amp and is running better....as before.

janisk · Fri Nov 20, 2009 3:52 pm

we are still working on the problem, cannot promise results soon thought.

od3m · Sun Nov 22, 2009 3:54 pm

hey,

i´m new to mikrotik boards.
i ran alix2c3 before.

i got the same problems.... rb450g reboots sometimes...
hope you´ll find a solution.

ccr · Tue Dec 01, 2009 12:18 am

I have a bunch of 450's out in the field using 24v power supplies. Not working right at all. Rebooting. Losing their config. Is 24v power supplies the problem? Will 18v fix it? What a mess.

enderx · Tue Dec 01, 2009 12:28 am

ccr, what are you powering yours with?

These are the power supplys I've tried:
1. 28v 0.8amp the one pictured above.
2. ubiquity nanostation2 12v 1amp
3. Tycon Power Systems 5port switch 24v 5amp version

The nanostation2's is the only one that seems stable, which is odd because it seems like a cheap wall wart.

ccr · Tue Dec 01, 2009 12:42 am

ccr, what are you powering yours with?

These are the power supplys I've tried:
1. 28v 0.8amp the one pictured above.
2. ubiquity nanostation2 12v 1amp
3. Tycon Power Systems 5port switch 24v 5amp version

The nanostation2's is the only one that seems stable, which is odd because it seems like a cheap wall wart.

Hello

The 450g Meta Routers I have been having problems with were using the 24v .8amp pictured above. Exact same.
I am now going to try some 18v to see if that helps.
Pretty much the Meta Routers/450g have been in complete meltdown mode; dying everywhere; losing config. Read this thread and figured it was probably the power supply. Going to try 18v and if it works great;

It looks like 24v is what is having problems.

roc-noc.com · Fri Dec 04, 2009 1:40 am

Thanks for pointing me to this thread Drew. I just read through it and have some comments to post.

I have worked in electronics including product design since the early 1980s. My company designed and manufactured electronic computer products successfully for a number of years before I became an ISP. Some of the designs were my own and I was involved in all of the designs.

Before we jump the gun and blame the power supply, lets look at a couple of facts.

1) Metarouter appears to have some stability issues. We are not sure if it is version specific or environment specific but the complaints in this thread all relate to running an application in metarouter under various ROS versions with various amounts of RAM, and on various RouterBoards.

2) While it doesn't appear to be a cure, there have been some differences noticed in the frequency of the crash when changing to various voltage power supplies.

3) This thread was originally opened as a question about the RB450G with Metarouter but has spanned several routerboards including RB450G, RB750, RB433, and RB493. I am going to stay focused on just the RB450G as it has enough RAM and CPU power to handle most virtual applications. Some of the others mentioned do not.

Problem #1 looks like software or resource problem. I think the focus of this problem should remain there.

I have a problem with fact #2. The RB450G and most of the RouterBoards from the past two years (and all RB4xx boards) use the same power supply circuit. They use a Richtek RT9206 power controller which takes 4.75 to 28vdc in and puts out 3.3vdc.

I found the data sheet here:
http://www.cheertech.com.tw/RichTek%20C ... 206-09.pdf

This should remain the same whether we are feeding it 12vdc or 24vdc. The power capacity only needs to be sufficient to run the RB450G at full load. I suspect that would be about 3 watts since that is what the RB493AH is supposed to use. The RB450G has 4 fewer ports but Gigabit ethernet instead of 10/100. I call that an equal power tradeoff. Any power supply from 8vdc to 28vdc with at least 3 watts of capacity should work fine to power it.

<putting on my Mikrotik distributor hat> The 18POW is one of our best selling products. We have sold about 800 RB450G-complete (with case and power supply) this year that included the 24volt, 19 watt 18POW. We are only one of the 12 US Mikrotik distributors and there are many other Mikrotik distributors around the world. That means many thousands of these have been sold and deployed.

We have been selling these since the beginning of 2009. This is the first that I have heard of a stability problem with the RB450G and 18POW supply and most of my RB450G customers are using that combination.

The missing part in the failure discussed in this thread is Metarouter with operwrt running a script or application.

Again, I am thinking.... software problem.

Sorry to bore you with my opinion but I think this thread needed some new blood.

Many thanks,

Tom

gcakici · Fri Dec 04, 2009 11:54 am

We're using same configuration and never seen such problems with openwrt. (this is for honoring the RB450G)

What we see is when we change the configuration of a service in openwrt such as network configuration or syslog configuration while they're running, services becomes sometimes unstable (sometimes stable) or after rebooting our modifications gone and files turns back to original states. So we stop the service first then change the configuration and when we think it's all done we reboot the system with fresh configuration. This is because some services don't die exactly the way we want in openwrt.

Regards
Gokalp

ccr · Fri Dec 04, 2009 6:56 pm

Tom

I didn't read your post but I am leaning towards version problem. 3.x and Metarouter don't work together correctly. I have quite a bit running on 4.3 so far without trouble. I will let you know...

janisk · Mon Dec 07, 2009 8:52 am

lets wait for 4.4 and see if that version solves your problem with hanging (rebooting) routers.

QpoX · Mon Dec 07, 2009 9:10 am

lets wait for 4.4 and see if that version solves your problem with hanging (rebooting) routers.

When is 4.4 due?

Mon Dec 07, 2009 10:26 am

this week

QpoX · Fri Dec 11, 2009 2:36 pm

this week

Like today or before Sunday?

Fri Dec 11, 2009 3:48 pm

ok, monday ! apparently I'm not so good at predictions.

QpoX · Fri Dec 11, 2009 4:20 pm

ok, monday ! apparently I'm not so good at predictions.

Yes i did notice that

Chupaka · Fri Dec 11, 2009 4:44 pm

let's buy a fortune-teller's magical crystal globe by the next MUM and present it to Normis ))

QpoX · Fri Dec 11, 2009 5:25 pm

let's buy a fortune-teller's magical crystal globe by the next MUM and present it to Normis ))

Madam Normis?

QpoX · Mon Dec 14, 2009 8:31 pm

ok, monday ! apparently I'm not so good at predictions.

Normis you really need to get your crystalball clean'd!

QpoX · Wed Dec 16, 2009 11:14 am

ok, monday ! apparently I'm not so good at predictions.
Normis you really need to get your crystalball clean'd!

Hmm... bump?

Wed Dec 16, 2009 12:02 pm

We are working on some remaining issues. This week is 99.99%

Wed Dec 16, 2009 12:03 pm

however, I can't vouch for that "let's see if it fixes your rebooting router", as I don't know if you submitted this issue to support and if it was fixed ...

janisk · Wed Dec 16, 2009 12:52 pm

it should fix rebooting router problem, at least my test routers have not rebooted after installing version with patches. Sadly other problems in that build prevent me for giving out some pre-release versions.

QpoX · Wed Dec 16, 2009 1:44 pm

it should fix rebooting router problem, at least my test routers have not rebooted after installing version with patches. Sadly other problems in that build prevent me for giving out some pre-release versions.

Just as long that the GPS problem is fixed (as i was promised since 4.1) edontcare about anything else

But is it possible to get a pre version that has the GPS fixed?
Since i've been without a internal NTP server for over 2 month's now...

Wed Dec 16, 2009 1:46 pm

there were other more serious issues in pre-releases. we are testing newer builds now, but those are not ready to be given out.

nik247 · Mon Dec 21, 2009 10:47 am

I upgrated my test 450G to RoS 4.4, but problem with rebooting with enabled OpenWRT in Metaroter not fixed.
My RB450G rebooted after 20 minutes. I can sent supout file.

janisk · Mon Dec 21, 2009 11:46 am

in log - rebooted by watchdog timer?

none of test routers hanged for me at least. and are up and running since patched version was built.

nik247 · Mon Dec 21, 2009 12:14 pm

Yes. In log present next:
[nik@mt-test] > log print
jan/01/2002 03:00:04 system,error,critical router was rebooted without proper shutdown by watchdog timer

What does it mean - "patched version"?

Mon Dec 21, 2009 12:15 pm

he means v4.4 that is on the web. maybe you have some other issue now? contact support

nik247 · Mon Dec 21, 2009 1:42 pm

I disabled watchdog timer and RB450G (RoS 4.4) stop rebooting, but sometimes RB450G freezes and stop ANY network activity. I will contact support.

webasdf · Mon Dec 21, 2009 6:34 pm

Running openwrt r17779, w/ 4.4 firmware on RB450G...rebooted itself in 17 minutes...

janisk · Wed Dec 23, 2009 9:31 am

what are you doing in those metarouter guests?

nik247 · Wed Dec 23, 2009 4:35 pm

Not doing anything at all. Now I disabled VM in Metarouter.

bkb · Mon Dec 28, 2009 2:07 am

I just run some kind of tests on a brand new 450g with ROS 4.4 and openwrt in metarouter and I thought, the results could be helpful.

I tried different openwrt image versions with no difference.
In every case the watchdog rebooted the system in under 2h uptime.
When I disabled the watchdog however, the system ran 20h straight, until I aborted the test.
While running without watchdog, the system seemed to hang for about 30 - 60 seconds from time to time.
While in the "hanging" state, the system did not answer ICMP packets (RouterOS or OpenWRT) and also did not react to key presses on the serial console.
After the said time of up to 60 seconds, the system ran normally, but nagios (running inside the OpenWRT/Metarouter) showed extremly high cpu usage.

Regards,
bkb

Edit: Checked out r16424 of OpenWrt and built it, because it is the same Version your image in the Wiki is, but the packets on openwrt.org are missing, so anyways...
I built r16424 and tried that, with watchdog and ROS 4.4.
I installed all my packets which I had on the newer OpenWrt Version (see above).
I have 8h of uptime and going, so that looks quite good!
I will abort the test when I reached 24h uptime without watchdog-reboot and tell you about that.

janisk · Wed Dec 30, 2009 11:56 am

if you can, run test for several days, because some times it took more than that to reproduce the problem. At least one i could reproduce.

Also, if possible remove all additional configuration/packages that are installed on the router (best case, leave just system package) and see if problem manifests it self.

bkb · Thu Dec 31, 2009 10:03 pm

ok, unfortunately the system rebooted after about 20h of uptime and then pretty regularly about every 2 hours.
I really thought, that using the older version of OpenWrt would turn out to make a difference.

I am hoping that the there will be a fix for that problem...

- bkb

McCJay · Sun Jan 03, 2010 8:38 pm

I am experiencing similar issues, I have tried the following, all with the same sporadic crashes:

* RouterBoards: RB450G, RB411U, RB433UAH
* RouterOS versions: v4.2, v4.3, v4.4
* Only installed System package, also tried with only System & Routerboard packages
* OpenWRT builds: Official from the wiki, I also built R18199
* default OpenWRT config as well as several variations stripping out as much extra funtionality as I could, removed all kernel modules, extra packages
* 12v, 24, 48v PSU's with and without poe
* disabled/enabled watchdog

It seems the OpenWRT VM crashes without anything consistant in the logs, like the virtual hardware is locking up entirely at random. If the vm is left entirely idle it seems to increase the stability, got 2 days uptime in one case, but as soon as there's activity (network traffic, shell access) it doesn't stay up for longer than a few hours, often hanging every few minutes.

Please let me know if you need any further information/testing on this. I'm eager to get a stable OpenWRT VM running.

McCJay · Mon Jan 04, 2010 12:46 am

Just noticed that it seems the VM's lockup is not permanent. Instead of disaling and enabling the metarouter, I just left it in it's frozen state and it reanimated some time later(not sure how long, I was away from my pc for a few hours).

The first message was:

Bus error

Hope this helps...

congo · Mon Jan 04, 2010 12:40 pm

While in the "hanging" state, the system did not answer ICMP packets (RouterOS or OpenWRT) and also did not react to key presses on the serial console.
After the said time of up to 60 seconds, the system ran normally, but nagios (running inside the OpenWRT/Metarouter) showed extremly high cpu usage.

I have the same problem with OpenWRT in RB450G MetaRouter on v4.4 (upgraded from v3.28 which also had the issue). I noticed the high CPU load inside the OpenWRT from the system load then logging in after the hang.

gcakici · Tue Jan 05, 2010 11:48 pm

Up for 20 days with my 450G and 3.30 RouterOS.

uptime: 2w6d5h33m11s
version: "3.30"
free-memory: 186524kB
total-memory: 257364kB

Inside metarouter

root@xxxxxxxx/# uptime
18:48:09 up 19 days, 23:58, load average: 0.00, 0.00, 0.00

jkoko · Wed Jan 06, 2010 12:03 am

Hi, can you public your metarouter image ?

Up for 20 days with my 450G and 3.30 RouterOS.

uptime: 2w6d5h33m11s
version: "3.30"
free-memory: 186524kB
total-memory: 257364kB

Inside metarouter

root@xxxxxxxx/# uptime
18:48:09 up 19 days, 23:58, load average: 0.00, 0.00, 0.00

jkoko · Wed Jan 06, 2010 12:09 am

Hi,
can you public your metarouter image ?
Thank

Up for 20 days with my 450G and 3.30 RouterOS.

uptime: 2w6d5h33m11s
version: "3.30"
free-memory: 186524kB
total-memory: 257364kB

Inside metarouter

root@xxxxxxxx/# uptime
18:48:09 up 19 days, 23:58, load average: 0.00, 0.00, 0.00

congo · Wed Jan 06, 2010 1:40 am

Up for 20 days with my 450G and 3.30 RouterOS.

uptime: 2w6d5h33m11s
version: "3.30"
free-memory: 186524kB
total-memory: 257364kB

Inside metarouter

root@xxxxxxxx/# uptime
18:48:09 up 19 days, 23:58, load average: 0.00, 0.00, 0.00

I have just downgraded from 4.4 to 3.30 to check, but the issue remains. It is not about the uptime, but every few minutes both the RouterOS and the OpenWRT will go to "sleep". Does not reboot (at least not very often...).

janisk · Wed Jan 06, 2010 11:11 am

older version that 4.4 will have other bug that was fixed when you router just crashed and was rebooted by watchdog timer.

I am not clear how to induce that state of freezing reliably, all i have now is sit and wait. Also, on router that freezes from time to time - try to put it under load, make it do something in host, and see if it still freezes from time to time. (Run like bandwidth-test to itself).

congo · Wed Jan 06, 2010 12:16 pm

older version that 4.4 will have other bug that was fixed when you router just crashed and was rebooted by watchdog timer.

It's back on 4.4 now. This is just a test box serving my IP phones, so I can test any version.

I am not clear how to induce that state of freezing reliably, all i have now is sit and wait. Also, on router that freezes from time to time - try to put it under load, make it do something in host, and see if it still freezes from time to time. (Run like bandwidth-test to itself).

If you wish, I can provide you with the info how I configured this board, even give remote access to it if that helps. Just started a bbandwidth test to localhost, will write if it helps.

EDIT: It did not help. Just this time it was not freezing for minutes, but rebooting. The logs show:

jan/02/1970 01:00:19 system,error,critical System rebooted because of kernel failure
jan/02/1970 01:00:19 system,error,critical router rebooted without proper shutdown, probably power outage

janisk · Wed Jan 06, 2010 12:43 pm

well, first bug did hang board completely, so only chance was to reboot the router (power-cycle it if watchdog timer was disabled) second bug i am trying to reveal in details - will not hang your router completely, but it will freeze it for some time and then router will continue to work. that is, as far as i have tested and seen results.

also, limit BT to some values making cpu load above 50 but below 70 or 80 percent.

congo · Wed Jan 06, 2010 6:47 pm

also, limit BT to some values making cpu load above 50 but below 70 or 80 percent.

Today btest was running for about an hour (~60% CPU), it did not hang during that. But after the Winbox session failed (maybe some network problem) and btest stopped, it hang again. Also it did not fail yet during a call was made through the Asterisk running in OpenWRT - or maybe I was lucky every time. I will try this with a long test call and report back.

EDIT: Is it possible, that there is no "complete hang", so router only hangs for a few minutes, but people are not patient enough and restart?

EDIT 2: After more than 1 hours of the test call, Winbox session and SSH to the OpenWRT failed, and the phone went deaf, but after a few minutes came back. Maybe in the case of the btest, the winbox failed because of a hang?

McCJay · Wed Jan 06, 2010 7:26 pm

EDIT: It did not help. Just this time it was not freezing for minutes, but rebooting. The logs show:
Code: Select all
jan/02/1970 01:00:19 system,error,critical System rebooted because of kernel failure
jan/02/1970 01:00:19 system,error,critical router rebooted without proper shutdown, probably power outage

I get identical log messages when running both guest and host under load. uptime varies between a few hours and a few minutes.

RB411U
ROS v4.4
OpenWRT r18199

McCJay · Tue Jan 12, 2010 9:18 am

Any updates from Mikrotik on the status of this bug?

I submitted a support request (as suggested in this thread ) over a week a ago without any response.

Would be nice to know whether it's being worked on and if so whether a fix might be included in the next update.

janisk · Tue Jan 12, 2010 9:55 am

It is worked on, if it where no, i would not have asked for support files.

congo · Mon Mar 01, 2010 12:57 am

Hi!

Is there any update on this issue?

webasdf · Wed Mar 10, 2010 9:04 pm

I second that question. It has been two months since last response. Firmware 4.6 has come out, but nothing in changelog about this issue. This forum post has been very quiet too. Is this issue still being worked on?

Thanks.

janisk · Thu Mar 25, 2010 12:34 pm

could anyone check if that problems you have on RB450G also occur on RB433AH.

webasdf · Fri Mar 26, 2010 5:39 pm

We tested the 433AH in our lab yesterday using 24VDC. It has rebooted at least once since yesterday. So far uptime is 23 hours.

I have also used 12VDC on the 433AH. When running 12VDC the 433AH reboots about once every 3 to 4 days.

janisk · Mon Mar 29, 2010 10:26 am

you where running OpenWRT in metarouter. What interface type you use, as in, dynamic, static or static physical?

webasdf · Mon Mar 29, 2010 4:15 pm

The interface type is dynamic on both RB433AH and RB450G.

congo · Sat Apr 24, 2010 10:46 pm

Today I upgraded my RB450G to v4.7 because of the MetaROUTER fix mentioned in the changelog. Interestingly 4.7 for RB4xx is not available for me on the web, but I changed the version number in the URL of 4.6, and it worked. (One mirror served a corrupted file with the correct size so I had to download it twice.)

The frequent lockups went away, so after running fine for a few hours, I turned watchdog back on (which was disabled as per a request from support). Sadly, another few hours later the router rebooted itself again, and the log says watchdog timer. Also, with nearly all the reboots, the few last recently edited configuration files in the OpenWRT metarouter are truncated to zero-length.

I opened a ticket with support, and am waiting for a response. Also turned off watchdog, and hoping for the best...

ps. My interface type is also dynamic.

Update: The hangs continue, just they appear every few hours instead of every few minutes. This issue has probably not been fixed.

Xymox · Tue May 04, 2010 2:52 am

I believe this is the exact issue I am experiencing on 5.0 Beta 2. It got ALOT worse on 5.0 B2. I had hardly noticed it on 4.6-4.9 but on 5.0B2 it fails at most in 1 hour or less.. With the watchdog turned off it hangs the router but does recover by itself after some period of time.

This is on a 450G. My power supply is doing 14.6Vdc and has 3 amps available.

Im using a Mikrotik image for the meta router.

I have a thread on this at http://forum.mikrotik.com/viewtopic.php ... 2&p=206394 and have sent support requests.

My watchdog timer does not generate autosup for these events..

congo · Tue May 04, 2010 10:51 am

Running 4.9 now. I added a Netwatch to the metarouter guest. Funny.

may/03 22:22:40 system,info monitoring new host by admin
may/03 22:22:41 script,info OpenWRT guest is up.
may/03 22:23:26 script,info OpenWRT guest is down.
may/03 22:28:42 script,info OpenWRT guest is up.
may/03 22:34:20 script,info OpenWRT guest is down.
may/03 22:36:40 script,info OpenWRT guest is up.
may/03 22:41:18 script,info OpenWRT guest is down.
may/03 22:48:40 script,info OpenWRT guest is up.
may/03 22:53:16 script,info OpenWRT guest is down.
may/03 22:58:07 script,info OpenWRT guest is up.
00:13:06 script,info OpenWRT guest is down.
00:22:48 script,info OpenWRT guest is up.
01:02:44 script,info OpenWRT guest is down.
01:06:32 script,info OpenWRT guest is up.
01:39:14 script,info OpenWRT guest is down.
01:43:02 script,info OpenWRT guest is up.
02:26:36 script,info OpenWRT guest is down.
02:34:36 script,info OpenWRT guest is up.
02:38:38 script,info OpenWRT guest is down.
02:41:48 script,info OpenWRT guest is up.
03:16:00 script,info OpenWRT guest is down.
03:24:14 script,info OpenWRT guest is up.
03:34:04 script,info OpenWRT guest is down.
03:36:36 script,info OpenWRT guest is up.
03:42:45 script,info OpenWRT guest is down.
03:46:58 script,info OpenWRT guest is up.
04:03:36 script,info OpenWRT guest is down.
04:15:50 script,info OpenWRT guest is up.

I opened another ticket on 24. apr. regarding this issue. Still no reply...

webasdf · Tue Jun 15, 2010 11:47 pm

Had a weird one today. During the night, the metarouter on one of our Tiks seems to have deleted itself...apparently after one of the watchdog reboots. This is only a theory because this is CPE equipment that one of our resellers sold to a customer. So, anything is possible

This is the first time I have seen this.

Anybody else see this behavior?

reverged · Wed Jun 16, 2010 4:29 am

I have hesitated to ask, but is there any update for this issue?

Either way, would it be too much to ask for weekly or monthly updates?

It has been quite a while since anyone from MT posted a status.....

webasdf · Thu Jul 01, 2010 6:05 pm

I believe this is the exact issue I am experiencing on 5.0 Beta 2. It got ALOT worse on 5.0 B2. I had hardly noticed it on 4.6-4.9 but on 5.0B2 it fails at most in 1 hour or less.. With the watchdog turned off it hangs the router but does recover by itself after some period of time.

This is on a 450G. My power supply is doing 14.6Vdc and has 3 amps available.

Im using a Mikrotik image for the meta router.

I have a thread on this at http://forum.mikrotik.com/viewtopic.php ... 2&p=206394 and have sent support requests.

My watchdog timer does not generate autosup for these events..

I read your post and it was really good troubleshooting. Way to dig in there

I see in that post that when you disabled NTP on 5.0B2 you were satisfied that MR and ROS were running fine. Is this still the case? I was previously running 4.5 with same reboot issues. I tried to disable all un-used ROS packages, clean up openWRT (disable all daemons I wasn't using) and disable ROS NTP. Reboot issues remain. I see from the changelogs that there was mention of a metarouter fix in 4.7.

So, now I'm doing the same sort of test (although not as detailed as your previous tests) with ROS 4.10 and OpenWRT MR. ROS 4.10, all unused ROS packages disabled, openwrt cleaned up and NTP disabled. Just began test yesterday. Results aren't ready yet. Testing on 2 routers in production and 1 in the lab. 2-RB433AH (24v) and 1-RB450G (12v). Incidentally, the 433AHs are a little more stable, but if they have enough load from users I've seen them reboot quite often (upwards of 5 times per day). Even w/o load they still reboot a couple times per week.

Another thing I am considering trying is to re-compile with the OpenWRT MR 1.2 patch. I am currently using an image I compiled with the 1.1 patch. Not sure if that will make any difference.

webasdf · Mon Jul 12, 2010 4:27 pm

Well, 4.10 and OpenWRT in the configuration above was a little more stable. Previously on the 2 production routers, I was getting 1-2 days uptime. Now, I get 4-5 days uptime. So, I think it's an improvement. There seemed to be no difference in my configuration whether NTP was enabled or not.

What I did notice on a different router at a really busy site with the same config was that when the CPU is already really busy about 30-40% and MR is placed on top of it, the CPU gets pegged out to 100% and stays there. After about 10-15 minutes the router reboots via watchdog. That particular test was on a 433AH board running 24vdc. Had to disable MR at that location. I think it would be nice to be able to tell metarouter that you want it to only consume a specific percentage of CPU resources. This way, metarouter would not take too many cpu resources.

Has anybody recompiled OpenWRT with the new 1.2 patch? Is it any better? Wondering if I should give that a try next.

pkelly1603 · Thu Aug 26, 2010 6:32 am

I have a lot of Metarouters in production, all running a 2nd routerOS on 450g's. We have stopped putting them in production and have started to replace some of them with other solutions because of these problems.

I have tried assigning different amounts of memory to the metarouter, tried different power supplies, tried different routerOS versions (currently running 4.10). Still these continue to reboot themselves. Version 4.9 and 4.10 do seem more stable than previous versions as they seem to reboot themselves instead of needing manual power cycles. If I disable watchdog, they lock up.

Mikrotik, any update?

Thanks!

pkelly1603 · Thu Aug 26, 2010 10:15 pm

Also, have tried:
- different firmware versions, currently running version 2.26.
- turning off NTP client (on both virtual and physical routers)

reverged · Mon Sep 06, 2010 12:42 am

I have been waiting for this issue to be resolved and have had to give up and move on.

But, I do have this to add:
Is this problem directly related to time keeping by the metarouter?

The reason I suggest this is I had a very similar problem with a vmware machine. The virtual servers would occasionally and randomly peg the cpu. During this time, no traffic would pass. The only cure was a virtual machine reset.

What I found was that time was not being kept, and in fact, time would sometimes move backwards as least as reported by `date`. These machines did not have the proper ntp settings as recommended by vmware.

In particular, I have found that only the vmware.pool.ntp.org servers are reliable and 'tinker panic 0' is required in ntp.conf.
If I move away from the vmware.pool.ntp.org servers in about 2 weeks time I get a lockup.
vmware states that large time errors can be expected and ntp would typically choke on such a large error if 'tinker panic 0' is not set.

Does this problem only occur with openwrt images or also mikrotik metarouters?

If the answer to that is 'yes' than what is different about time keeping of the 2 metarouters?

With all that being said, is it possible this or something similar could be the problem?

I would truly like to see a reliable OpenWRT metarouter for the 450G.

janisk · Mon Sep 06, 2010 9:58 am

that happens with both OpenWRT and RouterOS virtual routers. And it sometimes happens with timekeeping turned off, and with timekeeping turned on.

If talk about virtualized RouterOS, then most stable results are if you keep installed packages to the minimum (like system package only)

pkelly1603 · Tue Sep 07, 2010 5:56 pm

Is there a fix in the works for this?

webasdf · Wed Nov 03, 2010 5:40 am

Word is v5 firmware will fix. However, I haven't had a chance to test it yet.

pkelly1603 · Fri Jan 14, 2011 12:13 am

I have an RB450g that has been running on 5.0rc7, 2.29 firmware for a few weeks. Same problem

MMM MMM KKK TTTTTTTTTTT KKK
MMMM MMMM KKK TTTTTTTTTTT KKK
MMM MMMM MMM III KKK KKK RRRRRR OOOOOO TTT III KKK KKK
MMM MM MMM III KKKKK RRR RRR OOO OOO TTT III KKKKK
MMM MMM III KKK KKK RRRRRR OOO OOO TTT III KKK KKK
MMM MMM III KKK KKK RRR RRR OOOOOO TTT III KKK KKK

MikroTik RouterOS 5.0rc7 (c) 1999-2010 http://www.mikrotik.com/

(324 messages not shown)
jan/01/1970 18:00:25 system,error,critical router was rebooted without proper shut
d
by watchdog timer

jan/01/1970 18:00:25 system,error,critical router was rebooted without proper shut
d
own by watchdog timer

webasdf · Fri Jan 21, 2011 6:02 pm

Bummer

ett2tre · Tue Mar 08, 2011 6:36 pm

Still no news? An update about what progress (if any) is being made would be nice.

enzof · Wed Mar 09, 2011 5:38 pm

Hi all,
I need to run a web server on an RB450G to allow a quick web configuration wizard for the routerboard.

In the past I did the job by using a java setup wizard application. Now I have migrated the code to PHP and I have put it "in the hands" of lighttpd under OpenWRT.

Everything works almost fine but the stability, and, indeed, the routerboard reboots (and/or crashes) very often.

I did a lot of tests with no success.

By reading this thread I realized that the issues I experienced could be related to some kind of bug (power supply???) rather than a RouterOS or an OpenWRT misconfiguration.

Is there someone who has solved this issue? No one has finally cutted the "Gordian Knot"?

My configuration is:

- Roterboard model: RB450G

- Operating System: routerOS 5.0rc11

- OpenWRT: KAMIKAZE (bleeding edge, r25398) with lighttpd (1.4.28) and php (5.3.4) (

Thank you in advance.

janisk · Thu Mar 10, 2011 8:39 am

remove all the unnecessary packages can bring down the crash numbers a lot. Just think what packages you need and ones you can do without. And it is not PSU related, it just requires good one, as any other device.

enzof · Thu Mar 10, 2011 5:42 pm

Thank you Janisk for your kind reply.

remove all the unnecessary packages can bring down the crash numbers a lot. Just think what packages you need and ones you can do without.

Do you mean the unnecessary packages in RouterOS, in OpenWRT or both?

Do you know if someone already solved the "reboot issues" by removing these unnecessary packages?

And it is not PSU related, it just requires good one, as any other device.

I normally use the TW-AC24-10 Power Supply 24V 1.0A EU.
Of course, if I deactivate the metarouter the RB is really stable.

Anyway, I would like to better understand what exactly arises the reboot issue...memory leak? cpu load? resource access collison?

Any suggestion about the amount of memory to allocate for OpenWRT metarouter? I did several tests allocating memory in the range 32-128MB. I experienced some issue by using 128MB because of the router had not enough memory to run properly.

Thanks again.

janisk · Fri Mar 11, 2011 3:16 pm

as my tests show - then uptime increases.

congo · Sun Mar 13, 2011 6:00 pm

I gave up, and stopped using MetaRouter for anything. It is interesting though, that some people in this forum still recommend running stuff in MetaRouter for production use.

janisk · Mon Mar 14, 2011 9:55 am

if i have metarouter with uptime of more than 2 weeks, and only reason why it is not higher is version updates. Then why not?

Sob · Thu Mar 24, 2011 7:11 pm

Unfortunately, not everyone is so lucky. Allow me to sum up my frustration with metarouter.

About two months ago I received request for recursive DNS resolver with DNSSEC validation. MikroTik is clearly not going to provide anything like that in near future, if ever. No problem, OpenWRT with Unbound running as MR would be perfect solution.
As I was new to MR and not expecting any problems, I started with installed ROS 4.5 or so (at RB450G). Router didn't stay alive for long enough to even allow me to finish the configuration. I upgraded to latest and greatest stable ROS at the time (4.16), but nothing changed. I went through forum and found this thread. Oh well, just my luck that I must get the only model that does not work well, I thought. So the resolver went to other computer that was fortunately also running 24/7.

Another case where MR would be useful came up yesterday. This time with brand new RB493G and ROS 4.17 (I understood that it was mostly RB450G with problems, so why not). And really, it was much better. Except for one strange MR lockup at the beginning (I pretended that I didn't see it), it appeared stable. There were no other problems for the whole time I was playing with it (around three hours). Then I left it running. Today I connected to host router again and oops, there was reboot by watchdog timer in the log. I talked to colleague who was sitting near the router and he said it was beeping (= rebooting) the whole day. Great. So I upgraded ROS to 5.0rc11, just to lose all hope, when first reboot came two minutes after starting MR. The only difference is that instead of "watchdog timer" the reason is now called "cause 1". The router is going to production use tomorrow, without MR of course. I don't have the nerve to explain to customer, that everything under fifty reboots a day is ok.

In both cases the routers were basically idling. Just doing the role of NAT router for three users and 4 Mbits in first case. And completely nothing in second. Installed packages were near the bare minimum (advanced-tools, dhcp, ipv6, ppp, routerboard, security and system). I could probably get rid of advanced-tools and not sure about routerboard, but if I had to touch anything else, I could as well throw out the whole ROS, because those features are required. But anyway, why should the number of packages influence anything? Even if I had every single existing package installed, it should still work.
I could also blame my OpenWRT images, I mean, they were self-compiled, not tested by anyone else, it's possible that there's something wrong with them. But then again, it's not guest OS that's crashing, it's host system. I'm no virtualization expert, but if guest system in VirtualBox on my desktop computer crashed host system, I'd say there's something horribly wrong with VB. Is it any different with MR?
Workarounds like disabling watchdog timer are no solution. If I undestand it correctly, it may sometimes prevent unnecessary reboots, but can also cause the router to freeze completely, which is unacceptable.

I wish myself more luck next time.

reverged · Fri Mar 25, 2011 7:09 am

if i have metarouter with uptime of more than 2 weeks, and only reason why it is not higher is version updates. Then why not?

I believe that is the problem - most of us could not get 2 weeks of uptime with MR running.

I too gave up on MR some time ago. I ended up using a $50 Open-Mesh router reflashed with Backfire to do a simple network monitoring task.

MR could have easily done this task, but it was not stable. The customers could not afford reboots on the main router.

janisk · Thu Mar 31, 2011 3:36 pm

have any of you RB433AH available to try these configurations, just to see if there is a difference.

Also, if possible, can any of you try to disable watchdog on host and run some monitoring software inside metarouter (with virtual ethernet not had port attached to connect host/guest)

Sob · Mon Apr 04, 2011 1:00 am

I can't do much. I don't have any RB433x. Currently I can play only with one remote RB450G, but I must be very careful. Reboots are acceptable outside of busy hours, lockups are not (so no playing with watchdog). Currently it still has ROS 4.16, but this week I should be able to get it upgraded to ROS 5.0.

I can share my OpenWRT images if it might help. One is trunk build from half of december and the other one is from trunk about week old. I have .config for second one. I get the same results with both.

Steps to reproduce:
- import image with 32MB RAM and unlimited disk
- don't add any network interface
- start MR
- open console
- run top in console

The console freezes in matter of seconds, minutes at most. Sometimes even before it gets to command prompt. Reboot by watchdog timer follows shortly after. Automatic Supout is enabled in Watchdog, but doesn't produce anything.

QpoX · Mon Apr 04, 2011 4:58 am

Another case where MR would be useful came up yesterday. This time with brand new RB493G and ROS 4.17 (I understood that it was mostly RB450G with problems, so why not). And really, it was much better. Except for one strange MR lockup at the beginning (I pretended that I didn't see it), it appeared stable. There were no other problems for the whole time I was playing with it (around three hours). Then I left it running. Today I connected to host router again and oops, there was reboot by watchdog timer in the log. I talked to colleague who was sitting near the router and he said it was beeping (= rebooting) the whole day. Great. So I upgraded ROS to 5.0rc11, just to lose all hope, when first reboot came two minutes after starting MR. The only difference is that instead of "watchdog timer" the reason is now called "cause 1". The router is going to production use tomorrow, without MR of course. I don't have the nerve to explain to customer, that everything under fifty reboots a day is ok.

I'm have the same problem on a RG450G did try 4.17 then 5.0rc11 and now 5.0 stable, still reboots. And also writes the same in the logs as you have.
Nice to hear, others having this problem

ett2tre · Fri Apr 08, 2011 10:38 am

remove all the unnecessary packages can bring down the crash numbers a lot. Just think what packages you need and ones you can do without. And it is not PSU related, it just requires good one, as any other device.

Thank you for your answer! Disabling unneeded packages does indeed seem to help.

I'm running ROS 4.16 on a RB450G. The OpenWRT inside metarouter is r18199 as provided by nik247 in the "Metarouter images" thread.

Like you suggested I disabled all ROS packages except routeros-mipsbe, system, and routerboard. This increased the stability greatly, but the router still rebooted ~1-2 times/day with a reference to the watchdog timer. When I stress tested the VM with an infinite loop bash script it was active for about 4hrs at maximum CPU load. Previously that would have killed the VM within minutes.

I have now disabled watchdog and my OpenWRT uptime is at 14hrs and counting (there is no significant load on the system other than a httpd serving some continuously updating cgi scripts for monitoring).

Last time I was tinkering with this problem I remember disabling watchdog led to (temporary) router lockups instead of the reboots. As I'm not sure this will show in the logs of either the RB or in OpenWRT I will write a small ping script to see whether the VM stays up or if the "uptime" in OpenWRT is only how long the VM thinks it's been up.

Is there any other way of detecting these lockups? I am rather proficient on the linux side of things but not so much with ROS, so I apologise if this is a matter of simply "reading the logs."

Good to see the problem is being worked on.

janisk · Fri Apr 08, 2011 4:31 pm

idea is to monitor host 450G from outside and from guest, to see freezes happen internally too, as it is not clear yet. that is, if it is possible to reaech host system from guest when from outside there are no response (6second freeze as i see them)

ett2tre · Fri Apr 08, 2011 6:06 pm

idea is to monitor host 450G from outside and from guest, to see freezes happen internally too, as it is not clear yet. that is, if it is possible to reaech host system from guest when from outside there are no response (6second freeze as i see them)

My uptime in OpenWRT is now 20 hours with heavy load (infinite loop bash script) for at least 5 of those hours. There has been one freeze lasting about 10 minutes, with the host and guest becoming responsive at the same time and without anything in the error logs of either. There has also been two instances of the guest not responding for a very short time (less than 10 seconds), I guess that could be the kind if 6 second freeze you refer to.

I will tweak my monitor script to run inside the guest and monitor the host. Will post results once I have them.

pkelly1603 · Mon Apr 11, 2011 5:26 pm

I don't know if this helps but...

I noticed a router w/ metarouter that was rebooting like crazy. I logged in when the metarouter had locked up (but router was still responding). Before it rebooted I looked at the metarouter through the virtual console. It spit out a bunch of strange information (and rebooted shortly after).

Oops[#1]:
Cpu 0
$ 0 : 00000000 10008200 0000004c c0387d90
$ 4 : c0387d90 c03e5cf0 0000000f 00000001
$ 8 : 00000076 00000000 00000001 00000000
$12 : c0355a90 00000000 178cb0c1 c03e5ce8
$16 : c03e5cf0 0000000f c03e5d08 0000004c
$20 : c0d73000 c03e5cf8 c03e5dd0 c0cb1b80
$24 : c03e5d40 000186a0
$28 : c03e4000 c03e5cb0 c03e5cf8 c01d61b4
Hi : 00000000
Lo : 00000000
epc : 8d53df8c 0x8d53df8c
Not tainted
ra : c01d61b4 send_request+0x78/0x94
Status: 10008203 KERNEL EXL IE
Cause : 10000010
BadVA : 8d53df8c
PrId : 0001800a (MIPS 4Kc)
Process swapper (pid: 1, threadinfo=c03e4000, task=c03e0000, tls=00000000)
Stack : c03e5d4c 0000004c c0c2a140 c0cb1b80 00000001 00000001 c03e5cf0 0000000f
c0cc5000 c03e5d08 c01d6cec c02ed030 c0198fdc c01d61b4 c0d71000 c03e5d4c
00000002 00000000 00000001 64657600 00008001 c0d6f000 00000000 00000000
0000037e 00000000 00000800 000041ed 00000001 00000000 00000000 00000000
4b4c5e18 00000000 4b4c5e18 00000000 0000004a 00000000 00001000 00000000
...
Call Trace:
[<c01d6cec>] mfs_lookup+0xb8/0x12c
[<c0198fdc>] d_alloc+0x34/0x208
[<c01d61b4>] send_request+0x78/0x94
[<c018cad4>] do_lookup+0xe4/0x1e4
[<c018e2d8>] __link_path_walk+0x10c/0x1134
[<c018e650>] __link_path_walk+0x484/0x1134
[<c020a88c>] vsnprintf+0x584/0x5fc
[<c01a94e0>] propagate_mnt+0x1bc/0x1e8
[<c018f350>] path_walk+0x50/0xb8
[<c018f518>] do_path_lookup+0x10c/0x148
[<c018f518>] do_path_lookup+0x10c/0x148
[<c0184914>] get_empty_filp+0x78/0x120
[<c018e2d8>] __link_path_walk+0x10c/0x1134
[<c01901c8>] __path_lookup_intent_open+0x60/0xbc
[<c019023c>] path_lookup_open+0x18/0x24
[<c018c504>] path_put+0x18/0x44
[<c0190464>] do_filp_open+0x9c/0x810
[<c01260d4>] release_console_sem+0x230/0x2b8
[<c019c8ac>] alloc_fd+0x74/0x154
[<c0180e04>] do_sys_open+0x6c/0x118
[<c01267c8>] printk+0x24/0x30
[<c01015f4>] init_post+0x30/0xfc
[<c0107080>] kernel_thread_helper+0x10/0x18
[<c0107070>] kernel_thread_helper+0x0/0x18

Code:<4>unaligned data access at c010a76c
unaligned data access at c010c654
Unhandled kernel unaligned access[#2]:
Cpu 0
$ 0 : 00000000 10008200 00000000 00000000
$ 4 : 8d53df80 00001161 00000001 c0389174
$ 8 : 00000d0a c03bf82e 00000010 63303130
$12 : 00002201 00000000 178cb0c1 c03e5ce8
$16 : c03e5aa8 8e270000 00000000 00000000
$20 : 00000008 c0310000 c03e5dd0 c0cb1b80
$24 : 00000010 000186a0
$28 : c03e4000 c03e5a80 c03e5cf8 c010c5b4
Hi : 00000000
Lo : 00000000
epc : c010c654 do_ade+0x1b4/0x3c0
Not tainted
ra : c010c5b4 do_ade+0x114/0x3c0
Status: 10008202 KERNEL EXL
Cause : 00000010
BadVA : 8d53df80
PrId : 0001800a (MIPS 4Kc)
Process swapper (pid: 1, threadinfo=c03e4000, task=c03e0000, tls=00000000)
Stack : 00001137 c010a76c c0340000 8e270000 fffffffa 8d53df80 00000000 fffffffd
c0104de0 0000004c c038446e 0000000c c0340000 00000004 10008200 c0126748
00000000 10008200 00000000 00000001 c0346de0 c03e5a2d 00000000 c0389174
6f64653a 00000003 00000001 436f6465 00002201 00000000 178cb0c1 c03e5ce8
fffffffa 8d53df80 00000000 fffffffd 00000008 c0310000 c03e5dd0 c0cb1b80
...
Call Trace:
[<c010c654>] do_ade+0x1b4/0x3c0
[<c0104de0>] ret_from_exception+0x0/0x24
[<c010a76c>] show_code+0x80/0x114
[<c010b528>] show_registers+0x98/0xb0
[<c010b5c4>] die+0x84/0x108
[<c010c820>] do_ade+0x380/0x3c0
[<c0104de0>] ret_from_exception+0x0/0x24

Code: 00431024 14400054 00000000 <88920000> 98920003 24020000 080431a5 0000
0000 8f830018
---[ end trace 8f3d94c9bc041f2f ]---
Kernel panic - not syncing: Attempted to kill init!

pkelly1603 · Mon May 09, 2011 7:24 pm

Is there going to be a fix for this or will metarouters always have these bugs?

janisk · Fri May 13, 2011 9:43 am

not all boards has this problem with metarouter. And yes there will be a fix for the issue eventually.

liquidcz · Mon May 23, 2011 10:34 pm

not all boards has this problem with metarouter. And yes there will be a fix for the issue eventually.

Im get caught too, by buing RB450G for virtualization.

So,.... please, can you publish list of 100% compatible routerboards for virtualization? I had to decide witch RB i had to buy for it.

pkelly1603 · Tue May 24, 2011 9:31 pm

not all boards has this problem with metarouter. And yes there will be a fix for the issue eventually.

What routerboards actually work consistently with metarouter? Any model I have ever tried it on is unstable.

alafangla · Tue Jun 28, 2011 3:01 pm

hi, as i see i am next person having this problem

i bought RB450G in january 2011

virtualized openwrt
tried 5 images all with same problems - reboot of entire RB (by watchdog) or openwrt freezes
tried routeros v4.17, v5.5 - doesnt help
tried at 400MHz and 680MHz - doesnt help
cpu, ram, flash utilization of RB is low (also in openwrt)
tried to make simple config (acting as switch) - doesnt help
tried to disable most packages - help a little

tried many combinations of ram and flash alocation for openwrt - doesnt affect uptime
tried 24v/0.8a power supply and 24/1a POE - same behavior
out of ideas

maybe somebody should try to compile another OS than openwrt

janisk · Wed Jun 29, 2011 4:02 pm

What routerboards actually work consistently with metarouter? Any model I have ever tried it on is unstable.

i have running RB433AH, RB1000, RB800 on my table that are not rebooting. All of them have at least one metarouter guest running that is RouterOS. Also, all have 32MB of RAM assigned, since 16MB with all packages installed is not enough.

petrn · Wed Jun 29, 2011 6:41 pm

What routerboards actually work consistently with metarouter? Any model I have ever tried it on is unstable.
i have running RB433AH, RB1000, RB800 on my table that are not rebooting. All of them have at least one metarouter guest running that is RouterOS. Also, all have 32MB of RAM assigned, since 16MB with all packages installed is not enough.

I had to move to RB800 too from R450G - i needed to run two metarouters - RouterOS , RB450G keeps rebooting after 15 - 20 mins, no autosupout or any other output. Tryed V4.17 and few V5.x before 5.3

pkelly1603 · Fri Jul 15, 2011 4:31 pm

What routerboards actually work consistently with metarouter? Any model I have ever tried it on is unstable.
i have running RB433AH, RB1000, RB800 on my table that are not rebooting. All of them have at least one metarouter guest running that is RouterOS. Also, all have 32MB of RAM assigned, since 16MB with all packages installed is not enough.

Are your test routerboards doing anything or just sitting there? I think it has something to do with stress on them that makes them reboot because I see a lot of variation in the amount of time it takes for them to reboot. I tried the same settings as you have on a 433AH and still have the same problems (I have even tried with all unnecessary packages disabled also).

alafangla · Wed Jul 20, 2011 2:24 am

hi,
i own rb450g
do you even assume what should cause this problem?
i am sure it is NOT psu related
i have already tried 24V/0.8A (20W) and 19V/4.74A (90W) psu and 24V/1A (24W) POE - uptime was never longer than 10-15 minutes (whole routeros restart or openwrt "freezes")
also i read here about disabling all unnecessary packages
i left enabled only "system" and "routerboard" (that 2 cant be disabled)
the uptime raises from 10-15 minutes to 1 hour
i tried at least 4 different openwrt images and uptimes was almost same
all of this tried with ROS4.17 then ROS5.5
i need at least 2 weeks
so again, do you even assume what should cause this problem?
is it HW or SW related?
if you have some ideas, i can test or send anything you want

janisk · Fri Jul 22, 2011 10:57 am

if resources are available (router has few % of cpu left and there is ram) i have seen no difference in reboot frequency with or without load. Even simple usage patterns did not cause it to reboot more.

Reboots usually where done by watchdog, disabling it - revealed that router freezes from time to time.

At the moment idea is that problem is software related, but has to be tested on different hardware (like RB433AH - same cpu, decent amount of RAM). And that problem is, that something does not like MetaROUTER being ran on the RB450G.

pkelly1603 · Fri Jul 29, 2011 4:33 pm

I believe it is software related. I have the same problems on 433AH as 450g.

Also, I have noticed that routers with worse connections (higher packet loss) seem to reboot more frequently.

janisk · Mon Aug 01, 2011 12:22 pm

watchdog may reboot your router if it does not get the time-slice to do the "check if I am alive" and that is not a problem, since router is loaded 100% and for some long time, meaning load on the router is too big in current configuration and should be looked at and optimised if possible.

You should check with router that is not fully loaded (like 80% - 95 % load) and you should see that RB433AH will stop rebooting.

Last week i removed RB433AH that where running old version of RouterOS with several MetaROUTER guests running with 8 month uptime. Also, there might be other problems that causes reboots in your case, since i ran minimal feature set on the board to ensure that all the problems are caused by metarouter on the router.

alafangla · Thu Aug 04, 2011 3:20 pm

something new?
still having this problem, and metarouter was only reason why i didnt bought ubiquiti routerstation pro
it is possible to return this device to reseller, because metarouter is key feature for me, and it is not working?

janisk · Tue Aug 09, 2011 2:03 pm

you have to contact reseller and then the same question.

pkelly1603 · Tue Aug 16, 2011 5:43 pm

It sounds to me like Mikrotik is going to deny the existence of this problem. Then they don't have to deal with it!

janisk · Wed Aug 17, 2011 11:57 am

you should learn to read. Problem is known and admitted long time ago. Since there is no reliable way to reproduce and causes are not known problem still persist. RB540G has the problem, while RB433AH with the same CPU does not. Only dependency for MetaROUTER is the CPU.

If someone wants to return the router only because it does not have working feature, he has to contact one who have sold it and make arrangements.

alafangla · Fri Sep 02, 2011 10:30 am

some suggestions to solve this problem?

janisk · Mon Sep 05, 2011 11:40 am

to sum it all up on RB450G:
1) choose other board, even it is MIPSBE
2) install less packages (minimal set for the task)
3) disable watchdog, if lucky, you will encounter short freezes, and everything will continue to work.

Your experience may very. Situation can change after reboot.

alafangla · Fri Sep 09, 2011 7:49 pm

your advice is to buy another board with same problem

janisk · Mon Sep 12, 2011 10:12 am

your advice is to buy another board with same problem

my advice is to read carefully

. RB433AH have the same CPU but does not have same MetaROUTER problem.

alafangla · Tue Sep 13, 2011 11:54 am

you dont know what cousing this problem, so how you know rb430ah dont freeze
it is same with rb450g, not all are freezing

janisk · Tue Sep 13, 2011 1:39 pm

as far as i have tested eventually all RB450G have frozen. As devs are explaining, only parts that metarouter uses are CPU and RAM. Testing RAM and CPU under load did not show any problems with the parts. So no, I do not know where is the problem. If i knew, i could point devs and that would have been fixed long long long time ago.

What is written in this thread that some RB450G can freeze with larger interval some with shorter, and that interval can change.

QpoX · Tue Sep 13, 2011 6:12 pm

Ever since i got RB450G i had the metarouter problem... And since then i've been trying to find the problem.
Now my testing show's that it is a mr-network problem.

If i add 8 RouterOS metarouter's with no network connection, it runs fine for weeks, not even a single problem.
But if i add a interface (dynamic or static) to just one of the metarouter's, i starts to freak out and get unstable!
And then i can't connect to the metarouter(s) or the RB450G even not on the serial port!
Then after some time (all from a few min to hours/days) it comes back on the network and serial, then i can connect and disable the interface, then it runs fine for weeks again...

Now does who have a RB450G, can i get you to try this?
Clean install on RB450G and then add 8 Metarouters with RouterOS (no network interface) and let something like the dude monitor it (just to see if ping works).
Then let it run for a week or two.
If it still runs with no problems, try and add just one interface to a metarouter.
Then let it run for as long as it can (or just up to 1-2 weeks), then write here, with your experience.

janisk · Wed Sep 14, 2011 2:07 pm

after reading this in the morning i set up RB450G with metarouter configured as follows:

/metarouter add name=mr1 memory=32
/interface virtual-ethernet add
/metarouter interface add virtual-machine=mr1 type=static static-interface=ether5
/metarouter interface add virtual-machine=mr1 type=static static-interface=ether1

connected ether5 with ether1 using wire.

[admin@450G-2] > sy resource print 
                   uptime: 3h57m29s
                  version: 5.6
              free-memory: 207336KiB
             total-memory: 257120KiB
                      cpu: MIPS 24Kc V7.4
                cpu-count: 1
            cpu-frequency: 680MHz
                 cpu-load: 4%
           free-hdd-space: 464132KiB
          total-hdd-space: 520192KiB
  write-sect-since-reboot: 132
         write-sect-total: 1324790
               bad-blocks: 1.5%
        architecture-name: mipsbe
               board-name: RB450G
                 platform: MikroTik

pinging the hosts (since host is one that do not respond. (looking at max RTT since packets could stay in hosts receive buffer and the replied when router unfreezes)

no problems yet

will leave for some time.

edit: it was not crashing with metarouter running w/o interfaces for some time w/o crashing as described. Also, i have OSPF running on the router.

QpoX · Mon Sep 26, 2011 6:07 pm

after reading this in the morning i set up RB450G with metarouter configured as follows:
Code: Select all
/metarouter add name=mr1 memory=32
/interface virtual-ethernet add
/metarouter interface add virtual-machine=mr1 type=static static-interface=ether5
/metarouter interface add virtual-machine=mr1 type=static static-interface=ether1
connected ether5 with ether1 using wire.
Code: Select all
[admin@450G-2] > sy resource print 
                   uptime: 3h57m29s
                  version: 5.6
              free-memory: 207336KiB
             total-memory: 257120KiB
                      cpu: MIPS 24Kc V7.4
                cpu-count: 1
            cpu-frequency: 680MHz
                 cpu-load: 4%
           free-hdd-space: 464132KiB
          total-hdd-space: 520192KiB
  write-sect-since-reboot: 132
         write-sect-total: 1324790
               bad-blocks: 1.5%
        architecture-name: mipsbe
               board-name: RB450G
                 platform: MikroTik
pinging the hosts (since host is one that do not respond. (looking at max RTT since packets could stay in hosts receive buffer and the replied when router unfreezes)

no problems yet

will leave for some time.

edit: it was not crashing with metarouter running w/o interfaces for some time w/o crashing as described. Also, i have OSPF running on the router.

I don't use OSPF on it, just a clean install and setup with metarouter.
But i think it is funny that i on my RB450G can replicate this problem time after time and even after a netinstall...
Maybe it is a faulty board then?
The only thing i wanted it to do, was run metarouters, but stacking RB750 seems to be a better idea...

timberwolf · Mon Sep 26, 2011 6:29 pm

The only thing i wanted it to do, was run metarouters, but stacking RB750 seems to be a better idea...

I've come to the same conclusion...
The 256MB of RAM in the RB450G are a total waste of ressources(and money) without Metarouter.

janisk · Tue Sep 27, 2011 10:21 am

running metarouter on board that before was the worst candidate if talking about stability of MetaROUTER, so the stats:

[admin@450G-2] > sy resource print 
                   uptime: 1w6d1m55s
                  version: 5.6
              free-memory: 207392KiB
             total-memory: 257120KiB
                      cpu: MIPS 24Kc V7.4
                cpu-count: 1
            cpu-frequency: 680MHz
                 cpu-load: 4%
           free-hdd-space: 464132KiB
          total-hdd-space: 520192KiB
  write-sect-since-reboot: 419
         write-sect-total: 1325077
               bad-blocks: 1.5%
        architecture-name: mipsbe
               board-name: RB450G
                 platform: MikroTik

[admin@450G-2] > sy package print 
Flags: X - disabled 
 #   NAME                                               VERSION                                              SCHEDULED              
 0   system                                             5.6                                                                         
 1   hotspot                                            5.6                                                                         
 2   routerboard                                        5.6                                                                         
 3   ipv6                                               5.6                                                                         
 4   ppp                                                5.6                                                                         
 5   security                                           5.6                                                                         
 6   mpls                                               5.6                                                                         
 7   wireless                                           5.6                                                                         
 8   advanced-tools                                     5.6                                                                         
 9   option                                             5.6                                                                         
10   routing                                            5.6                                                                         
11   ntp                                                5.6                                                                         
12   dhcp                                               5.6

[admin@450G-2] > metarouter print 
Flags: X - disabled 
 #   NAME                                                                      MEMORY-SIZE     DISK-SIZE     USED-DISK STATE        
 0   mr1                                                                             32MiB     unlimited        277kiB running

[admin@450G-2] > metarouter interface print 
Flags: X - disabled, A - active 
 #   VIRTUAL-MACHINE                                    TYPE    STATIC-INTERFACE                                   VM-MAC-ADDRESS   
 0 A mr1                                                static  ether5                                             02:1B:7E:B1:B7:E2
 1 A mr1                                                static  vif1                                               02:16:16:90:EF:0E

[admin@450G-2] > interface virtual-ethernet print 
Flags: X - disabled, R - running 
 #    NAME                                                                                          MTU ARP        MAC-ADDRESS      
 0  R vif1                                                                                         1500 enabled    02:8B:7B:3C:33:B0

ether1 and ether5 is connected by wire

running metarouter stats:

[admin@MikroTik] > sy resource print 
                   uptime: 1w6d6m18s
                  version: 5.6
              free-memory: 20776KiB
             total-memory: 29708KiB
                      cpu: MIPS 4Kc V0.10
                cpu-count: 1
            cpu-frequency: 680MHz
                 cpu-load: 1%
           free-hdd-space: 464128KiB
          total-hdd-space: 464405KiB
  write-sect-since-reboot: 0
         write-sect-total: 0
               bad-blocks: 0%
        architecture-name: mipsbe
               board-name: RB MetaROUTER
                 platform: MikroTik

[admin@MikroTik] > interface print 
Flags: D - dynamic, X - disabled, R - running, S - slave 
 #     NAME                                                                                  TYPE               MTU L2MTU  MAX-L2MTU
 0  R  ether1                                                                                ether             1500
 1  R  ether2                                                                                ether             1500

ping from neighbor router looks like this:

10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 1ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 1ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 1ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 0ms  
10.5.45.5                                  56  63 0ms  
    sent=80 received=80 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=2ms

changes. before router was set up it had bad caps (the green ones that where bulged up) changed. after that nothing has been done. time difference in host and guest uptime is time to make printouts and pasting them here and doing some rearranging.

barkas · Tue Sep 27, 2011 10:57 am

You can't seriously try to tell us that there's nothing wrong.

timberwolf · Tue Sep 27, 2011 11:04 am

Janis,

we have 3 brand new 450G in our test-setup, ALL of them are practically useless regarding stability.

janisk · Tue Sep 27, 2011 3:47 pm

You can't seriously try to tell us that there's nothing wrong.

all i did was report back to persons interested on how the test is going. Obviously - bad. Problem did not manifest it self, but it should have done that. At least i was hoping for that to happen.

Maybe someone could try to check this:

[admin@450G-2] /interface ethernet switch> print 
Flags: I - invalid 
 #   NAME                        TYPE         MIRROR-SOURCE                      MIRROR-TARGET                      SWITCH-ALL-PORTS
 0   switch1                     Atheros-8316 none                               none                               yes

All i need is to make precise scenario to recreate the problem in relatively short time.

timberwolf · Tue Sep 27, 2011 3:55 pm

Well in my case, the setting was set to switch-all-ports NO. But I thought you could see this from the last suppouts I sent a while ago.
I will do another test as time allows.

janisk · Tue Sep 27, 2011 3:59 pm

trying to have more open conversation, so if someone who wants to help out with same problem could fill in. I am unhappy with the testing results - problem is - no scenario on how to recreate it hence, nothing useful to report to devs. It just starts to freeze when it gets "bad mood" or something.

timberwolf · Tue Sep 27, 2011 4:05 pm

Janis, lets see if I got this right.
1.) You test with a RB450G.
2.) You have setup your switch to switch all ports.
3.) It runs stable for you.

What if you change switch all ports to no?
In my case problems startet right away.

janisk · Tue Sep 27, 2011 4:23 pm

set to switch-all-ports=no lets wait for the results.

if you set to switch all ports=yes, do the problem stop for you?

edit: left running and sending ping requests to the router. will leave it for the night.

timberwolf · Tue Sep 27, 2011 5:05 pm

I started a quick test at my live router.
I added one Metarouter with one local dynamic interface.
ROS 5.6
Uptime at the moment 5w3d4h53m30s.
I will report back in tomorrow.

EDIT: switch all ports was set to no, I changed it to yes.

QpoX · Tue Sep 27, 2011 5:27 pm

set to switch-all-ports=no lets wait for the results.

if you set to switch all ports=yes, do the problem stop for you?

edit: left running and sending ping requests to the router. will leave it for the night.

Did just change mine from "yes" to "no" and has been running fine for about 5mins now (4½min longer than last test).
And the dude is on the case

*EDIT* Now 2hours later and it still runs fine... i will let it run for a day or two (maybe a week), then try the old settings and see if it fails again.

barkas · Wed Sep 28, 2011 12:14 am

nope, died on me after 2 hours

QpoX · Wed Sep 28, 2011 2:26 am

nope, died on me after 2 hours

Mine did run for 7hours! then the dude told me that is was down, then 2mins later it was up again.
5mins after it came up again, it went down for about 5mins and came up.
And has now been running for 1½ hours with no downtime.

janisk · Wed Sep 28, 2011 8:31 am

no crashes/freezes during the night

timberwolf · Wed Sep 28, 2011 11:09 am

2 Metarouters, just pinging to the host, stable since yesterday 15:00 UTC.
Each Metarouter doesn't show a single loss, max ping rtt 13-14ms.

/metarouter
add disabled=no disk-size=32000kiB memory-size=32MiB name=chaos
add disabled=no disk-size=32000kiB memory-size=32MiB name=chaos2
/metarouter interface
add disabled=no dynamic-bridge=none dynamic-mac-address=02:1B:7E:B1:B7:E2 \
    type=dynamic virtual-machine=chaos vm-mac-address=02:45:5F:D0:20:B1
add disabled=no dynamic-bridge=none dynamic-mac-address=02:52:9C:ED:DC:EF \
    type=dynamic virtual-machine=chaos2 vm-mac-address=02:A7:68:6A:E4:9B

/ip address
add address=192.168.101.1/24 disabled=no interface=vif1 network=192.168.101.0
add address=192.168.102.1/24 disabled=no interface=vif2 network=192.168.102.0

I will change the setup now, to using an externel ethernet port.

timberwolf · Wed Sep 28, 2011 11:34 am

Conclusion by now: As both barkas and QpoX both have crashes regardless the switch switch-all-ports setting, I think we can rule that out.
Significant difference between my setup and barkas setup is, that my Metarouter does nothing more than just pinging over one interface, and barkas does full MPLS tests.
Janis maybe you could do some MPLS tests between Metarouter and host or some load tests using ipsec? Anything that touches ethernet/ip stacks and/or produces some load...

QpoX · Wed Sep 28, 2011 2:15 pm

Here is my Log:

rb450g-Clipboard01.jpg

janisk · Wed Sep 28, 2011 2:19 pm

i have had router with metarouter freeze up without MPLS. I have one suspect that i am testing out, it crashed once without debug MetaROUTER, now router sits at devs with debug version of MetaROUTER. That is thanks to new board that has similar structure as RB450G where debug mode or MetaROUTER can be monitored (it cannot be done or RB450G).

Fingers crossed it crashes again.

some details - Router was just idling same as metarouter, and then it rebooted.

timberwolf · Wed Sep 28, 2011 2:45 pm

Ok, my new Metarouter test crashed after about 3 hours.

Short sketch of the config:
1 Metarouter, 32MB RAM&Disk
2 Virtual Interfaces, one dynamic as bridge port, one static to ether2.
Pinging from Metarouter to host.
Pinging from another System connected at ether2 to Metarouter.
Only things configured in Metarouter where one IP per interface and System name.

Packages:

/system package print
Flags: X - disabled
 #   NAME                    VERSION                    SCHEDULED
 0   routeros-mipsbe         5.6
 1 X ipv6                    5.6
 2   system                  5.6
 3   mpls                    5.6
 4 X hotspot                 5.6
 5 X wireless                5.6
 6   routerboard             5.6
 7   advanced-tools          5.6
 8   security                5.6
 9   ppp                     5.6
10   routing                 5.6
11   dhcp                    5.6
12   ntp                     5.6

Switch all ports ist still set to yes.

I'am sorry, but that is about all I can supply as tests by now.

timberwolf · Wed Sep 28, 2011 3:00 pm

That is thanks to new board that has similar structure as RB450G where debug mode or MetaROUTER can be monitored (it cannot be done or RB450G).

Your devs forgot to include the AR7116 in the JTAG scan chain on the RB450G?

QpoX · Thu Sep 29, 2011 2:00 am

Here is my final log of this test... so it's no working solution...
Now all metarouter interfaces have been disabled again, and it runs stable, but no network to the virtual machines.

RB450G_Clipboard02.jpg

timberwolf · Tue Oct 11, 2011 2:46 pm

@MT/janisk:
Any progress to report?

robertpenz · Sat Oct 15, 2011 2:14 pm

I also would be really interested in the progress .....

sigxcpu · Sat Oct 15, 2011 8:31 pm

Unfortunately, who is in charge in solving the problem has no interest

janisk · Mon Oct 17, 2011 9:03 am

Unfortunately, who is in charge in solving the problem has no interest

just stop spamming the forums.

xdemon · Fri Oct 21, 2011 5:05 pm

Any updates about that problem? Reboots for me with metarouter, but works perfectly stable without. RB493G, 5.7

reverged · Sat Oct 22, 2011 12:05 am

Does anyone know if this problem has been seen/reported on RB750GL?

I have, yet again, a useful purpose for a metarouter.

timberwolf · Sat Oct 22, 2011 11:07 am

I guess you only could try yourself, there is extremely less feedback on Metarouters altogether.
And, without wanting to bash MT, there seems to be no clear understanding of the cause on MIPSBE, so every MIPSBE based board should be viewed as affected. Even a small software update might tip those boards over the edge.

sigxcpu · Sun Oct 23, 2011 1:19 am

Unfortunately, who is in charge in solving the problem has no interest
just stop spamming the forums.

If people are asking for progress and we don't get any answer, why would you characterize my comments as spam?

@xdemon: I've never reached 24 hours uptime on 750GL with MetaROUTER activated (with OpenWRT guest). I don't know about ROS guest, didn't test. And btw, the VM was doing very light DNS traffic for home LAN.

barkas · Sun Oct 23, 2011 1:45 am

I don't think it's traffic related. I don't really use mine except for testing now and then and it reboots often enough.

janisk · Mon Oct 24, 2011 11:01 am

unfortunately it is all I can disclose. and even then some of info is just a guess

so, there are reports of metarouter hanging on RB450G and RB493G. Anyone with RB435G or RB411AH can try metarouter on it?

for me RB433AH does not crash. Now stuck with RB450G that does not reboot

timberwolf · Mon Oct 24, 2011 11:19 am

I can only repeat and clarify what I said in earlier posts:

The whole MIPSBE series should be considered unusable in combination with Metarouter.
Especially with the information available from MT and the fact that they aren't capable of fixing this problem in such a long time.

@MT:
This is not intended as pun, just as a warning to everyone planing on using Metarouter on MIPSBE based MT boards.
And don't start repeating that Metarouter is stable on "some" MIPSBE boards, this is just plain luck and might change with every change/release you make.

xdemon · Mon Oct 24, 2011 6:58 pm

It doesn't stable even with routeros virtualized. Crashed after 10 mins with router reboot. metarouter-fs used 40-80% of cpu first 3 mins. I have autosupout.rif file, but i can't find enything useful in it.

UPD: looks like it working after reboot.

timberwolf · Mon Oct 24, 2011 9:43 pm

@xdemon:
Crashes may occur anywhere between 1minute to nearly 1day. The rate increases as you do more work inside the MR.

janisk · Tue Oct 25, 2011 9:54 am

It doesn't stable even with routeros virtualized. Crashed after 10 mins with router reboot. metarouter-fs used 40-80% of cpu first 3 mins. I have autosupout.rif file, but i can't find enything useful in it.

UPD: looks like it working after reboot.

send that supout.rif to support.

reverged · Thu Oct 27, 2011 5:38 am

I can only repeat and clarify what I said in earlier posts:

The whole MIPSBE series should be considered unusable in combination with Metarouter.
Especially with the information available from MT and the fact that they aren't capable of fixing this problem in such a long time.

@MT:
This is not intended as pun, just as a warning to everyone planing on using Metarouter on MIPSBE based MT boards.
And don't start repeating that Metarouter is stable on "some" MIPSBE boards, this is just plain luck and might change with every change/release you make.

Is this problem isolated to MIPSBE or all BE processors? LE architectures do not have the problem?

If so, those are huge clues.

janisk · Thu Oct 27, 2011 9:46 am

MIPSLE boards do not support metarouter, hence - they do not suffer from this problem. As far as i know various RB4XXG models do suffer from this problem (RB493G, RB450G, RB435G). Never seen this problem on RB433AH and similar.

timberwolf · Thu Oct 27, 2011 11:01 am

@janis:
What he meant where LE architectures in general, not necessarily MIPS-LE, I guess.
And yes, this could have an influence but at a point where it's already to late out of a developers perspective.

janisk · Thu Oct 27, 2011 11:11 am

le, is x86 and mipsle, both of the does not have MetaROUTER.

and this problem is not out of devs perspective. Just there is not a reliable way to reproduce it. As sometimes something starts to happen after 2 weeks other time, restarts happen once in 10 minutes.

reverged · Fri Oct 28, 2011 4:54 am

What he meant where LE architectures in general, not necessarily MIPS-LE, I guess.

Yes

le, is x86 and mipsle, both of the does not have MetaROUTER.

Interesting. I did not know le, and in particular x86, doesn't support metarouter! This saves me a hard lesson.

janisk · Fri Oct 28, 2011 9:05 am

on x86 you go KVM to virtualize.

Rivera · Tue Nov 01, 2011 4:28 pm

Can anyone run 'while true;do dmesg -c;done' and see what happens? And please leave that command running in openwrt (for example in background)

Rivera · Tue Nov 01, 2011 4:48 pm

And yes, there is something i discovered in process of testing:

1) netconsole. There is no error logs sent via network - only standart boot data.
2) There is no crashdumps in openwrt machine. Basically everything is okay with openwrt.
3) i tried watching dmesg, and there is no error messages too.
4) Mikrotik MetaRouter (with RouterOS machine) sometimes hangs too.
5) And... sometimes reboots of router (physical, not virtual one)

So... this is 100% RouterOS fault. I sent supout file to mikrotik.

P.S. if we only had access to more verbose logs...

barkas · Tue Nov 01, 2011 5:03 pm

P.S. if we only had access to more verbose logs...

But we don't, because the machine crashes hard.

Rivera · Fri Nov 11, 2011 3:31 pm

v 5.8, still broken. Also crashes host machine.

Rivera · Fri Nov 11, 2011 9:47 pm

Some interesting info. I enabled writing debug logs to disk, and tried to catch some messages.

Of course i haven't found any useful info, but...

There were no virtual machine freezes
Instead, router crashes hard. I always saw that first virtual machine freezes, and then router crashes (if you do not disable VM via /metarouter _before_ router crashes - if you will be fast and disable VM router will not crash)
I tried that, well, somewhere around 10 times. VM _do not freeze_. Router crashes anyway...

Maybe that somehow related to filesystem?

rviteri · Fri Nov 18, 2011 5:56 pm

I am just using metarouter so I can load OpenVPN into OpenWRT to use UDP as the transport protocol. If MT could just add the UDP option for OpenVPN instead of only TCP I bet it would solve some problems out there.

mitjax · Thu Dec 29, 2011 1:07 pm

Any news on this RB450G + Metarouter + OpenWRT problem?
I am running RB450G v5.11 and still the board crashes.

Mikrotik?

timberwolf · Tue Jan 03, 2012 6:44 pm

I wouldn't hold my breath.
Maybe a x86 Board like Soekris net6501 with KVM will be more usable; I didn't try.
I just use MikroTik/ROS for testing and home usage and gave up all planned and realized productive installations.

alafangla · Tue Jan 17, 2012 11:39 pm

unfortunately, we already paid for mikrotik board, so why are you wondering about no support...?

this isn`t good way to atract new customers, shame on you MT. i bought RB450G, it was my first and last product from you.

elgo · Wed Jan 18, 2012 11:38 am

I just use MikroTik/ROS for testing and home usage and gave up all planned and realized productive installations.

unfortunately, we already paid for mikrotik board, so why are you wondering about no support...?
this isn`t good way to atract new customers, shame on you MT. i bought RB450G, it was my first and last product from you.

I couldn't agree more

Iliasla · Thu Feb 02, 2012 6:08 am

Perhaps the problem is really in miscalculations in the design of power converters on the routerboard 450G. With a standard external power supply DVE DSA-20P-20 24V/0.8A router, when running metarouter, was unstable and often restart at arbitrary times, even in idle mode. ROS 3x, 4x, 5x - no difference. However, with the simple old transformer power supply 12V/0.5A router works fine for more than a day, with a test load in the form of Ping Speed between the router and metarouter (average 163 Mbps at a reduced up to 400 MHz cpu clock).
By the way, lowering the CPU clock from 680 MHz to 400 MHz did not affect the instability.
When using low voltage power supply, became less warm coils and capacitors in both the power converters to 3.3V on routerboard, which is an indirect proof of engineering miscalculation. And stronger heated coil, which supplies the switch chip and memory. In this scheme fits the fact, that router is a bit more stable in the mode "switch-all-ports=no".
IMHO, the designers of this model were to use a more powerful power converter, similar to that used in RB433x.
Another is the idea of replacing the electrolytic capacitors on the solid polymer capacitors. It is not excluded, that this may also solve the problem.

ps Forgive me my nice Google English

janisk · Thu Feb 02, 2012 9:51 am

the frequency of reboots of the board is not connected to PSU in any way, of course, board has to have enough resources. It was tested at full CPU load.

peson · Thu Feb 02, 2012 10:42 am

I guess you only could try yourself, there is extremely less feedback on Metarouters altogether.
And, without wanting to bash MT, there seems to be no clear understanding of the cause on MIPSBE, so every MIPSBE based board should be viewed as affected. Even a small software update might tip those boards over the edge.

It's not just the MIPSBE, I have the same behavior with PPC platform, RB1000 and RB1100AH tested.
My bet is that the MetaROUTER software is unstable.
Have anyone configured the guest interfaces in dynamic mode only?
I'm wondering if the setup of static interfaces is not correctly configured in the underlaying Linux kernel.
I was convinced that static interfaces is a HW bypass directly to the guest, but I don't think so any more.

janisk · Fri Feb 03, 2012 8:36 am

what configuration you have? All problems about MR on PPC routers was resolved by adjusting configuration

peson · Fri Feb 03, 2012 9:27 am

what configuration you have? All problems about MR on PPC routers was resolved by adjusting configuration

I have a support ticket that you are familiar with.
/Paul

reverged · Sat Feb 04, 2012 10:16 pm

what configuration you have? All problems about MR on PPC routers was resolved by adjusting configuration

Can you elaborate on this? What specific configuration changes resolve MR issues on PPC routers?
Is there a wiki of sorts?

I had given up on Metarouter until mipsbe becomes stable. But if ppc is stable....

I'm installing an RB1100AH next week and could use MR for lots of things - if ppc Metrouter is stable.

timberwolf · Sat Feb 04, 2012 11:43 pm

Just don't configure anything and everything will be almost fine.

peson · Mon Feb 06, 2012 11:20 am

Just don't configure anything and everything will be almost fine.

Timberwolf, so your opinion is not to use MR at all, at least not until a stable version is available?

timberwolf · Mon Feb 06, 2012 11:32 am

Timberwolf, so your opinion is not to use MR at all, at least not until a stable version is available?

Yes, that is exactly my opinion.
And my impression is, that this will be never.

peson · Mon Feb 06, 2012 11:46 am

Timberwolf, so your opinion is not to use MR at all, at least not until a stable version is available?
Yes, that is exactly my opinion.
And my impression is, that this will be never.

So, your saying MR in general on all platforms including PPC platform?
And you're not agree on the statement? :
http://forum.mikrotik.com/viewtopic.php ... 11#p301963

Do you have any experience of ROS x86 KVM stability?