MetaROUTER stability issues on certain MIPSBE and PPC boards

reverged · Thu Sep 06, 2012 4:59 am

...
I have also noticed that once the "bad" router gets stuck in a freeze-up pattern, occasionally the health stats show some real bizarre numbers which get "stuck" there. For example, right now as I am typing this, the "bad" router is in the middle of one of its freeze-up loops, and "/system health print interval=1" is stuck at 4.3V @ 13C.
...

I'm seeing the same thing.
I can get this to happen by constantly rebooting the mr.
Sometimes it will stay stuck with status of "rebooting" and health values are crazy.

NathanA · Thu Sep 06, 2012 8:03 am

I'm seeing the same thing.

Are you running 5.21rc1?

-- Nathan

janisk · Thu Sep 06, 2012 12:15 pm

also, to clear things up a bit - cause for MetaROUTER crashes on MIPS and PPC are not connected in any way. Patches supplied by devs are completely different.

Also, as indicated by users in this thread regarding RB450G and i can add about RB1100AH - not all problems are resolved and we are still looking into it.

NathanA · Thu Sep 06, 2012 12:22 pm

janisk,

Thanks for the update. If you still need to borrow one of my "bad" boards where the problem is easily reproducible, just let me know.

-- Nathan

janisk · Thu Sep 06, 2012 1:29 pm

one more update how MetaROUTER works to clear up some confusion:

MetaROUTER actually has its own hypervisor (it's tiny ~64kb in size) and each guest & host runs under it as separate task. Host actually looks just like a guest to hypervisor except it has access to all the hardware.

Every task (including host) gets 1/300s timeslice unless it has nothing to do and yields CPU to other task. Guests could be waiting for host response, host could be waiting for interrupts. In short, when they wait for something, they do not use CPU.

Hypervisor gets loaded in just before you start first guest. When hypervisor gets started it takes over controll and puts running kernel with its current state as the first task. At that point host kernel does not controll CPU & memory fully anymore, but it never notices that.

Hypervisor provides communication channels between it's tasks. Those are used by guests to communicate with hosts. Virtual ethernet interfaces are implemented this way. Even when you assign physical interface to guest, host simply bridges every packet received on this port to specific guest virtual interface.

Guests do not have access to the raw flash, instead they use some kind of fake NFS to get access to their files that reside in host flash. But host is the one that runs fileserver for each of the guests. And that means if guest wants to access some file it will have to sleep till host will be able to handle its request.

guy1with1mr1problems · Thu Sep 06, 2012 2:54 pm

Ok we have good news - we are very happy to report that after putting a RB435G to a pretty extreme stress, Metarouter performed pretty slow but was stable, which is most important for us.

But when kernel was compiled with unionfs support:
we got a 'unable to handle kernel paging request' crash almost immediately after /sbin/init started. This also about 70% of the time crashed the routerboard. When it did, it could only be recovered by unplugging the rb, watchdog didn't do anything. Since unionfs has a very bad stability history and especially since the kernel metarouter patches work with is so old, I think this is probably unionfs's fault.

So it looks like when the guest kernel misbehaves and does something it shouldn't, routerboard crashes hard.

But, apart from it being very slow (about 10 times difference from the exact same system running on the board directly), it was stable without unionfs !

We tested with the same kernel openwrt uses (compiled by hand to also include some other stuff we need + our metafs fix).
The system under that is an eglibc-based cross linux from scratch (mostly http://trac.cross-lfs.org/). - only difference is that we compiled everything as softfloat - our specific cflags: with -Os -march=mips24Kc -msoft-float -fomit-frame-pointer. We also use a tiny statically compiled C program instead of init that mounts a squashfs image and pivots root into it and loads /sbin/init - to save a lot of space. This works very well.

We tested php 5.3 (in fpm mode) + apc on top of nginx - both nginx and php had one worker thread each - and it performs pretty well, although external process spawns could be faster.

The system was compiled natively on a RB433UAH, then reconfigured, minimized, put into squashfs and distributed to the RB435G.

Now, since proof of concept was a success, we'll be porting gentoo to metarouter and report how it works.
We'll also be doing a gentoo image for some ppc boards, so we'll report then if the ppc console fix works.

Thanks for your great work !

guy1with1mr1problems · Thu Sep 06, 2012 2:57 pm

Also janisk, could you please ask how memory protection works? Specifically, do the guests have 'their own' fully functional MMUs or not? We're asking because we're not sure if we should compile various stack protections, optimize for mmu-less system, etc.
Any answer much appreciated, thanks.

janisk · Thu Sep 06, 2012 3:54 pm

Also janisk, could you please ask how memory protection works? Specifically, do the guests have 'their own' fully functional MMUs or not? We're asking because we're not sure if we should compile various stack protections, optimize for mmu-less system, etc.
Any answer much appreciated, thanks.

Guests and host that run under hypervisor do not have access to any of privileged instructions, they can't disable/enable interrupts set up MMU, etc. They run as superuser (MIPS) or userspace (powerpc). All the privileged instructions are emulated, and kernel itself never notices that, i.e., every privileged instruction generates trap to hypervisor.

In reallity all those instructions are translated by JIT to other code that emulates its behavour on first trap. This translator is actually greedy, once it needs to translate one privliged instruction it translates all of them that are in its neighborhood. The translation result gets cached in hypervisor memory, so next time trap occurs it doesn't need to translate them again.

All CPU MMU gets emulated by hypervisor, mapping guests physical addresses to real physical addresses. Only host has 1:1 mapping for its physical addresses.

You should defenitly compile kernel as for mips 4k with MMU.

timberwolf · Thu Sep 06, 2012 3:55 pm

janisk
Thank you very much for sharing those details with us. With this knowledge we shouldn't see that much jitter in an unloaded system if each task gets a 1/300s timeshare. Your devs might whant to check if the health monitoring ISR code disables interrupts globally and if there is a race condition with the systick ISR for the hypervisor. This would need to be done on machine instruction level, maybe the compiler reorderd or miscompiled an assembly sequence without your knowledge. Just my thoughts.

janisk · Thu Sep 06, 2012 4:06 pm

you should consider how many steps can be done when ICMP echo travels from one guest to another. And if packet is not processed in one time-slice (it arrives at the end) how long it can be delayed.

guy1with1mr1problems · Thu Sep 06, 2012 4:13 pm

Also janisk, could you please ask how memory protection works? Specifically, do the guests have 'their own' fully functional MMUs or not? We're asking because we're not sure if we should compile various stack protections, optimize for mmu-less system, etc.
Any answer much appreciated, thanks.
Guests and host that run under hypervisor do not have access to any of privileged instructions, they can't disable/enable interrupts set up MMU, etc. They run as superuser (MIPS) or userspace (powerpc). All the privileged instructions are emulated, and kernel itself never notices that, i.e., every privileged instruction generates trap to hypervisor.

In reallity all those instructions are translated by JIT to other code that emulates its behavour on first trap. This translator is actually greedy, once it needs to translate one privliged instruction it translates all of them that are in its neighborhood. The translation result gets cached in hypervisor memory, so next time trap occurs it doesn't need to translate them again.

All CPU MMU gets emulated by hypervisor, mapping guests physical addresses to real physical addresses. Only host has 1:1 mapping for its physical addresses.

You should defenitly compile kernel as for mips 4k with MMU.

Thanks that's excellent info ! That's precisely what we did and it works

timberwolf · Thu Sep 06, 2012 4:16 pm

you should consider how many steps can be done when ICMP echo travels from one guest to another. And if packet is not processed in one time-slice (it arrives at the end) how long it can be delayed.

Ok, then lets try some math:
I have running 3 tasks: 1 host and 2 guests
Each task gets arround 3,33ms (1/300s).
So from the point of view of one task, it may take 3,33ms * 2 = 6,66ms between two invokations.
If a ping travels from the host to the guest it may take worst case 3,33ms(host time) + 3,33ms time for an unrelated guest + 3,33ms time for the target guest = 9,99ms to get the request and the same amount to send the response, so we end up at 2*9,99ms = 20ms. Thats only a triple of the 60ms I saw.
And this would be a full load scenario, where each task gets its full 3,33ms share but my router was idling.

Any ideas? Am i missing something`?

reverged · Thu Sep 06, 2012 6:06 pm

I'm seeing the same thing.
Are you running 5.21rc1?

-- Nathan

Yes and latest fw.

(A little out of sequence in this thread.)

insect · Fri Sep 07, 2012 5:28 pm

My RB1100AH still rebooting with RC1

janisk · Tue Sep 11, 2012 3:31 pm

good news everyone, new patches has landed in MIPS MetaROUTER implementation. new build can be downloaded here:
http://www.mikrotik.com/download/share/ ... .21rc1.npk

it would be very helpful if you could try this out on your routers with MetaROUTER under and w/o load.

Note: this update is only for MIPS

timberwolf · Tue Sep 11, 2012 3:39 pm

Version 5.21 RC1 is this correct?
The last version also was RC1.

janisk · Tue Sep 11, 2012 4:13 pm

correct, new build is still called the same, the difference is, package build date is going to be 11.09.2012

barkas · Tue Sep 11, 2012 4:23 pm

I can no longer establish openvpn tunnels with this new build. I can not even see it trying to establish a tcp connection when it says it is dialing.

EDIT: please increment something next time.

- I can not post a bug unambiguously
- I can't rollback since I can't get the old download anymore
- I can't see if the update has cleanly applied or even applied at all

guy1with1mr1problems · Tue Sep 11, 2012 6:19 pm

I can no longer establish openvpn tunnels with this new build. I can not even see it trying to establish a tcp connection when it says it is dialing.

EDIT: please increment something next time.

- I can not post a bug unambiguously
- I can't rollback since I can't get the old download anymore
- I can't see if the update has cleanly applied or even applied at all

I think you should be able to downgrade to latest official stable and then reapply the first mr fixes

barkas · Tue Sep 11, 2012 11:24 pm

I can no longer establish openvpn tunnels with this new build. I can not even see it trying to establish a tcp connection when it says it is dialing.

EDIT: please increment something next time.

- I can not post a bug unambiguously
- I can't rollback since I can't get the old download anymore
- I can't see if the update has cleanly applied or even applied at all
I think you should be able to downgrade to latest official stable and then reapply the first mr fixes

Yes. That is, if I still had that one.

guy1with1mr1problems · Wed Sep 12, 2012 1:42 am

I can no longer establish openvpn tunnels with this new build. I can not even see it trying to establish a tcp connection when it says it is dialing.

EDIT: please increment something next time.

- I can not post a bug unambiguously
- I can't rollback since I can't get the old download anymore
- I can't see if the update has cleanly applied or even applied at all
I think you should be able to downgrade to latest official stable and then reapply the first mr fixes
Yes. That is, if I still had that one.

those builds are marked release candidates for a good reason you know.. don't try to blame anyone but yourself, you won't succeed.

NathanA · Wed Sep 12, 2012 6:07 am

barkas,

Contact me off-forum: nathana@fsr.com -- I still have a copy of the old one I can hook you up with.

I would just try a downgrade from the current rc1 to the old one and see if it takes. In addition to the version number, NPKs also have a package build date/timestamp that shows under "/system package print" and I suspect this is what RouterOS pays attention to for upgrades and downgrades.

-- Nathan

EDIT:

- I can't see if the update has cleanly applied or even applied at all

I agree with your other two points, but there were several indicators that the update applied successfully, if you were paying attention:

1. "/system package print detail" -- as previously hinted, look for "build-time". Newest NPKs are dated with TODAY'S date.
2. RouterBoard took just as long to reboot as when it normally updates.
3. Serial console clearly displayed the updating progress.
4. NPK file was no longer on the filesystem after the reboot (when update is unsuccessful, NPK is left alone; it is deleted when applied successfully)
5. "/log print" showed no update errors after reboot, and in fact showed the following:

00:00:13 system,info verified routeros-mipsbe-5.21rc1.npk 
00:00:14 system,info installed routeros-mipsbe-5.21rc1 
00:00:14 system,info router rebooted

All of these point to a successful upgrade.

NathanA · Wed Sep 12, 2012 9:18 am

janisk,

I've been testing out the September 11 build for the past 30 minutes, and I think you guys may have done it this time!! It is absolutely rock-solid for me so far on one of my "bad" 450G boards, even when running my OpenWRT+Asterisk guest! And just minutes before upgrading it to this build, I was still merrily reproducing the bug on the older build running on the same board. It's a night-and-day difference.

I haven't yet stress-tested it...I've merely allowed OWRT to boot and Asterisk to start up, but they aren't doing much. I'll try to get around to installing this build onto another 450G soon, and then generate some artificial CPU and network load between them.

Woo-hoo!

-- Nathan

timberwolf · Wed Sep 12, 2012 9:39 am

NathanA,

I admire your optimism, but lets wait a few days.

Lets hope the effects barkas is seeing aren't somehow related to MR fixes, but just the side product of some other RC1 changes.

Jan

janisk · Thu Sep 13, 2012 10:57 am

openvpn issue should be something completely un-related to MetaROUTER.

Rivera · Thu Sep 13, 2012 5:49 pm

i can confirm that metarouter works flawlessly with latest build on my RB493G.
However when i try to use nginx to proxy_pass some streaming audio, i experience timeouts. Tried with same config on my home server, looks like it's metarouter problem...

insect · Thu Sep 13, 2012 6:24 pm

I don't know why is it happening (bigger load perhaps?), but the previously mentioned RB1100AH is rebooting 3, or 4 times a day now... We can barely handle this downtime... Is there anything we can do to help the bug resolving? Next month, we will have a new important third partner who will use this router and we are not able to avoid using the metarouter feature.

Thank you in advance!

barkas · Thu Sep 13, 2012 6:31 pm

openvpn issue should be something completely un-related to MetaROUTER.

Upgrading ate my default route, for some unfathomable reason. After replacing it, it did work again.

janisk · Fri Sep 14, 2012 9:39 am

I don't know why is it happening (bigger load perhaps?), but the previously mentioned RB1100AH is rebooting 3, or 4 times a day now... We can barely handle this downtime... Is there anything we can do to help the bug resolving? Next month, we will have a new important third partner who will use this router and we are not able to avoid using the metarouter feature.

Thank you in advance!

unfortunately there are no good news this time regarding powerpc

timberwolf · Sat Sep 15, 2012 11:14 am

I've startetd some simple tests right now.
One thing I did notice immediatly: The health values now are a little crazy, my voltage reading now jumps from 12 to 14 to 20 Volts randomly and the temperature readings jump from 58 to 68 randomly

janisk, could you provide a little insight on how the health readings work? I really think you should provide an option to disable the voltage.ko kernel module, people who do not need voltage readings can then disable it, this would lower the IRQ load on the system significantly.

I will continue testing, although I just got a roundtrip time of 970ms on one MR, 99ms on the other MR and 90ms on the host. EDIT: Which might have been caused by another component and not the RB450G.

Lets see how worse it can get.

Its to early to see if the longer lockups are gone.

guy1with1mr1problems · Sat Sep 15, 2012 1:24 pm

I've startetd some simple tests right now.
One thing I did notice immediatly: The health values now are a little crazy, my voltage reading now jumps from 12 to 14 to 20 Volts randomly and the temperature readings jump from 58 to 68 randomly

janisk, could you provide a little insight on how the health readings work? I really think you should provide an option to disable the voltage.ko kernel module, people who do not need voltage readings can then disable it, this would lower the IRQ load on the system significantly.

I will continue testing, although I just got a roundtrip time of 970ms on one MR, 99ms on the other MR and 90ms on the host. EDIT: Which might have been caused by another component and not the RB450G.

Lets see how worse it can get. Its to early to see if the longer lockups are gone.

Could you try loading your system in another way - for example importing a .tar.gz into metarouter (which loads cpu on 100%) and looking at voltage values during that import? That way you can tell if regular load causes the voltage values to go freaky. ADCs built into microcontrollers for example usually perform worse when the chip is under load. They usually recommend running conversions in some 'adc noise reduction' cpu mode (which is like underclocked or sleep with oscillator disabled for just a while, etc). Also there could be some noise issues on the power lines that could get worse under load. Even though I think its not likely it would affect the voltage readings THAT much, maybe its worth a try ?

timberwolf · Sat Sep 15, 2012 1:39 pm

Just got an 12second hang

The host and all MR(which seems logical) froze in that time. Still the same simple test setup with no load.
EDIT: All ICMP requests got answered after this time, however.

MT I beg you to give us the option to turn off health monitoring so we could at least try without voltage.ko and GPIO IRQs without (unsupported) patching of an official image.

timberwolf · Sat Sep 15, 2012 1:50 pm

I've startetd some simple tests right now.
One thing I did notice immediatly: The health values now are a little crazy, my voltage reading now jumps from 12 to 14 to 20 Volts randomly and the temperature readings jump from 58 to 68 randomly

janisk, could you provide a little insight on how the health readings work? I really think you should provide an option to disable the voltage.ko kernel module, people who do not need voltage readings can then disable it, this would lower the IRQ load on the system significantly.

I will continue testing, although I just got a roundtrip time of 970ms on one MR, 99ms on the other MR and 90ms on the host. EDIT: Which might have been caused by another component and not the RB450G.

Lets see how worse it can get. Its to early to see if the longer lockups are gone.
Could you try loading your system in another way - for example importing a .tar.gz into metarouter (which loads cpu on 100%) and looking at voltage values during that import? That way you can tell if regular load causes the voltage values to go freaky. ADCs built into microcontrollers for example usually perform worse when the chip is under load. They usually recommend running conversions in some 'adc noise reduction' cpu mode (which is like underclocked or sleep with oscillator disabled for just a while, etc). Also there could be some noise issues on the power lines that could get worse under load. Even though I think its not likely it would affect the voltage readings THAT much, maybe its worth a try ?

I am only running ROS MR guests, and I think this fluctuations come from the way MT implemented health monitoring. As health monitoring causes arround 200interrupts/s on "GPIO" i think they are using external senors. But i don't know any protocol which would require that much interrupt servicing to implement.

guy1with1mr1problems · Sat Sep 15, 2012 2:15 pm

I've startetd some simple tests right now.
One thing I did notice immediatly: The health values now are a little crazy, my voltage reading now jumps from 12 to 14 to 20 Volts randomly and the temperature readings jump from 58 to 68 randomly

janisk, could you provide a little insight on how the health readings work? I really think you should provide an option to disable the voltage.ko kernel module, people who do not need voltage readings can then disable it, this would lower the IRQ load on the system significantly.

I will continue testing, although I just got a roundtrip time of 970ms on one MR, 99ms on the other MR and 90ms on the host. EDIT: Which might have been caused by another component and not the RB450G.

Lets see how worse it can get. Its to early to see if the longer lockups are gone.
Could you try loading your system in another way - for example importing a .tar.gz into metarouter (which loads cpu on 100%) and looking at voltage values during that import? That way you can tell if regular load causes the voltage values to go freaky. ADCs built into microcontrollers for example usually perform worse when the chip is under load. They usually recommend running conversions in some 'adc noise reduction' cpu mode (which is like underclocked or sleep with oscillator disabled for just a while, etc). Also there could be some noise issues on the power lines that could get worse under load. Even though I think its not likely it would affect the voltage readings THAT much, maybe its worth a try ?
I am only running ROS MR guests, and I think this fluctuations come from the way MT implemented health monitoring. As health monitoring causes arround 200interrupts/s on "GPIO" i think they are using external senors. But i don't know any protocol which would require that much interrupt servicing to implement.

I see.. maybe those are spurious interrupts ? Could you test regular load as I said? If those interrupts were noise induced, maybe you can somehow get the same results without metarouter ? It is possible that removing the voltage.ko module actually means that interrupts on some gpio line(s) are masked out and they don't happen, this could be plausible.

guy1with1mr1problems · Sat Sep 15, 2012 2:32 pm

I just found out that at least on some routerboards, they use LM87 as their sensor, which is i2c + alert irq line. It can generate interrupts when set alert values are passed, so any big noise on power lines or near that chip could make bad stuff happen.

timberwolf · Sat Sep 15, 2012 6:59 pm

I see.. maybe those are spurious interrupts ? Could you test regular load as I said? If those interrupts were noise induced, maybe you can somehow get the same results without metarouter ? It is possible that removing the voltage.ko module actually means that interrupts on some gpio line(s) are masked out and they don't happen, this could be plausible.

This interrupt load is "normal" on RB450G and some other boards and has been reported by others, this has already been discussed in this thread, I think before you participated. I can't recall having had freezes with a RB450G without metarouter. I won't do any more tests, I'm through with everything that made sense.

janisk · Tue Sep 18, 2012 2:06 pm

Just got an 12second hang The host and all MR(which seems logical) froze in that time. Still the same simple test setup with no load.
EDIT: All ICMP requests got answered after this time, however.

this happened once or you could make it happen more than once? Also, could you post build date and time of system package?

Router i have one RB450G running 2 MR with static VIFs bridged on host ping each other and PC pinging host router. did not have any freezes since i posted here. And that is on lightly modified router giving around 1000 GPIO interrupts a second. Unchanged router with just 2 metarotuers running without HW changes too does not have problems. So this is important, how many freezes you have and how often you have them and exact length of freezes.

MT I beg you to give us the option to turn off health monitoring so we could at least try without voltage.ko and GPIO IRQs without (unsupported) patching of an official image.

health monitoring can report wrong values if router is under load, ambient temperature changes from one router was calibrated in. Newer routers have these problems resolved by implementing different logic and circuitry to measure voltage/temperature.

timberwolf · Tue Sep 18, 2012 2:43 pm

Just got an 12second hang The host and all MR(which seems logical) froze in that time. Still the same simple test setup with no load.
EDIT: All ICMP requests got answered after this time, however.
this happened once or you could make it happen more than once? Also, could you post build date and time of system package?

Router i have one RB450G running 2 MR with static VIFs bridged on host ping each other and PC pinging host router. did not have any freezes since i posted here. And that is on lightly modified router giving around 1000 GPIO interrupts a second. Unchanged router with just 2 metarotuers running without HW changes too does not have problems. So this is important, how many freezes you have and how often you have them and exact length of freezes.

This happened once after arround 3-4 hours uptime, I can't however afford more testing time right now, sorry. Regarding the build date of the system package: I did a netinstall with your latest RC1 download so the build date will be waht you expect. The board isn't running right now so I can't verify this however.

MT I beg you to give us the option to turn off health monitoring so we could at least try without voltage.ko and GPIO IRQs without (unsupported) patching of an official image.
health monitoring can report wrong values if router is under load, ambient temperature changes from one router was calibrated in. Newer routers have these problems resolved by implementing different logic and circuitry to measure voltage/temperature.

First, this answer doesn't answer anything from my question/request.
Second, this doesn't help me and others much.

Also which boards are you talking about, newer version of RB450G?
And I am not talking about little deviations but big fluctuations which are clearly an error.

janisk · Tue Sep 18, 2012 3:25 pm

values reported can vary wildly if running metarouter guests. Just ignore readings given by '/system health' in this case.

barkas · Tue Sep 18, 2012 5:08 pm

values reported can vary wildly if running metarouter guests. Just ignore readings given by '/system health' in this case.

So disable voltage.ko when metarouter is enabled if they are worthless anyway and tend to cause unpredictable system behaviour.

reverged · Tue Sep 18, 2012 11:59 pm

values reported can vary wildly if running metarouter guests. Just ignore readings given by '/system health' in this case.
So disable voltage.ko when metarouter is enabled if they are worthless anyway and tend to cause unpredictable system behaviour.

I agree with barkas here. Either make it an option or disable it automatically when MR is enabled.

@janisk:
I find this information troubling.
MT knows the values can "vary wildly if running metarouter", but leaves the routine running?
If you know why the values are varying wildly, then presumably you can address it.
If you don't know why the values are varying wildly then it is best to remove the "feature" until you figure it out.
Does MT know why the values vary wildly?

I too have seen lockups and also had profiler hang when running 2 ROS mr's. It's difficult to reproduce....

I see also that metarouter-fs consumes high CPU (up to 80%) when mr is first enabled or rebooted. This is easy to reproduce.

janisk · Wed Sep 19, 2012 9:52 am

yes, we know why values reported can be wrong, that is why design of monitoring circuitry and logic behind it was changed.

Explanation on how metarouter works you can find on this very thread in one of my posts - due to that values can vary wildly under heavy load.

about metarotuerfs load - new routeros or guest data has to be red from the storage device as there is no other way on how to load it into RAM

please give more details about lockup - how long, how often, what PSU did you use?

timberwolf · Wed Sep 19, 2012 10:07 am

Ok I see where this is heading, again.
Well that's it, I'm done with MetaROUTER, MikroTik do whatever you want. I'm not paid for testing.

janisk · Wed Sep 19, 2012 1:12 pm

if i can show to the devs it happening again on my or other test routers someone will look at it.

Now i have 2 MR running on RB450G and sending trafficto each other. Then PC is sending ICMP echo to host router since it was posted that with latest 5.21rc1 saw another freeze.

insect · Mon Sep 24, 2012 12:10 pm

Hi,

Any news for PPC boards? It hasn't rebooted for 2 days, but today it did THREE times in a row. I'm still running RC1. I'm a bit desperate that it won't be fixed soon. If you can declare an estemate, even if it cost us extra, I can manage this problem with another router and an additional rack space, but please if this is the case, tell me to not expect this fix in a week, or two.

Thanks!

janisk · Mon Sep 24, 2012 2:00 pm

it seems that ppc metarouter reboots now are connected to packets-per-second received by the router. More packets - more frequent reboots. We are currently working on the issue, but it may take some time. As it is unknown yet what exactly causes the issue. That is all i can comment at this moment on this matter.

about mips metarouter - i had a moment where response time was 12129.041ms so we will look into it. As we now know that there still is some issue that can happen rarely

insect · Mon Sep 24, 2012 2:32 pm

thank you for the info... first step will be to de-virtualize the router receiving the bigger load...

timberwolf · Tue Sep 25, 2012 8:39 am

about mips metarouter - i had a moment where response time was 12129.041ms so we will look into it. As we now know that there still is some issue that can happen rarely

Oh really? Of course there is an issue! What cause do you think I would have for making this up?!

janisk · Tue Sep 25, 2012 10:09 am

Oh really? Of course there is an issue! What cause do you think I would have for making this up?!

Bad part is - it took week to manifest itself. Anyway, "blackout period was those 12 seconds.

Did you have one or several guests running?

timberwolf · Tue Sep 25, 2012 10:12 am

Bad part is - it took week to manifest itself. Anyway, "blackout period was those 12 seconds.

I took about 3-4 hours on my setup.

Did you have one or several guests running?

Two, as I wrote earlier.

Rivera · Wed Oct 03, 2012 4:33 pm

Does this patches (for mipsbe) included in 6.0rc1?

janisk · Fri Oct 05, 2012 9:34 am

yes, all of the changes regarding MetaROUTER changes done for 5.21rc1 are in 6.0rc1

guy1with1mr1problems · Tue Oct 16, 2012 1:00 pm

I just got a crash on powerpc (rb800) with both 6.0rc1 and 5.21.

When I load the 'malicious' image, the board prints this on its serial port

BUG(1): unhandled unknown exception!
pc=0x30023064 cpc=0x80000f64 lr=0x10013750 ctr=0x30023108
regs=0xfc926f60 cr=0x30000093 xer=0xc000006b
r0=0x10013750 r1=0x7fffc640 r2=0x00000000 r3=0x7fffc710 
r4=0x00000000 r5=0x00000000 r6=0x00000000 r7=0x1001eb40 
r8=0x0000000c r9=0x10030000 r10=0x100324e8 r11=0x7fffc600 
r12=0x30000093 r13=0x10038018 r14=0x7dfdbf7b r15=0xe9ffe5ef 
r16=0xffda7cbc r17=0xffbfa775 r18=0x7ff7fe5d r19=0xcf7ffffe 
r20=0xffdb7eff r21=0xdfbb7dbf r22=0xbffbbef8 r23=0x00000000 
r24=0x7fffc958 r25=0x1001cd20 r26=0x7fffcfe7 r27=0x10001f08 
r28=0x10002420 r29=0x00000000 r30=0x7fffca14 r31=0x00000002 
dear=0x10032004 msr=0x0002d000(0002d000) esr=0x00000080

and then hangs. This happens every time.
by trial and error, we determined this happens immediately when busybox(uclibc) with hardfloat and '-march=powerpc' runs.

Also another problem: Metarouter sets kernel parameters on powerpc. Unfortunately, in regular kernels, this makes setting custom kernel parameters inside the image impossible, as they always get overriden by those Metarouter sends. We had to modify the kernel sources to force it to accept our boot parameters.

guy1with1mr1problems · Tue Oct 16, 2012 1:11 pm

I think I might have an idea. Looks like our code mistakenly uses binary incompatible fpu instructions, it should be using fpe. But still, it definitely shouldn't crash the host, maybe just the vm kernel.

janisk · Tue Oct 16, 2012 2:52 pm

can you share the image, so i could get it?

guy1with1mr1problems · Tue Oct 16, 2012 2:57 pm

can you share the image, so i could get it?

no problem, but due to its proprietary nature, could we agree on exchange via e-mail or something like that? It's a single 3.3 MB .tar.gz file

janisk · Tue Oct 16, 2012 3:00 pm

then send it to support@mikrotik.com

guy1with1mr1problems · Tue Oct 16, 2012 3:05 pm

then send it to support@mikrotik.com

mail sent from my personal account, its subject is: 'powerpc metarouter image that crashes RB800' (without the quotes). Thanks.

alfil · Thu Oct 18, 2012 11:20 am

...
We did various tests on RB450G boards, most of them were conducted by NathanA. The conclusions for this board so far are:
1.) The powersupply does have an influence but isn't the cause.
2.) Disabling hardware monitoring by a hack seems to improve the stability, assumably because of much lowered IRQ load.
3.) It seems as there has been no significant progress by the MT devs, maybe with point 2 there will be.
UPDATE: It more and more seems to narrow down to the GPIO ISR(s) as NathanA reports that high IRQ load from the switch chip doesn't seem to cause issues.
...

About work on the RG450G.
I have more than half year steadily working the OpenWrt Kamikaze - With X-Wrt Extensions 8.09.
And at first I had constant rebooting. But after I drill the case of RG450G and mount the fan all the problems were solved.

janisk · Thu Oct 18, 2012 1:20 pm

there where different methods on how to delay the hangs, like using 12V PSU instead of 24V that reduced interrupt count.

Same as removing voltage.ko driver that removes the cause of the hang not the reason. But still with removed driver router still could hang, as that was not the only interrupt generator, but it does that at correct frequency to cause the problems.

Wazza · Thu Oct 18, 2012 2:15 pm

Has any progress actually been made with PPC systems?

I've recently upgraded 2 RB1100AH units, to 5.21, thinking that it "might help", and we're only running a couple (2 ROS metarouters), and I'm still seeing daily reboots, under very lightly loaded traffic conditions...

Wazza

janisk · Mon Oct 29, 2012 9:32 am

over last week a lot of work went into metarotuer on mips. It would be really nice if you could try and see if this still hangs/freezes for you. I ran tests over the weekend (55hours) and did not get eny problems.

So if any of you can put it somewhere, that would be highly appretiated.

Here the link for new download:
http://www.mikrotik.com/download/share/ ... .22rc1.npk

regarding ppc metarotuer - fixed crash when FPU exception was rised.

xbojer · Wed Oct 31, 2012 2:08 am

What changed in RouterOS v6 regarding metarouter on mipsbe? MR seems to work fine on my rb2011uas-2hnd-in on versions 5.xx but trying any rc of 6.0 causes router crash on adding metarouter...

janisk · Wed Oct 31, 2012 11:32 am

we will fix that

timberwolf · Mon Nov 12, 2012 6:24 pm

over last week a lot of work went into metarotuer on mips. It would be really nice if you could try and see if this still hangs/freezes for you. I ran tests over the weekend (55hours) and did not get eny problems.

So if any of you can put it somewhere, that would be highly appretiated.

Here the link for new download:
http://www.mikrotik.com/download/share/ ... .22rc1.npk

regarding ppc metarotuer - fixed crash when FPU exception was rised.

I will do some tests, one thing I noticed right after starting is that health vlaues now are totally bogus. I think this was to be expected
Lets see how it will behave in longterm testing.

janisk · Tue Nov 13, 2012 11:32 am

unfortunately due to how metarouter operates (one of my older posts in this thread) health monitoring values can be off the mark.

timberwolf · Tue Nov 13, 2012 11:38 am

unfortunately due to how metarouter operates (one of my older posts in this thread) health monitoring values can be off the mark.

Yes I know, that's why me and other people recommended disabling health monitoring completely.

janisk · Tue Nov 13, 2012 2:53 pm

first ppc metarouter has to start work, and resolve issues there, maybe afterwards someone will make this cosmetic change.

timberwolf · Tue Nov 13, 2012 2:55 pm

first ppc metarouter has to start work, and resolve issues there, maybe afterwards someone will make this cosmetic change.

Understandable

timberwolf · Tue Nov 13, 2012 6:28 pm

I am sorry, but i have to report the 12 second hang as still NOT fixed.
Still exactly the same behaviour like in 5.21rc2 and 5.21, whole system freezes for 12 seconds and resumes afterwards like nothing happened.
Therefore I request MT to finally provide an image without hardware monitoring for testing, lets just test if my and other RB450Gs then work without those hangs...

EDIT: I did some math because I find those 12 seconds suspicious and discovered that 12,89 seconds is exactly the time it takes for a 32bit counter, which is incremented every 333µs (or with 3000 Hz), to roll over. You say the MR hypervisor ticks every 300Hz maybe your devs use a scaled 3000 Hz timer for this. This is to accurate to be a coincidence, don't you think?

janisk · Wed Nov 14, 2012 3:55 pm

how often these happen? can you describe configuration, mine was like this:

1 metarouter running with 1 dynamic vif and 1 static vif interface. both bridged with physical port. in network only usual broadcast chatter. No freezes so far.

timberwolf · Wed Nov 14, 2012 4:27 pm

how often these happen?

It now happened once in 24h .

can you describe configuration, mine was like this:

1 metarouter running with 1 dynamic vif and 1 static vif interface. both bridged with physical port. in network only usual broadcast chatter. No freezes so far.

The complete config without any steps left out was asd following:

On host
--------
1.) Fresh netinstall setup.
1a.) enable IPv6, disable hotspot and wireless packages.
2.) Create bridge with ether1 in it.
3.) Add dhcp client on host.
4.) Create 2 metarouters each with 32mb ram.
5.) Connect each metarouter with one dynamic vif to bridge.
6.) Start metarouters and add dhcp client to ether1 inside each metarouter.

On unrelated MT system
-----------------------
Ping host from above and both metarouters

On unrelated Mac OS X box
--------------------------
Ping host from above and both metarouters

janisk · Thu Nov 15, 2012 9:00 am

off, to test some metarotuers. Thank you for details of your setup.

timberwolf · Thu Nov 15, 2012 9:10 am

off, to test some metarotuers. Thank you for details of your setup.

I wish you luck. The bug seems to be related to the scheduling of the hypervisor I think. An overflow (maybe with interrupt) seems to "wake" the scheduler up after the normal tick has been somehow blocked. Your devs should be able to diagnose this better then me or you, I guess.

janisk · Thu Nov 15, 2012 11:01 am

there is more than this, as normal tick, happens every 333us (microsecond) in case of 12.88 second delay these are 38K missing ticks. and at least 10K missing ticks that would bring us back to time-slice of main guest of hypervisor with 2 guests running, as hypervisor sees them as 3 guests where one of them has bit more privileges.

timberwolf · Thu Nov 15, 2012 11:10 am

Hmm your are right with your 38k ticks, I made a mistake in my calculation. Never do math aber 10:00pm

I will come back to you, as soon as get this 32bit counter thing figured out correctly.

EDIT: Is there any chance, that you use a 32bit integer as a timestamp with microsecond accuracy? Cause what I originally calculated was 2^32us / 333us which gives 12,89s. I can't figure out the rest, as I didn't implement your scheduler or timing logik.

janisk · Thu Nov 15, 2012 11:25 am

i will relay this to devs. Maybe they can come with something useful.

timberwolf · Fri Nov 16, 2012 11:06 pm

EDIT: Merged post with next post.

timberwolf · Sat Nov 17, 2012 9:55 am

In the meantime I started conducting the exact same tests with a RB2011UAS-RM but with the following modifications:
1.) ROS 5.21
2.) Host an MRs use static IPs
3.) Host runs NTP client

On first sight, I got the same freezes after just 4 hours but with differing packet losses and timeouts,to rule out any errors in the test setup I restarted the test and setup a dedicated RB450G for monitoring in addition to the Mac OS X machine. I will report back, but please be so kind and also setup the same tests with a RB2011UAS if your ressources allow.

--

Ok here are the results of running this setup for one night, which don't look very good:
192.168.10.100 ist the host, .101 is the first and .102 the second MetaRouter.

Pings from RB450G
----------------
HOST: IP 192.168.10.100
sent=37938 received=37924 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=191ms

MR1: IP 192.168.10.101
sent=37936 received=36951 packet-loss=2% min-rtt=0ms avg-rtt=2ms max-rtt=485ms

MR2: IP 192.168.10.102
sent=37939 received=37925 packet-loss=0% min-rtt=0ms avg-rtt=3ms max-rtt=82ms

Pings from Mac OS X box
---------------------
HOST: IP 192.168.10.100
39006 packets transmitted, 39006 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.255/3.841/13489.331/152.880 ms

MR1: IP 192.168.10.101
39003 packets transmitted, 38076 packets received, 2.4% packet loss
round-trip min/avg/max/stddev = 0.757/382.016/757900.732/15178.624 ms

MR2: IP 192.168.10.102
38997 packets transmitted, 38997 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.805/5.513/13495.470/152.992 ms

As ROS discards pings after a timeout and Mac OS X doesn't the results differ. The Host and MR2 seemed to freeze for the same amount of time 13.5seconds(+/- 1s), MR1 showed a different behaviour of dropping pings totally.

I will restart the tests with ROS5.22rc1.

--

Same behaviour with 5.22rc1, this time it happened twice within just 20 minutes.

Pings from RB450G
----------------
HOST: IP 192.168.10.100
sent=1263 received=1237 packet-loss=2% min-rtt=0ms avg-rtt=1ms max-rtt=995ms

MR1: IP 192.168.10.101
sent=1261 received=1234 packet-loss=2% min-rtt=0ms avg-rtt=2 max-rtt=753ms

MR2: IP 192.168.10.102
sent=1262 received=1235 packet-loss=2% min-rtt=0ms avg-rtt=2ms max-rtt=672ms

Pings from Mac OS X box
---------------------
HOST: IP 192.168.10.100
1241 packets transmitted, 1241 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.311/167.836/14015.345/1252.018 ms

MR1: IP 192.168.10.101
1241 packets transmitted, 1241 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.827/166.600/14138.749/1238.100 ms

MR2: IP 192.168.10.102
1242 packets transmitted, 1242 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.786/161.610/13734.863/1209.999 ms

So MetaRouters on RB450G and RB2011UAS is still broken. I doubt that if I would tester further MIPS-BE boards I would find one which is working flawless.
As this issue even affects the RB2011 boards with a new CPU, there really must be a bug hidden inside MetaRouter (hypervisor) code which causes trouble on any platform. And as it always is the same timing behaviour it all points to an error inside some timing routine.

payday · Mon Nov 19, 2012 11:41 pm

I have a problem with MetaROUTER on RB2011UAS-2HnD v6rc3, too. I have set up 3 MetaRouters for testing:

 #   NAME                 MEMORY-SIZE     DISK-SIZE     USED-DISK STATE        
 0   OpenWRT-1                  20MiB      32768kiB       8132kiB running         
 1   RouterOS-1                 24MiB      32768kiB        219kiB running         
 2   OpenWRT-2                  20MiB      32768kiB       4790kiB running

OpenWRT-1 is r18199, downloaded from: http://openwrt.pantele.com.ua/trunk/metarouter/
and OpenWRT-2 is r30812, from: http://mirrors.muarf.org/openwrt-custom ... r/mr-mips/
RouterOS-1 is just regular MR as name says.

Their interfaces are all dynamic connected to one bridge:

 #   VIRTUAL-MACHINE         TYPE    STATIC-INTERFACE         VM-MAC
 0 A OpenWRT-1               dynamic                          02:89:
 1 A RouterOS-1              dynamic                          02:80:
 2 A OpenWRT-2               dynamic                          02:9D:

Configuration on all of them is just basic: IP numbers and gateway on interfaces. I set up Tools\Netwatch for every of them (ping every 5 seconds) and every time MetaRouter starts or stops responding, it is logged. Example of Netwatch script that I created for every MR:

add down-script=":log warning \"- - - - - - - RouterOS is not responding\"" host=192.168.0.3 interval=5s up-script=\
    ":log warning \"- - - - - - - RouterOS is OK\""

And now my log is:

19:58:25 script,warning - - - - - - - RouterOS is OK 
20:05:58 script,warning - - - - - - - RouterOS is not responding 
20:13:31 script,warning - - - - - - - RouterOS is OK 
20:15:28 script,warning - - - - - - - RouterOS is not responding 
20:22:20 script,warning - - - - - - - RouterOS is OK 
20:24:18 script,warning - - - - - - - RouterOS is not responding 
20:32:50 script,warning - - - - - - - RouterOS is OK 
20:33:03 script,warning - - - - - - - RouterOS is not responding 
20:34:21 script,warning - - - - - - - RouterOS is OK 
20:34:28 script,warning - - - - - - - RouterOS is not responding 
20:34:55 script,warning - - - - - - - RouterOS is OK 
20:35:33 script,warning - - - - - - - RouterOS is not responding 
20:36:56 script,warning - - - - - - - RouterOS is OK 
20:37:03 script,warning - - - - - - - RouterOS is not responding 
20:41:15 script,warning - - - - - - - RouterOS is OK 
20:41:23 script,warning - - - - - - - OpenWRT-1 is not responding 
20:43:10 script,warning - - - - - - - OpenWRT-1 is OK 
20:43:28 script,warning - - - - - - - OpenWRT-2 is not responding 
20:46:45 script,warning - - - - - - - OpenWRT-2 is OK 
20:47:08 script,warning - - - - - - - RouterOS is not responding 
20:47:58 script,warning - - - - - - - RouterOS is OK 
20:48:23 script,warning - - - - - - - RouterOS is not responding 
20:50:26 script,warning - - - - - - - RouterOS is OK 
20:51:33 script,warning - - - - - - - OpenWRT-2 is not responding 
20:53:22 script,warning - - - - - - - OpenWRT-2 is OK 
20:58:33 script,warning - - - - - - - RouterOS is not responding 
21:01:18 script,warning - - - - - - - RouterOS is OK 
22:01:43 script,warning - - - - - - - RouterOS is not responding 
22:01:51 script,warning - - - - - - - RouterOS is OK 
22:12:08 script,warning - - - - - - - RouterOS is not responding 
22:13:15 script,warning - - - - - - - RouterOS is OK

And on night time:

02:25:28 script,warning - - - - - - - RouterOS is not responding 
02:25:35 script,warning - - - - - - - RouterOS is OK 
02:35:58 script,warning - - - - - - - OpenWRT-2 is not responding 
02:36:10 script,warning - - - - - - - OpenWRT-2 is OK 
02:37:53 script,warning - - - - - - - RouterOS is not responding 
02:38:17 script,warning - - - - - - - RouterOS is OK 
04:08:38 script,warning - - - - - - - OpenWRT-2 is not responding 
04:09:17 script,warning - - - - - - - OpenWRT-2 is OK 
04:24:13 script,warning - - - - - - - RouterOS is not responding 
04:26:32 script,warning - - - - - - - RouterOS is OK 
06:19:43 script,warning - - - - - - - RouterOS is not responding 
06:26:06 script,warning - - - - - - - RouterOS is OK 
07:56:08 script,warning - - - - - - - RouterOS is not responding 
07:57:58 script,warning - - - - - - - RouterOS is OK 
08:44:13 script,warning - - - - - - - RouterOS is not responding 
08:44:35 script,warning - - - - - - - RouterOS is OK 
10:42:33 script,warning - - - - - - - OpenWRT-2 is not responding 
10:44:10 script,warning - - - - - - - OpenWRT-2 is OK 
12:39:23 script,warning - - - - - - - OpenWRT-2 is not responding 
12:41:13 script,warning - - - - - - - OpenWRT-2 is OK 
13:32:08 script,warning - - - - - - - OpenWRT-2 is not responding 
13:32:47 script,warning - - - - - - - OpenWRT-2 is OK 
15:27:28 script,warning - - - - - - - RouterOS is not responding 
15:29:03 script,warning - - - - - - - RouterOS is OK

Definitely it happens more often when there is load on the router (e.g. downloading files from internet) and when MetaRouter is not responding to pings, also it's console is not responding - it freezes.
I have red this thread and I don't see any solution for this isue. Will it be fixed? Or am I doing something wrong?

timberwolf · Tue Nov 20, 2012 8:03 am

Will it be fixed? Or am I doing something wrong?

To keep the answer short, we all hope it will be fixed anytime soon as it is an open issue since 2 years(if I'm not mistaken). And no I'am quite sure you aren't doing anything wrong.

janisk · Tue Nov 20, 2012 9:54 am

payday - use of 5.22[rc1] will yield a bit better results. however it is not yet completely fixed.

timberwolf - during weekend and national holiday i have generated quite a lot of useful stuff for devs to work on regarding metarouter.

timberwolf · Tue Nov 20, 2012 10:31 am

timberwolf - during weekend and national holiday i have generated quite a lot of useful stuff for devs to work on regarding metarouter.

Thanks, I must admit, right now I'm quite disappointed that RB2011 isn't working either.

janisk · Fri Nov 23, 2012 8:52 am

anyone running MetaROUTER on MIPSbe routers - your OS version of choice at the moment of release is 5.22

timberwolf · Fri Nov 23, 2012 8:57 am

Are you indicating, that you might have fixed it?

janisk · Fri Nov 23, 2012 9:14 am

atm - I'd say - less likely to happen. No problematic events on debug metarouter have happened for a long time. At le

timberwolf · Fri Nov 23, 2012 9:55 am

atm - I'd say - less likely to happen. No problematic events on debug metarouter have happened for a long time. At le

Hmm "less likely", what's the developers point of view? No offense, but this sounds like "we have no idea if we did fix it", don't you agree?

janisk · Fri Nov 23, 2012 10:37 am

switch from one guest to another does not happen over one instruction, so, when guest gets distracted in the switch event it could go bananas. Now this issue has been addressed and router containing pached hypervisor did not crash in over 18h+ period.

So, all i can say now is - it is likely to be fixed as there where periods in time that router did not crash even without a patch. Anything more conclusive i can say only after weekend.

timberwolf · Fri Nov 23, 2012 10:44 am

Thank you for the additional information.
I have one question left for now, you talk about crashes, what about those freezes which I've even seen on a RB2011UAS?

janisk · Fri Nov 23, 2012 10:52 am

these actually are freezes, router will continue to work after some time. Called them crashes due to way this was changed in debug version - i get registry dump and stacktrace after each event and guest is restarted each time this happens, sometimes host was also restarted. So, control router just freezes and continues as if nothing has happened. Last real reboots was in 5.20, at least for routers i am testing on.

timberwolf · Fri Nov 23, 2012 1:02 pm

Good sounds promising. I will retest with 5.22 as soon as it is available, as I happen to have some spare time at the moment.

timberwolf · Fri Nov 23, 2012 11:25 pm

Ok janisk here come the results for the 5.22 release version.

RB450G
Got a freeze after about 5 hours, this time 11,9seconds, affecting the host and both MR instances, so no significant change, just -1second(luck?).

RB2011UAS
Still the same strange behaviour I got with 5.22rc1. Host froze for about 10seconds, one MR instance froze for about 12,6seconds and the other MR instance lost some packet totally and/or was unresponsible for about 330seconds(!).

Sorry, not fixed and not even really better.

payday · Mon Nov 26, 2012 12:09 am

I have just downgraded my RB2011UAS-2HnD from 6rc3 to 5.22, started two metarouters (RouterOS and OpenWRT) and set up Netwatch. I will report, tomorrow.

janisk · Mon Nov 26, 2012 10:51 am

thanks for the updates. I also got a freeze. So we are continuing on this issue.

payday · Mon Nov 26, 2012 9:36 pm

03:59:08 script,warning - - - - - - - OpenWRT-2 is not responding 
04:03:36 script,warning - - - - - - - OpenWRT-2 is OK 
04:23:58 script,warning - - - - - - - OpenWRT-2 is not responding 
04:27:01 script,warning - - - - - - - OpenWRT-2 is OK 
08:42:53 script,warning - - - - - - - OpenWRT-2 is not responding 
08:44:57 script,warning - - - - - - - OpenWRT-2 is OK 
09:19:08 script,warning - - - - - - - RouterOS is not responding 
09:23:35 script,warning - - - - - - - RouterOS is OK 
10:31:28 script,warning - - - - - - - OpenWRT-2 is not responding 
10:32:20 script,warning - - - - - - - OpenWRT-2 is OK 
10:32:33 script,warning - - - - - - - RouterOS is not responding 
10:33:23 script,warning - - - - - - - RouterOS is OK 
11:24:23 script,warning - - - - - - - OpenWRT-2 is not responding 
11:27:38 script,warning - - - - - - - OpenWRT-2 is OK 
11:32:33 script,warning - - - - - - - RouterOS is not responding 
11:33:10 script,warning - - - - - - - RouterOS is OK 
15:09:23 script,warning - - - - - - - RouterOS is not responding 
15:15:15 script,warning - - - - - - - RouterOS is OK 
16:37:08 script,warning - - - - - - - RouterOS is not responding 
16:37:17 script,warning - - - - - - - RouterOS is OK 
17:03:13 script,warning - - - - - - - RouterOS is not responding 
17:10:03 script,warning - - - - - - - RouterOS is OK 
19:18:58 script,warning - - - - - - - RouterOS is not responding 
19:19:20 script,warning - - - - - - - RouterOS is OK 
19:39:43 script,warning - - - - - - - OpenWRT-2 is not responding 
19:45:35 script,warning - - - - - - - OpenWRT-2 is OK

There is no change on mine, also. Problem persists.

timberwolf · Tue Nov 27, 2012 9:14 am

There is no change on mine, also. Problem persists.

Thanks for your verification effort!

payday · Thu Nov 29, 2012 10:28 pm

I have also noticed that short (about 10 seconds) freeze of whole device (not only MetaRouters), many times yesterday, when using console a lot. Traffic is not forwarded during that break and console does not respond.
I monitor some hosts behind MT router. Normally they are all green and when freeze happens all go red for few seconds.

timberwolf · Mon Dec 03, 2012 8:27 am

I have also noticed that short (about 10 seconds) freeze of whole device (not only MetaRouters), many times yesterday, when using console a lot. Traffic is not forwarded during that break and console does not respond.
I monitor some hosts behind MT router. Normally they are all green and when freeze happens all go red for few seconds.

Yes, this is the "expected" behaviour for now, freezes are arround 12-14 seconds and affect the whole system running MR instances.

timberwolf · Tue Dec 18, 2012 8:22 am

@MT
Any christmas presents planned?

PastuhMedvedey · Wed Dec 19, 2012 1:28 pm

Metarouter freezes when loading the base system more than 40-50 per cent.
When using metarouter on RB2011, metarouter hangs on version 5.22.
After a while, when the load on the base system fall below 10 percent, metarouter restores normal operation.
After you upgrade to 6.0rc5, the problem remained.

timberwolf · Wed Dec 19, 2012 4:15 pm

Metarouter freezes when loading the base system more than 40-50 per cent.
...

We are not even at the point, where Metarouter works in an idle situation. So I guess it's no suprise that a freezes with load.

payday · Wed Jan 09, 2013 10:17 pm

From http://forum.mikrotik.com/viewtopic.php?f=15&t=55286 :

yes, of course we are working on these issues. Unfortunately MetaROUTER feature is neither simple to debug or simple to fix issues found.

Can we help you in solving this issue? I am a former Assembler programmer with some OS programming experience, and maybe there are some more MetaROUTER users that can halp analyze the problem. Don't get me wrong. I'm not saying your developers are poor but some fresh look on the problem may be helpful.

MetaROUTER functionality is a big selling point in RouterOS, greatly expanding abilities of MikroTik devices. This is very important feature that makes me choose MikrotTik over other manufacturers. I am at the point where I have to decide whether I can rely on MetaROUTER functionality or not - in future implementations. I am working on replacing HP AP's and HP MSM Wireless controllers with MikroTik devices (as a cheaper alternative) and as long as MR feature is not reliable I can not "invest" in this idea. I am afraid that problems mentioned in this thread will persist for another half of the year or maybe one day you will say that it can't be fixed at all.
I was talking to my manager about great features of RouterOS (trying to convince him) and finally I tried to show him MetaROUTER and it was not responding.

For me it is kinda funny: If you are that good that you developed MetaROUTER then it should be easy for you to fix some small issues.

otgooneo · Thu Jan 10, 2013 10:04 pm

+1 on payday

infused · Mon Jan 14, 2013 11:33 am

Just to chime in. I bought Mikrotik because of Metarouter. Waiting for it to be ported to dual cores (CCR). Otherwise I will eventually have to go with a Juniper, which I'd rather not do.

timberwolf · Thu Jan 17, 2013 9:01 am

Well some fresh optimism in this thread can't do any harm.

timberwolf · Thu Feb 07, 2013 2:44 pm

I guess this topic is an general hold again? Mikrotik, care to comment on current status?

Fraction · Mon Feb 25, 2013 11:52 am

Has anyone tried to change power supply for RB2011UAS-2HnD-IN and tested its impact to MetaROUTER stability? Is it even possible to change it without losing warranty?

janisk · Mon Feb 25, 2013 3:24 pm

yes, it is possible. It is allowed to open cases of RouterBOARD devices. Just use quality 12V adapter to power the router.

timberwolf · Mon Feb 25, 2013 4:00 pm

Has anyone tried to change power supply for RB2011UAS-2HnD-IN and tested its impact to MetaROUTER stability? Is it even possible to change it without losing warranty?

This isn't even worth a try, as the MR issues clearly aren't powersupply dependent. But if you want to waste your time, like we all did in the past, you are welcome.

janisk · Mon Feb 25, 2013 4:42 pm

there where difference in intrrupts when used different PSUs, so that guess was not without merit, just for some bad stuff happened less frequent.

Fraction · Tue Feb 26, 2013 5:55 pm

Ok, I stole PSU from TP-Link TL-WR1043ND (12V 1,5A) and RB2011 is again up and running. Next I'll set up my test-OpenWRT MetaROUTER again and maybe generate some load for that and we'll see what happens..

Fraction · Tue Feb 26, 2013 9:00 pm

Ok, I stole PSU from TP-Link TL-WR1043ND (12V 1,5A) and RB2011 is again up and running. Next I'll set up my test-OpenWRT MetaROUTER again and maybe generate some load for that and we'll see what happens..

Well, not a significant difference.. I think freeze periods are a bit shorter than earlier, but anyway them still exists.

Edit: My RB2011 was while ago rebooted by Watchdog (first time ever). I think it was overly good timing to be just coincidence, so I quickly changed back to original PSU.

barkas · Wed Feb 27, 2013 11:45 am

Please don't go the powersupply way again, that has been thoroughly tested and been found to not be the problem.

otgooneo · Wed Feb 27, 2013 5:03 pm

When the Metarouter will stable? Please MT guys, say something shiny. I`ve very good solutions and ideas using Metarouter. But stability issue doesn`t allow me to do cool things......

NathanA · Wed Feb 27, 2013 9:51 pm

I wonder why I'm not seeing this issue so far?

Not only have all of my own phone calls been going through an RB450G for the past few months, but we also have a few customers now that we have installed similar systems for (RB450G that handles internet routing -- PPPoE client + NAT -- and runs Asterisk inside of MetaROUTER with all phone calls being trunked via VoIP) and so far they have no complaints either. If this were happening to them, they would be getting random periods of dead air > 10s in duration in the middle of phone calls, and I expect their PPPoE tunnel would drop, too, after timing out on our side (11 seconds is an eternity!). But neither of these things are happening...

Full disclosure: I still distrust the interrupt-crazy health monitoring subsystem on the 450G, so even on 5.22 and 5.23, I have been ripping out voltage.ko on any systems we deploy.

I'll see if we have a 2011 kicking around somewhere that I can run some tests on; I haven't bothered to play with MetaROUTER on one of those.

-- Nathan

Fraction · Wed Feb 27, 2013 10:09 pm

I didn't tried that voltage.ko -trick, it felt too arduous to find out how to do that and give it a shot.
ROS version was 5.24.

timberwolf · Wed Mar 06, 2013 10:27 am

I wonder why I'm not seeing this issue so far? ...
... Full disclosure: I still distrust the interrupt-crazy health monitoring subsystem on the 450G, so even on 5.22 and 5.23, I have been ripping out voltage.ko on any systems we deploy....

Well that is because it is most likely a faulty ISR which kills the system, and ripping out voltage.ko lowers the IRQ load significantly. So what are you wondering about again, I simply don't get it?

I'll see if we have a 2011 kicking around somewhere that I can run some tests on; I haven't bothered to play with MetaROUTER on one of those

Testet it with the same setup I've been using for the RB450G, result was the same with slightly different timing behaviour.

otgooneo · Mon Apr 08, 2013 10:29 am

Hey MT, I know you have a lot of work with ROSv6.0, but how about Metarouter stability issue?

NathanA · Thu Apr 11, 2013 5:43 am

Well that is because it is most likely a faulty ISR which kills the system, and ripping out voltage.ko lowers the IRQ load significantly. [...] Testet it with the same setup I've been using for the RB450G, result was the same with slightly different timing behaviour.

I finally got around to testing on a 2011 (L-IN).

MetaROUTER isn't even close to being stable on it. I will note the following:

Interrupt load on a 2011 isn't nearly as high as it is on a 450G. voltage.ko monitoring is not done on a 2011L-IN (/system health print == blank). Only things generating interrupts are the 2 switch chips and "beeper".

I ran my tests on a brand-new RB2011L-IN that I then upgraded to 5.24. I also tried it on the stock 5.20 install before upgrading. I was running my Asterisk MetaROUTER image, the same one I run on 450Gs without a problem.

The problems I observed are these:

1) At random intervals (usually after being up for only a few minutes), the MR guest stops responding for a long, long time (a few minutes usually). No IP traffic, no console response. Eventually it will start responding again, but usually it only works again for a few seconds before it hangs for several more minutes. I have seen it eventually break out of this death spiral and then work for a long period of time again.

2) The metafs driver acts...weird. Sometimes certain branches of the filesystem tree will just outright disappear, and only come back after rebooting the guest. I had the contents of /usr/share disappear on me twice while testing my image on this 2011. If you did an 'ls -la /usr/share', it wouldn't even show the '.' and '..' directories.

3) I tried to generate a supout while the hanging problem (#1) was occurring. On the RB2011L-IN, the RouterOS host never hangs: only the guest does. However, if the guest is hung, and you try to make a supout, the supout process itself hangs and never completes. (The rest of the router continues to be responsive. CPU goes to 100% for a while during supout generation, then eventually settles back into single-digits. Supout progress, though, fails to advance.)

Thus, I cannot recommend anybody use MetaROUTER on the 20xx models. I'm mystified why the behavior is so startlingly different from the 450Gs, but obviously nobody at MikroTik bothered to actually test MetaROUTER on this model before shipping it.

-- Nathan

timberwolf · Thu Apr 11, 2013 11:22 am

Hi Nathan,

thanks for also testing this. Using only MT Metarouter instances wasn't as bad as you described, but also practically unuasable. I also would have expected MT to test this...
And as this topic gets ignored by MT again...well, I won't repeat myself.

gamespb · Thu Apr 11, 2013 12:20 pm

Hi, I just RB2011UAS-2HnD 6.0rc13 OpenWRT - Linux version 2.6.31.10 (bencoh@procyon) (gcc version 4.6.3 20120201 (prerelease) (L
inaro GCC 4.6-2012.02) ) #4 Mon Mar 5 12:16:12 CET 2012 and freez.
In openwrt "top" shows the CPU utilization on the lan interface, not the underlying processes.
if /metarouter interface leave empty, MR have only loopback. No Freez!(top show - CPU: 0% usr 1% sys 0% nic 98%) .
This may be due to the user interface to switch openwt promiscuous mode? tell me how to check this?

otgooneo · Thu Apr 11, 2013 6:48 pm

uhhhh... I strongly wish that NathanA`s Asterisk image be worked stable. If I`m correctly remember, my RB450G hanged too. But it`s nice to hear that RB450G has no issue. I will test it on RB450G again and will post here.

otgooneo · Wed Apr 17, 2013 11:07 am

Tested with RB450G ROSv5.20. Result is same.

How to solve this issue...^*(&()$^^%$%^#$##$%%$#&*

[admin@PBXTest450G] > log print
00:00:06 system,info router rebooted 
00:00:15 interface,info ether5 link up (speed 1000M, full duplex) 
00:00:59 system,info,account user admin logged in from 192.168.123.149 via winbox 
00:02:56 system,info monitoring new host by admin 
00:02:57 script,warning Metarouter UP 
00:58:05 system,info,account user admin logged out from 192.168.123.149 via winbox 
02:37:32 script,warning Metarouter Down 
02:41:06 script,warning Metarouter UP 
02:56:14 system,info,account user admin logged in from 192.168.123.149 via winbox 
02:56:25 system,info,account user admin logged out from 192.168.123.149 via winbox 
02:57:00 script,warning Metarouter Down 
03:01:51 script,warning Metarouter UP 
03:12:34 system,info,account user admin logged in from 192.168.123.149 via winbox 
03:26:31 script,warning Metarouter Down 
03:38:06 script,warning Metarouter UP 
03:38:37 system,info,account user admin logged out from 192.168.123.149 via winbox 
04:04:57 system,info,account user admin logged in from 192.168.123.149 via winbox 
04:05:08 system,info,account user admin logged in from 192.168.123.149 via telnet

janisk · Wed Apr 17, 2013 1:15 pm

please try these new buiilds:
http://www.mikrotik.com/download/share/ ... .25rc1.npk
http://www.mikrotik.com/download/share/ ... .25rc1.zip

http://www.mikrotik.com/download/share/ ... .0rc14.npk
http://www.mikrotik.com/download/share/ ... .0rc14.zip

they contain metarouter fixes.
corresponing cangelog entry:
*) metarouter: fixed occasional lockups on mipsbe boards;

otgooneo · Wed Apr 17, 2013 1:27 pm

Wooooow!!!
I will test right now and report here back.

payday · Wed Apr 17, 2013 2:09 pm

Wooooow!!!
I will test right now and report here back.

I can't test it until friday, so I am waiting for your reports.
Try to put heavy load on router - this makes lockups happen more often.

otgooneo · Thu Apr 18, 2013 6:40 am

Pretty good results. That`s very very satisfied for me:
RB2011 with v5.25rc1 - No interruption during 14 hours. Average CPU utilization is 36%. At this rate, on v5.24, interruption was 5-6 times.
RB450G with v5.24 - No interruption during 16 hours. Tomorrow I will test v5.25rc1 on 450G.

Unfortunately I have only 1 SIP client. Hope result is same using many SIP clients.

gamespb · Thu Apr 18, 2013 10:00 am

set up - http://www.mikrotik.com/download/share/ ... .0rc14.npk

ethernet ports have stopped working(reset to default does not help, try eth1), just wifi worked.
check metarouter not.

autosupout.rif created how i try
http://kuznecova.net/autosupout.rif
http://kuznecova.net/autosupout.old.rif

RB2011UAS-2HnD

timberwolf · Thu Apr 18, 2013 10:11 am

please try these new buiilds:
http://www.mikrotik.com/download/share/ ... .25rc1.npk
http://www.mikrotik.com/download/share/ ... .25rc1.zip

http://www.mikrotik.com/download/share/ ... .0rc14.npk
http://www.mikrotik.com/download/share/ ... .0rc14.zip

they contain metarouter fixes.
corresponing cangelog entry:
*) metarouter: fixed occasional lockups on mipsbe boards;

Will eventually test this today or tomorrow evening, many thanks for your update.

otgooneo · Fri Apr 19, 2013 8:39 am

RB2011withv5.25rc1 works 2 days without Metarouter issue. Starting to test RB450G using v5.25rc1.

payday · Fri Apr 19, 2013 10:42 am

RB2011withv5.25rc1 works 2 days without Metarouter issue. Starting to test RB450G using v5.25rc1.

Thanks for great info. I can't wait to test it myself. Today, in the evening I will test v6.0rc14.

gamespb · Fri Apr 19, 2013 11:38 am

Hello.
i have RB2011UAS-2HnD
downgrade to 5.25rc1
New firmware has changed the MAC address on the other!!!

janisk · Fri Apr 19, 2013 11:46 am

downgraded from 6.0rc? and how your interfaces where configured? static vif interfaces?

telepro · Fri Apr 19, 2013 3:56 pm

Have tested v5.25rc1 on 3 RB951G units for ~ 40 hours each. Would have expected to see 6 or more failures on each during that interval. Have found none. Thanks much.

gamespb · Fri Apr 19, 2013 9:57 pm

downgraded from 6.0rc? and how your interfaces where configured? static vif interfaces?
Yep from 6.0rc14
interface vif1 dynamic

[admin@Kuznecova3] > interface bridge port pr
Flags: X - disabled, I - inactive, D - dynamic 
 #    INTERFACE               BRIDGE               PRIORITY  PATH-COST    HORIZON
 0    eth1-welcome-to      Welcome                  0x80         10       none
 1    eth5-welcome-from    Welcome                  0x80         10       none
 2  I wlan1                   Welcome                  0x80         10       none
 3  D vif1                    Welcome                  0x80         10       none

Now i try MetaRouter 25rc1
2 hours - good fly

gamespb · Fri Apr 19, 2013 10:04 pm

OpenWrt r30812
[ 0.000000] Linux version 2.6.31.10 (bencoh@procyon) (gcc version 4.6.3 20120201 (prerelease) (Linaro GCC 4.6-2012.02) ) #4 Mon Mar 5 12:16:12 CET 2012

top show :

Mem: 7336K used, 14736K free, 0K shrd, 0K buff, 2412K cached
CPU:   0% usr   1% sys   0% nic  98% idle   0% io   0% irq   0% sirq

nic 98% - i only ping use

payday · Sat Apr 20, 2013 2:16 am

Wow!
6 hours of heavy testing and no freeze at all! It is ROCK SOLID!

I think it's occasion for party!

timberwolf · Sat Apr 20, 2013 4:30 pm

I started tests with 5.25rc1 on RB2011 a few hours ago, I am using the same test setup like before, so time will tell. It didn't hnag in the last 5h but I got that result before. So lets wait a little before we declare this fixed. Right now I was brave enough to also update a more important RB450G to 5.25rc1, so lets see what we will get there.

payday · Sat Apr 20, 2013 9:02 pm

Mine (6.0rc14) is working fine since yesterday. When using previous version I had about 20 hangs a day.

otgooneo · Mon Apr 22, 2013 5:45 am

OpenWrt r30812
[ 0.000000] Linux version 2.6.31.10 (bencoh@procyon) (gcc version 4.6.3 20120201 (prerelease) (Linaro GCC 4.6-2012.02) ) #4 Mon Mar 5 12:16:12 CET 2012

top show :
Code: Select all
Mem: 7336K used, 14736K free, 0K shrd, 0K buff, 2412K cached
CPU:   0% usr   1% sys   0% nic  98% idle   0% io   0% irq   0% sirq
nic 98% - i only ping use

Seems it`s not. As your top show, the NIC is 0% and IDLE is 98%. It`s normal.

RB450G with ROSv5.25rc1, without Metarouter interruption last 2 days. Hope V5.25rc1 all other functions are same as Metarouter without any issue.

timberwolf · Mon Apr 22, 2013 9:15 am

Still looking good for me too, will upgrade another system soon. Keeping my fingers crossed, but it looks very promising this time.

Wazza · Mon Apr 22, 2013 2:04 pm

All,

So I don't suppose anyone has had any luck with PPC based systems?
This all sounds very promising, but I'd really really like to see this working in a PPC environment too.

I've updated a couple of test RB1100AH's. I'm not going to hold my breath, but here's hoping...

Thanks.

Wazza

janisk · Mon Apr 22, 2013 2:23 pm

ppc currnetly has no changes. So it is still crashable. As it is completely different from MIPS implementation so they are looked at separately.

gamespb · Mon Apr 22, 2013 7:50 pm

Hello All.
Point to the working mirrors Metarouter?

After change from http://downloads.openwrt.org/snapshots/ ... ps/package
to: src/gz snapshots http://downloads.openwrt.org/snapshots/ ... s/packages

Now error: Package zsh version 5.0.2-1 has no valid architecture, ignoring

or could share newest image OpenWrt to MetaROUTER?

otgooneo · Tue Apr 23, 2013 6:58 am

ppc currnetly has no changes. So it is still crashable. As it is completely different from MIPS implementation so they are looked at separately.

Hi Janisk, any expected date for fixing issue of PPC? Also do you have any plan to implement metarouter for multicore PPC? tile?

NathanA · Tue Apr 23, 2013 8:00 am

A lot of credit appears to go to majbthrd, who came up with a reliable way of reproducing the hangs on mipsbe: http://forum.mikrotik.com/viewtopic.php?f=15&t=58936

I find it interesting, though, that in that thread, RB2011 are talked about as if they don't have any problems with MetaROUTER, even though I can easily reproduce the problem myself on my RB2011L-IN running 5.24.

I tried 5.25rc1, and I can confirm that even on the RB2011L-IN, there are no more freezes! Awesome! Also, I'll have to run some tests, but I think CPU utilization as shown by the host (/system resource print) when the guests are busy is more accurate now. In the past, I could load down the guests with tasks that would bring the system to a crawl, but CPU utilization would not be showing anything near 100%.

However, I am still seeing the same bizarre problems with MikroTik's MetaFS. I don't know whether the bug is in the host or the guest code. Testing on my RB2011L-IN after several reboots of my Asterisk guest, I saw...

1) the contents of /usr/share completely disappear almost every time, shortly after bootup.
2) the contents of /usr/lib/asterisk/static-http disappear (and thus the Asterisk GUI was unable to find its files)
3) the contents of /etc/init.d disappear in the middle of bootup (so it did not complete booting properly)

And once, I was in /etc/init.d trying to change the file mode of one of the startup scripts (I wanted to prevent Asterisk from starting on next boot to test a theory I had), and although the files were listing, they could not be accessed; chmod told me "chmod: asterisk: Invalid argument". Worked fine after another reboot.

This almost never happens on my RB450G boards, *although* I have seen similar symptoms on an RB450G after using SCP to copy a large file over to the MetaROUTER guest (the file was larger than the amount of RAM I assigned to the guest, and 'top' showed that during the copy, all of either the page cache or the filesystem cache had been consumed). Whenever I did this, the guest would begin to act very...wonky, and similar things would occur (entire directories would disappear, trying to execute commands would result in a system hang or "Segmentation fault", and so on). This made me wonder if the problem was memory-related (after all, 450G has 256MB of memory and I always assign my Asterisk guest half of that), and that's why I wanted to prevent Asterisk from starting on boot-up on my 2011. What I found is that it does help to reduce the occurrence of the problem on the RB2011, but doesn't eliminate it completely (when /etc/init.d disappeared while init was in the middle of running through the boot-up scripts, it disappeared before it reached the Asterisk one, so Asterisk never even managed to start that time).

On the 450G, whenever I would copy a large file to it over the network, I would carefully watch the consumption of available RAM, and periodically issue an 'echo 3 > /proc/sys/vm/drop_caches' to free up any memory that could be freed. That did seem to help. Interestingly, on the 2011 board, when I do that, it almost immediately causes the guest to go sideways...from that moment on, executing any command will either hang the guest or produce "Segmentation fault" (the guest is still responsive, though...console prints typed characters, it responds to pings, etc., so it's not the normal "hanging" problem, which I really do think has finally been fixed for good). It must be a problem with the pagecache because I get the same behavior if I echo '1' instead of '3' (see http://linux-mm.org/Drop_Caches).

Whatever is causing this (bug in Linux kernel in guest, bug in guest MetaFS driver, bug in RouterOS host), it's really annoying.

-- Nathan

timberwolf · Tue Apr 23, 2013 8:28 am

I find it interesting, though, that in that thread, RB2011 are talked about as if they don't have any problems with MetaROUTER, even though I can easily reproduce the problem myself on my RB2011L-IN running 5.24.

Different CPU? Im not into MIPS assembly but locking/pipelining might be a totally different beast when you compare 24K to 74K MIPS...

otgooneo · Tue Apr 23, 2013 8:47 am

2) the contents of /usr/lib/asterisk/static-http disappear (and thus the Asterisk GUI was unable to find its files)

That issue occurs in my case also timezone, hostname configurations lose after non-proper reboot. So I added custom script into rc.local:

echo GMT-8 > /etc/TZ

ntpclient -c 1 -s -h 202.131.224.136

hostname RouterBoard_PBX

echo "[general]
enabled = yes
bindaddr = 0.0.0.0
bindport = 8088
enablestatic = yes
redirect = / /static/config/index.html
[post_mappings]
backups = /usr/lib/asterisk/gui_backups
moh = /usr/lib/asterisk/moh" > /etc/asterisk/http.conf

udhcpc

NathanA · Tue Apr 23, 2013 8:57 am

Quick update:

Whenever the contents of a directory "go missing", it appears that I can fix the problem without rebooting by 'echo 2 > /proc/sys/vm/drop_caches'. Sometimes I need to run a 'sync' before doing so.

On RB2011, telling the kernel to flush the pagecache by echoing '1' or '3' (which is essentially an instruction to perform both 1 and 2) to drop_caches immediately makes the guest unusable. This doesn't happen on 450G, even when the guest is configured to have the same amount of RAM (in this case, 32MB).

Here's another weird one: when I import an image into MetaROUTER, and then reboot the RouterBOARD, RouterOS creates a bunch of directories in the root directory of all my guests. They are called /cm, /logs, /pckg, /run, /store, and /vstore. They are all empty. Those directories do not exist in my original compressed tarball, and are not being made by any program included in the image. If I check the guest immediately after importing and before rebooting the router, they are not present, and they do not show up simply by my shutting down and rebooting the guest. If I remove the directories and then reboot RouterOS, they will come back.

-- Nathan

EDIT: I deleted the VM and re-imported it on the 2011. All the same problems remained. However, after performing a Netinstall of 5.25rc1 on my RB2011, the problems seemed to have gone away. Could this be a YAFFS problem??

EDIT 2: Spoke too soon. Reconfigured the router the way it was before, and same problems are all back. Probably going to take me a little bit to track down the culprit.

luminescentsimian · Tue Apr 23, 2013 9:59 am

Looking good on RB750GL running some openwrt image running unbound, openconnect, and quagga to do DNSSEC validation and OSPF routing for my Cisco work VPN. I haven't noticed any stalling or reboots and quagga doesnt seem to be crashing and generating gibberish OSPF information like it had been.

[admin@Mikrobob] > /system resource print
uptime: 2d3h44m51s
version: 6.0rc14
build-time: Apr/19/2013 15:00:13
....
board-name: RB750GL

janisk · Tue Apr 23, 2013 2:55 pm

i would say that all of the credit goes to majbthrd as he created image that what it does is wait for interrupt and when that occurs, wait again. That allowed recreate freezing problem in matter of seconds, instead of yours that what it was previously.

Also, i was corrected by devs - there are changes for PPC metarouter too as result of this, so it would be useful if PPC was tried out too on more configuration but my test setups.

NathanA · Tue Apr 23, 2013 4:38 pm

i would say that all of the credit goes to majbthrd as he created image that what it does is wait for interrupt and when that occurs, wait again. That allowed recreate freezing problem in matter of seconds, instead of yours that what it was previously.

Very true. I actually meant that credit is split between majbthrd (for the image) and MetaROUTER developers (for knowing what to do with the image).

I am curious: did it turn out to be a simple problem to fix, after you had majbthrd's image to test with? Are you allowed to talk about what the fix ended up being?

Also, i was corrected by devs - there are changes for PPC metarouter too as result of this, so it would be useful if PPC was tried out too on more configuration but my test setups.

Good to know! I will break out the RB1100s again.

-- Nathan

NathanA · Wed Apr 24, 2013 3:13 pm

That issue occurs in my case also timezone, hostname configurations lose after non-proper reboot. So I added custom script into rc.local:

That's a slightly different issue than the one I'm currently fighting, I think. I know others have seen files that were not properly closed (by, say, Asterisk) become lost if the application is still running and the MetaROUTER instance isn't cleanly shut down. That's what is causing your http.conf file to disappear.

What I'm talking about is files becoming temporarily inaccessible or even temporarily hidden. When this happens, the files miraculously reappear if I either reboot the MetaROUTER, or flush the inode cache ('sync; echo 2 > /proc/sys/vm/drop_caches').

In addition, I have also noticed that after extended/prolonged filesystem activity (which also gives the Linux VM subsystem quite a workout), some sort of memory corruption is occurring that is non-reversible once it happens; the only recourse at that point is to reboot the MetaROUTER. This can happen if I copy a file to the MetaROUTER that is larger than the RAM I have allocated to it, or if I read in a file that is larger than the allocated RAM, or 9 times out of 10 on an RB2011, when I flush the pagecache ('sync; echo 1 > /proc/sys/vm/drop_caches'). After doing that, if I try to execute any program in the MetaROUTER, the program will either hang the MetaROUTER, or will immediately quit with "Segmentation fault".

It seemed to be a memory-constraint-related issue at first, and I sought ways of either disabling or limiting both the page and inode caches in the Linux VM subsystem, but apparently as of 2.6.x, this is not possible. It is possible to tune the 'swappiness' (how much weight the VM subsystem gives to clean pages in the RAM cache vs. running code in RAM), but not to instruct it to limit the page cache to only a percentage of available physical RAM. Linux VM is made to consume all possible RAM and then only drop clean pages if RAM is needed by something else. In theory, this should not impact the user negatively, and should even improve overall performance. I was concerned that the fabled Linux "OOM killer" was incorrectly kicking in rather than having clean pages dumped from the cache after available RAM was filled.

So I have tried several things to see if any of them affect the problem:

I tried eliminating the 'tmpfs' RAM-disk partition (at /tmp), just in case it was causing issues and holding onto RAM that it shouldn't be.
I tried making a small swapfile* and setting '/proc/sys/vm/swappiness' to 0, just in case a bug in the VM subsystem was causing it to behave incorrectly when no swapfile was present.
I tried going back to the older MikroTik MetaROUTER patch for OpenWRT (1.1, which only supports MIPSBE) and the older kernel that it supports (2.6.27) just in case there was a bug in either the newer patch (1.2) or the newer kernel (2.6.31).
I took the time to create a working 2.6.35 kernel that boots in a MetaROUTER using patches for 2.6.35 that I got from MikroTik a while back.

No change: the system continued to have both randomly "disappearing files" and memory corruption that coincided with filesystem activity (even if it was just filesystem reads, not writes).

Interestingly, it happens within a few minutes of running my guest on top of an RB2011L-IN, but it is very difficult to make a MetaROUTER guest unstable in this way on a RB450G! (...although it does happen, just very infrequently.) I even gave the RB450G guest the same amount of RAM (less, even) as I had configured on the RB2011L guest, and it didn't flinch. It takes a long time before the same kind of memory corruption surfaces on the 450G. The only way I can reliably reproduce the problem on a 450G is when I SCP a 200+ megabyte file over to the guest. Most of the time, after that copy is finished, it starts acting the same way that the 2011L does after I flush the page cache on it.

At this point, I have to believe that this is a problem in the MetaROUTER implementation itself somehow. I will see about putting together a test suite that I can submit as a ticket to MikroTik support. It's going to be tricky because the problem doesn't always manifest itself the same exact way every time...

-- Nathan

* making a swapfile on top of MetaFS is not easy, because MetaFS just maps files in your MetaROUTER to files in the RouterOS host's filesystem, and on MIPSBE, the host filesystem is YAFFS, which can't be used to hold a swapfile because you can't allocate a large block of contiguous free space ('swapon' actually told me "swapfile has holes" when I tried). So I had to fake it out by building a kernel with VFAT support, creating a file large enough to hold the swapfile, formatting it as VFAT, mounting the file via loopback, and then creating the swapfile on THAT. It's not worth it, and the only reason I went to the trouble was to test a theory. There is no good reason to use swap space inside of a MetaROUTER guest; "don't try this at home."

janisk · Wed Apr 24, 2013 5:08 pm

on RB2011RM with 32MB assigned to guest (openwrt) copied over 74MB file. encountered no problems. What i am doing wrong?

Also, if you run RouterOS as guest, does copying of large file causes you any issues?

otgooneo · Wed Apr 24, 2013 6:46 pm

That's a slightly different issue than the one I'm currently fighting, I think. I know others have seen files that were not properly closed (by, say, Asterisk) become lost if the application is still running and the MetaROUTER instance isn't cleanly shut down. That's what is causing your http.conf file to disappear.

What I'm talking about is files becoming temporarily inaccessible or even temporarily hidden. When this happens, the files miraculously reappear if I either reboot the MetaROUTER, or flush the inode cache ('sync; echo 2 > /proc/sys/vm/drop_caches').

Ohh this issue is too complicate to understand for me

. My issue was simple than this issue.
Sorry that asking off topic question, I want to learn to compile own asterisk image by myself. How did you make your PPC, MIPSE asterisk images? Did you posted step-by-step guide anywhere?

NathanA · Thu Apr 25, 2013 12:42 am

Sorry that asking off topic question, I want to learn to compile own asterisk image by myself. How did you make your PPC, MIPSE asterisk images? Did you posted step-by-step guide anywhere?

I did post a step-by-step of how I built my original OpenWRT "Kamikaze" 8.09.2 image for MetaROUTER, here: http://forum.mikrotik.com/viewtopic.php ... 81#p311681. I suggest that any questions related to this subject be redirected to that thread, or perhaps that a new thread be started.

-- Nathan

NathanA · Thu Apr 25, 2013 6:34 am

on RB2011RM with 32MB assigned to guest (openwrt) copied over 74MB file. encountered no problems. What i am doing wrong?

I don't know, but I managed to come up with a way to reproduce the bug 100% of the time (for me). Try following these steps and see if you get the same results:

Fresh Netinstall of 5.25rc1 to RB2011L. No default config.
Copy the MikroTik OpenWRT image (http://www.mikrotik.com/download/metaro ... rootfs.tgz) to the router.
Import it and give it 32MB of RAM.
After it starts up, open the console, and press Enter to get a shell prompt.

Verify you can perform simple tasks without any problems (run 'top', get a listing of the root directory, etc.):

root@OpenWrt:/# ls /
bin      etc      lib      overlay  rom      sbin     tmp      var
dev      kernel   mnt      proc     root     sys      usr      www

Create a large (16MB), empty file in the root directory:

root@OpenWrt:/# dd if=/dev/zero of=/testfile bs=16M count=1
1+0 records in
1+0 records out

Now try to do some of those same simple tasks again:

root@OpenWrt:/# ls /
Segmentation fault
root@OpenWrt:/# top
Segmentation fault
root@OpenWrt:/# ifconfig
Segmentation fault
root@OpenWrt:/# reboot
Segmentation fault

As you can see, after exercising the filesystem (and the page cache) with a large write operation, the system becomes unstable. Any EXISTING processes are still running and continue to run, but you cannot start any new ones.

Also, if you run RouterOS as guest, does copying of large file causes you any issues?

I don't know; good question. I have not tried it. The same problem might exist with RouterOS guests, but you might not be able to see them because my understanding is that RouterOS largely consists of a single, monolithic process (Nova?). Since any existing processes that are already running seem to be unaffected, I'm not sure whether you would notice any problems or not. Hopefully, though, my steps above with OpenWRT makes it possible for you and the developers to reproduce the problem, and thus to fix it.

Thanks,

-- Nathan

payday · Thu Apr 25, 2013 11:38 pm

I have tested my OpenWRT instance and everything works fine:

BusyBox v1.19.4 (2012-11-09 20:36:06 CET) built-in shell (ash)
Enter 'help' for a list of built-in commands.

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 ATTITUDE ADJUSTMENT (Bleeding Edge, r31411)
 -----------------------------------------------------
  * 1/4 oz Vodka      Pour all ingredients into mixing
  * 1/4 oz Gin        tin with ice, strain into glass.
  * 1/4 oz Amaretto
  * 1/4 oz Triple sec
  * 1/4 oz Peach schnapps
  * 1/4 oz Sour mix
  * 1 splash Cranberry juice
 -----------------------------------------------------

/> ls
bin      etc      logs     pckg     root     store    usr      www
cm       kernel   mnt      proc     run      sys      var
dev      lib      overlay  rom      sbin     tmp      vstore

/> dd if=/dev/zero of=/testfile bs=16M count=1
1+0 records in
1+0 records out

/> ls /
bin       etc       logs      pckg      root      store     tmp       vstore
cm        kernel    mnt       proc      run       sys       usr       www
dev       lib       overlay   rom       sbin      testfile  var

I have intentionally included OpenWRT banner to show you the version of OpenWRT that I am using. Obviously top and ifconfig also run normal. I am using v6rc14 and it is really stable. Try upgrading your firmware and try different release of OpenWRT.

NathanA · Fri Apr 26, 2013 2:41 am

I have tested my OpenWRT instance and everything works fine: [...] I have intentionally included OpenWRT banner to show you the version of OpenWRT that I am using. Obviously top and ifconfig also run normal. I am using v6rc14 and it is really stable. Try upgrading your firmware and try different release of OpenWRT.

Hey, thanks for testing. As you can see from previous posts of mine, I have tried a bajillion different OpenWRT images and Linux kernel versions and they all behave the same way, which is why I was convinced it was a RouterOS host bug/problem. However, I would be interested in trying out your image. Can you post it somewhere for me to download? Also, what kernel version are you running ('uname -a') and what version of the MikroTik MetaROUTER Linux kernel patches did you use to build your kernel?

UPDATE: I just tried my tests on 6.0rc14 with all of the same OpenWRT images, and I don't have the problem! In fact, the guest kernel's pagecache/VM subsystem seems to behave completely differently when the guest is run on 6.0rc14, even though the guest is the exact same image and kernel as I was running under 5.25rc1. So definitely something changed in MetaROUTER itself in 6.0. (If I copy a large file over, it doesn't seem to consume the entire remaining bits of the guest's RAM allotment, for example. I can also flush the clean pages from the pagecache without causing instability in the OpenWRT guest now as well.)

The problem is that I cannot, in good conscience, install BETA software on my customer's devices. 6.0 still has its share of other problems, and I'm not just using the routers to run MetaROUTER on...they do a bunch of other non-MetaROUTER-related tasks as well. So I really need for this to work under 5.x if at all possible. Also, it looks like 6.0 with the same set of default NPK packages enabled uses up a lot more RAM; I didn't have enough free RAM on the RB2011L with the default packages on 6.0 to give my guest 32MB. On 5.x, I have 32MB and then some on top of that available with default packages.

-- Nathan

timberwolf · Mon Apr 29, 2013 1:21 pm

Just a quick update on the ROS MR front.
I didn't have a single freeze or other issue since I installed 5.25rc1 on my RB450G and RB2011.
Although I didn't conduct any further tests, this feels pretty stable as for now. I will try to do some more elaborate tests if my time allows.

janisk · Tue Apr 30, 2013 9:17 am

thank you all for update. If you see issues, please write here or to support. We are checking out NathanA claim about files going missing till MR guest reboot.

NathanA · Tue Apr 30, 2013 11:59 am

We are checking out NathanA claim about files going missing till MR guest reboot.

Thank you. I am not sure, but I suspect that the missing files problem and the "segmentation fault" problem are both related to each other. As I documented before, I can reproduce the "segmentation fault" problem with 5.25rc1 on a RB2011L using MikroTik's own OpenWRT image with the steps I provided. Have you been able to reproduce it on your end?

I haven't come up with a reliable way of reproducing the missing files problem, but I can make it happen more often than not using an OpenWRT image I made with Asterisk on it, on an RB2011. After Asterisk automatically starts on bootup and consumes a good amount of available RAM, it is very likely to happen. If you want a copy of that image, I would be happy to provide it to you.

-- Nathan

janisk · Tue Apr 30, 2013 3:34 pm

it would be nice to get link or a file with asterisk image.

NathanA · Tue Apr 30, 2013 4:14 pm

it would be nice to get link or a file with asterisk image.

You can download the current version of my MIPSBE image here: http://www.nconx.com/~nathan/ast-owrt-m ... ips_b4.tgz. I give it 32MB of RAM when I import it.

Many times, shortly after bootup, on my RB2011L, 'ls -la /usr/share' will show nothing. Then if I 'sync; echo 2 > /proc/sys/vm/drop_caches', usually after that 'ls -la /usr/share' will show the subdirectories.

I will continue to work on finding a way to consistently reproduce the problem, in case you cannot replicate my issue. (So far, though, the instructions I wrote out earlier [here] reproduce "segmentation fault" for me on RB2011L every time.)

Thanks again,

-- Nathan

otgooneo · Fri May 03, 2013 9:43 am

Sorry that asking off topic question, I want to learn to compile own asterisk image by myself. How did you make your PPC, MIPSE asterisk images? Did you posted step-by-step guide anywhere?
I did post a step-by-step of how I built my original OpenWRT "Kamikaze" 8.09.2 image for MetaROUTER, here: http://forum.mikrotik.com/viewtopic.php ... 81#p311681. I suggest that any questions related to this subject be redirected to that thread, or perhaps that a new thread be started.

-- Nathan

Thanks Nathan, Is there any possible way just change asterisk default configuration files of the image, without rebuilding it? I tried to use 7-ZIP. It allow me to change files of the image without extracting it. But when I upgraded asterisk configuration files of the image, image become unable to import into metarouter. Metarouter says "Archive is too short"

Second thin is, yesterday I had power issue and router rebooted. But /etc/asterisk/users.conf file has broken. Just became blank file. All other files "/etc/asterisk/extensions.conf, /etc/asterisk/extconfig.conf" are OK. How can I automate process "backup users.conf file every day and replace original users.conf by this backup file after every reboot occurs"

NathanA · Fri May 03, 2013 10:35 am

is there any possible way just change asterisk default configuration files of the image, without rebuilding it? [...]

This is the wrong thread for this discussion. I posted my response to the other thread.

-- Nathan

otgooneo · Thu May 16, 2013 7:15 am

Hi MT Support,
Sometimes metarouter disappears after non proper shutdown. Just no metarouter. Metarouter interface is still there but assigned to unknown Virtual machine. What it can be?

NathanA · Thu May 16, 2013 9:57 am

New issue: presence of a MetaROUTER on 5.25 can cause host in certain cases to take much longer to complete tasks than it normally would.

Example: I added a new MetaROUTER to a 450G that a customer had. Instantly, CPU on 450G shot up to 100% and stayed there until I disabled the MetaROUTER. Profiler showed "console" process taking up the majority of the CPU time (70%+) when this was happening.

I traced the issue to a Scheduler event that was kicking off every 10 seconds, and performing a search of the routing table (example: '/ip route find dst-address~"192.168.1.126/32"'). Routing table was roughly 700 entries long ('/ip route print count-only'). While MetaROUTER was running and the CPU was at 100%, I disabled the Scheduler event and killed all active jobs ('/system script job') that it spawned, and CPU went back to normal.

I then tried executing the same '/ip route find' search from a terminal window, both with and without a running MetaROUTER. Without any MetaROUTER running, the routing table search took 1 second (maybe 2, at the most). With a MetaROUTER running, routing table search took almost 15 seconds, and "console" process was maxing out the CPU the entire time.

-- Nathan

timberwolf · Thu May 16, 2013 2:04 pm

Nathan, just to be sure, you are talking about a OpenWRT MetaROUTER or a ROS MetaROUTER?
I'll have to try and reproduce this with a ROS MetaROUTER.

ayufan · Thu May 16, 2013 2:06 pm

New issue: presence of a MetaROUTER on 5.25 can cause host in certain cases to take much longer to complete tasks than it normally would.

Example: I added a new MetaROUTER to a 450G that a customer had. Instantly, CPU on 450G shot up to 100% and stayed there until I disabled the MetaROUTER. Profiler showed "console" process taking up the majority of the CPU time (70%+) when this was happening.

It's not something new. Use of 1 MetaROUTER roughly decreases performance of host by 50%. Use of 2 even more. Despite the fact that MetaROUTER is idle and not doing anything at all, just idling ;)

Kamil

timberwolf · Fri May 17, 2013 8:21 am

New issue: presence of a MetaROUTER on 5.25 can cause host in certain cases to take much longer to complete tasks than it normally would.

Example: I added a new MetaROUTER to a 450G that a customer had. Instantly, CPU on 450G shot up to 100% and stayed there until I disabled the MetaROUTER. Profiler showed "console" process taking up the majority of the CPU time (70%+) when this was happening.
It's not something new. Use of 1 MetaROUTER roughly decreases performance of host by 50%. Use of 2 even more. Despite the fact that MetaROUTER is idle and not doing anything at all, just idling

Kamil

I can't acknowledge this, ROS MetaROUTER instances don't seem to consume a significant amount auf CPU cycles when idle. Of course this will change if the instances get real load.

NathanA · Wed May 22, 2013 1:45 am

It's not something new. Use of 1 MetaROUTER roughly decreases performance of host by 50%. Use of 2 even more. Despite the fact that MetaROUTER is idle and not doing anything at all, just idling

We are not talking about 50%, we are talking about a factor of 10x or more. BIG DIFFERENCE.

If what you say were true, then a task which would normally take about 1 second without a MetaROUTER guest running would take roughly 2 seconds with a MetaROUTER guest. I'm seeing 10-15 seconds!

-- Nathan

ayufan · Wed May 22, 2013 12:44 pm

We are not talking about 50%, we are talking about a factor of 10x or more. BIG DIFFERENCE.

If what you say were true, then a task which would normally take about 1 second without a MetaROUTER guest running would take roughly 2 seconds with a MetaROUTER guest. I'm seeing 10-15 seconds!

-- Nathan

I didn't test performance of guests. I see how works the hypervisor (base RouterOS). It's twice slower than usual EVEN if guest MetaROUTER is idle.

NathanA · Wed May 22, 2013 1:12 pm

I didn't test performance of guests. I see how works the hypervisor (base RouterOS). It's twice slower than usual EVEN if guest MetaROUTER is idle.

I am not talking about guests either. I am saying that the HOST is TEN times slower when a guest is running (idling), not TWO times slower. The task that normally takes 1 second on the HOST will take 10-15 seconds on the HOST when a guest is running and idling.

-- Nathan

timberwolf · Wed May 22, 2013 1:20 pm

We are not talking about 50%, we are talking about a factor of 10x or more. BIG DIFFERENCE.

If what you say were true, then a task which would normally take about 1 second without a MetaROUTER guest running would take roughly 2 seconds with a MetaROUTER guest. I'm seeing 10-15 seconds!

-- Nathan
I didn't test performance of guests. I see how works the hypervisor (base RouterOS). It's twice slower than usual EVEN if guest MetaROUTER is idle.

The hypervisor hands out shares of CPU time every 3,33ms if I'm not mistaken, the guests can and do yield back if there is nothing to do.
So what fact you claim to know exactly makes you think, there is a penalty of 50%?

ayufan · Wed May 22, 2013 2:44 pm

The hypervisor hands out shares of CPU time every 3,33ms if I'm not mistaken, the guests can and do yield back if there is nothing to do.
So what fact you claim to know exactly makes you think, there is a penalty of 50%?

Ok. I will later today or tomorrow put some tests. So it happens that I have spare RB450G which can handle 7 MetaROUTERs.

janisk · Mon May 27, 2013 3:55 pm

please note that all RouterOS instances are guests, only one of them have exclusive permission to access actual hardware of the RouterBOARD. Anything else is same for all of them.

Thit is posted here in the same place where it was told about that 1/300 second switch time between guests.

link to actual posts:
http://forum.mikrotik.com/viewtopic.php ... 54#p332654
http://forum.mikrotik.com/viewtopic.php ... 79#p332679

NathanA · Wed May 29, 2013 7:21 am

please note that all RouterOS instances are guests, only one of them have exclusive permission to access actual hardware of the RouterBOARD. Anything else is same for all of them.

janisk,

Thanks for the reminder of these details; I remember reading your posts after you made them, but had forgotten about these details in the context of the latest discussion.

I have to wonder if some of the latest changes meant to fix the IRQ issue(s) affected the timeslicing that the hypervisor engages in. I should be able to load an older version (say, 5.14 for MIPSBE or something around that time) and test to see if the same 10x performance hit happens to those older versions as well. If it does, then I guess that this huge performance hit just comes with the MetaROUTER territory on MIPSBE, although anything that MikroTik can do to try to improve that would be much appreciated. If older versions perform much better, though, then it's something that the developers should be aware of. Hopefully it is not a side-effect of recent MetaROUTER changes that are impossible to work around if we want the system to be stable.

-- Nathan

janisk · Wed May 29, 2013 4:15 pm

description there is what it should have. Variance from this is considered a bug or a problem that have to be resolved.

Also note that (as far as i understand) context switching takes place as usually to swap around and commands go through the hypervisor. Yes it uses cashing to make it faster, but it adds extra load.

NathanA · Thu May 30, 2013 6:57 am

Okay, I have tested 5.16 vs. 5.25 on an RB450G. They both behave the same. Not sure if it is a bug or not. Here is how to reproduce:

Netinstall 5.25, blank config/no defaults

Add ~1000 or so bogus routes to the default routing table:

[admin@MikroTik] > :for x from 0 to 3 do={:for y from 0 to 255 do={/ip route add dst-address=("192.168." . $x . "." . $y) gateway=127.0.0.1}}
[admin@MikroTik] > /ip route print count-only
1024

Try to traverse the routing table, and time how long it takes:
Code: Select all
```
[admin@MikroTik] > /ip route find dst-address~"1.2.3.4/32"
```
...with no MetaROUTER running, this will finish in about 1 second.
Now add a new MetaROUTER. RouterOS or imported does not matter; RAM size does not matter:
Code: Select all
```
[admin@MikroTik] > /metarouter add disabled=no
```
Traverse the routing table again while this MetaROUTER is running and doing nothing (0-1% CPU), and time how long it takes:
Code: Select all
```
[admin@MikroTik] > /ip route find dst-address~"1.2.3.4/32"
```
...with 1 MetaROUTER running, this will finish in about 15 seconds!!!

If you have "/tool profiler" running, you will see the "console" process taking up all CPU time on the host during the routing table traversal.

-- Nathan

chojrak11 · Wed Jun 19, 2013 2:33 am

Code: Select all
[admin@MikroTik] > /ip route find dst-address~"1.2.3.4/32"
...with 1 MetaROUTER running, this will finish in about 15 seconds!!![/*][/list]

If you have "/tool profiler" running, you will see the "console" process taking up all CPU time on the host during the routing table traversal.

-- Nathan

I can confirm this exact scenario and observations. On one of my 493 running RouterOS 6.1 it takes about 45 seconds, not 15 seconds, to complete the command above, while the MetaRouter is running (and I waited until that MetaRouter completely started after creation prior to issuing this command in "host"). I repeated the command after shutting down the MetaRouter and it still took more than 30 seconds to complete. After removing the MetaRouter, things went back to normal (1-2 secs).

janisk · Wed Jun 19, 2013 9:37 am

it seems that this is it. I can confirm that there is a slowdown. As host and guests both are same turf - being guests, it does not matter where you run this script. However adding extra guests does not increase the run-time.

NathanA · Fri Nov 07, 2014 4:39 pm

Sorry to drag this thread out of retirement, but has anybody been experiencing random reboots of RB450G with recent versions of RouterOS (6.20, for example) when using MetaROUTER?

I just opened up a ticket (#2014110766000602) with MikroTik, so hopefully I will hear back from them next week, but last week I updated an RB450G from 6.11 that was running a MetaROUTER with no problems to 6.20. I did this primarily to get the latest CAPsMAN before implementing CAPsMAN on this client's network (13 APs, all RB751Ui), and now, after the upgrade, the 450G reboots randomly every few hours (although sometimes it happens only after a few minutes). ARGH.

This time, it is not a watchdog timer going off and rebooting the router, it is "kernel failure". At this point, I don't know for sure that the MetaROUTER is even involved -- it could be CAPsMAN, or some other part of 6.20 that is buggy, and not MetaROUTER! -- but if anybody else out there has upgraded a 450G that they are using MetaROUTER on and didn't do anything except upgrade it (configuration is exactly the same), and yours is also now rebooting with "kernel failure", it would be interesting to compare notes. At least I would know what the culprit is...

Thanks,

-- Nathan

NathanA · Mon Nov 17, 2014 1:03 pm

ppc currnetly has no changes. So it is still crashable. As it is completely different from MIPS implementation so they are looked at separately.

Hey janisk,

Did anything ever change with PPC MetaROUTER, as far as stability goes, since you said this a year ago? Did the devs ever find and fix the problems that made PowerPC MR crash? I tried to avoid using PowerPC for MetaROUTER because of this, but now I am testing it out again on RB1100 boards with RouterOS 6.20 and it looks like I am still able to crash it. Results are not 100% conclusive yet, though, so I am going to continue to run tests.

Here is the crazy part: I applied the same basic unsupported workaround that I came up with for RB1100AHx2 (using uniprocessor kernel to enable MetaROUTER) to a RB850Gx2, and not only does it work, but so far, MetaROUTER on a RB850Gx2, which is unsupported, seems to be more stable than MetaROUTER on a RB1100, which is supported! So it is possible that there is something specific to RB1100 that needs to be looked into. If I can manage to get the 850Gx2 to crash, though, then I guess we can conclude that it is a general issue with MR.

-- Nathan

NathanA · Wed Nov 19, 2014 12:24 pm

ppc currnetly has no changes. So it is still crashable. As it is completely different from MIPS implementation so they are looked at separately.
Hey janisk,

Did anything ever change with PPC MetaROUTER, as far as stability goes, since you said this a year ago? [...] I am testing it out again on RB1100 boards with RouterOS 6.20 and it looks like I am still able to crash it. Results are not 100% conclusive yet, though, so I am going to continue to run tests.

I can confirm that I can still crash and reboot RB1000 and RB1100 boards (I tried both) when transmitting network traffic to a MetaROUTER VM running on a RB1000-series board.

At the same time, my "hacked" RB850Gx2 runs MetaROUTERs like a champ, and I cannot crash it. It just goes and goes.

Kill me now.

-- Nathan

flashberry · Thu Nov 27, 2014 11:27 am

I have the same unstable behaviour as NathanA.

We have about 30 RB951Ui-2HnD devices, all of them have hostpot functionality and we use metarouter with OpenWRT on every device, to filter content with transparent proxy. We tried router firmwares from 6.7 till 6.22 and different build of OpenWRT.

All devices have almost the same configs (SSIDs and hotspot files are the only different things). Routers had different factory firmwares (from 6.0RC till 6.15) and we have several router with "freeze effect". Freezes take place when Metarouter is started and have some network activity. Both Metarouter and Mikrotik systems aren't responding during this time, and we have about 12 seconds of freeze and 20 seconds of normal work after it. We tried different network configs, different firmwares, openwrt images and have no success.

Usually we have issues with routers came was bought with 6.15. Looking for help, because we couldn't scale without overcomimg this problem.

NathanA · Mon Dec 01, 2014 4:25 am

I have the same unstable behaviour as NathanA.

Hmm, actually it doesn't sound exactly the same...you are complaining about frequent freezes on a MIPSBE board, while I was complaining in my most recent messages about PPC boards rebooting themselves while running MetaROUTERs. So very different. Your frequent freezing problem, though, is not unlike some of the problems reported earlier in this thread, most of which have been resolved for most people for many, many versions now...

We have about 30 RB951Ui-2HnD devices, [...]

Now that's interesting. Most people here have been running MetaROUTERs on RB450G boards, primarily because of the generous complement of RAM and NAND flash. But, like I said, the symptoms you are reporting sound very similar to the symptoms that people who used to have problems with MR on 450G boards would complain about. Most of the sporadic performance and lockup problems on the 450G were fixed long ago, but it would not surprise me to learn that other boards still had quirky issues specific to them (for example, it used to be that 450Gs were unstable while other boards such as the 433AH were stable running the same RouterOS version).

However, about a week ago, I played with a MetaROUTER on a RB951Ui, and didn't experience any issues like the ones you brought up.

Usually we have issues with routers came was bought with 6.15. Looking for help, because we couldn't scale without overcomimg this problem.

So are you saying that some 951Ui routers have the problem, and others don't, and it's usually the newer shipments that have problems?

It's possible that there are slight hardware differences/updates being made to newer shipments of the 951 series by MikroTik that maybe could explain the different behavior you are seeing with older vs. newer units. For example, this thread would seem to suggest that 951G boards being shipped today differ in subtle ways from older ones; the same thing could also be true of the 951Ui: http://forum.mikrotik.com/viewtopic.php?f=2&t=86802

-- Nathan

NathanA · Mon Dec 01, 2014 4:29 am

I just opened up a ticket (#2014110766000602) with MikroTik, so hopefully I will hear back from them next week, but last week I updated an RB450G from 6.11 that was running a MetaROUTER with no problems to 6.20. I did this primarily to get the latest CAPsMAN before implementing CAPsMAN on this client's network (13 APs, all RB751Ui), and now, after the upgrade, the 450G reboots randomly every few hours (although sometimes it happens only after a few minutes). ARGH.

This time, it is not a watchdog timer going off and rebooting the router, it is "kernel failure". At this point, I don't know for sure that the MetaROUTER is even involved -- it could be CAPsMAN, or some other part of 6.20 that is buggy, and not MetaROUTER!

So just to update everyone, as far as the 450G/MIPSBE problem goes, it turns out that my 6.20 "kernel failure" reboots must have been CAPsMAN related and were in fact fixed in later versions, since the problem has gone away ever since I upgraded to 6.23rc7. It may have been MetaROUTER + CAPsMAN causing this in conjunction with each other, because I also tried running CAPsMAN on another MIPSBE board (RB951Ui) for a while, and it never rebooted, but I wasn't running MetaROUTERs on it at the time either. But what's important is that problem seems to be fixed (*knock on wood*).

So thank you to MikroTik for that.

PPC issues are still a thing.

-- Nathan

otgooneo · Wed Dec 10, 2014 8:19 pm

Hi Nathan, How are you?
I`m waiting for a too long MT to implement Metarouter on the tile chipset. Do you any news or guess on that? Imagine metarouters on CCRs, will be very nice.

NathanA · Fri Dec 12, 2014 6:01 am

I`m waiting for a too long MT to implement Metarouter on the tile chipset. Do you any news or guess on that?

Only MikroTik can know their own plans or progress. I suggest you ask them.

Imagine metarouters on CCRs, will be very nice.

Right now I think it would just be nice if MetaROUTER worked on the platforms it is already advertised to be supported on.

-- Nathan

otgooneo · Sat Jan 10, 2015 8:31 am

Got reply from MT support. They say

For tileX architecture virtualization features will most likely come in one ROS v7 builds. We should be able to provide more info on that when v7 will go into beta testing stage.