RB4011 and RB1100 AHx4 "bricks" randomly

marisspringis · Mon Jun 03, 2019 1:18 pm

Hi everyone,
since the last week of may, strange thing has happened to Router Boards which i manage.
issue - Router simply bricks, what i mean by that is - you cannot connect to router in any way (stays on logging in and nothing more happens), APs that are connected lose all config from dude, SNMP stops working and so on.
in the same time, from computers which are connected to switch, internet is working. also i can ping that router
this can be resolved only by hard reset (take power cable off/on)
issue has happened only on RB4011 and RB1100AHx4 Dude edition
Router Os - 6.44.3
previously this has never happened.
these RB are in different countries.

so far this has happened only once but with every RB4011 we have and one RB1100 AHx4
does anyone else has seen this?

ccardenas · Mon Jun 03, 2019 1:48 pm

Hello! Do you have bridges in your network implementation? How many hosts are passing traffic among these "random bricking" devices? Could you provide us more info? So we can understand your problem and we'll be able to help you better.

We have a similar issue, and we are suspecting about the bridge host table size and (possible) memory exhaustion problem. It only happens in new arm devices (RB4011 and RB1100x4).

Symptoms are loss of connectivity, manageability and it's impossible to access the device in any way, but it keeps working as a switch. After a reboot (unplug power cable/ replug) all begin to work fine and we can see a lot log lines like: snmp, warning timeout while waiting for program XX (where xx is a variable two digit number)

Regards

marisspringis · Mon Jun 03, 2019 2:43 pm

Hi ccardenas,
yes, we have 3 bridges in RB4011 and in RB1100AHx4.
hosts in RB4011 most of the time are - 30-40
in RB1100 not more than 5 connected directly, it is used as dude server for monitoring.

one more thing to point that problem is in these RB4011 is that we have a lot of RB2011 with the same config, and they work perfectly, without any problems. also ROS version in all ar is the 6.44.3

symptoms are identical to yours.

marcin21 · Mon Jun 03, 2019 4:32 pm

Itr seems that cpu is getting exhausted over time.
this particular ARM based WAP60g is 149d up.

ccardenas · Mon Jun 03, 2019 5:45 pm

one more thing to point that problem is in these RB4011 is that we have a lot of RB2011 with the same config, and they work perfectly, without any problems. also ROS version in all ar is the 6.44.3

Hello!! Yes, totally true. RB2011 and RB1100Hx2 in the same place, in the same network, in the same situation and nothing happens, they never block. We've opened a support ticket to Mikrotik and they told us to plug a serial cable and wait the device to block, then try to access it via console and make a supout, but we have a couple of them with cables attached and now they never block!

Other devices within the network keep blocking randomly. In the meanwhile we have scheduled a reboot (lame solution, but it saves the day) at nights a couple of times a week until we find the real problem, but it seems that some process inside the routerboard hangs or collapses the memory, making the another processes fall in cascade and block the access to the device.

If someone is experiencing the same problem, please share with us, maybe we can find a hint in the meanwhile, until I can get a good supout file and send to Mikrotik support.

Regards!!

Dude2048 · Mon Jun 03, 2019 6:38 pm

I have a RB1100ahx4 Dude edition which has the same behavior. What happens is that the memory hogs and the device will become inaccessible. I have a script that reboots the device when 70% is used. During the times that it is inaccessible I tried to make a supout, via console, but that didn't work.

kehrlein · Thu Dec 05, 2019 12:24 pm

Same issue here with 3 different RB1100AHx4. The situations happens with < 7 days uptime.
During the incident, the devices don't accept new SSH, Winbox, SNMP or PPTP connections to the router itself.
Also logins via CLI aren't possible (serial connection is possible; login doesn't work). We tried to have a running serial connection to the affected devices. If the issue occurs, we are able to type commands into the cli, but generating the supout or doing real actions (e.g. initiate a reboot) doesn't work.
Other traffic goes through the router smoothly. The issue can temporary be fixed by powering off an on the device. Temporary solution is a scheduled reboot.

meshnet · Thu Dec 05, 2019 9:05 pm

Starting to see this issue also on 4011s. 6.44.1 on the last one it happened to.
Power cycle restores all functionality..
SNMP polling becomes very intermittent right before this happens, pointing to the CPU issues..
Only bridge configured on the devices is an empty bridge for loopback.

Dude2048 · Fri Dec 06, 2019 12:01 pm

Same issue here with 3 different RB1100AHx4. The situations happens with < 7 days uptime.
During the incident, the devices don't accept new SSH, Winbox, SNMP or PPTP connections to the router itself.
Also logins via CLI aren't possible (serial connection is possible; login doesn't work). We tried to have a running serial connection to the affected devices. If the issue occurs, we are able to type commands into the cli, but generating the supout or doing real actions (e.g. initiate a reboot) doesn't work.
Other traffic goes through the router smoothly. The issue can temporary be fixed by powering off an on the device. Temporary solution is a scheduled reboot.

What version are you using.

quackyo · Fri Dec 06, 2019 1:36 pm

Does everybody experiencing this run Dude on the device?
I have had the exact same issue on my RB1100AHx4 shortly after setting up Dude. Happened 3 or 4 times over a month or two before i connected the dots. Disabled Dude server and no hickup since (6-8 months ago...).

kehrlein · Fri Dec 06, 2019 1:38 pm

Same issue here with 3 different RB1100AHx4. The situations happens with < 7 days uptime.
During the incident, the devices don't accept new SSH, Winbox, SNMP or PPTP connections to the router itself.
Also logins via CLI aren't possible (serial connection is possible; login doesn't work). We tried to have a running serial connection to the affected devices. If the issue occurs, we are able to type commands into the cli, but generating the supout or doing real actions (e.g. initiate a reboot) doesn't work.
Other traffic goes through the router smoothly. The issue can temporary be fixed by powering off an on the device. Temporary solution is a scheduled reboot.
What version are you using.

Latest bad experience was with 6.45.7.
Time for experience with 6.46 was too short.

After some mails with the MikroTik support during the last month, I finally got this answer:

Unfortunately, this problem seems to be caused by a hardware issue.
Please contact the seller and return the router for warranty repairs, if the router is still covered by it. You can refer to this ticket number - SUP-3012.

So now I am talking to the reseller about refund.

kehrlein · Fri Dec 06, 2019 1:40 pm

Does everybody experiencing this run Dude on the device?
I have had the exact same issue on my RB1100AHx4 shortly after setting up Dude. Happened 3 or 4 times over a month or two before i connected the dots. Disabled Dude server and no hickup since (6-8 months ago...).

I had the issues on several devices without running the Dude.

Bolle · Fri Dec 06, 2019 2:49 pm

Hi !

Same here with RB4011iGS+5HacQ2HnD und RB1100Dx4.

Both devices ´freezes´ several times a week.
The RB1100 sometimes two or three times a day.

I tryied with ROS 6.45.7 and 6.46beta59.

marcin21 · Mon Dec 16, 2019 1:38 pm

Anyone has found a solution to this problem ?
Maybe SNMP disable?

I've got problematic 4011 and since few days it started to die.

aoakeley · Sat Dec 21, 2019 3:18 pm

I have two 1100AHx4 with this issue, and an open case with Mikrotik (Ticket#2019100722004559). I have just been able to connect in with a serial cable and generate a supout file.

I am not running Dude on them.
SNMP is enabled though.

FreeVoip · Sat Dec 21, 2019 4:49 pm

Hello!.

Same problem here!!!!! viewtopic.php?f=2&t=154859&sid=9cf5b071 ... 23dad9a667

I don't know what else to do. It already happened to me 4 times in the week.
There is no high CPU consumption, no RAM consumption or anything strange.

It is a very big problem for me.

FreeVoip · Sat Dec 21, 2019 4:57 pm

It is incredible that since June this was reported and nobody from Mikrotik did anything.

In my case they suggested connecting via serial cable but I am 800 km from my node. It is impossible for me to do that. It is easier to restart it but it is not very serious for an internet provider to cut the service every day at the same time. Needless to say, if it "hangs" at 10 am I have to wait until the next day to access again.

marcin21 · Sat Dec 21, 2019 8:31 pm

since last problem, I 've got winbox session opened on screen a this problemtic 4011 is up and running for 5d+
I wonder if it has something in common with
viewtopic.php?f=3&t=142298

aoakeley · Mon Dec 23, 2019 3:34 am

It is incredible that since June this was reported and nobody from Mikrotik did anything.

I suspect that they did not have enough information to replicate the issue and develop a fix. Like you many of my devices were a long way away, and they asked to have a serial cable plugged in. This was difficult, but I now have one connected and have sent a supout to support.

With a few more people reporting the issue now, hopefully they will be able to find a fix.

marcin21 · Mon Dec 23, 2019 11:59 pm

I wouldn't be so optimistic regarding fast tracking the problem down.
it seems that very similar issue with rb4011
viewtopic.php?f=3&t=142298
was reported in dec'18, and in april'19 first response from Mikrotik staff that they recognise problem and working on solution,
and it seems that they haven't solved it till now.
maybe there is some serious flaw in hardware, as we see it above

Unfortunately, this problem seems to be caused by a hardware issue.
Please contact the seller and return the router for warranty repairs, if the router is still covered by it. You can refer to this ticket number - SUP-3012.

either the way i've got my winbox opened on my problematic 4011 and 7d12h uptime. maybe winbox opened does the trick? I doubt it but whatever, when it works.
Merry Xmas

luciansilviu · Fri Dec 27, 2019 4:14 pm

Hi!

Just a quick "me too". I have had this issue with two boards RB4011iGS+ running on 6.45.4.

reapster · Sat Dec 28, 2019 8:03 pm

Hey there,

We also see this on our RB1100AHx4 (Currently 6.45.7, and inaccessible), every few weeks it exhibits this behaviour. Cannot access via Winbox/Telnet/SSH, but routing remains fine (thankfully!). This unit does almost no traffic (< 10Mbps), and has no special configuration, its not even doing SNMP. About 10 firewall rules, usual masquerade NAT and I think a DHCP server, very basic setup.

All functionality returns after a power cycle, sadly not a feasible long term solution though. We have quite a few Mikrotik units deployed, but will have to start moving away from Mikrotik if there's no real solution to this.

aoakeley · Thu Jan 02, 2020 2:57 pm

We have quite a few MikroTik units deployed, but will have to start moving away from Mikrotik if there's no real solution to this.

We have hundreds (thousands??) of MikroTik units in the field, and only 2 exhibiting this issue. So don't throw the baby out with the bath water.

After obtaining supout.rif files by console cable, MikroTik have organised warranty replacement for the two 1100 units exhibiting this issue. Can't ask for more than that.
We have not seen this issue with any 4011 units.

Andy

allevot · Fri Jan 03, 2020 3:18 pm

Hi,

We have several RB4011 and exactly the same happens to us. Randomly some MikroTik is no longer accessible by management, and in Dude appears the services "memory, disk and cpu" down, but the ping goes well.
The services work correctly, but there is no way to log in to the equipment and the only way to be able to have management is to restart it.

For more info we have VRRP, Bridge, Tunnel GRE, ACLs, BGP, Queues, SNMP and traffic is 20-50Mbps.

Regards,

aoakeley · Sat Jan 04, 2020 2:56 am

We have several RB4011 and exactly the same happens to us. Randomly some MikroTik is no longer accessible by management, and in Dude appears the services "memory, disk and cpu" down,

Regards,

- Use a console cable to get a supout.rif file while the router is in the non responsive state (might as well do this when you lodge the ticket, as they will ask you to do this anyway). To generate the suport.rif you will probably need to have the console cable connected and be logged on before the router craps out.
- Log a ticket with support@mikrotik.com.au
- if there are any autosupout.rif in the file list send these also

Good luck.

allevot · Thu Jan 09, 2020 10:43 am

Hi,

We are unable to make a supout.rif because we are unable to log in into the device in any way.
Someone from MikroTik support can help us? It's a big problem for our company.

Thanks,

r00t · Thu Jan 09, 2020 6:51 pm

If router is still running, you could try to generate supout by watchdog script that:
1) checks if router can ping some IP that should normally be reachable at all times (gateway, some other LAN device)
2) if not (interfaces are frozen, down, IP not accessible) generate supout
3) reboot the router
Run it every minute and hope it works when the issue happens.
But if router is not even accessible by console cable, it's likely completely frozen (CPU/ROS) and above script would not run anyway...

Some more debugging options for ROS would be great, like redirecting kernel syslog to console port at all times.

inteq · Fri Jan 10, 2020 4:31 pm

Only one RB4011 (without WiFi) out of 12 crashed once with some process stuck.
None of RB1100AHx4 or RB1100AHx4 Dude Edition out of 19 crashed so far.
Also, bricking can happen to Mikrotiks, but it did not happen to me (yet) and if a power reset fixes it, it did not happen to you (yet).

jdejansb · Sat Jan 11, 2020 12:07 am

... and I was just about buying a 4011 . . . should I do that or not

.. it's for a main network gateway, PPPoE, dude, . . .

aoakeley · Mon Jan 13, 2020 5:18 am

... and I was just about buying a 4011 . . . should I do that or not .. it's for a main network gateway, PPPoE, dude, . . .

I have plenty running that have no issue. I think the devices with this issue are very few.

tquibell · Sat Jan 18, 2020 7:41 am

Same Problem as OP I think.

Not using any fancy routing. Only doing basic Firewall/NAT/Mangle and DHCP Server and Client. Some static routes, etc. Nothing else.

RouterOS Long Term 6.44.3 would cause RB4011 to randomly lock up. First few times a reboot did the trick, uptimes averaged about 60 days between lockups. The router would remain visible in Winbox but otherwise I could not communicate with it in any fashion. IP addresses would become 0.0.0.0 etc. Terminal unresponsive. Have not confirmed whether the built in serial console was also unresponsive. Finally on the last go-round the board bricked.

On boot up was able to see that the board was hanging on loading kernel. Used Netinstall to wipe the NAND and reflash the Firmware and RouterOS. Voila. Moved to Stable 6.46.2 to see if this thing will survive with a different version.

Will report back after some light stress testing and a few weeks of uptime (hopefully alot more).

Mikhalich · Mon Jan 20, 2020 7:05 pm

Have same problem on RB4011 without WiFi.
System stops forward client packets but I can connect via winbox to WAN-port.
System runs script make supout.rif every 2-3 minutes but can not finish it.
Device did not execute any commands via console but respond on command via winbox.
Tools-Profile shows 100% load 1 of 4 core and 77% unclassified load.
Uptime was about 60 days. ROS 6.46.1

After reboot via winbox system runs normally.

Maggiore81 · Sat Jan 25, 2020 7:12 pm

Hello
I have a 4011 that had the same behaviour described. I replaced it with another 4011, same problem!
Ethernet ports simply stop responding, then in the logs there were "waiting for progam xx" , sometimes 20, sometimes other number as described.
I tried with watchdog to one ip to reboot automatically, with no help.

I solved completely planning a system reboot at 05:30:00 every day.
Then with the latest 6.44.6, every few day.
zero issues since then

omidkosari · Sat Feb 01, 2020 3:05 pm

Hello,

I want to share my experience with same issue. This bug exists for years with several different device types like RBxx ,CCR1009 , CCR1036 , x86 Server , etc. I had several tickets with mikrotik support but unfortunately they were not able to find the problem . The only common feature in all of those routers was PPPoE concentrator (pppoe server).

The only current workaround is to schedule reboot the router everyday. Even with rebooting everyday the problem may appear randomly each few weeks.

Maggiore81 · Sat Feb 01, 2020 6:20 pm

In my setup there were not any pppoe services, just plain routing with simple queues.

omidkosari · Sun Feb 02, 2020 8:56 am

We have to find a common point. After reading this topic I am suspicious about two new points.

1-Do you have this issue on a device which is not monitored by Dude ?
Please also take a look at following topic viewtopic.php?f=2&t=114365

2-Do you have this issue on a device which is not using Queues?

Maggiore81 · Sun Feb 02, 2020 9:31 am

Hello. All my devices are monitored by dude. And have queues. I have one without queues and it works flawlessy.

omidkosari · Sun Feb 02, 2020 9:58 am

In my devices like yours, all of the buggy devices have queues. Almost each pppoe user which connects to router will have dynamic queue from radius server. I think the issue happens when a specific dynamic queue will change (add or remove) on the router. The main problem to solving this issue is that , the issue is not reproducible by mikrotik staff.

Please describe your router configs and your thoughts about the situation which the issue may happen.

Bolle · Sun Feb 02, 2020 10:44 am

We have to find a common point. After reading this topic I am suspicious about two new points.

1-Do you have this issue on a device which is not monitored by Dude ?
Please also take a look at following topic viewtopic.php?f=2&t=114365

2-Do you have this issue on a device which is not using Queues?

I have a RB 4011 monitored by Dude, and a 1100DX4 wich is the Dude Server.
On both devices are only the default Interface Queues.
ROS 6.46.2
And both devices freezes

marcin21 · Sun Feb 02, 2020 4:15 pm

my 4011 has no pppoe-server, nor dynamic queues and yet still stalls.
although it does lot of nat, close to 1gbps, also BGP, and it is monitored by dude and snmp.

inteq · Sun Feb 02, 2020 6:46 pm

No queues or pppoe server/client.
Not even NAT. Just routing.
2nd time this one crashes and reboots

mt.png

Bolle · Mon Feb 03, 2020 5:07 am

today the RB4011 freezes after 11 hours.
No ROS management via Dude-server, no telnet, no Winbox.
but the WLAN is still running...
Dude recorded a cpu spike at 20% cpu usage, normal is 2% - 4%.

gtx Bolle

AlenDV · Mon Feb 03, 2020 5:09 pm

Hi everyone,

Our RB1100AHx4 running with ROS 6.44.6 yesterday froze in a remote Data Center, so we had to power recycle it, since then all works fine. The router is in use for several months, this seem to be the second or third time it's failing this way. Before RB1100AHx2 was in use with the same config - no issues in several years.

I disabled relatively newly added options, particularly standard queues (did not remove the rules, just disabled) and SNMP - most probable suspects. Will see if that helps.

P.S. Also updated ROS to 6.45.8.

allevot · Fri Feb 07, 2020 10:31 am

Hi!

Since one month ago, no MikroTik have bricked. I have 50 of them and they still with the same RouterOS 6.44.6.
The only thing that I changed on all of them its the gateway of every route. Before the change, the gateway had the interface and now they have the IP of the next hop.

I'm not sure this is the solution but it's fine at the moment.

Best regards,

seho · Fri Feb 07, 2020 2:36 pm

I also got a report from customer that is running two RB1100AHx4 as VPN concentrators (L2TP over IPSec+ IPSec IKEv2) in a VRRP Setup.

The VRRP Master device stopped accepting VPN connections, but unfortunately the VRRP has continued working - it stayed master device and the fail-over to the slave device wasn't performed. They restarted the master device and then everything has worked again.

SSH and Winbox access were not working was reported.

I want to build at least a workaround for them maybe by using a script, that at least the slave device will automatically take over. Maybe by increasing the VRRP interface priority of the slave device.

The problem looks to me like the posted Problem. But currently I got no idea how to detect that failure in a reliable way.

Can someone who was able to connect to the device using the serial port give a hint how to detect that the problem has occured?

From what I read:
1) SSH not working
2) Winbox not working

Is the device responding to ICMP pings?
Responding to ARP-Requests?

Every hint is appreciated.

It also thinkable for me to run the detection on a external application. So really every hint is welcome, not only the one which were detectable using scripting.

Thanks in advance.

Kind Regards,
Sebastian

netflow · Sat Feb 08, 2020 12:25 am

Well you just gave the solution:

The VRRP Master device stopped accepting VPN connections

So just try to open a VPN connection

oukltdtech · Mon Feb 17, 2020 2:55 pm

We had a brand new RB4011 stop responding last week after around 6 hours of putting it in. Winbox sat at "Logging In".

It had a couple of bridges on it we had set so we could monitor traffic going through the ports. Other than that, just an IP. It was updated to 6.46.3 before we put it in production.

The unit still bridged traffic and so didn't take our site down however it needed a hard reboot to fix.

Awaiting response from Miktorik Support.

allevot · Mon Feb 24, 2020 5:35 pm

I have configured Watchdog, but when MikroTik was bricked, the watchdog don't work.
Is very stressful, we have Rb4011 in differents sites...

Staff MikroTik please, can you find a solution for this problem?

Thanks a lot.

Maggiore81 · Mon Feb 24, 2020 5:44 pm

Schedule a nightly reboot each day.

allevot · Mon Feb 24, 2020 5:50 pm

There are services 24*7, i can't reboot it every day...

Filip92 · Wed Feb 26, 2020 12:19 am

I have the same problem.

Device: RB1100AHx4 Dude Edition
Configuration:
- running dude server with ~320 routerboards, ~390 services (ping)
- routing 11 static routes
- firewall 7 filter rules, 14 NAT rules, ~1k connections
- no queues
- running FTP server for auto mikrotik's backup
- 3 bridges and ~ 550 hosts in network

The device was work good, and around after half year this issue appeared...

I had a serial connection, but it was also suspended. In my console window i was see:

LOOPER: read_raw read failed: EOF
died with signal

I have Logging rules: critical, error and info - action: disk. In log file is nothing special, only:
Feb/25/2020 11:03:48 system,info,account user dude logged in from xx.xx.xx.247 via dude
Feb/25/2020 11:04:25 system,info,account user dude logged out from xx.xx.xx.247 via dude
----- BRICK --------
Feb/25/2020 05:28:12 system,error,critical router was rebooted without proper shutdown

Maggiore81 · Wed Feb 26, 2020 8:35 am

There are services 24*7, i can't reboot it every day...

I see. But if you prefer bricking... it is better a 30 sec outage for reboot in the night....

inteq · Wed Feb 26, 2020 9:05 am

There are services 24*7, i can't reboot it every day...

I see. But if you prefer bricking... it is better a 30 sec outage for reboot in the night....

We prefer a fix. Rebooting every night ain't one. I hope you don't do that to your users.

Maggiore81 · Wed Feb 26, 2020 9:12 am

I do. At 04 am ti 04.00.30 is not an issue...
I know that is not a fix but works. Eventually I replaced the 4011 with a ccr1009.

omidkosari · Thu Feb 27, 2020 10:15 am

We prefer a fix. Rebooting every night ain't one. I hope you don't do that to your users.

We all prefer to solve this problem. Our reports and complains to mikrotik staff did not solve the problem so we try to find a workaround.

seho · Thu Feb 27, 2020 12:49 pm

I got some supouts from our customer which has reported the problem with non responding device. One with a device running in non error state and one in error state.

Because of an NDA i'm currently not allowed to share the supout with MikroTik.

I had a look to the supout.rif locally on my pc using Mikrotik RIF decoder (https://github.com/farseeker/go-mikrotik-rif)

Unfortunately i'm pretty unexperienced interpreting all the stuff written out.

But theres something looking very suspicious to me is the "== SECTION profile", the device got a very high unclassified part in the cpu usage.
When I look at the same entries of device which is not in failure state the values are much lower.

== SECTION profile
NAME                    CPU        USAGE
total                                 0%

total                                 0%

total                                 0%

firewall                            1.2%
management                          3.1%
traffic-accou...                    2.6%
unclassified                       21.5%
total                              28.4%

NAME                    CPU        USAGE


management                1         0.5%
traffic-accou...          1           3%
unclassified              1        16.5%
cpu1                                 20%

firewall                  0           0%
management                0          10%
traffic-accou...          0          23%
unclassified              0       146.5%
cpu0                              179.5%
management                1         0.5%
traffic-accou...          1           3%
unclassified              1        16.5%
cpu1                                 20%
firewall                  2        25.5%
networking                2         0.5%
management                2        14.5%
unclassified              2           6%
cpu2                               46.5%
firewall                  3           3%
management                3           3%
unclassified              3           0%
cpu3                                  6%

Could someone who encountered the same problem, and have a supout of the device in failure, please have a look at a supout.rif in failure state? And report back if the section at their failure device has a similar high "unclassified" cpu usage.

Maybe this has nothing to do with the problem at all. I read that the unclassified could came from encryption.

allevot · Thu Feb 27, 2020 4:59 pm

One of my devices has an autouport.rif, it was created on 24-02-2020, the same day Mikrotik bricked.
The Supout.rif Reader shows this:

MEMORY
MemTotal:        1034864 kB
MemFree:          966468 kB
Buffers:               0 kB
Cached:            27688 kB
SwapCached:            0 kB
Active:            10596 kB
Inactive:          22656 kB
Active(anon):       5664 kB
Inactive(anon):       64 kB
Active(file):       4932 kB
Inactive(file):    22592 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          5636 kB
Mapped:             5508 kB
Shmem:               160 kB
Slab:              22976 kB
SReclaimable:       3192 kB
SUnreclaim:        19784 kB
KernelStack:         672 kB
PageTables:          412 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      517432 kB
Committed_AS:      11020 kB
VmallocTotal:    1024000 kB
VmallocUsed:        6696 kB
VmallocChunk:     973848 kB
CPU
Processor	: ARMv7 Processor rev 4 (v7l)
processor	: 0
BogoMIPS	: 2793.47

processor	: 1
BogoMIPS	: 2793.47

processor	: 2
BogoMIPS	: 2793.47

processor	: 3
BogoMIPS	: 2793.47

Features	: swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc0f
CPU revision	: 4

Hardware	: AnnapurnaLabs Alpine (Device Tree)
Revision	: 0000
Serial		: 0000000000000000

angriukas · Fri Feb 28, 2020 1:31 pm

10 days ago installed two RB4011iGS+ devices.
Firewall, 3 bridges, CAPsMAN with two SSID's, EoIP between both devices, few queues.
No bricks, no reboots, no issues at all.
RoS and firmware: 6.45.8

fposavec · Fri Feb 28, 2020 1:52 pm

[XXX@YYY] > system resource print 
             uptime: 35w2d21h45m51s
            version: 6.44.3 (stable)
         build-time: Apr/23/2019 12:37:03
   factory-software: 6.43
        free-memory: 904.0MiB
       total-memory: 1024.0MiB
                cpu: ARMv7
          cpu-count: 4
      cpu-frequency: 1400MHz
           cpu-load: 9%
     free-hdd-space: 418.4MiB
    total-hdd-space: 512.3MiB
  architecture-name: arm
         board-name: RB4011iGS+
           platform: MikroTik

no issue at all.
BGP routing, NAT, DHCP...

Overide · Fri Feb 28, 2020 5:49 pm

We've been dealing with this issue since we began deploying 4011's

We use virtual bridges, pppoe, DCHP and ospf. not mush else. this issue has never occurred for us on any other hardware platform.

StubArea51 · Fri Feb 28, 2020 6:09 pm

We've run into this with a few clients that run the 4011 and have used the following as a workaround while we work on a ticket with MikroTik. It seems to be more stable in the few we have tested.

Set the CPU frequency to 1200mhz

system routerboard settings set cpu-frequency=1200

ingdaka · Fri Feb 28, 2020 8:13 pm

I have 3 of them and I've not faced this problem never! Uptime more than 1 year running without any problem with normal 700Mbps to 1G of traffic!

anschluss · Mon Mar 02, 2020 4:09 pm

We've run into this with a few clients that run the 4011 and have used the following as a workaround while we work on a ticket with MikroTik. It seems to be more stable in the few we have tested.

Set the CPU frequency to 1200mhz
Code: Select all
system routerboard settings set cpu-frequency=1200

Thanks very much for this helpful hint! The RB4011 has been acting up along the lines described in this thread, with intervals of running seemingly normal for weeks, then days, then hours. Until I downclocked the CPU, the device started jettisoning all routing functions even after just a minute's connection, such as a phone call, a VPN or remote desktop connection.

cpbruton · Mon Mar 02, 2020 10:10 pm

I have encountered this issue with two different RB1100AHx4s. Both routers became inaccessible after a few weeks deployed. Forwarding and OSPF continued to work fine so we had no interruption in our network traffic, but could not log into the routers via web, ssh, mac-telnet, or direct console connection. A physical power cycle restored access.

Both routers have watchdog enabled but only one of them generated a partial autosupout.rif file, with the following message at the top:

--- nv::message --------
died with signal 6 on Sun Feb 23 15:02:12 2020

Both were running 6.45.8 (long-term). I downgraded to 6.44.6 before finding this thread, but it sounds like people have encountered this on older versions too.

I will try the suggestion to reduce CPU frequency to 1200MHz.

aoakeley · Tue Mar 03, 2020 4:03 pm

After obtaining supout.rif files by console cable, MikroTik have organised warranty replacement for the two 1100 units exhibiting this issue.

And yet another 1100AHX4 with this issue.
The previous 2 units which were replaced under warranty have not had a recurrence of the issue.

omidkosari · Thu Apr 02, 2020 6:40 pm

Hello

I see that your router did a reboot because of Kernel Watchdog.

For debugging you should turn off watchdog and test again.
"/system watchdog set watchdog-timer=no"

Connect now to your router over serial console, make sure that you have accessed RouterOS command line interface and leave console running.

Now router either:

1) might be stuck (freeze). If that happens, then you have to generate a supout file on the router through a serial console. Now you can reboot the device.

2) might be stuck (freeze) and become unavailable over a serial console. If that happens, then reboot the router and then generate a supout file.

3) will reboot. If that happens, then after reboot manually generate a supout file.

Send supout file and full serial console output (within text file) to us for investigation.

Before that, I suggest that you upgrade to the latest "stable" version (if there is an actual bug, then it might be already fixed).

Best Regards

This is a reply from mikrotik support. Has anyone done this ?

aoakeley · Fri Apr 03, 2020 10:00 am

This is a reply from mikrotik support. Has anyone done this ?

Yes. see my previous posts

jebz · Fri Apr 03, 2020 2:36 pm

And yet another 1100AHX4 with this issue.
The previous 2 units which were replaced under warranty have not had a recurrence of the issue.

.
Have you had any feedback on what the fault was?

aoakeley · Mon Apr 06, 2020 11:05 am

Have you had any feedback on what the fault was?

.
No. No feedback. Have replaced 3 under warranty now.

And I have one of the ones that was previously replaced, and had been running fine since end of Jan now inaccessible again. So either there is a systemic issue in this model or there is something in my config screwing them up. I am really not doing anything complex though, they are all just NAT routers for branch offices. I have hundreds of routers with very similar config and the only ones I am seeing issues with are the 1100AHx4 Models.

See screenshot of as many authentication attempts as I could get on one screen, yet DNS forwarding is still working as are general routing duties.

jebz · Mon Apr 06, 2020 3:40 pm

See screenshot of as many authentication attempts as I could get on one screen, yet DNS forwarding is still working as are general routing duties.

I've experienced issues with connecting with Winbox on occasions. In a few cases Webfig was still available so I disabled Winbox access and re-enable Winbox which restored Winbox access. On one occasion neither was accessible although routing was still operational. This required a reboot to regain access.

CheshirCa · Thu Apr 09, 2020 2:44 am

That night, one 1100AHx4 hung. Winbox stuck at "logging in". Webfig - cannot authenticate, wrong user or password.
Work as PPTP VPN server and DUDE server

akarag · Sun Apr 12, 2020 10:48 am

SSH and Winbox access were not working was reported.

I want to build at least a workaround for them maybe by using a script, that at least the slave device will automatically take over. Maybe by increasing the VRRP interface priority of the slave device.

The problem looks to me like the posted Problem. But currently I got no idea how to detect that failure in a reliable way.

Can someone who was able to connect to the device using the serial port give a hint how to detect that the problem has occured?

From what I read:
1) SSH not working
2) Winbox not working

Is the device responding to ICMP pings?
Responding to ARP-Requests?

Every hint is appreciated.

It also thinkable for me to run the detection on a external application. So really every hint is welcome, not only the one which were detectable using scripting.

Thanks in advance.

Kind Regards,
Sebastian

Hello, I would like to add my case as well regarding the well known problem with RB1100. We have several RB1100 and we have seen that 2 of them have similar behavior as the one described in this thread. Regarding the behavior, in every case the routing continued to work without any downtime, however the SNMP, winbox access, SSH access, even MAC based access was not possible, Regarding the access, seems that there is a common process that controls the access and this process failed. However, i am not sure regarding the SNMP if has some common points with the login process.
Currently we are at the latest version 6.46.5.Some of these routers support critical systems in a vrrp configuration while others support backbone. We depend on Mikrotik to deal with this annoying issue. So far we have confirmed that unclassified processes in Tool/profile have increased before the random brick of the device.

If the problem persists, would you recommend another Mikrotik device, meybe in the series of CCRs ? Are there any reports of the same problem in CCR series?

aoakeley · Thu Apr 16, 2020 11:42 am

Currently we are at the latest version 6.46.5.Some of these routers support critical systems in a vrrp configuration while others support backbone. We depend on Mikrotik to deal with this annoying issue. So far we have confirmed that unclassified processes in Tool/profile have increased before the random brick of the device.

That's interesting. I will keep my eyes open for that. How many processes are you talking about before it crashes? I have just checked a few different routers (CCR and MIPSBE and 1100 and CHR) and they all have a handful of unclassified processes.

If the problem persists, would you recommend another Mikrotik device, meybe in the series of CCRs ? Are there any reports of the same problem in CCR series?

I have not seen this issue on any other Mikrotik device.

akarag · Fri Apr 24, 2020 9:51 am

Ooops it did it again!

Once again one of our main RB1100 routers that support critical business functions, was bricked in terms of access to winbox. We cannot login to the router, MAC discovery is not working. However, routing is working. What we observed is that SNMP is not working, however netflow information is transmitted to the NMS.

It is getting really annoying issue for us.

Is there any fix for this issue for RB1100AHx4 or should we consider move to another appliance or vendor?

As soon as I have access to winbox, I will create a supout and send it to mikrotik for further investigation.However, this will be accomplished after a router restart through power disconnection

inteq · Fri Apr 24, 2020 8:36 pm

And again on a RB4011iGS+RM

crash.png

nje431 · Sat Apr 25, 2020 1:41 am

We are having the same problems as the OP. I got the same response from support that it's failed hardware. We bought 50 of the RB1100AHx4 and have deployed about 13 so far with 5 units each having done a management lockout. So I'm having a hard time believing a hardware failure. It's a pretty dismal failure rate if it is. Thank goodness for managed PDUs.

I'm trying different configuration changes, but so far nothing good to report.

From what I gather here, it doesn't sound like there is a magic software version yet. The one post I saw that piqued my interest was reducing the clock rate. For those that have tried it, what were the results?

Thanks

anschluss · Sat Apr 25, 2020 4:26 am

From what I gather here, it doesn't sound like there is a magic software version yet. The one post I saw that piqued my interest was reducing the clock rate. For those that have tried it, what were the results?

No more problems since reducing the CPU freq to 1200 Mhz; running for two months now.

nje431 · Sat Apr 25, 2020 5:18 pm

No more problems since reducing the CPU freq to 1200 Mhz; running for two months now.

That sounds promising. Thanks! I just hope I didn't jinx you

Overide · Wed Apr 29, 2020 12:04 am

1200mhz tested and has not fixed the issue for us. So far no clue what triggers the issue

msatter · Wed Apr 29, 2020 12:45 am

Looking for a long time at the 4011 and I afraid that it will stay with looking at and not buying.

thsun · Wed Apr 29, 2020 4:34 am

I have many of RB4011 and RB1100 AHx4, some of them encountered the same issue in this week. No matter use 6.46, 6.46.5 and 6.45.8.
Currently, I scheduled an on/off snmp every 6 hours and also set cpu to 1200Mhz. Hope it help.

/system schedule
add interval=6h name=ResetSNMP on-event="{\r\
\n/snmp\r\
\nset enabled=no;\r\
\n:delay 3s;\r\
\nset enabled=yes;\r\
\n:log warning \"Reset SNMP to prevent ROS encounter timeout while waiting program XX\"\r\
\n}" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=apr/28/2020 start-time=09:20:04

wojo · Wed Apr 29, 2020 7:15 am

This concerns me but running a RB4011iGS+ right now at stock frequency with no issue for months under Gigabit load. Will be watching this thread for updates.

sarada · Wed Apr 29, 2020 9:49 am

Hi

Maybe the NTP package is not enabled?

For me, too, due to the ntp package, the ARM architecture cAP ac was restarted randomly.
Once I turned off the ntp package, it has been stable ever since.

Link that can be resolved: viewtopic.php?t=153988

inteq · Fri May 08, 2020 6:53 am

Setup Dude to monitor CPU on all 4011s
So far only two have issues but tired of this.
Starting to replace all 4011. Not worth the trouble.

cpu.png

maloi3390 · Fri May 08, 2020 9:48 am

Hello everyone! I have two RB1100DX4 and both devices are brick every week, max uptime was 30 days. Support says to send device to warranty, but I don't believe in fix this problem...

Maggiore81 · Fri May 08, 2020 10:22 am

As I told you, I just scheduled a nightly reboot.
I cant believe that there are people that cant reboot the router nightly till the fix is out.
I understand that is not a solution but it is a full working fix.
today went out the new longterm!

nmt1900 · Fri May 08, 2020 11:08 am

Now I understand why I did not end up finding this thread before.

I am having same kind of issues with 2 devices - one is RB1100x4 and another is RB1100Dx4 - and these are only ones plagued with problems because these are only ones updated to 6.45.8 from 6.44.6 (only long-term releases in use for customer locations, but even this strategy is backfiring sometimes).

Both suffer from "unclassified" processes taking over one CPU core and sometimes loss of all static routing thus no remote access except via IPsec tunnel before its' death.

It is hard to believe this being strictly hardware issue if it appeared only after upgrade to 6.45.x version - but then again my issue might be something else - only common denominators being 6.45.x version and 1100x4 - which shares same CPU with 4011 series. Go figure...

Filip92 · Fri May 08, 2020 11:39 am

I have RB1100AHx4 Dude Edition with Factory firmware 6.38.4, currently working on version 6.42.12 (long-term). This issue began after a year of impeccable work... Maybe this CPU has a special feature that breaks under warranty.

PS. My device is running from 14 days with 1200MHz and still working

maloi3390 · Fri May 08, 2020 2:36 pm

At now I try to replace 1100Dx4 to CHR because work with this issue 24/7 impossible. I want to be change all service and users to CHR and than try to test 1100Dx4 with RoS7beta5. I believe that this broblem with the brick device fix in future in new RoS version.

nje431 · Mon May 25, 2020 9:48 pm

I may have found something. While it's a bit too early to know for sure, the fix is simple enough and safe to try.

First, what didn't work. I tried dropping the CPU clock, but that didn't work for me. Then I disabled the SNTP client, but that just changed how the failure presented itself from random management lockouts to random reboots.

Last weekend, following up on a tip in another thread, I started testing Ethernet speed/duplex settings on the 5 routers that have given me the most trouble. I had set a few to a mix of fixed 100/full and auto 1000/full only ports; some to auto 100/full only and auto 1000/full only ports; and one to a mix of fixed 100/full and default autonegotiate settings. Both the first 2 groups have been stable over the last week, while the 3rd had a random reboot.

Yesterday, while researching the Ethernet chipset for these routers, I came across the manufacturer's sales sheet. It explicitly states that 1000/half is not supported, yet the defaults in the ROS autonegotiate settings is to enable advertising that combo. Since the chipset doesn't support that, it's an unnecessary setting. I went through all 28 of our deployed 1100's yesterday evening and turned 1000/half off on all ports. This morning when checking the logs, the system was unusually stable overnight. Not even any of the EoIP drop-outs that I sometimes see.

Anyway, I'd be curious to see what others results are when they disable 1000m/half.

Cheers

msatter · Mon May 25, 2020 10:19 pm

If you look in the status tab you will see that 1000 half is not advertised by the 4011. However it is a test worth to see if also disabling in the settings brings a improvement or a solution.

nje431 · Mon May 25, 2020 10:32 pm

Yes, I noticed that. My suspicion is that ROS is asking the chip to do something it's not capable of, and creating an out of bounds condition. That in turn is causing unexpected behaviour. That's my current working theory anyway.

jebz · Tue May 26, 2020 2:46 am

I have RB1100AHx4 Dude Edition with Factory firmware 6.38.4, currently working on version 6.42.12 (long-term). This issue began after a year of impeccable work... Maybe this CPU has a special feature that breaks under warranty.

Open a case with Mikrotik and send a Supout.rif . It's best to have it fixed under warranty.

nje431 · Tue May 26, 2020 3:09 pm

Actually, I did. A couple times. That answer is always the same. You have bad hardware, return it to the seller for warranty (not Mikrotik, like most manufacturers would do). We couldn't get the Amazon seller to even respond to one with a bad power supply out of the box. Plus with so many failing this way, that would indicate a real quality control problem. No, I suspect this problem looks like a hardware failure because the hardware is being ask to do what it's not designed to do. But that's just a guess at this point.

marckan · Mon Jun 01, 2020 9:44 pm

And again on a RB4011iGS+RM
crash.png

Hi,

What should i configure to get result as above? i mean automatically reboot after crash

regards,
M

inteq · Tue Jun 02, 2020 12:50 pm

And again on a RB4011iGS+RM
crash.png
Hi,

What should i configure to get result as above? i mean automatically reboot after crash

regards,
M

Not 100% sure, but I guess that is handled by Watchdog under System.

Meins · Tue Jun 02, 2020 1:43 pm

Same isuse here. I also has two router that stop working from time to time. One about one every month, one crash for 3-4 days.

bash777 · Fri Jun 05, 2020 9:37 pm

I have RB1100AHx4 Dude Edition with Factory firmware 6.38.4, currently working on version 6.42.12 (long-term). This issue began after a year of impeccable work... Maybe this CPU has a special feature that breaks under warranty.

PS. My device is running from 14 days with 1200MHz and still working

I have RB1100AHx4 Dude same issue
different version
%(

jebz · Fri Jun 12, 2020 5:03 am

We are having the same problems as the OP. I got the same response from support that it's failed hardware. We bought 50 of the RB1100AHx4 and have deployed about 13 so far with 5 units each having done a management lockout.

.
Have you returned any failed RB1100AHx4 that Mikrotik have diagnosed as failed hardware? Did Mikrotik repair the RB1100AHx4 and provide a cause or component identity?

nje431 · Fri Jun 12, 2020 6:51 am

No we haven't.
A) Because we couldn't get the seller to even respond to an out of the box power supply failure. And Mikrotik won't handle the warranty directly.
B) I really don't believe it's a hardware problem.

Actually, we've deployed a good majority of them, and with some configuration changes, we've got them pretty stable. In fact it's been a couple months without a single management lock out, where we had about one a week with just a handful deployed before. I attribute that to (mostly) disabling the SNTP client. I say mostly because we use the scheduler to enable it for 3 minutes at startup, and then for 20 seconds, once a day.

After the SNTP change, the random management lock outs were replaced by random reboots. Disabling 1000/half under ethernet auto-negotiation seemed to greatly reduce the reboots, although that may be a placebo effect. There hasn't been enough time to be sure.
As I mentioned before, the ethernet chips in these units don't support 1000/half, so no capabilities lost. I ask support about that, but they said the autosupout files didn't indicate an ethernet switch chip problem. When I ask what it did indicate, they said they couldn't tell for sure. Figure that one out.....

I'm also experimenting with a 4am reboot that's triggered when there has been a "reboot without proper shutdown ". My experience tells me that Mikrotiks sometimes aren't stable after an improper reboot. Especially power cycle reboots, and also software updates without an additional clean reboot.

Anyway, that's where we're at for the moment.

Cheers

pamela · Tue Jun 16, 2020 2:00 am

I have about 5 pieces 1100ahx4 and 50 pieces of 4011 on our network, and failure rate is like 90%. After few days or weeks, they just brick. I am very pissed off that mikrotik is pretty much ignoring this issue. we just stopped buying this piece of s...t because sending technicians repeatedly to reboot this junk already cost us a lot of money and time. Mikrotik wake up

jebz · Wed Jun 17, 2020 1:15 am

After few days or weeks, they just brick. I am very pissed off that mikrotik is pretty much ignoring this issue.

.
Please send Mikrotik supout and return them for repair if Mikrotik diagnose the issue as hardware issue. The more reports and investigations the better it will be long term.

inteq · Wed Jun 17, 2020 3:00 am

After few days or weeks, they just brick. I am very pissed off that mikrotik is pretty much ignoring this issue.
.
Please send Mikrotik supout and return them for repair if Mikrotik diagnose the issue as hardware issue. The more reports and investigations the better it will be long term.

Sending supout is in vain. They will just reply with a standard "Connect a serial cable to this device, open serial console and make sure that you have successfully connected to RouterOS CLI.
Now leave device running and console opened until the problem appears again. After the problem appears, copy serial console output to a text file."
My opinion: non existent quality control to lower manufacturing costs. At least I hope so, because if they pay for QC and still have so many defects, that is wasted money.

jebz · Wed Jun 17, 2020 10:53 am

Sending supout is in vain. They will just reply with a standard "Connect a serial cable to this device, open serial console and make sure that you have successfully connected to RouterOS CLI.

.
No I don't think sending supout in this case is in vain. More feedback will result in improvement.

inteq · Wed Jun 17, 2020 11:18 am

Sending supout is in vain. They will just reply with a standard "Connect a serial cable to this device, open serial console and make sure that you have successfully connected to RouterOS CLI.
.
No I don't think sending supout in this case is in vain. More feedback will result in improvement.

If that feedback is accepted, of course.
Try sending a supout if you have this problem and see Mikrotik's response. Then tell me if you still think sending it results in any improvement for you.

jebz · Wed Jun 17, 2020 2:16 pm

If that feedback is accepted, of course.
Try sending a supout if you have this problem and see Mikrotik's response. Then tell me if you still think sending it results in any improvement for you.

.
I've been though the process and currently waiting on longer term evaluation. At this point it has been an improvement and potentially resolved.

Maggiore81 · Wed Jun 17, 2020 6:49 pm

I have permanently solved the issue with a nightly reboot.
I cannot imagine a network that cannot reboot in the middle of the night for a couple of seconds (less than 20)
MT is not resolving the issue, IMHO.
thank you

jacuinde · Thu Jun 18, 2020 10:20 am

Got another 4011 on 6.45.8 that died out of the sudden and created a support file with the following

--- nv::message --------
died with signal 6 on Tue Jun 16 14:33:48 2020

Used as a VPN server mostly and less than 50mbps of bandwidth, no mpls or mangling, NAT stuff too

inteq · Sat Jun 20, 2020 7:34 pm

And a new RB1100AHx4 started crashing. 2nd time today.

crash.png

Raice · Sun Jun 21, 2020 6:58 pm

I'm not sure I've got the same problem, but one guy from Telegram group suggested this.
I have RB1100AHx4 Dude Edition, ROS 6.47
3 WAN connections, NAT, some IPSec, DHCP Server (~2000 DHCP clients), SNMP, BGP (private ASN). Today one of my technicians noted that DNS stopped working. I checked - 1 core is 100%, 30% - unclassified, SSH is fine, Winbox is fine. All services except DNS Server is fine.

What do you think? Is it the same problem?

nje431 · Sun Jun 21, 2020 7:30 pm

Who knows? But the forum thread is focused on management lockouts where Winbox, SSH, Telnet, and SNMP, all stop responding, but revenue traffic continues to flow. Your symptoms seem to be counter to this. But in my dealing with Mikrotik support, I get the impression that they have no idea what the problem is, as it's so random. Your symptoms may just be a different expression of the same problem.

I would suggest you contact support, as it may be something different that they can easily diagnose.

Cheers

thsun · Mon Jun 29, 2020 10:59 am

No we haven't.
A) Because we couldn't get the seller to even respond to an out of the box power supply failure. And Mikrotik won't handle the warranty directly.
B) I really don't believe it's a hardware problem.

Totally agree your opinion, after downgrade routeros to 6.43.7, the problem seems disappear.

nje431 · Mon Jun 29, 2020 3:08 pm

Rolling back ROS has been my latest test. I've got a few on 6.43.16 and so far, so good. But it's too early to celebrate just yet. My other experiments with scheduled SNTP and disabling 1000/half on 6.45.9 have settled things down considerably, but hasn't been 100% effective. So time will tell.

Cha0s · Mon Jun 29, 2020 3:54 pm

I've got the same issue with multiple RB4011's.
Completely locked out, while still forwarding traffic.

My common denominator on the boards that lock me out, is the use of IPsec.
More specifically the devices that crash are VPN concentrators that allow IPsec sessions from anyone in the world.

One of the offending devices has crashed even without being in use. It was configured, powered on, accessible from the internet (ONLY UDP 500, 4500 + ipsec-esp, ipsec-ah) but not put in production yet.
And yet it crashed while sitting there, idle, for months.

But on boards with or without IPsec that are not accessible from everyone I haven't had any lockouts.

I too doubt this is a hardware defect.

But that's easily disputed:
Has anyone gotten their board replaced? And if so, has that solved the problem?

rmatthew34 · Mon Jun 29, 2020 3:58 pm

Out of curiosity, how many of you are running OSPF on those routers? I have seen a few people experience the issue and all have that configured. We are experiencing it on a single 4011 and while I thought I had it completely disabled, I did find some legacy config still enabled. Will test and see.

nje431 · Mon Jun 29, 2020 10:42 pm

No OSPF

But:
EoIP w/AES128 GCM
VLANs
Static routes

aoakeley · Tue Jun 30, 2020 9:34 am

I have received through the grapevine what is the apparent cause of this. Not sure if I am allowed to share it though, and don't want to start gossip if it turns out not to be true.

Maybe Normis or someone from Mikrotik can come in here and make an official statement?

I'll give it a week or so and then if nothing official from Mikrotik I'll have a think about spilling the beans.

aoakeley · Tue Jun 30, 2020 9:36 am

I too doubt this is a hardware defect.
But that's easily disputed:
Has anyone gotten their board replaced? And if so, has that solved the problem?

Yes, Replaced
Yes, Solved the problem
Also, Sometimes replaced multiple times before finally fixed.

hamster922 · Thu Jul 09, 2020 9:30 am

We had the same problem with several RB4011 and we tried many tips, but randomly the "brick" came again.

The only thing the helped (now uptime over 80 days without any problem) was to set the cpu to 1,2 GHz...

Running 6.46.4

Maybe that helps others

jacuinde · Sun Jul 19, 2020 9:46 pm

FW 6.47, all ports without half duplex and clock speed at 1GHZ and still it happened, were going to rollback to RB3011 because of this issue and stop buying RB4011

ubikrotik · Mon Jul 20, 2020 2:53 am

Hi, we have about 100 rb4011 deployed in our network.

THIS PROBLEM suck a lot you now that?

MIKROTIk team do you have a fix for us.?

Kampfwurst · Mon Jul 20, 2020 10:28 pm

i have 3 RB1100Ahx4.

All 3 have problems. One had a load of 50% (2 Cores stuck at 100%).
One has a winbox lockout. Only reboot helped.
One is restarting every 4-5 days.

mikrotik should stop to produce new devices. They should get the software fixed

msatter · Tue Jul 21, 2020 12:39 am

Any thoughts on this question?

viewtopic.php?f=3&t=151046#p806892

nje431 · Tue Jul 21, 2020 3:55 am

None of our problem child 1100x4 routers are currently running L2TP in any flavor.

BTW - So far, I'm seeing promising results rolling back the 1100x4's to 6.43.16. I've been running 23 routers at 6.45.9 and 23 routers at 6.43.16 for the last couple weeks. Only the 6.45.9 routers have farbled so far ( 2 of them about a week ago). I rolled those 2 back to 6.43.16, and they've been stable since. Still not ready to declare victory yet though.

Kampfwurst · Tue Jul 21, 2020 9:46 am

some of my RB1100AHx4 running a L2TP server with IPsec. I have now disabled the SNTP. Maybe it helps. Now running the 6.45.9. Its for me unbelivible why mikrotik brings out a long term version "version is the most stable release without new features, just most important fixes." and this version has so many problems with stability

ivn · Wed Jul 29, 2020 11:16 pm

We have the same problem on 2 of 4 RB1100ahx4. It occurs only on 2 routers which have l2tp server with ipsec.
Also we had several times when only l2tp server stopped working so I could login to the router, disable/enable l2tp server and everything continued to work.
It seems to me that those two problems are connected somehow.
Or course I have sent supout file to Mikrotik support several times, but just got a recommendation to RMA our device.

Kampfwurst · Thu Jul 30, 2020 8:52 am

i tested at the moment with 2 RB1100ahx4 the Version 6.45.9 and i disabled the sntp and enable it once a day for 90sec for sync. since 10 days no problem with the RB1100ahx4

i got the same answer from the support. I should RAM the device. The best for the support team would be a bot to send automatically this answer. I newer got any other solution from the support. very poor

inteq · Thu Jul 30, 2020 10:07 am

And another RB4011 with 100% CPU usage on one core without traffic, just waiting to crash.
And another useless response from support with "netinstall blah blah blah" after I sent them supout.rif
I guess up time and letting the client actually use the devices they pay for is an unknown for Mikrotik, for quite a while now.

faxxe · Thu Jul 30, 2020 8:00 pm

Even if they don't believe it yet, it will take revenge

-faxxe

Filip92 · Thu Jul 30, 2020 10:20 pm

Wow 41 days uptime without crash... @1200MHz

maloi3390 · Fri Jul 31, 2020 12:11 pm

Dear team of Mikrotik! When we can see fix for this problem? If you can't to solve this, you must stop to sale this products! My RB1100Dx4 was able to work 5 days after last reboot.

jebz · Fri Jul 31, 2020 2:13 pm

It's over 14,000km to return my device to the factory. It's still not resolved. Then 14,000km back. I have a RB1100AHx4Dude edition without Dude enabled to test for this issue. This sucks!

wojo · Fri Jul 31, 2020 3:48 pm

Well, it happened again for me on my RB4011iGS+. Woke up to a nearly frozen router, been at 100% dropping nearly all traffic and pings since 2am. Had to unplug to get it back up and running.

About to call it quits on the RB4011.

xparanoik · Fri Jul 31, 2020 6:13 pm

I have a new RB1100AHx4 purchased back in late May or early June that has been running non stop since I racked in the datacenter (about 60 days or so).

It has the following features enabled:

BGP
One Ethernet bridge
SNTP client
SNMP
Firewall rules on input chain
Winbox/SSH
Daily backup script

The router isn't heavily loaded in terms of traffic just yet. Not sure if I just been lucky with this unit or I happen to not use a setting that causes these issues. Hopefully this helps

inteq · Fri Aug 14, 2020 4:23 pm

A new RB4011, bought last August, just died. No link on any port. Reset not working. Just keeping track here

later edit:
How I unbricked this RB4011

The RB had no link on any port.
Reset to factory defaults did nothing.
Could not put it into netinstall mode.

So I took it apart to check for visible damage on the board. Nothing detected.
Tried to put it in netinstall again with the board outside the case and success! The RB had link and was visible inside netinstall.
Not wanting to loose the settings, I thought I would give it a shoot, reboot and everything will be fine. No such luck.
Only after about 25-30 retries to put the RB into netinstall again I was successful, meaning only on the last try I had link on port 10 and I could netinstall.
Everything seems OK now.
Might help some one. Just keep trying to put it into netinstall multiple times even if you have no link.

jonthorpe · Tue Aug 25, 2020 2:39 am

Mikrotik - any word on how to resolve this?

I have an RB4011 which needs a hard reboot every two months due to loss of management (and a few other functions, but flow seems ok) for the reasons described here. It's doing:
BGP with BFD
PPPoE Client
VRRP
L2TP/IPSEC

There's no NTP and SNMP is disabled. I have just:
1. Set the CPU clock to 1200
2. Run /interface ethernet set [find] advertise=10M-half,10M-full,100M-half,100M-full,1000M-full

We'll see if that helps, but who knows.

I would be less concerned if the watchdog timer rebooted the router when this condition occurs, however it doesn't. Given how long people have reported this issue without resolution or even the means to troubleshoot further, I'm losing faith.

I understand leaving a serial console permanently attached to the router might help, however I'm not in a position to do this.

Tue Aug 25, 2020 8:18 am

I am considering to give a chance to rb4011 and thinking about to buy some of them. When reading this and the second thread about the freezing of rb4011, I would like to know if it is really hardware problem or not. If yes then which serial number interval is affected and which for sure is not.

I am not willing to netinstall each month, schedule daily reboot, lower the cpu frequency or cripple the ethernet port negotiation. I am willing to use vpn tunnels, vlans, SNMP, WiFi and all other common things like usually. I need the devices to work and not to play the RMA games with the distributors.

Normis, please give the clear official statement to the world about the rb4011. I am not going to buy any rb4011 without this.

Thank you.

Tue Aug 25, 2020 3:00 pm

RB1100AHx4
serving circa 100 person at company branch = Internet + VPNs to main site for NAS access = 110 days uptime

RB4011
circa 50 persons + VPN for homeoffice + VPN to branch + normal routing, firewoling etc = 36 days uptime as we have had to power down our company due to power maintenance. Previously up time was over 120 days

nje431 · Tue Aug 25, 2020 7:01 pm

Approximately (50) 1100AHx4 routers deployed now, and after 30 days not so much as a hiccup. The reason? Rolling back to 6.43.16. That is not an option on new routers, as I discovered, since you can't roll back beyond the factory installed version, and the last one we bought was at 6.45.x.

One interesting occurrence that happened a month ago, while still running some sites on 6.45.9. We are running bridged/encrypted EoIP between 2 hub sites and remote sites. One of our hub sites at a DC had an ISP interruption lasting just a couple minutes at the most. At the same moment the interruption started, one of our remote sites locked up. If not related, it was an oddly timed coincidence.

While communicating with the Mikrotik support tech, asking about the possibility of rolling the new router back, I mentioned how the older software was stable and related the episode I just described. They seemed intrigued, but stated they still thought it was a hardware issue in their opinion. Previously when I had ask what hardware was failing, they said they didn't know, the failures were too random. How they could discount software problems, I don't know. It sure looks like software from my viewpoint. Or at a minimum, a software intolerance of a hardware hiccup. Reading back through everyone else's experience, it's hard to pin down a specific culprit though. While we don't have a lot of sites that don't connect to the hubs with encrypted EoIP, I'll note that the few sites that are just bridged microwave passthrough/routed drops, no encryption or EoIP, have never given us a problem.

Anyway, that's our experience so far.

P.S. has anyone tried v7 yet, and had any of these problems? Enquiring minds want to know.

43north · Sun Sep 13, 2020 8:04 pm

I have received through the grapevine what is the apparent cause of this. Not sure if I am allowed to share it though, and don't want to start gossip if it turns out not to be true.

Maybe Normis or someone from Mikrotik can come in here and make an official statement?

I'll give it a week or so and then if nothing official from Mikrotik I'll have a think about spilling the beans.

Well what do you know?

Filip92 · Mon Sep 14, 2020 10:58 pm

Hello everyone

my RB1100AHx4 Dude Edition with routing, bridge'ing and The Dude Server is still running from 88 days. ROS v.6.42.12 and CPU @ 1200MHz.

Hominidae · Tue Sep 15, 2020 5:57 pm

I just received a RB4011iGS+RM ... the extra sticker on the package said "RB4011iGS+RM - new" and under System - Routerboard, the dialoge states "Revision: r2".
Shipped with Firmware revision "6.45.9 stable"

...will monitor closely and report how this thing will perform its duties.

wojo · Tue Sep 15, 2020 6:03 pm

I just received a RB4011iGS+RM ... the extra sticker on the package said "RB4011iGS+RM - new" and under System - Routerboard, the dialoge states "Revision: r2".
Shipped with Firmware revision "6.45.9 stable"

...will monitor closely and report how this thing will perform its duties.

Be really interesting to see some internal pics. We could compare with rev1 and see what any visible differences could be.

Kampfwurst · Tue Sep 15, 2020 6:19 pm

I just received a RB4011iGS+RM ... the extra sticker on the package said "RB4011iGS+RM - new" and under System - Routerboard, the dialoge states "Revision: r2".
Shipped with Firmware revision "6.45.9 stable"

...will monitor closely and report how this thing will perform its duties.
Be really interesting to see some internal pics. We could compare with rev1 and see what any visible differences could be.

I would like to see if the have changed the hardware

Hominidae · Tue Sep 15, 2020 7:07 pm

I just received a RB4011iGS+RM ... the extra sticker on the package said "RB4011iGS+RM - new" and under System - Routerboard, the dialoge states "Revision: r2".
Be really interesting to see some internal pics. We could compare with rev1 and see what any visible differences could be.
I would like to see if the have changed the hardware

So, can you guys confirm that yours state "r1" when checking in "System -> Routerboard" ?
I checked some unboxing videos but couldn't see any of the presenters go there,
I must admit, I was getting worried, as I found this thread only after I did the purchase. .

wojo · Tue Sep 15, 2020 8:53 pm

I just received a RB4011iGS+RM ... the extra sticker on the package said "RB4011iGS+RM - new" and under System - Routerboard, the dialoge states "Revision: r2".
Be really interesting to see some internal pics. We could compare with rev1 and see what any visible differences could be.
I would like to see if the have changed the hardware
So, can you guys confirm that yours state "r1" when checking in "System -> Routerboard" ?
I checked some unboxing videos but couldn't see any of the presenters go there,
I must admit, I was getting worried, as I found this thread only after I did the purchase. .

Everything just says CCR1009-7G-1C-1S+ on mine, no mention of r1 or r2.

pacmanfan · Tue Sep 15, 2020 8:59 pm

No mention of revision on my RB4011; I purchased it about 9 months ago.

/system routerboard> print
       routerboard: yes
             model: RB4011iGS+
     serial-number: B8F40BE81E39
     firmware-type: al2
  factory-firmware: 6.44.5
  current-firmware: 6.47.2
  upgrade-firmware: 6.47.2

msatter · Tue Sep 15, 2020 9:41 pm

If there is no specific mention of the revision then you can assume that you have the first revision. Look also at the factory firmware number can be an indication but then you have to know the version that was shipped with the second revision.

Hominidae · Tue Sep 15, 2020 9:54 pm

...this is from my new device:

[admin@MT-RB4911] /system routerboard> print       
       routerboard: yes
             model: RB4011iGS+
          revision: r2
     serial-number: D4480C22538F
     firmware-type: al2
  factory-firmware: 6.45.9
  current-firmware: 6.45.9
  upgrade-firmware: 6.45.9

Edit: /system/resource states a different version of factory firmware/software, though:

/system resource> print   
             uptime: 4h26m34s
            version: 6.45.9 (long-term)
         build-time: Apr/30/2020 10:25:34
   factory-software: 6.44.6
        free-memory: 965.2MiB
       total-memory: 1024.0MiB
                cpu: ARMv7
          cpu-count: 4
      cpu-frequency: 1400MHz
           cpu-load: 0%
     free-hdd-space: 425.4MiB
    total-hdd-space: 512.3MiB
  architecture-name: arm
         board-name: RB4011iGS+
           platform: MikroTik

msatter · Wed Sep 16, 2020 12:03 am

The build-time refects the build-time of the software and not the hardware.

Hominidae · Wed Sep 16, 2020 12:07 am

...hmmm, maybe not important in this topic, but what is the difference between "factory-firmware" and "factory-software" ?

Nevertheless, my RB4011 carries the revision info in its hardware (routerboard) firmware/config.

Chupaka · Thu Sep 17, 2020 1:33 am

Software is RouterOS, firmware is... boot loader.

ilkogd · Fri Oct 02, 2020 11:03 am

Hello,

I'm planning to change the FreeBSD server working as a gateway, firewall, and VPN for a small company with RB4011 or RB1100. Reading this topic for months I'm wondering is this a good choice, because in these days it is very important for network infrastructure to be rock solid. So my question is what is the failure rate for these devices at the moment, is it very high as discussed in this topic. In this case, the better choice maybe is CCR1009 if there are no problems with this device, but it is double the price of RB4011.

Any opinion will be in help. Thanks.

msatter · Fri Oct 02, 2020 11:13 am

Ask before you buy if you will receive revision 2 of the device.

viewtopic.php?f=2&t=149062&p=820138#p817223

igkhv · Fri Oct 02, 2020 12:48 pm

I have two RB1100AHx4. One transferred to the latest LT firmware, the second to 7.1 beta 2. For two weeks now there are no freezes. We used to hang out every three days.

Hominidae · Sat Oct 03, 2020 11:14 pm

I just received a RB4011iGS+RM ... the extra sticker on the package said "RB4011iGS+RM - new" and under System - Routerboard, the dialoge states "Revision: r2".
Shipped with Firmware revision "6.45.9 stable"

...will monitor closely and report how this thing will perform its duties.

...update: the RB4011 is running fine, uptime now 15days....two WAN interfaces, 3x1GBit Access-Ports on the same switch (2nd switch empty) and a 1m SFP+ AOC to a CRS326-24G-2S+-RM....6 VLANs, CapsMan also enabled and running, no Dude ....no problems at all.
I am a happy camper ;-)

maloi3390 · Thu Oct 15, 2020 11:21 am

I just received a RB4011iGS+RM ... the extra sticker on the package said "RB4011iGS+RM - new" and under System - Routerboard, the dialoge states "Revision: r2".
Shipped with Firmware revision "6.45.9 stable"

...will monitor closely and report how this thing will perform its duties.
...update: the RB4011 is running fine, uptime now 15days....two WAN interfaces, 3x1GBit Access-Ports on the same switch (2nd switch empty) and a 1m SFP+ AOC to a CRS326-24G-2S+-RM....6 VLANs, CapsMan also enabled and running, no Dude ....no problems at all.
I am a happy camper ;-)

My congratulations! At now i've try last long-term.

si458 · Mon Oct 26, 2020 11:18 am

ill put my 2pence worth in if i may

im having the exact same issue with our RB1100AHx4 for over the past year,

it would run fine for a few weeks, then randomly lock up, only way of fixing it was to ask staff onsite to unplug/plug both power leads
(i colored the power leads to make it easier for them)

I would get SYSLOG errors galore about process crashing or missing,
i was then unable to access the unit via winbox,web,ssh, even raw console with serial lead!)
i also noticed no traffic was flowing through it either

the only thing i spotted that was WEIRD was any UDP traffic was working no problem ???
(our syslog server in datacenter was getting all the syslog events from the router no problems and we could access our CCTV as well using port forwarding via udp?)
it was only TCP and ICMP that was affected?

we have just ordered a replacement RB4011iGS+RM and hoping this will fix the issue of a replacement unit (we dont need the redundant psu as we have batteries and a generator on site lol)

tommytnt · Thu Nov 12, 2020 11:40 pm

Hello. I have three pieces RB1100AHx4, one RB4011, three RB1100AHx2, one RB3011, eight RB750Gr3, and few HAP AC^2. I can't say for sure about HAP AC^2, since its uptime has not exceeded 100 days for me yet. But other devices works perfectly, except RB1100AHx4 and RB4011. They freeze every ~100 days, and I cannot connect to them and all they do when they freeze is respond to pings. Only power reset can help. I tried set CPU frequency to 1200mHz, but nothing chnged. Also, I tried to leave the frozen router for a week. Nothing has changed, within a week he was responding to pings, I could not connect, and he did not record a graph of the load on the CPU and RAM (and any other graphs).
All devices are located in different countries and have approximately the same settings.

Here is my config (I only delete senstive data and some settings like queues, which definitely do not affect freezes, because without them it also freezes):

mikrotik_freezes.rsc

Hope this helps you find and fix the bug.

nje431 · Sat Nov 14, 2020 8:21 pm

Interesting.

TLDR: I have narrowed our issue down to software releases sometime following 6.43.16, but starting prior to 6.45.8, and using IPsec. Somewhere between these 2 releases is when Mikrotik changed the security package to require the DHCP package.

Originally we were running our network of ~50 routers using a wide area layer 2 with IPsec EoIP tunnels. After upgrading to 6.45.8 is when we started seeing the problem. Support said it was a hardware issue and to warranty replace the defective routers. But of course by that time they were out of warranty. Support also hadn't been able to say what hardware failed, which destroyed my confidence in their answer. That tells me they haven't figured it out yet. And yes, I submitted multiple tickets for different routers with the same response.

I noticed early on, that the problem seemed to only effect routers connected to the Internet and therefore running the tunnels. I tried changing the IPsec tunnels from AES to Blowfish, but it didn't seem to fix the problem. I tried various other things, but rolling back to 6.43.16 worked. But then I received a new router that couldn't roll back so I figured I should continue looking for a new workaround.

We were planning a change to a layer 3 routed network with OSPF anyway, so I moved everything back to 6.45.9 and changed the tunnels to IPsec/AES GRE. But within a week the problem returned. I then put half the network on unencrypted GRE, and the other half on Microsoft encrypted L2TP tunnels. Both solved the problem. So right now we are running the L2TP MPPE tunnels system-wide.

When I say the problem is resolved for us, I need to caveat that. I have seen the problem rarely, but everytime it's been within 24 hours of making a configuration change, and not having done a soft reboot. Once within about 2 minutes after logging out. And I have seen unexplained reboots on rare occasions.

In your case I didn't see IPsec listed, but I did notice OPVN AES, so maybe that? Or this may be reaching a bit, but maybe something in DHCP? I only recently (a month ago?), added a DHCP server to an open Ethernet port on each of our routers for the convenience of plugging in a service laptop, so it's rarely used. And we are not running any other DHCP. Everything else in the network is static, including ISP IPs.

Cheers

nje431 · Sat Nov 14, 2020 10:33 pm

Well, so much for that. I hadn't much more than hit submit, than I heard from one of our techs that one of the routers had stopped responding to SNMP overnight, so he did a remote power cycle to bring it back. This is making me all that much more suspicious of that DHCP package. Time for more testing :-)

si458 · Sat Nov 14, 2020 10:51 pm

Hi nje431

Out of curiosity, how do u power cycle them remotely?

We currently have one power port plugged into a ups and the other is plugged directly into the main power wall socket

If I could power cycle it remotely myself it would save so much time!?

Regards

Simon

jebz · Sun Nov 15, 2020 2:21 am

Well, so much for that. I hadn't much more than hit submit, than I heard from one of our techs that one of the routers had stopped responding to SNMP overnight, so he did a remote power cycle to bring it back. This is making me all that much more suspicious of that DHCP package. Time for more testing :-)

.
You need to create a Supout package file just after a failure and open a support case. As Mikrotik have said many times only they can determine the cause of each individual incident. The more that the RB1100AHx4 issue is officially reported with evidence the more attention it will get. All manner of packages have been blamed for this issue ie. ntp, ospf etc.

si458 · Sun Nov 15, 2020 9:46 am

Hi jebz
The problem you have is the device locks up!
So you can’t create a supout file!?
As you have no way of accessing the device to generate it!?
All you can do is power cycle the router and then generate a supout file but that’s not helpful as it’s then a working router in there eyes
Regards
Simon

jebz · Sun Nov 15, 2020 10:46 am

All you can do is power cycle the router and then generate a supout file but that’s not helpful as it’s then a working router in there eyes

.
There's still potentially good evidence captured after a power cycle. ie. there may be remaining settings that are from a previous configuration causing a compatibility clash. All gathered observations add to the pool that may make resolving the issue faster.

fr0zonza1 · Sun Nov 15, 2020 12:39 pm

We had this issue for months!
We started capturing pcaps 24/7 and caught massive storms of "RSTP" traffic flooding interfaces just before the lock ups occurred.
This traffic was infact RoMon packets.
We disabled RoMon on all routers and the issue stopped.

This is reproducible..... It is worse on devices running 10Gig interface speeds.
We would literally bring a new router online with 10G optics and we would have various 4011 and CCR routers reboot all at the same time across the network, Lockup or even interfaces would flap.
This is on a large fully routed WISP network with OSPF and PPPoE- NO bridging or large L2 domains anywhere.

psipek · Mon Nov 16, 2020 9:36 am

Hi guys,

I have two RB4011´s, while the first one is revision 1 and running flawlessly for months, the second is freshly installed (3 days ago) revision 2 and locked up yesterday. I´ve had to powercycle to make it work.

They both have two WAN connection, no IPSec, few NAT and filter rules. I use SNMP to monitor them with Zabbix. It is really no heavy load, around 30 PCs behind them, they work just as a gateway for small office network. Both running on 6.47.x

During the brick:
no internet connection on the site
Winbox ask me to turn legacy mode as the router does not accept encrypted connection
Winbox in legacy and SSH keeps "logging in" but no success. I tried to put wrong password but it does not even check it.
No device discovery to connect through MAC
No Winbox connection through MAC as I tried to put MAC manually
I still was able to remote connect to the Windows server on remote site via the other site which is connected with dedicated fibre cable between two cities, AD replication worked as usually between both sites.
Ping to the internal IP of Mikrotik was pretty normal from the internal LAN

After powercycle everything is smooth and running, I enabled watchdog but as I see in other posts, it will not help. I also applied the 1200 MHz trick, and will monitor the situation.

The reason why I am posting this is that board revision 2 does not mean you are safe from this issue. The fun part is that I installed this RB4011 in exchange to RB750 after years of duty with few KB of memory left and CPU almost all the time on 100% usage.

si458 · Mon Nov 16, 2020 11:55 am

Hi guys,

I have two RB4011´s, while the first one is revision 1 and running flawlessly for months, the second is freshly installed (3 days ago) revision 2 and locked up yesterday. I´ve had to powercycle to make it work.

They both have two WAN connection, no IPSec, few NAT and filter rules. I use SNMP to monitor them with Zabbix. It is really no heavy load, around 30 PCs behind them, they work just as a gateway for small office network. Both running on 6.47.x

During the brick:
no internet connection on the site
Winbox ask me to turn legacy mode as the router does not accept encrypted connection
Winbox in legacy and SSH keeps "logging in" but no success. I tried to put wrong password but it does not even check it.
No device discovery to connect through MAC
No Winbox connection through MAC as I tried to put MAC manually
I still was able to remote connect to the Windows server on remote site via the other site which is connected with dedicated fibre cable between two cities, AD replication worked as usually between both sites.
Ping to the internal IP of Mikrotik was pretty normal from the internal LAN

After powercycle everything is smooth and running, I enabled watchdog but as I see in other posts, it will not help. I also applied the 1200 MHz trick, and will monitor the situation.

The reason why I am posting this is that board revision 2 does not mean you are safe from this issue. The fun part is that I installed this RB4011 in exchange to RB750 after years of duty with few KB of memory left and CPU almost all the time on 100% usage.

i had a suggestion from one of our other tech guys who uses mikrotik
the watchdog doesnt appear to be doing its job correctly and rebooting the device when it locks up
he suggested i create a script that runs inside the schedular every say 15mins
all it does is ping an external ip (i spotted we couldnt ping external sites when it was locked up) and when it notices its down
it then creates a supout file then tries to reboot the router itself

anybody up for trying it?

P.S: you can actually use the netwatch tool to ping an external ip every so often with a timeout and then in there, create the supout file and reboot the device

sjoukes · Mon Nov 16, 2020 6:11 pm

We've got about 5 different RB4011's how have crashed in the past few months.
1 continued IP routing (but no DNS, capsman,winbox etc) the others stopped all functionality all together but you could ping them locally.
Some crashed after one or two months the most recent one crashed within one week after it was installed.

All setups use CAPSMAN
3 are dual WAN with failover
2 are single WAN
All setups use VPN tunnels
RoMoN is disabled on all devices
There are no services available via WAN except for (IP filtered) incoming VPN's.
We've set the CPU to 1200 MHz allready after the first crash.
It is present in multiple software versions including 6.47.7

A autosupout file is generated when the device stops working but it is 0kb in all cases.
When they've crashed we see the folowing behaviour.
All VPN tunnels are dropped.
The device remains pingable
Webservices don't respond but accept connections
Winbox does not responds but accepts connections
SSH does not responds but accepts connections.
CAPSMAN stops working
Graphs are not updated (there is no data while this is happening)
DNS/DHCP server etc all stops working.

The watchdog timer of the device is not working and the error is not recoverable
You can only get the device back again via a powercycle.

The autosupout was generated while the device was not working.
The normal supout was created after an engineer powercycled the device.

2020-11-10 - autosupout.PNG

nje431 · Tue Nov 17, 2020 3:53 pm

Hi nje431

Out of curiosity, how do u power cycle them remotely?

We currently have one power port plugged into a ups and the other is plugged directly into the main power wall socket

If I could power cycle it remotely myself it would save so much time!?

Regards

Simon

We've used several devices. Most sites have the WTI managed PDU with 8 controlled ports, or the Blackbox rebranded WTI unit. A few sites use the 2 port Webswitch. Some networked UPS's can reboot a port as well. Since the router always continues to pass traffic, this works. If it didn't, all the managed PDUs have a "Ping, no Answer" reboot option.

The Webswitch is economical, but my biggest problem with it, is if you want to change IP settings, it requires a power cycle reboot, so you can't do that remotely.

As far as a support case, I've given up on those guys after multiple cases. All I ever hear is "It's bad hardware". Even when I point out how older ROS didn't have near the problem, they stick to their story. They're like a broken record. And when ask what hardware is broken, they don't know. So if you want to open a support case, be my guest.

jebz · Wed Nov 18, 2020 1:51 pm

As far as a support case, I've given up on those guys after multiple cases. All I ever hear is "It's bad hardware". Even when I point out how older ROS didn't have near the problem, they stick to their story. They're like a broken record. And when ask what hardware is broken, they don't know. So if you want to open a support case, be my guest.

If the unit is under warrantee ship it back for repair. You may be surprised what hardware faults can do. I'm happy I did. In my case disabling the Dude on an RB1100AHDx4 stopped the 1month full crash reboot. But on the same ROS version with same config on a new RB1100AHDx4 no crashes after 35days with the Dude enabled.

si458 · Wed Nov 18, 2020 1:56 pm

As far as a support case, I've given up on those guys after multiple cases. All I ever hear is "It's bad hardware". Even when I point out how older ROS didn't have near the problem, they stick to their story. They're like a broken record. And when ask what hardware is broken, they don't know. So if you want to open a support case, be my guest.
If the unit is under warrantee ship it back for repair. You may be surprised what hardware faults can do. I'm happy I did. In my case disabling the Dude on an RB1100AHDx4 stopped the 1month full crash reboot. But on the same ROS version with same config on a new RB1100AHDx4 no crashes after 35days with the Dude enabled.

we dont use the dude in any which way only snmp externally
and sadly its now outta the 1 yr warranty, and we have no replacement router! (accounts wont allow us to buy another one)
so we cant just ship it bk for repair as it could take weeks

asam · Wed Nov 25, 2020 4:40 pm

I really like the form factor of 4011 series and considering to buy one.
As I google for the 4011 problem, I came across this thread and so sad to read so many users have problem with it.

Have you consider if the unit is under attack ? Check this out https://www.cvedetails.com/vulnerabilit ... teros.html

psipek · Wed Nov 25, 2020 4:55 pm

Hi Asam,

it is definitely not attack issue. I have hundreds of mikrotiks around and only RB4011 had this problem. The issue disappeared for me after applying CPU clock = 1200 MHz (10 days uptime now, compared to every second day crash before change).

Also some people here said they saw the issue in their lab environment without being online...

si458 · Wed Nov 25, 2020 4:57 pm

I really like the form factor of 4011 series and considering to buy one.
As I google for the 4011 problem, I came across this thread and so sad to read so many users have problem with it.

Have you consider if the unit is under attack ? Check this out https://www.cvedetails.com/vulnerabilit ... teros.html

i have considered it in the past as when i torch our external interface i do see hundreds of connects from itself to itself
and after looking at our traffic flow server, it appears the router is suddenly talking to itself on every single port under the sun?
and it seems to stem from somebody outside talks to a port then the router itself trys talking to ALL of its ports under the sun?

i also always use the latest software releases, generally 1 week after its released on the rss feed anyways

aoakeley · Mon Nov 30, 2020 12:48 pm

I have received through the grapevine what is the apparent cause of this. Not sure if I am allowed to share it though, and don't want to start gossip if it turns out not to be true.

Maybe Normis or someone from Mikrotik can come in here and make an official statement?

I'll give it a week or so and then if nothing official from Mikrotik I'll have a think about spilling the beans.
Well what do you know?

Well as there seems to be no official post in here. This is what I have been told, I do not know if it is true; but all my faulty devices have been replaced.

"We have finally gotten more information regarding the issues with these devices, it doesn't look like it is an issue with you set up but instead it was an issue with the devices them selves.
Mikrotik has let us know that the problem was due to a 'low quality' component installed in the routers but unfortunately there is no way to determine what routers would be effected by this as not all of the same component installed at the time was 'low quality'
Since they have found the issue they have been able to resolve the issue by swapping to a different component. They also went on to say that only the 1100 and the 4011's have been reported to have this issue an other devices should not be effected as well as they have apologized for any inconvenience this could of caused. "

Mon Nov 30, 2020 3:02 pm

I just checked all correspondence you had with MikroTik, and it was not said that there is a "low quality component" or anything like that. The staff asked you to provide more information, upgrade, and if upgrade doesn't help, send the unit for inspection, since it is covered by warranty. No conclusion about hardware components was made in your ticket.

For others following this topic, please upgrade to 6.47.8 or newer, there are relevant fixed that will improve or remedy any issues described in this topic.

Chupaka · Mon Nov 30, 2020 9:58 pm

I just checked all correspondence you had with MikroTik, and it was not said that there is a "low quality component" or anything like that.

Looks like he quoted the answer from his reseller or something.

aoakeley · Tue Dec 01, 2020 4:56 am

I just checked all correspondence you had with MikroTik, and it was not said that there is a "low quality component" or anything like that.
Looks like he quoted the answer from his reseller or something.

Correct. Who were quoting Mikrotik (or so they say). But nice to nee Normis finally make an appearance in this thread after 1.5 years.

aoakeley · Tue Dec 01, 2020 4:59 am

For others following this topic, please upgrade to 6.47.8 or newer, there are relevant fixed that will improve or remedy any issues described in this topic.

I still have one device very intermittently showing this issue, I will upgrade that unit and see how it goes.

nmt1900 · Sun Dec 06, 2020 11:27 pm

I am having same kind of issues with 2 devices - one is RB1100x4 and another is RB1100Dx4 - and these are only ones plagued with problems because these are only ones updated to 6.45.8 from 6.44.6
...

As I also received suggestion from Mikrotik, that hardware issue might be at question here, then now it's time update on the situation: one device was downgraded to 6.44.6 and other upgraded to 6.46.7. Downgraded device has uptime of 200 days and upgraded one 70 days - both in operation without any issues at all - while neither one was not able to survive a week without reboot before that...

sjoukes · Mon Dec 14, 2020 11:31 am

A short update;
We've update all RB4011's to 6.47.8 apart from one, which was still running on 6.47.7.
Until now we had not crashes with 6.47.8.
Yesterday the 6.47.7 router (RB4011) crashed and stopped accepting connections (internal and external) with an uptime of about 2 weeks.
An autosupout file of 0kB was present. We've now updated it to 6.47.8 to see if this resolves the issue.

Cha0s · Sun Jan 03, 2021 1:56 pm

Has anyone with an r2 revision unit experienced any problems?

Nothing mentioned here in this thread actually works for more than a few months (at best).

So, has r2 resolved the issues we are seeing? If yes I can start replacing boards left and right. I am tired of this...

FezzFest · Sun Jan 03, 2021 3:23 pm

We have an RB4011r2 that crashed recently (routing traffic but web/ftp/ssh/telnet/winbox were unavailable), running 6.47.4 with an uptime of about 8 weeks. Updated it to 6.47.8, we'll see how it goes.

nje431 · Sun Jan 03, 2021 5:06 pm

We've been running our 50 or so 2 year old RB1100AHx4 routers on 6.47.8 with IPSec and DHCP for about a month now with zero problems. Before, about a week is all we could get with either of those turned on.

si458 · Sun Jan 03, 2021 5:25 pm

I’ve updated the four of ours we have to 6.48 and no issue so far as well
The one router that was having the issue hasn’t rebooted in 2 weeks (fingers crossed!)

psipek · Sun Jan 03, 2021 11:16 pm

I have two Rev.1 and two Rev.2. On all of them I experienced this issue. After the 6.47.8 update no freeze so far. To be honest, I still have the CPU clock set to 1200 MHz, as I am not in the mood to break the things again (3 units are pretty far away from my current location).

I hope the issue was really resolved

Cha0s · Tue Jan 05, 2021 5:54 pm

I still have the CPU clock set to 1200 MHz, as I am not in the mood to break the things again (3 units are pretty far away from my current location).

I've set everywhere the cpu to 1200MHz and while that reduces the frequency of the problem, it does not eliminate it.
And having to drive 2 hours to get to the top of a mountain just to power-cycle a router, is not my type of fun.

Interestingly, on the WiFi version of RB4011, I've never had a single crash (running at default 1400MHz).

kobuki · Wed Jan 06, 2021 10:41 pm

For others following this topic, please upgrade to 6.47.8 or newer, there are relevant fixed that will improve or remedy any issues described in this topic.

I'm experiencing the same issue on RB1100AHx4. As I see the 6.47.8 version is in the stable branch, but I only run the long term one. I've burnt myself with stable several times in the past so I'd avoid it if possible. Do you plan backporting this stability fix into the long term branch?

sindy · Thu Jan 07, 2021 10:07 am

They will either backport it or 6.47.x will become the long-term branch, whichever comes first.

jebz · Thu Jan 07, 2021 11:14 am

I'm experiencing the same issue on RB1100AHx4. As I see the 6.47.8 version is in the stable branch, but I only run the long term one. I've burnt myself with stable several times in the past so I'd avoid it if possible. Do you plan backporting this stability fix into the long term branch?

I suspect an ARM microcode or similar issue. Can you try upgrading to the fixed 6.47.8 ROS and firmware, then down grade to your desired long term version.

I'm running ROS 6.45.9 Long Term on a new RB1100AHx4DE after having trouble with my initial RB1100AHx4DE on the same version restarting itself at about 1 month intervals. The new RB1100AHx4DE is running well with 84days uptime.

kobuki · Thu Jan 07, 2021 1:08 pm

I'm running ROS 6.45.9 Long Term on a new RB1100AHx4DE after having trouble with my initial RB1100AHx4DE on the same version restarting itself at about 1 month intervals. The new RB1100AHx4DE is running well with 84days uptime.

So 6.45.9 is fine for you? I don't have much room for experiments as it's a live router in a DC. I don't want to downgrade if the fix in the 6.47.8 version works.

jebz · Thu Jan 07, 2021 2:37 pm

So 6.45.9 is fine for you? I don't have much room for experiments as it's a live router in a DC. I don't want to downgrade if the fix in the 6.47.8 version works.

.
For me ROS 6.45.9 is best for DUDE on my RB1100AHx4DE.

grlmutiajr · Wed Jan 20, 2021 2:20 am

Hi team,

Just a "me too" post. We've got two sites with a RB4011iGS+ (arm) running on 6.47.2 and 6.47.8. Only happened once for each of the router and so far, hard reboot restores it back. No internet for any end hosts but because we have a L2TP tunnel back to the router, we can access the backbone through their loopback addresses. Telnet, winbox or ssh weren't working even through the backbone. We've got at least 4 bridges running, OSPF, VLANs and VPLS tunnels, hotspot, PPPoE, DHCP. Nothing special

Please see attached photos from the SNMP. Cheers

maloi3390 · Thu Jan 28, 2021 9:16 pm

Finally i'm replaced the constantly freezing RB1100Dx4 on CCR1009, at now i can experementally to firmware and test 1100 on different versions ROS. Mikrotik thank you for head hurt!

kobuki · Thu Jan 28, 2021 9:45 pm

FWIW, I've set up a nightly restart of the affected router and it offers a workaround, as the issue hasn't repeated yet since then. I's not ideal and not a real solution but causes no problems at the moment. I'm still hoping for a backport of the fix in the LTS series.

dejanjak · Thu Feb 04, 2021 12:02 am

After almost a year (or more than a year) working with absolutely no problems, couple of days ago rb4011iGS+ factory reset itself (there's no way someone could've done that since no login from external or customer network is possible)... and today it froze (power-cycle did help).

It's on 6.46.7 and running at 1200MHz for a year ...

I'm trying to find out if there are external (physical) problems (switches, servers, ...) but so far - nothing.

... just adding to the story, and I'll keep following - I really don't need "bigger" (read - much more expensive) device but it's not convenient to have 400 customers down over a router "glitch" :(

Cheers!!

Mirek2asn · Thu Feb 04, 2021 11:03 am

I have same problem.

RB4011iGS+
r2
factory firmware 6.45.8
rOS v. 6.47

It was not possible to log in to it. But otherwise it worked in this state for over 14 days. We had to restart manually yesterday.

ms712004 · Wed Feb 24, 2021 3:46 am

same issue running 1100AHx4 using 6.45.8

asam · Sun Mar 14, 2021 12:41 pm

it's march 2021, please update your findings with the new ROS version :)

thank you