RB750 kernel failure, out of memory and rebooting after upgrade to 7.1

MultiTricker · Wed Dec 08, 2021 11:46 am

Hello,
I tried to upgrade one router in our network to 7.1, but unfortunately, it ended up badly.

After reboot, there was high packet loss (90+%) to the router and to a few people behind it. Winbox hung up on "logging in". I was able to login via SSH a few times for a short amount of time, so I could get to these last messages:

Press F1 for help
(12904 messages not shown)
jan/02/1970 00:00:47 system,error,critical kernel failure in previous boot
jan/02/1970 00:00:47 system,error,critical out of memory condition was detected
jan/02/1970 00:02:21 system,error,critical,,,,,,,,,radius error while running customized default configuration script: std failure: timeout (13)
jan/02/1970 00:06:14 system,error,critical login failure for user admin from 78.156.32.131 via winbox
jan/02/1970 00:00:45 system,error,critical router was rebooted without proper shutdown
jan/02/1970 00:00:46 system,error,critical kernel failure in previous boot
jan/02/1970 00:00:46 system,error,critical out of memory condition was detected
jan/02/1970 00:02:19 system,error,critical,,,,,,,,,radius error while running customized default configuration script: std failure: timeout (13)

I also managed to get routerboard info:

/system/routerboard/print
       routerboard: yes
             model: 750
     serial-number: 3B0202CE9968
     firmware-type: ar7240
  factory-firmware: 2.38
  current-firmware: 6.47.1
  upgrade-firmware: 7.1

Previous ROS was as current-firmware states 6.47.1. I replaced router with new hEX (RB750Gr3) on 7.1 with exactly the same configuration and it works. Configuration is very simple - few IP addresses on two interfaces, simple queues and some /ip firewall filter:

# dec/05/2021 03:42:12 by RouterOS 6.47.1
# software id = XEF5-JKG3
#
# model = 750
# serial number = 3B0202CE9968
/interface ethernet
set [ find default-name=ether1 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=ether2 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=ether3 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=ether4 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=ether5 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
/interface list
add exclude=dynamic name=discover
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=ETH
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=3des
/queue type
add kind=pfifo name=ethr pfifo-limit=100
/queue simple
add burst-limit=39321600/39321600 burst-threshold=10/10 burst-time=20s/20s \
    max-limit=32768k/32768k name="id=43" queue=ethr/ethr target=\
    84.244.105.166/32
add burst-limit=272793600/272793600 burst-threshold=10/10 burst-time=20s/20s \
    max-limit=227328k/227328k name="id=50" queue=ethr/ethr target=\
    78.156.44.238/32
add burst-limit=272793600/272793600 burst-threshold=10/10 burst-time=20s/20s \
    max-limit=227328k/227328k name="id=158" queue=ethr/ethr target=\
    78.156.44.230/32
add burst-limit=61440k/61440k burst-threshold=10/10 burst-time=20s/20s \
    max-limit=51200k/51200k name="id=463" queue=ethr/ethr target=\
    78.156.44.235/32
add burst-limit=61440k/61440k burst-threshold=10/10 burst-time=20s/20s \
    max-limit=51200k/51200k name="id=580" queue=ethr/ethr target=\
    78.156.44.232/32
add burst-limit=272793600/272793600 burst-threshold=10/10 burst-time=20s/20s \
    max-limit=227328k/227328k name="id=1168" queue=ethr/ethr target=\
    172.16.14.101/32,84.244.105.162/32
add burst-limit=61440k/61440k burst-threshold=10/10 burst-time=20s/20s \
    max-limit=51200k/51200k name="id=3087" queue=ethr/ethr target=\
    84.244.105.168/32
add burst-limit=272793600/272793600 burst-threshold=10/10 burst-time=20s/20s \
    max-limit=227328k/227328k name="id=4184" queue=ethr/ethr target=\
    78.156.44.234/32
add burst-limit=272793600/272793600 burst-threshold=10/10 burst-time=20s/20s \
    max-limit=227328k/227328k name="id=4324" queue=ethr/ethr target=\
    78.156.44.236/32
add burst-limit=86016k/86016k burst-threshold=10/10 burst-time=20s/20s \
    max-limit=71680k/71680k name="id=4683" queue=ethr/ethr target=\
    78.156.44.229/32
add burst-limit=61440k/61440k burst-threshold=10/10 burst-time=20s/20s \
    max-limit=51200k/51200k name="id=6980" queue=ethr/ethr target=\
    172.16.14.102/32
add burst-limit=61440k/61440k burst-threshold=10/10 burst-time=20s/20s \
    max-limit=51200k/51200k name="id=7182" queue=ethr/ethr target=\
    172.16.14.103/32
add burst-limit=61440k/61440k burst-threshold=10/10 burst-time=20s/20s \
    max-limit=51200k/51200k name="id=7774" queue=ethr/ethr target=\
    172.16.14.104/32
add burst-limit=86016k/86016k burst-threshold=10/10 burst-time=20s/20s \
    max-limit=71680k/71680k name="id=3096" queue=ethr/ethr target=\
    78.156.44.226/32
add burst-limit=24576k/24576k burst-threshold=10/10 burst-time=20s/20s \
    max-limit=20480k/20480k name="id=6331" queue=ethr/ethr target=\
    172.16.14.120/32
add burst-limit=61440k/61440k burst-threshold=10/10 burst-time=20s/20s \
    max-limit=51200k/51200k name="id=6220" queue=ethr/ethr target=\
    172.16.14.106/32
add burst-limit=61440k/61440k burst-threshold=10/10 burst-time=20s/20s \
    max-limit=51200k/51200k name="id=1472" queue=ethr/ethr target=\
    172.16.14.126/32
/snmp community
set [ find default=yes ] addresses=192.168.0.0/21
add addresses=192.168.0.0/22 name=dohled
/system logging action
set 0 memory-lines=100
set 1 disk-lines-per-file=100
/ip neighbor discovery-settings
set discover-interface-list=discover
/ip address
add address=192.168.0.50/24 interface=ether1 network=192.168.0.0
add address=172.16.14.97/27 interface=ether2 network=172.16.14.96
add address=78.156.44.225/28 interface=ether2 network=78.156.44.224
add address=84.244.105.161/28 interface=ether2 network=84.244.105.160
/ip firewall filter
add chain=forward comment=Established connection-state=established
add action=accept chain=forward comment="idip=8320" dst-address=\
    84.244.105.166
add action=accept chain=forward comment="idip=8320" limit=32768,32768:packet \
    src-address=84.244.105.166 src-mac-address=20:AA:4B:57:C3:15
add action=accept chain=forward comment="idip=10444" dst-address=\
    78.156.44.238
add action=accept chain=forward comment="idip=10444" limit=\
    227328,227328:packet src-address=78.156.44.238 src-mac-address=\
    4C:5E:0C:78:97:B8
add action=accept chain=forward comment="idip=5957" dst-address=78.156.44.230
add action=accept chain=forward comment="idip=5957" limit=\
    227328,227328:packet src-address=78.156.44.230 src-mac-address=\
    D4:CA:6D:F1:D0:6F
add action=accept chain=forward comment="idip=16358" dst-address=\
    78.156.44.235
add action=accept chain=forward comment="idip=16358" limit=51200,51200:packet \
    src-address=78.156.44.235 src-mac-address=54:83:3A:D5:F2:8D
add action=accept chain=forward comment="idip=11031" dst-address=\
    78.156.44.232
add action=accept chain=forward comment="idip=11031" limit=51200,51200:packet \
    src-address=78.156.44.232 src-mac-address=4C:5E:0C:04:91:33
add action=accept chain=forward comment="idip=12137" dst-address=\
    172.16.14.101
add action=accept chain=forward comment="idip=12137" limit=\
    227328,227328:packet src-address=172.16.14.101 src-mac-address=\
    54:E6:FC:B9:53:A5
add action=accept chain=forward comment="idip=6976" dst-address=\
    84.244.105.162
add action=accept chain=forward comment="idip=6976" limit=\
    227328,227328:packet src-address=84.244.105.162 src-mac-address=\
    54:E6:FC:B9:53:A5
add action=accept chain=forward comment="idip=9765" dst-address=\
    84.244.105.168
add action=accept chain=forward comment="idip=9765" limit=51200,51200:packet \
    src-address=84.244.105.168 src-mac-address=E8:DE:27:25:CB:37
add action=accept chain=forward comment="idip=6828" dst-address=78.156.44.234
add action=accept chain=forward comment="idip=6828" limit=\
    227328,227328:packet src-address=78.156.44.234 src-mac-address=\
    D4:CA:6D:F4:43:B0
add action=accept chain=forward comment="idip=7147" dst-address=78.156.44.236
add action=accept chain=forward comment="idip=7147" limit=\
    227328,227328:packet src-address=78.156.44.236 src-mac-address=\
    54:E6:FC:AF:D0:2B
add action=accept chain=forward comment="idip=10445" dst-address=\
    78.156.44.229
add action=accept chain=forward comment="idip=10445" limit=71680,71680:packet \
    src-address=78.156.44.229 src-mac-address=4C:5E:0C:69:C5:29
add action=accept chain=forward comment="idip=12309" dst-address=\
    172.16.14.102
add action=accept chain=forward comment="idip=12309" limit=51200,51200:packet \
    src-address=172.16.14.102 src-mac-address=E4:8D:8C:65:96:7B
add action=accept chain=forward comment="idip=12789" dst-address=\
    172.16.14.103
add action=accept chain=forward comment="idip=12789" limit=51200,51200:packet \
    src-address=172.16.14.103 src-mac-address=98:DE:D0:AB:C7:2F
add action=accept chain=forward comment="idip=14007" dst-address=\
    172.16.14.104
add action=accept chain=forward comment="idip=14007" limit=51200,51200:packet \
    src-address=172.16.14.104 src-mac-address=A4:2B:B0:CB:12:A1
add action=accept chain=forward comment="idip=5184" dst-address=78.156.44.226
add action=accept chain=forward comment="idip=5184" limit=71680,71680:packet \
    src-address=78.156.44.226 src-mac-address=E4:8D:8C:6A:0C:7C
add action=accept chain=forward comment="idip=10918" dst-address=\
    172.16.14.120
add action=accept chain=forward comment="idip=10918" limit=20480,20480:packet \
    src-address=172.16.14.120 src-mac-address=4C:5E:0C:C2:2F:BB
add action=accept chain=forward comment="idip=10683" dst-address=\
    172.16.14.106
add action=accept chain=forward comment="idip=10683" limit=51200,51200:packet \
    src-address=172.16.14.106 src-mac-address=C4:6E:1F:DE:DE:29
add action=accept chain=forward comment="idip=16522" dst-address=\
    172.16.14.126
add action=accept chain=forward comment="idip=16522" limit=51200,51200:packet \
    src-address=172.16.14.126 src-mac-address=00:31:92:3A:F2:1F
add action=drop chain=forward comment=OSTATNI
/ip ipsec policy
set 0 dst-address=0.0.0.0/0 src-address=0.0.0.0/0
/ip proxy
set cache-path=web-proxy1
/ip route
add distance=1 gateway=192.168.0.1
/ip service
set telnet address=192.168.0.0/21 disabled=yes
set ftp address=192.168.0.0/21 disabled=yes
set www address=192.168.0.0/21 disabled=yes
set ssh address=192.168.0.0/21
set www-ssl address=192.168.0.0/21
set api address=192.168.0.0/21
set winbox address=192.168.0.0/21
set api-ssl address=192.168.0.0/21 disabled=yes port=8728
/snmp
set enabled=yes
/system clock
set time-zone-autodetect=no
/system identity
set name=ETH
/system ntp client
set enabled=yes primary-ntp=192.168.0.6

Hope this help to identify what caused this.

phalaaxx · Thu Dec 16, 2021 10:34 pm

I second that - same router, same problem. Although I was able to remotely downgrade back to 6.49.2 - seems like there is a high CPU usage after reboot and if it does not reboot again in a minute or two it's relatively safe to log in from Winbox (I had to try to connect several times because of the "logging in" timeout problem).
According to router log it was unstable for the exact same reason (out of memory), although memory usage seems to be the same with both 6.49.2 and 7.1 on RB750. However there's a difference in memory usage between 6.49.2 and 7.1 in hAP ac router - 7.1 uses approximately 15MB more memory, so probably RB750 does not properly report memory usage.
Profiling shows that most of the time cpu0 has highest usage and is usually followed by management (especially when I try to connect to the router from SSH or another Winbox, or while opening/closing windows in Winbox - these peak cpu usage to 100%).

phalaaxx · Tue Dec 28, 2021 7:52 pm

Today I tried with 7.1.1 - seems to be much more stable than 7.1 but probably still not quite enough for anything other than testing purposes.

MultiTricker · Mon Jan 03, 2022 10:28 am

Thank you for reply.

I still wait for reaction or at least some new major version with more bugfixes which could mean "we got this".

It might be bad luck, but it was second RouterBoard I upgraded back then and that happened. I do not have confidence to continue with similarly configured devices.

studd · Tue Jan 11, 2022 3:11 am

I've also experienced the same problems with RB751U-2HnD RouterOS upgrade to 7.1.1. But after that also I couldn't downgrade back to 6.49.2 Stable. But I finally solved this problem! I downloaded 6.48.6 Long-term MIPSBE all_packages-mipsbe-6.48.6.zip and unpacked from there ONLY necessary packages: dhcp, ppp, security, system, wireless. routeros-mipsbe itself has these packages + a bit more. See https://wiki.mikrotik.com/wiki/Manual:System/Packages description of packages on wiki page. Then I scheduled routeros-7.1.1 package to uninstall inside WebFig web-interface through my tablet and uploaded above mentioned necessary packages into Files through WebFig interface and clicked there Downgrade and confirmed additional reboot. After router reboot I got downgrade to 6.48.6 Long-term MIPSBE necessary stable packages and everything started to work again without problems so far.

MultiTricker · Fri Jan 14, 2022 11:14 am

Thank you for sharing this info.

I can netinstall that board for sure, but still - I wish to know that what caused this issue was fixed, so probably I do not encounter that on any other board upgrades.

studd · Fri Jan 14, 2022 5:37 pm

You are welcome.

I can netinstall that board for sure, but still - I wish to know that what caused this issue was fixed, so probably I do not encounter that on any other board upgrades.

I didn't use netinstall utility. I used Webfig interface in the browser. I had to spent several days with that router, because I couldn't downgrade routeros 7.1.1 to 6.49.2 Stable normally.
I think that this problem is caused by insufficient RAM, so router has to restart periodically in a loop after watchdog detects that RAM became too low to work and router has to be rebooted. Insufficient RAM may be caused by some kind of incorrect working script or something else of such kind. Maybe some kind of config changes case incorrect config option for 7.1.1 and that provoke memory leak and/or high CPU load and unstable work of the router itself. I found at least 1 incorrect option. It was marked in red color. I think it was "Default Route Distance" property which was auto set in routeros 6.x config equal to 0, but in routeros 7.1.1 that same option equality to 0 cases error and marked in red color by interface. But, after setting that option to 1 or other small integer number I still couldn't get very stable work of the router on routeros 7.1.1 and still couldn't downgrade back to 6.49.2 Stable until I found the above mentioned solution.

phalaaxx · Sat Feb 19, 2022 10:47 am

Last night I tried to upgrade to 7.1.2, things seem to be worse than before - I was actually not able to log in with winbox.
I had to downgrade (back to 6.49.3) with scp and ssh.
Seems to me it is a bad thing to try to login immediately after boot - this way chances for random reboots are much higher. I had to give it at least a couple of minutes after reboot before I attempt anything. Occasionally it reboots by itself a few seconds after boot on its own but if it survives long enough (a couple of minutes at least) chances are it will be able to downgrade properly. Also from past experience - winbox upload seems to be much slower than scp and cpu load is higher, therefore chances of random reboots increase - use scp and ssh if possible.
I suppose I was lucky not to brick the router and be able to downgrade remotely, but I suppose if did not have ssh enabled I would have needed to resort to other means to downgrade it (not remote anyway). However I share my experience here because it might be useful for people who are in similar situation - trying to downgrade remotely.

pe1chl · Sat Feb 19, 2022 12:15 pm

It is probably better not to try to upgrade such old and small devices, especially when you are not using any of the newer features.
(and also not when you were using features that are not yet stable in v7)

It really is not an upgrade to just do for the fun of it. And certainly not on a remote router.

phalaaxx · Sun Feb 20, 2022 12:25 pm

It is probably better not to try to upgrade such old and small devices, especially when you are not using any of the newer features.
(and also not when you were using features that are not yet stable in v7)

It really is not an upgrade to just do for the fun of it. And certainly not on a remote router.

Actually WireGuard seems to be a nice to have feature, especially for old and small devices like RB750. Given I can't do that right now I'm switching to RB760iGS, but I'll probably continue to follow up on this topic and experiment with RouterOS v7 since I have absolutely no other use for the old RB750 device.

MultiTricker · Tue Mar 01, 2022 2:19 pm

It is probably better not to try to upgrade such old and small devices, especially when you are not using any of the newer features.
(and also not when you were using features that are not yet stable in v7)

I get it and common sense says the same. But they said in the newsletter that even an historical devices can benefit better performance with RoS 7, you get the idea that it's not a problem. After they released v7 even for those devices, I expect to get it working if not with initial stable release, then after some time for sure.

I have no problem with having devices like RB750 deprecated for RoS 7. Well, RB750 isn't that bad, if you consider other like RB433

Wish luck to developers with getting the most types of their routerboards compatible, bug free and working without problem. They already did a tons of great work.

RB750 kernel failure, out of memory and rebooting after upgrade to 7.1

RB750 kernel failure, out of memory and rebooting after upgrade to 7.1

Re: RB750 kernel failure, out of memory and rebooting after upgrade to 7.1

Re: RB750 kernel failure, out of memory and rebooting after upgrade to 7.1

Re: RB750 kernel failure, out of memory and rebooting after upgrade to 7.1

Re: RB750 kernel failure, out of memory and rebooting after upgrade to 7.1

Re: RB750 kernel failure, out of memory and rebooting after upgrade to 7.1

Re: RB750 kernel failure, out of memory and rebooting after upgrade to 7.1

Re: RB750 kernel failure, out of memory and rebooting after upgrade to 7.1

Re: RB750 kernel failure, out of memory and rebooting after upgrade to 7.1

Re: RB750 kernel failure, out of memory and rebooting after upgrade to 7.1

Re: RB750 kernel failure, out of memory and rebooting after upgrade to 7.1

Who is online