Missing RouterOS configuration after a reboot on very rare occasions

Sat Apr 02, 2022 9:57 pm

Hello, all of our trusted MikroTik users and fans. This time here is a topic from MikroTik asking for some assistance.

We are aware that there is some kind of an issue in v7 of RouterOS when in some very, very rare cases and under unclear circumstances part of the router configuration is lost. We are not referring to missing IPsec configuration or missing routing tables. These issues are already resolved. Currently, we have tried everything that we can imagine and have not managed to reproduce this problem in order to resolve it.

We are asking for some help here. If you have experienced such a problem (after an upgrade or reboot) then please let us know about it. Please write an e-mail to support@mikrotik.com and send a list of RouterOS versions which you have used on this router ever before and are using now. Let us know which part of the configuration was lost. Provide supout file (generated when the configuration is lost) and if it is a possibility backup file (before the configuration was lost). If you do send a backup, then please also let us know how to access the router (username, password, IP address, interface to use).

Try to notice if the configuration indeed was lost or simply was returned back to the configuration which was present on the router in the past.

Please keep this topic strictly related to this problem and provide information only if you are 100% sure that the configuration indeed was lost. For example, it was not lost due to a safe mode being enabled, some other administrator making changes on the router, etc. Report only if you are 100% sure that this is the case of missing configuration.

P.S. We do believe that the issue might be already resolved and affects only routers which have had some "bad" RouterOS version installed on them in the past.

Thank you very much already ahead of time!

rextended · Sun Apr 03, 2022 12:03 am

I notice that problem happen sometime (also on previous 6.x versions) if the SSH host key (on devices with unused SSH service) certificate is invalid.
And also that those devices generate corrupted backups.

error creating backup file

The soluction, for me, is do "/ip ssh regenerate-host-key" on problematic device and, after that, the backup and/or upgrade happen without lost the config.
Usually are devices with first software installation older than 6.44 and never netinstalled again, but only updated.
This happen also if security package is disabled on the past software version (and enabled or not later), before upgrade to 7.x

benlg · Sun Apr 03, 2022 11:34 am

Thank you very much already ahead of time!

And thank you for being so transparent, deserves consideration !
Had to be said :)

ahmedelbarbary · Sun Apr 03, 2022 1:07 pm

Thanks @strods, Since 7.1.2 everything well, using CHR on ESXI

fragtion · Sun Apr 03, 2022 1:22 pm

I experienced this type of config loss recently when installing/uninstalling The Dude on my RB5009 on 7.2rc5 (just before ARM64 support was fixed) (I don't think the dude had anything to do with the config loss). I saved a supout just before and after the config loss, which I attached to (now closed) support ticket SUP-78036. Maybe it can help you guys figure out what went wrong?

While the config loss issue is known to happen sometimes randomly during reboots (and perhaps even spontaneously during normal running operation, only to take effect after the next reboot) - it does seem more prone to occur during software updates... and now in my case, during a simple package installation/uninstallation procedure. Perhaps it has something to do with the way packages are deployed that could be overwriting flash where the config is stored and MT should pay attention to that side of things? IDK, just a wild guess here trying to help

Also:

P.S. We do believe that the issue might be already resolved and affects only routers which have had some "bad" RouterOS version installed on them in the past.

Is there any way to know if a given router is affected by some bad RouterOS version of the past? Even if we do or don't know, how can we clear/reset such influences so that our router only has traces of the latest version firmware influencing operation?? Surely you can agree that past versions of software should not influence current device software version and its operation ??

mkx · Sun Apr 03, 2022 1:33 pm

Is there any way to know if a given router is affected by some bad RouterOS version of the past?

I guess this is the very point of this thread ... to find out which old version went haywire (if that's indeed the case) and to try to fix it so that when mass of less ssvvy users start to upgradevfrom v6 to v7 they don't fail miserably.

fragtion · Sun Apr 03, 2022 1:41 pm

Is there any way to know if a given router is affected by some bad RouterOS version of the past?
I guess this is the very point of this thread ... to find out which old version went haywire (if that's indeed the case) and to try to fix it so that when mass of less ssvvy users start to upgradevfrom v6 to v7 they don't fail miserably.

We should really not be comfortable with the notion that past versions of software can influence current device operation on latest installed software version... I'm surprised there's not already more noise about this. To what extent does RouterOS store past version configuration and to what extent can that configuration (which should arguably be rendered inert) still be able to influence/affect device operation when running on a new version? Are we talking about actual configuration, or just the underlying disk partitioning/layout? Can a Netinstall fix all of this or not? If MikroTik wants our assistance with this, we need some guidelines on where to look and how this important aspect of device operation and configuration integrity, operates beneath the scenes

Is this somehow analogous to a Microsoft Windows Registry retaining entries from past OS revisions even if they are no longer in use? Surely if "/export" can provide a full, clean and sanitized output of the device configuration, then it should be possible for the router to clear extraneous/problematic configuration using similar functions to sanitize the current device config, negating it of any possible noise/interference? Furthermore, restoring that export on a device which has just been freshly wiped with "/system reset-configuration", should mean that this "registry" starts off with a clean state and that again, no prior installed RouterOS versions should have any effect on device stability going forward (from a config perspective). If it's a disk partitioning issue, then that is another story of course.. But in that case, Mikrotik should focus attention on comparing disk images structure from a device known to be prone to this config loss issue (such as my RB5009), against a similar device which has been freshly wiped with netinstall

I suppose one thing we can already know with certainty that the problem doesn't stem from RouterOS v6 or lower, because RB5009 can't run that anyway.

My money is on the problem existing in RouterOS v7 core software itself, in some new function somewhere, and that it still affects even latest versions and could strike anyone at any time regardless of past versions installed if they just meet all of the right conditions to trigger the bug... but that only a handful of people have deployed the specific config needed to trigger the bug, which is why it rarely occurs. But of course that is just my speculation which isn't likely to prove very useful, so I won't say any more on that until further research and investigation can be completed.

Optimistic that this will be fully resolved sooner than later as MikroTik are clearly taking this matter very seriously now, as they of course should

npeca75 · Sun Apr 03, 2022 7:35 pm

7.1.3 -> 7.1.5
rb3011

aprox 50% of static IPv4 DHCP lease entry was lost

could not find any logic
in every vlan (separate dhcp server) some entry was lost
every entry has same type of comment,
room number - device name
and no, no funny chars in dhcp lease or in comment
no logic at all

history ? 6.47.10 -> 7.1 -> 7.1.3 -> 7.1.5
no RC, ever

East2 · Mon Apr 04, 2022 11:05 am

HAP AC2 upgrade 7.1.3 - 7.1.5...All firewall filters clear...

brandaoeb · Mon Apr 04, 2022 11:07 am

Hi:
I recently make the move to 7.1.5 from 6.49.5 on a RB750 Gr3 All configs where messy or not present... It was working no dhcp (client and server no polls etc.. ), so i think i find the solution.
After some more tests i noticed no vlans?? After the upgrade router seems ok but if you do RoutherBoard upgrade, pufff its gone no dhcp etc.......

Solution
1 - Back to 6.49.5
2 - Save config for a file
3 - reset to factory settings (still 6.49.5)
4 - Upgrade to 7.1.5 OK
5 - Reset Factory settings OK
5 - RouterBoard Upgrade OK
7 - Reset Factory settings OK
8 - Upload config file (from 6.49.5) OK
Everything was working vlans dhcp l2tp/ipsec sstp ... etc !!!!
I do not know if all this resets are necessary but it works, i tested on another RB750 from a friend and it worked (of course different configs)
I do not know if this helps but my experience stays here
I will use this procedure in CCR1009 and others.

Tue Apr 05, 2022 11:41 am

Is it possible that in all of these cases there was a backup file generated on versions 7.0.x or older? And is it possible that at some point configuration was loaded from this backup file on v7.1.x or later?

fragtion · Tue Apr 05, 2022 12:07 pm

Is it possible that in all of these cases there was a backup file generated on versions 7.0.x or older? And is it possible that at some point configuration was loaded from this backup file on v7.1.x or later?

Is there any way to check this besides recalling from memory? I think I upgraded my rb5009 to newer version the moment I got it (arrived on 7.0.4) so it's unlikely that I would have done any config in that version which would have needed to be backed up and restored to the newer 7.x build, but that's months ago now and I, for one, can't recall for sure. Maybe someone else here can confirm this pattern

Chupaka · Tue Apr 05, 2022 1:24 pm

Okay, I repeated that on another CHR that I almost don't use, so should not be a problem to reproduce it for you. The router was upgraded to v7.2 first.

Create a WG interface named "wg-RedShield" (with an empty config) and add the following routing filter rule:

/routing/filter/rule/add chain=ospf-out rule="if ( gw-interface ether1 || gw-interface wg-RedShield ) { reject; } accept;"

Then reboot. A-a-and... It's gone

No more rule, just that WG interface. Some race condition while interfaces are not yet added during a boot?

UPD: If there are another rules (like "accept;") around - they are still here.

UPD2: I can reproduce it even without WG interface, with a simple rule:

if ( gw-interface ether1 ) { reject; } accept;

benlg · Tue Apr 05, 2022 1:36 pm

Good catch !
Sounds like 7.2.1 coming in the next hours :)

Tue Apr 05, 2022 2:38 pm

Okay, I repeated that on another CHR that I almost don't use, so should not be a problem to reproduce it for you. The router was upgraded to v7.2 first.

Create a WG interface named "wg-RedShield" (with an empty config) and add the following routing filter rule:
Code: Select all
/routing/filter/rule/add chain=ospf-out rule="if ( gw-interface ether1 || gw-interface wg-RedShield ) { reject; } accept;"
Then reboot. A-a-and... It's gone No more rule, just that WG interface. Some race condition while interfaces are not yet added during a boot?

UPD: If there are another rules (like "accept;") around - they are still here.

UPD2: I can reproduce it even without WG interface, with a simple rule:
Code: Select all
if ( gw-interface ether1 ) { reject; } accept;

This is a routing rule-specific issue which is being fixed at the moment. Not related to this global problem which we have not managed to reliably reproduce.

Tue Apr 05, 2022 2:40 pm

Currently, we are aware that if a version older than 7.1 was used, a backup file was generated and the router was upgraded, then all configuration can be gone. That problem is already fixed in v7.1. However, we can not in any way reproduce the missing configuration (random parts of it) without including this old version (without fix) it in the equation.

benlg · Tue Apr 05, 2022 2:52 pm

So, we must pay attention if we upgrade from RouterOS < 7.1 (including RouterOS 6 ?) to any other version >= 7.1 ?

rextended · Tue Apr 05, 2022 2:57 pm

From any occasion than you lost ALL the config, not only the routes...
Must be know: version before, version after and provide
backup before upgrade (not .rsc) and backup after the loss

Backup exports also Certificates Publica and Private, SSH host key, etc. than the normal .rsc do not export

Tue Apr 05, 2022 3:04 pm

So, we must pay attention if we upgrade from RouterOS < 7.1 (including RouterOS 6 ?) to any other version >= 7.1 ?

Yes, then you might lose configuration. And since we can not fix old versions, we can not do anything about that. The problem is resolved already and should not appear anymore when you upgrade from >=v7.1 to newer releases.

Also regarding missing route rules. It seems that we have fixed this issue, but they will be gone for one more time on upgrade to v7.3beta, etc. After that, when you add routing rules, they shold not disappear after reboot, upgrade, etc.

These previously mentioned issues have nothing to do with the problem which we try to reproduce. This problem also might actually not be real and probably all of the reports were caused by this version older than 7.1.

benlg · Tue Apr 05, 2022 3:12 pm

Thank you for clarification @strods 👍

IntLDaniel · Tue Apr 05, 2022 4:33 pm

So, we must pay attention if we upgrade from RouterOS < 7.1 (including RouterOS 6 ?) to any other version >= 7.1 ?
Yes, then you might lose configuration. And since we can not fix old versions, we can not do anything about that. The problem is resolved already and should not appear anymore when you upgrade from >=v7.1 to newer releases.

That IS NOT TRUE! Today I have upgraded 750G r3 from 7.2rc7 to 7.2 final and lost at least one interface configuration (had to restore backup config before upgrade). I had no chance (time) to do a backup after upgrade (with lost config) but I think you can do it by your self if I will send you my backup-before-upgrade (on 7.2rc7), you can put it to your 750G r3 and try to upgrade to 7.2 in your lab?

Tue Apr 05, 2022 4:44 pm

That IS NOT TRUE! Today I have upgraded 750G r3 from 7.2rc7 to 7.2 final and lost at least one interface configuration (had to restore backup config before upgrade). I had no chance (time) to do a backup after upgrade (with lost config) but I think you can do it by your self if I will send you my backup-before-upgrade (on 7.2rc7), you can put it to your 750G r3 and try to upgrade to 7.2 in your lab?

They key question here: HOW did you get to that 7.2rc7 version ?
Coming from ROS6 OR ROS7 <7.1 and then upgrade upgrade upgrade ?
Or netinstall from a version >=7.1 and then moving on to 7.2rc7 ??
I am going to assume the first option. And then it DOES apply, if I understood it well.

biomesh · Tue Apr 05, 2022 4:49 pm

They key question here: HOW did you get to that 7.2rc7 version ?
Coming from ROS6 OR ROS7 <7.1 and then upgrade upgrade upgrade ?
Or netinstall from a version >=7.1 and then moving on to 7.2rc7 ??
I am going to assume the first option. And then it DOES apply, if I understood it well.

Strods did not say a netinstall was required. He said an upgrade from >=7.1 to current versions should not have the issue, which was the case by IntLDaniel.

fragtion · Tue Apr 05, 2022 4:54 pm

I want to downgrade from 7.2 back to 7.2rc7, restore the last 7.2rc7 backup I did, and then try upgrade again to 7.2 to see if I get the same config corruption.
If I do get the same corruption again (in which case we know the issue is very much reproducible) then I want to try netinstall the same router to 7.2rc7, restore the same backup, and upgrade to 7.2 again. Then we know the problem is filesystem-related for sure, and can definitely be "cured" with netinstall.
The problem is, I have a support ticket open with Mikrotik and don't want to "take matter into my own hands" until given a formal instruction from Mikrotik support. I'm just concerned that Mikrotik support will be slow to respond to the ticket and then we delay/miss what could be a golden opportunity to corner and identify this bug even further. Any suggestions?

rextended · Tue Apr 05, 2022 5:03 pm

1) Downgrade
2) import previous backup of same version
3) Upgrade
if the problem happen:
4) Downgrade again,
5) import again the same previous backup of same version
6) change passwords and change (not delete) sensible settings
7) make backup called for example clean_before
8) Upgrade
if the problem happen again:
9) make backup called for example clean_after
10) make supout.rif
11) send both backups and supout.rif to support@mikrotik.com

fragtion · Tue Apr 05, 2022 5:07 pm

I'll try that. I guess I don't need to netinstall to figure out that the bug can be reproduced, only potentially cured, which we can find out later. Thanks @rextended.

rextended · Tue Apr 05, 2022 5:12 pm

If the bug can be reproduced on your machine WITHOUT netinstall, but using backups on another machine do not generate the bug,
ask a refund/new hardware and send the device to mikrotik (if MikroTik want) for forensic analisys.

The netinstall destroy any possibility to replicate the bug, because format the filesystem.
If the cause is a corrupted filesystem from previous versions (old than used before / after now)

fragtion · Tue Apr 05, 2022 5:25 pm

1. Downgraded from 7.2 to 7.2rc7. No config loss.
2. Upgraded back to 7.2. No config loss.
3. Downgraded back to 7.2rc7 again. No config loss.
4. Restored my 7.2rc7 backup which was made before even upgrading to 7.2 for the first time
4. Switched branch from testing to stable (as I did the first time I upgraded from 7.2rc7 to 7.2)
5. Upgraded back to 7.2. No config loss.

So it did not reproduce successfully :/ at least we tried.. back to the drawing board

Znevna · Tue Apr 05, 2022 5:27 pm

7.2 and 7.2rc7 are identical releases, little chance of corruption(?).

rextended · Tue Apr 05, 2022 5:31 pm

Is just to replicate upgrade from 7.2rc7 to 7.2 when the config is lost.
Probably, if the 7.2rc7 is not netinstalled, the corruption happen on previous versions and the 7.2 fix it, but the fix cause data lost....
Like with chkdsk sometime fix the windows disk, but you also lost the OS or your personal data,
and on RouterOS because fsck probably for some error do worst....

fragtion · Tue Apr 05, 2022 5:39 pm

Yes, @Znevna I was just trying to reproduce exactly what happened to me earlier (config corruption when upgrading from 7.2rc7 to 7.2) but not successful. So, maybe @rextended is right that it's been fixed in 7.2 at the cost/sacrifice of some lost config, which would explain why I could not reproduce it...
But I'm not sold on that. I think there's another reason why it corrupted originally but not while trying to reproduce. So I ask: Do we have any basis to know if 7.2 does anything to allegedly try fix the problem, that 7.2rc7 didn't? Forgive me if I just missed it but I don't think @strods specifically mentioned such. So if 7.2rc7 and 7.2 are in fact identical, then there is something else at play here and it probably isn't even filesystem-related... but I will stop speculating there.

rextended · Tue Apr 05, 2022 5:45 pm

Another speculation: Configuration database is on damaged/corrupted NAND / Flash area and fsck fix it marking as damaged block and deleting the config database?

fragtion · Tue Apr 05, 2022 5:50 pm

Another speculation: Configuration database is on damaged/corrupted NAND / Flash area and fsck fix it marking as damaged block and deleting the config database?

FWIW: 0.0 % Bad blocks, Check Installation reports "installation is ok". Maybe it wasn't ok before the first upgrade though. Who knows..

Znevna · Tue Apr 05, 2022 5:51 pm

I didn't see you mentioning 7.2rc7 -> 7.2 causing config corruption? Did you upgrade from 7.2rc7 to 7.2 and had config corruption?
7.2 doesn't contain any extra fixes that 7.2rc7 doesn't. As I've said earlier, they are identical releases built 1 day apart (30.03.2022 and 31.03.2022).
There are some changes in the .fwf files (RouterBoard Firmware) but that's it.

Amm0 · Tue Apr 05, 2022 5:54 pm

For example, it was not lost due to a safe mode being enabled, some other administrator making changes on the router, etc. Report only if you are 100% sure that this is the case of missing configuration.

P.S. We do believe that the issue might be already resolved and affects only routers which have had some "bad" RouterOS version installed on them in the past.

We need LTE MBIM support so forced to use V7... so been a "long time" V7 user & I can't recall a lost configuration FWIW. Various problems with V7 for sure, but this has NOT been one. Not saying their ain't potential bugs here... more just sharing that I haven't seen "lost config" in likely hundreds of various V7 beta/rc/"stable" upgrades, on half dozen different hardware types.

It's been our practice to upgrade to the latest V6 stable before going to any V7 version. That seem to have worked in our case. And, the only time we've need netinstall to recover a device is when upgrading cAPs from 6.41 directly to 7.x, which did get caught in a boot loop. Other than that, using the 5-10 second "reset-to-default" button method has always recovered from a bad V7 configuration WITHOUT netinstall.

But...Maybe Mikrotik should consider enforcing a restriction on upgrade path with a "minimum V6 version" that can upgrade? e.g. if V6 didn't have the "upgrade" channel, it should NOT be allowed to upgrade to V7...

DenisPDA · Tue Apr 05, 2022 6:15 pm

There are some changes in the .fwf files (RouterBoard Firmware) but that's it.

Возможно проблема в RouterBoard Firmware
There may be a problem with the RouterBoard Firmware

YO3IPT · Tue Apr 05, 2022 10:05 pm

Upgraded an RB4011iGS+5HacQ2HnD-IN from 7.1.5 to 7.2
In my configuration I have a bonding interface setup on ether2 and ether3 (balance xor) which is a trunk port to a CRS326.
After the upgrade, the interface disappeared completely. I reloaded a 7.1.5 back-up file on the 7.2 installation and the configuration restored normally.
I did not notice any other problem, IPSec identities remained.
After recovering the RB4011, I also did an upgrade to the CRS326, from 7.1.5 to 7.2, without any issues - the bonding interface was ok.
I also updated a couple of hAP ac2's as well, again without any problems.
In fact, until now, I only encountered problems on upgrade with RB4011. The upgrade from 7.1.3 to 7.1.5 caused an issue with the routerboard firmware. After updating RouterOS, on restart, the router did not come back up and was not accessible anymore. I did a reset to default and on further investigation I discovered that the routerboard firmware reverted to factory default version 6.45.1. I did the firmware upgrade and after that restored the device from back-up.

IntLDaniel · Wed Apr 06, 2022 9:28 am

That IS NOT TRUE! Today I have upgraded 750G r3 from 7.2rc7 to 7.2 final and lost at least one interface configuration (had to restore backup config before upgrade). I had no chance (time) to do a backup after upgrade (with lost config) but I think you can do it by your self if I will send you my backup-before-upgrade (on 7.2rc7), you can put it to your 750G r3 and try to upgrade to 7.2 in your lab?
They key question here: HOW did you get to that 7.2rc7 version ?
Coming from ROS6 OR ROS7 <7.1 and then upgrade upgrade upgrade ?
Or netinstall from a version >=7.1 and then moving on to 7.2rc7 ??
I am going to assume the first option. And then it DOES apply, if I understood it well.

YES, came from 6.49.2 to 7.x and then upgrade and upgrade.

npeca75 · Wed Apr 06, 2022 9:36 am

HAP ac2

netinstall 7.1.5
simple config
eth1 trunk, 3 pcs vlan
1 bridge, vlan filtering

device is used as dumb AP
every vlan have corresponding SSID on 2.4 and 5Ghz

static v4 addr on vlan254 for mgmn
static v6 addr on vlan254 for mgmn

upgrade to 7.2
static v4 default route is invalid
pointing to right interface, but red, "I"
deleted default route, created again, same values as before, working OK
static v6 is untouched
snmp, ssh keys, user, wifi security, everything OK, only default v4route gone mad

npeca75 · Wed Apr 06, 2022 10:03 am

... and second HAP ac2
same dumb AP config, only v4 and v6 vlan254 addr are different

update 7.1.5 -> 7.2
everything is intact, working fine

so, no, there is no logic

fragtion · Thu Apr 07, 2022 1:58 pm

I tried upgrading a different RB5009 and it upgraded software to 7.2 successfully. But when I rebooted to apply firmware upgrade (it was on 7.1.5 previously), the device came up unreachable, as bridge + some other cfg was lost, similar to my home unit.. I managed to log into the router from a non-bridged port...

So it's the firmware upgrade that broke it this time (maybe last time also, I can't recall for sure at which stage it broke last time)

Other devices upgraded without any issues whatsoever: 5x CHR, 1x RB4011iGS+5HacQ2HnD, 2x hEX, 1x hAP ac, 1x hAP ac lite, 1x mAP2n, 1x mAP lite, 1x SXT 5HnD, 1x RB433(from 7.1.5), 2x RB433(from 6.49.2), and 1x CRS106-1C-5S
Update: one clients hEX also unreachable after firmware update from 7.1.5 to 7.2 (software update successful). The firmware update seems to be causing the problem now ?

in my experience, bridges and wireguard interfaces seem to be most commonly affected by the corruption, along with some firewall rules

pawlisko · Fri Apr 08, 2022 10:30 pm

in my experience, bridges and wireguard interfaces seem to be most commonly affected by the corruption, along with some firewall rules

Absolutely. 100% right.

So I never experienced this behavior on RB1100x4AHDude. I just upgraded to RB5009 and I lost my config more than once. And I was perplexed. Via backup and some upgrades, I was able to recover but with certain issues, it requires me to change network to be able to connect.

1. Bridges - issues like hell. It just disappears later screw back-up as a certain portion of it is related to the bridge itself.
2. WG peers - I found that they will disappear as well (have 15 peers and 6 interfaces)
3. Firewall address lists - I have 21k entries there and ofter it will be reduced to 600 entries

I never had an issue with disappearing firewall rules but...

My symptoms stopped after using the backup procedure in WinBox.

I created a config (divided into 8 parts)
Reset RB to 0
Started with interfaces (going back where it stopped twice), then the other 7 parts (no issues).
Created backup via export. Reviewed backup that it is proper.
Created backup with WinBox
Reset RB to 0
Logged in - restored backup from WinBox
Since then - stable as a rock.
It even survived the upgrade from 7.1.5 to 7.2 (although I had prepared everything)
After upgrade - backup via export - comparing to 7.1.5 - no changes, created new backup via WinBox (just in case)
So far no problems

Also, I found that the MSTP bridge is more stable than RSTP (don't ask me why)
The biggest issue is so far restoring interfaces. It can randomly stop, and I need to find which line it stopped moving on

My code:

/interface bridge add name=LAN priority=0x7000 protocol-mode=mstp region-name=US-NY vlan-filtering=yes
/interface ethernet set [ find default-name=sfp-sfpplus1 ] name=WAN speed=100Mbps
/interface ethernet set [ find default-name=ether5 ] name="ZyXEL GS1920-24 Port 25" speed=100Mbps
/interface ethernet set [ find default-name=ether6 ] name="ZyXEL GS1920-24 Port 26" speed=100Mbps
/interface ethernet set [ find default-name=ether1 ] name="ZyXEL GS1920-24HP Port 23" speed=100Mbps
/interface ethernet set [ find default-name=ether2 ] name="ZyXEL GS1920-24HP Port 25" speed=100Mbps
/interface ethernet set [ find default-name=ether3 ] name="ZyXEL GS1920-24HP Port 26" speed=100Mbps
/interface ethernet set [ find default-name=ether4 ] disabled=yes name=eth4 speed=100Mbps
/interface ethernet set [ find default-name=ether7 ] disabled=yes name=eth7 speed=100Mbps
/interface ethernet set [ find default-name=ether8 ] name=eth8 speed=100Mbps
/interface 6to4 add comment="Hurricane Electric IPv6 Tunnel Broker" !keepalive local-address=*Private* mtu=1280 name=sit1 remote-address=*Private*
/interface wireguard add comment="KeepSolidVPN - Germany" disabled=yes listen-port=51821 mtu=1420 name=KeepSolidVPN-Germany private-key=*Private*
/interface wireguard add comment="KeepSolidVPN - France" disabled=yes listen-port=51822 mtu=1420 name=KeepSolidVPN-P2P private-key=*Private*
/interface wireguard add comment="KeepSolidVPN - Poland" disabled=yes listen-port=51823 mtu=1420 name=KeepSolidVPN-Poland private-key=*Private*
/interface wireguard add comment="KeepSolidVPN - UK" disabled=yes listen-port=51824 mtu=1420 name=KeepSolidVPN-UK private-key=*Private*
/interface wireguard add comment="VPN - Home Main Network" listen-port=52850 mtu=1420 name=wg0 private-key=*Private*
/interface wireguard add comment="VPN - Home Guest Network" listen-port=52860 mtu=1420 name=wg1 private-key=*Private*
/interface bonding add lacp-rate=1sec mode=802.3ad name="ZyXEL GS1920-24" slaves="ZyXEL GS1920-24 Port 25,ZyXEL GS1920-24 Port 26"
/interface bonding add lacp-rate=1sec mode=802.3ad name="ZyXEL GS1920-24HP" slaves="ZyXEL GS1920-24HP Port 23,ZyXEL GS1920-24HP Port 25,ZyXEL GS1920-24HP Port 26"
/interface list add name=WAN-list
/interface list add name=LAN-list
/interface list add comment="Own WireGuard VPN servers" name=WireGuard-Server
/interface list add comment="WireGuard VPN clients' connections" name=WireGuard-Clients
/interface list add name=WAN-interface-list
/interface list add name=WAN-Internet-list
/interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge nat add action=accept chain=srcnat
/interface bridge port add bridge=LAN interface="ZyXEL GS1920-24HP"
/interface bridge port add bridge=LAN interface="ZyXEL GS1920-24"
/interface bridge port add bridge=LAN interface=eth4
/interface bridge port add bridge=LAN interface=eth7
/interface bridge port add bridge=LAN interface=eth8
/interface detect-internet set detect-interface-list=all internet-interface-list=WAN-interface-list lan-interface-list=LAN-list wan-interface-list=all
/interface list member add interface=WAN list=WAN-list
/interface list member add interface=LAN list=LAN-list
/interface list member add comment="VPN - Home Main Network" interface=wg0 list=WireGuard-Server
/interface list member add comment="VPN - Home Guest Network" interface=wg1 list=WireGuard-Server
/interface list member add interface=WAN list=WAN-Internet-list
/interface list member add interface=sit1 list=WAN-interface-list
/interface list member add interface=WAN list=WAN-interface-list
/interface list member add interface="ZyXEL GS1920-24" list=LAN-list
/interface list member add interface="ZyXEL GS1920-24HP" list=LAN-list
/interface list member add interface=eth4 list=LAN-list
/interface list member add interface=eth7 list=LAN-list
/interface list member add interface=eth8 list=LAN-list
/interface list member add interface=wg0 list=WAN-interface-list
/interface list member add interface=wg1 list=WAN-interface-list
/interface list member add interface=KeepSolidVPN-Germany list=WireGuard-Clients
/interface list member add interface=KeepSolidVPN-P2P list=WireGuard-Clients
/interface list member add interface=KeepSolidVPN-Poland list=WireGuard-Clients
/interface list member add interface=KeepSolidVPN-UK list=WireGuard-Clients
/interface wireguard peers add allowed-address=*Private* interface=wg0 persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=*Private* interface=wg0 persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=*Private* interface=wg0 persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=*Private* interface=wg0 persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=*Private* interface=wg0 persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=*Private* interface=wg0 persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=*Private* interface=wg0 persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=*Private* interface=wg0 persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=*Private* interface=wg1 persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=*Private* interface=wg1 persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=*Private* interface=wg1 persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=0.0.0.0/0 comment="KeepSolidVPN - Poland" endpoint-address=*Private* endpoint-port=51820 interface=KeepSolidVPN-Poland persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=0.0.0.0/0 comment="KeepSolidVPN - UK" endpoint-address=*Private* endpoint-port=51820 interface=KeepSolidVPN-UK persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=0.0.0.0/0 comment="KeepSolidVPN - Germany" endpoint-address=*Private* endpoint-port=51820 interface=KeepSolidVPN-Germany persistent-keepalive=25s preshared-key=*Private* public-key=*Private*
/interface wireguard peers add allowed-address=0.0.0.0/0 comment="KeepSolidVPN - France" endpoint-address=*Private* endpoint-port=51820 interface=KeepSolidVPN-France persistent-keepalive=25s preshared-key=*Private* public-key=*Private*

Again few times I had to find out which line was executed prior WinBox being disconnected and running from that line. Bridges and WG peers were the main culprits.

sirbryan · Mon Apr 11, 2022 6:29 pm

I experienced missing configuration issues on 6.x just the other day. Unfortunately I was trying to get the customer's site back up and running, so I wasn't in a position to grab a bunch of support files.

I had an LHG-60 that was on 6.45.x(7?) with a pretty simple bridge between wlan60-1 and ether1. I wanted to get it to 6.49.x, which has been the best for all of my 60GHz gear.
I did the install remotely, and after realizing the radio wasn't coming back, I remembered I had had config-loss problems with this upgrade path before.

I drove on-site, got into the radio, and found the wlan60 interface completely missing. I backed the radio off to 6.47.10, and wlan60-1 came back, but was disabled and its config was corrupted. I fixed it, got it linked up, upgraded the firmware to 6.47.10, rebooted, then downloaded/upgraded to 6.49.5. When it rebooted, wlan60-1 had been renamed to wlan60-2, was enabled but no longer part of the bridge, had a MAC address of 00:00:00:00:00:01, and the wireless settings were again corrupted.

I had the MAC from the AP side, so I copied that over, fixed the SSID and password/key, and the radio came online. I upgraded the firmware, rebooted, and it's working fine now.

I’ve upgraded several radios in the field, but most of them went 6.45->6.46/6.47->6.48->6.49. Only those that started at 6.45 and went to directly to 6.48 or later experienced corruption, specifically on the wlan60 interface.

wojo · Tue Apr 12, 2022 5:46 am

I lost my bridge configurations on a CRS328 and my clock (defaulted to 1970) and most of my DHCPv4 static leases on my RB5009. This was around April 3rd or 4th, and I wasn't watching for it closely.

In all my cases it was not a revert to a prior, just random (e.g. 50% of DHCPv4 static leases that have been there forever) items gone. An entire default bridge config, etc.

For now on I will do the upgrades carefully watching and diff'ing the configs.

osc86 · Wed Apr 13, 2022 6:18 pm

I just have the case of missing parts of the configuration after updating a CCR2004 from 7.2 to 7.3b33. I took a verbose export and a supout, anything else I can do? I need to get the router up and running again asap. First thing I noticed is that all wireguard interfaces are gone, ether1 reset to defaults and the pppoe-client interface is gone. I made changes to ether1 and the pppoe interface just yesterday, but I didn’t make any changes to the wireguard interfaces. The peers seem to be still there this time. No need to mention that I’ll file a ticket once my internet is working again.

pawlisko · Wed Apr 13, 2022 7:52 pm

Try to notice if the configuration indeed was lost or simply was returned back to the configuration which was present on the router in the past.

Please keep this topic strictly related to this problem and provide information only if you are 100% sure that the configuration indeed was lost. For example, it was not lost due to a safe mode being enabled, some other administrator making changes on the router, etc. Report only if you are 100% sure that this is the case of missing configuration.

P.S. We do believe that the issue might be already resolved and affects only routers which have had some "bad" RouterOS version installed on them in the past.

Thank you very much already ahead of time!

I have configs for you (prior failure and post). Please contact me via DM or e-mail (pmajran @ gmail . com). I don't want to censor too much to post here.

IntLDaniel · Wed Apr 13, 2022 9:27 pm

I have updated my 750G r3 (mmips) from 7.2 to 7.2.1 and again lost some unspecified parts of config. Router did not have an access to the internet, red default gateway etc., I did not investigate what is missing and restore last backup just before the upgrade and everything OK again.

But later after the upgrade:
* updated my config - add some new scrips, add another interface list and some related changes to lists and internet detection (everything via WinBox 3.35 x64)
* did a reboot after all of my changes
* router lose some parts of configuration just only after the reboot (!) (mostly my last edited config was lost but also another objects in the same places where I did changes!)
* so reloaded backup before upgrade again and did another changes, did another new backup reboot again and everything seems to be OK, rebooted several times and still OK

So I did lost config also just via edit and reboot (without upgrade), so maybe the reason for lost config issue could be the WinBox 3.35 x64 itself ??

pawlisko · Wed Apr 13, 2022 10:22 pm

So I did lost config also just via edit and reboot (without upgrade), so maybe the reason for lost config issue could be the WinBox 3.35 x64 itself ??

I had very similar experiences but it does not matter if it is WinBox or WinFig or Terminal.

Basically for me to make it stick is to create a backup via WinBox and then Restore it. That makes it permanent.

I have all configs saved for MT people if they want it.

rtlx · Thu Apr 14, 2022 6:46 pm

I would like to add, that this problem is not exclusive to version 7.x.
There was the same problem on RouterBoard 2011 years ago - without any visible reason part of configuration was lost. It was even on version 5.x if I'm not mistaken, but it was exclusive to Routerboard 2011.
It was the only reason why we've stopped using RB 2011 and dropped them from our networks.

IntLDaniel · Fri Apr 15, 2022 12:03 pm

So I did lost config also just via edit and reboot (without upgrade), so maybe the reason for lost config issue could be the WinBox 3.35 x64 itself ??
I had very similar experiences but it does not matter if it is WinBox or WinFig or Terminal.
Basically for me to make it stick is to create a backup via WinBox and then Restore it. That makes it permanent.

I have all configs saved for MT people if they want it.

It sounds like a workaround - instead of reboot everytime do a Restore "just saved" configuration and do this step everytime before upgrade?

pawlisko · Fri Apr 15, 2022 5:39 pm

It sounds like a workaround - instead of reboot everytime do a Restore "just saved" configuration and do this step everytime before upgrade?

The proper procedure for config to stick:
1. Create backup via export
/export file=Backup.rsc terse show-sensitive

2. Review backup that it is good.
Important - when you are happy move to #3

3. Create a backup with WinBox (I do not encrypt nor password protect) - I use the name Backup.backup
4. Reset RB to 0 (no default config, no backup)
5. Log in via WinBox (using MAC address) and restore backup (from step 3 - Backup.backup) via WinBox procedure - it will then restart RB
6. RB stable as a rock - you can restart it without fear of losing config.

Any major change in config (more than 2-3 lines of code) or upgrade (here your experience may vary - sometimes it is ok, sometimes there is a config corruption) - start the procedure above (for changes) or reset to 0 and restore from the backup you already have via WinBox

Hope it helps

rextended · Fri Apr 15, 2022 6:04 pm

Ok, but instead: 1), 2), 3), netinstall, 5b) restore from .rsc, 6)

pawlisko · Fri Apr 15, 2022 6:15 pm

Ok, but instead: 1), 2), 3), netinstall, 5b) restore from .rsc, 6)

I had my issues with restore from .rsc - mainly with interfaces - problems with bridge, bonding, wireguard. Hence WinBox backup/restore works for me better.

rextended · Fri Apr 15, 2022 6:25 pm

I had my issues with restore from .rsc - mainly with interfaces - problems with bridge, bonding, wireguard. Hence WinBox backup/restore works for me better.

After 4000+ installations, .rsc is better than any type of backup, but these three conditions must be true:
A) Restore on the same device and keep the original MACs
B) Restore all necessary directories
C) Import SSH host key, Certificates, User Manager and Dude databases and files (.lng, .mib, etc.).

pawlisko · Fri Apr 15, 2022 8:15 pm

I had my issues with restore from .rsc - mainly with interfaces - problems with bridge, bonding, wireguard. Hence WinBox backup/restore works for me better.
After 4000+ installations, .rsc is better than any type of backup, but these three conditions must be true:
A) Restore on the same device and keep the original MACs
B) Restore all necessary directories
C) Import SSH host key, Certificates, User Manager and Dude databases and files (.lng, .mib, etc.).

I don't have such extensive background, but I prepared special .rsc files just in case in a form of backup.

interfaces.rsc - all things connected with interfaces including bridges, wireguard, bonding, etc.
after this is done I am doing the rest (split into 7 smaller files - IPv4, IPv6, DHCP,....., certificates, address lists, etc).

Many times interfaces will not finish so I need to find last successful line to restart manually.

But .rsc will not stick config for me. tried multiple times. only WinBox makes it for me.

Maybe also because I am not doing NetInstall... who knows

rextended · Fri Apr 15, 2022 8:19 pm

Some hints:
A) Apply the settings for the interface you use to connect with device... (like link speed, bridge, etc.) at the end...

B) When you reset or netinstall don't keep previous configuration and don't set the defaults, leave everything blank (no-default-configuration), or importing .rsc OBVIOUSLY cause errors ...

C) Do not mix or separate parts, the .rsc is on right order, for example, ip pool must be defined first, dhcp server later... If you invert the parts, dhcp server is not created

D) Keep this topic... on OP...

badwolf4561 · Tue Apr 19, 2022 7:44 pm

I get this issue with a CCR2004-1G-12S+2XS (v7)

I have (mainly) experienced corruption of the configuration after installing the Zerotier package (from the Extra's package). Installation and reboot is successful, however, after another reboot my config gets corrupt. Not all the config, but in the DHCP Server config. The DHCP interface changes from "LAN Bridge" to "*10". DHCP server becomes invalid. Winbox can no longer login to the router either via IP or Neighbour (MAC). I still have Console access (this is used to load a saved & good config backup).

I know for sure the latest corruption of config file occured when I installed Zerotier package on v7.2.1. I also remember the upgrade from v7.2rc5 to v7.2.1 performed flawlessly (no Zerotier installed).

laurinkus · Mon Apr 25, 2022 3:27 pm

Lost all bridges, all ipip interfaces, one ppp user and historical graphs after upgrading from 7.2 to 7.2.1 on hAP ac^2.

Tue Apr 26, 2022 10:17 am

Everyone, thank you very much for all the input. For your reports in the forum and support tickets. By putting our heads together and combining all of your reports we managed to reproduce this problem and seems that we have resolved it. The fix is included in v7.3beta37 and, of course, if it will be proven safe (we have tested it ourselves reliably, but would like to see the feedback from beta releases), then it will be included in stable releases as well.

Please note that the upgrade happens on the old version, not the version which you try to install. So the fix applies only when you try to upgrade FROM 7.3beta37 or a later version. Downgrade to older versions than 7.3beta37 still will be affected.

rextended · Tue Apr 26, 2022 10:36 am

@strods
Is possible, please, include this fix, and other various filesystem sanitization added on v7, also on v6 branch?
Thanks.

IntLDaniel · Tue Apr 26, 2022 11:06 am

It seems that the reason for config lost is not v6 but all v7 releases only? Maybe MikroTik should release some 100% working instructions to avoid config lost during upgrade on affected firmwares OR we have to wait for first v7 release which is not affected. The upgrage from any previous version should be safe?

rextended · Tue Apr 26, 2022 11:07 am

Is why I ask to put the fix also on v6, because rarely v6 to next v6 do full configuration lost...
And if the update process on v6 is already fixed when migrating to v7, the problem is gone...

IntLDaniel · Tue Apr 26, 2022 11:10 am

Really? I had never lost config during upgrade from v6 to v6. But if it is relevant, they shoud fix v6 as well.

fragtion · Tue Apr 26, 2022 11:12 am

Woohoo !! That's massive news, Congratulations guys, if it turns out to be true then this is quite a milestone & gamechanger for v7 stability sentiments. Whichever dev found this gremlin deserves some reward!! :D
Obviously we're very curious what was wrong and how it was fixed (ie when it was first introduced / how many past versions were effected? this also ties in to @rextended's request of course) .. if such information could be shared with the community. Also now that the bug has been isolated, maybe we could know what conditions could trigger it in theory (sw/fw update only, reboot, randomly spontaneously while device is operating normally? or could that be another issue causing those rare crashes?)
Through extensive user testing (myself included) we will know soon enough if that was the only bug causing this issue & whether it was definitively fixed or not , but optimism & enthusiasm is high right now! :D Thank you. Let the testing commence 😄 (plss don't disappoint 😅)

rextended · Tue Apr 26, 2022 11:31 am

Really? I had never lost config during upgrade from v6 to v6. But if it is relevant, they shoud fix v6 as well.

I try to explain myself better, as far as I understand.
The problem happen on upgrade process from v6 to v7, because the upgrade process used is not what is on new not installed v7 OS, but on previous already installed v6 OS.
Fixing also previous v6 OS before migrating from v6 to v7, solve the problem...

Tue Apr 26, 2022 11:58 am

This problem was related only to RouterOS v7 (v7 to v7 - upgrade, downgrade, extra package installation, package removal). It was triggered by internal algorithms of configuration maintaining processes and could not be workarounded or triggered whenever you want. A bunch of things had to be done in order to trigger the issue and additionally perfect timing was required. For example, our script required hundreds of upgrades to be made just to lose part of the configuration once - the same things are done over and over again until perfect timing is matched. So the answer here is simple - just upgrade to a fixed version and forget about it in the future.

rextended · Tue Apr 26, 2022 12:01 pm

Please let me explain if I understand.
Just (obviously after the various backups) upgrade from 6.4x to 7.3.x and not care about the problem?
Thanks

gstitt · Fri Apr 29, 2022 5:56 pm

Just to add more detail to this solved issue. I had this happen on 3 rb4011s in 3 different networks and 2 hapacs in two different networks, so it wasn't just a corner case.

laurinkus · Fri Apr 29, 2022 7:55 pm

When do you expect to release 7.3?

eworm · Sat Apr 30, 2022 12:02 pm

Guess that will take some time... Waiting for 7.2.2 instead.

Mon May 02, 2022 3:48 pm

Now the fix is also included in RouterOS v7.2.2.

epedersen · Sun Jul 24, 2022 9:23 am

I just upgraded my RB3011 from 7.3.1 to 7.4 and I lose all *password* settings for my ppp secret.
So, I had my ppp secret configured but I losed the password configuration on all of them.
Luckily I had a backup but damn what an ugly bug :(
I also noticed that this is not the first time this has happened to me, unfortunately the first time I didn't take note about the ROS version because I thought it was my mistake, but now I'm pretty sure it wasn't me...

Sun Jul 24, 2022 9:31 am

I just upgraded my RB3011 from 7.3.1 to 7.4 and I lose all *password* settings for my ppp secret.
So, I had my ppp secret configured but I losed the password configuration on all of them.
Luckily I had a backup but damn what an ugly bug :(
I also noticed that this is not the first time this has happened to me, unfortunately the first time I didn't take note about the ROS version because I thought it was my mistake, but now I'm pretty sure it wasn't me...

Advice from this post applies (pe1chl has written it quite nicely so no need to type it over :lol: ):
viewtopic.php?p=947349#p947288

Upgrading from version to version to version can sometimes causes nasty things to be left behind which can causes issues quite a bit later.
I also had to clean my Hex once after a pesky Wireguard bug. Reapplying the EXACT config on a clean device (netinstall) and the problem was gone !

epedersen · Sun Jul 24, 2022 10:03 am

Yes! a new step to take into account when upgrading...
Thanks for the tip

Who is online