Hello,

This question is directed to those among you who are using NUT (network ups tools) on their network along with Mikrotik devices.

As far as I know there is no NUT client for miktotik devices. In this context, what is your strategy for shuting down Mikrotik devices using nut ?

I was thinking about the following one: in "non-essential legs" of my network the "last standing device" (a NUT client) initiates in his shutdown script a proper shutdown of mikotik devices through ssh in the correct order. This would ensure that network communications between nodes in this leg are ensured as long as they are needed and that when everything's done, the last device shuts down mikrotik devices properly before shuting down itself. Only the devices which need to be up for a longer time are kept up, and mikrotik devices do not consumer battery for nothing.

From a theoretical viewpoint, this should work, but is there a better approach ?

Thank you in advance

From theoretical point of view, with certain UPSes and in certain conditions after shutting down router might remain in halted state until you manually cycle power[*]. MT staffers repeatedly claimed that cutting power without shuting down ROS should not be a problem. Which means it's better not to worry about proper shutdown procedure if you need ROS device get up&running after mains comes back.

[*] A few UPS models I have had experience with (select models by APC, TrippLite, Eaton) don't shut down upon received shutdown command if mains is on at that moment. Which means that if connected device started shutdown procedure due to low battery and as last action device sends shutdown command to UPS but mains returns while running shutdown procedure, UPS doesn't shutdown leaving device powered. In this moment it depends on how device continues the procedure. ROS devices (running v6, didn't test v7 yet) enter a halted state and only power cycle brings them back. Most other devices cut power to most of hardware requiring hitting the soft power switch or WoL magic packet to start again. Devices running "proper" OSes can be configured to do something else. My linux servers are configured to perform actual reboot if shutdown sequence was due to UPS battery depletion so if mains returns during shutdown sequence, they simply reboot. One could add wake-on-lan commands in boot procedureof NUT master but that only helps with WoL capable devices ... RB devices are not if I'm not mistaken.

Thank you for your answer.

ROS devices (running v6, didn't test v7 yet) enter a halted state and only power cycle brings them back.
One could add wake-on-lan commands in boot procedureof NUT master but that only helps with WoL capable devices ... RB devices are not if I'm not mistaken.

So if I understand correctly: no NUT client, no proper way to shutdown ROS devices in case of power loss and no proper way to boot them through reboot after powerloss or WOL after power returns and risks to have a device in a halted state in case of power return before complete UPS shutdown...

Well ok...

MT staffers repeatedly claimed that cutting power without shuting down ROS should not be a problem

Sorry, but "should" does not bring confidence. Nor do the logs which state that "improper shutdown occured".

I couldn't find any official statement ensuring that cuting power IS safe (and not "quite safe" as Normis wrote elsewhere on the forum. What does "quite" mean anway).
Further, on the wiki I found this in the FAQ (https://wiki.mikrotik.com/wiki/Manual:RouterOS_FAQ):

After power failure the MikroTik router is not starting up again
If you haven't shut the router down, the file system has not been unmounted properly. When starting up, the RouterOS™ will perform a file system check. Depending on the HDD size, it may take several minutes to complete. Do not interrupt the file system check! It would make your installation unusable.

My understanding is that it IS NOT safe at all. It means that in case of a power failure, as there are risks that multiple power failures occur in series, the first one may shut down all mikrotik devices, then after power returns as shutdown wasn't done properly, all devices launch a file check, and if power is lost again during this period and UPS shuts down the power then we get enough bricked devices to build a wall. Sounds like a shitty situation to me.

Yes we can "hope" to not be in such a situation, but "hope" is not a strategy isn't it ?

How do you people deal with it ?

So if I understand correctly: ... no proper way to shutdown ROS devices in case of power loss and no proper way to boot them through reboot after powerloss or WOL after power returns

You understood slightly wrong what I wrote. Indeed ROS doesn't have NUT client (nor master). However it's still possible to properly shut it down, only methods are somehow awkward, e.g. you have to use ssh command like you wrote, or possibly use API call (not sure if it's possible to exec shutdown via API though). However, shutdown leaves ROS device in halted state and only power cycle gets it running again. If power is indeed cut and mains outage continues, this is not a problem. After mains returns and UPS provides power to connected devices once again, ROS devices will boot just fine. Only in particular case I described (power is not cut after shutdown completed) there's a problem. Then you have to choose between two cases: get to site to cycle power to get things online again or netinstall the device. Depending on circumstances any of these cases is probably less distracting than the other. Also chances of happening either of them will help to decide (chances of bricking device due to improper shutdown seem to depend on device model and seems to have increased with latest versions of ROS. I've yet to experience one but that's my limited experience). It's up to you to decide what's best for you, I'm just mentioning as it seems not many people consider such possibility (it happened to me a few times in last couple of decades).

BTW, power cut during boot procedure is a big problem for any OS (not only ROS) and the only remedy is to use UPS that allows to set delay for enabling power output after outage ... it both allows partial battery recharge as well as skips short periods of mains availability which happen often if power outage is due to power line shorts (and the cause of blackout was not properly solved when grid operators try to restore operations).

HI!

Use proper UPS device that has IP-card ( LAN connection ), other options just real pain (USB/RS232).
There are tons of option on the IP-card for doing shutdown/restart/poweron based on triggers.

My linux servers are configured to perform actual reboot if shutdown sequence was due to UPS battery depletion so if mains returns during shutdown sequence, they simply reboot.

Can you elaborate as I'm thinking about how I should properly protect some Linux NUC servers.

The main point is from previous you noticed some UPS could send an alert to downstream servers because of failing upstream but fail to cut power downstream because upstream is back again ?
No way to enforce "cut power downstream 30 seconds after alerting no matter what" ?

What happens when your server is notified power is about to cut ? Instead of powering off, you reboot the server so that if power is really cut the reboot is interrupted somewhere and if the power if not cut (because upstream power is back), server is back online ?

My NUC boxes implement a BIOS After Power Failure option. Which one did you choose among Stay Off, Last State or Stay On ? I'm inclined to use Stay On when power is managed by UPS.