Trouble: Can't connect to ATL after update

utiker · Thu Sep 26, 2024 11:20 pm

Hello,

I am in a big trouble.

My setup is ATL -> hAP3 -> LAN.

What I did:

1. I have updated the hAP3:

/system/package/update/download
/system/routerboard/upgrade

Everything went fine and 7.16 works after the reboot.

2. I have updated the ATL:

/system/package/update/download
/system/routerboard/upgrade

Update was successful.
Then I rebooted.

RESULT: I can no longer connect to the ATL (I always use only SSH and that is the only enabled method):

ssh: connect to host 192.168.188.1 port 22022: Network is unreachable

I cannot even ping the ATL from the hAP3 (which powers it) or from the computer. I get packet-loss=100%. No configuration was touched before or after the update.

At the same time, I see that the LED of on the hAP3 port to which the ATL is connected is lit, i.e. there is connection on L1. From time to time that led goes dark for 2-3 seconds but then it is on again.

I tried connecting the ATL cable directly to the computer (using the PoE injector supplied with the ATL) but again - no connection is possible.

The problem:
============

My ATL is on a very high pole and it is impossible to reset it via the button. A reset would need unmounting of the whole construction.

I don't know why the update caused this but now I have no Internet connection because the ATL cannot be used. I am sending this using my phone Internet.

Please kindly advise how to proceed.

infabo · Fri Sep 27, 2024 1:44 am

There are reports of PoE issues in the 7.16 release topic.

Your ATL is probably configured with DHCP client and not with a static IP address? 192.168.188.1?
Instead of direct connection between your computer<->ATL try the following:

As described you power your ATL usually by HAp poE out port. Disable Poe out on the Ethernet port where your ATL is connected to (https://help.mikrotik.com/docs/display/ ... ut-offmode) or use a non-poe port. instead put the power injector in between HAP and ATL.

You first updated your HAP and the ATL afterwards. If there is really something fishy with Poe in latest release, your ATL maybe faced power interruption while updating. That would be worst case and could also explain why it is not reachable as well.

utiker · Fri Sep 27, 2024 11:03 am

Thank you for your quick reply.
OK, I found the thread you mention:

viewtopic.php?t=211157

Unfortunately, I don't understand from it what exactly I should do. Does it suggest that 7.16 is buggy? If yes, is downgrade to a good version possible?

Your ATL is probably configured with DHCP client and not with a static IP address? 192.168.188.1?

I have not touched it for a year (since purchased), so I don't have it all in my head. The IP address of the ATL is 192.168.188.1. I believe the rest is pretty much factory default.

I am not quite sure what is the difference between:

A) PoE power coming from the injector and Ethernet connected directly to the computer (not through hAP3)
B) PoE power coming from the injector and Ethernet connected to hAP3

Anyway, I followed your advice and disabled the PoE out on the hAP3 port and connected the power injector in between hAP3 and ATL.

[admin@MikroTik] > /interface/ethernet/poe/print detail   
 0 name="ether5" poe-out=off poe-priority=10 power-cycle-ping-enabled=no power-cycle-interval=none

Sadly, the result is still the same as in the OP - can't connect to the ATL. I also still see the LED of the ether5 port going dark for about 3 sec from time to time (not sure if this is normal, haven't paid attention before the updates).

your ATL maybe faced power interruption while updating.

All my equipment is UPS powered. There have been no surges or power outages during the update either.

So, what can I do?

infabo · Fri Sep 27, 2024 11:47 am

It is not about your general electrical power supply or your UPS. The POE controller on your HAP may unintentionally powercycled or something.

See changelog of 7.16

*) poe-out - upgraded firmware for SAMD20 PSE (AF/AT) controlled boards (the update will cause brief power interruption to PoE-out interfaces);

The difference between connecting your ATL to computer or HAP is pretty easy to understand: your HAP may act as an DHCP server, whereas your computer most probably not. So in case your ATL is configured as DHCP-client, then you are out of luck when connecting to your computer -> your ATL won't receive any IP address and will be not reachable in no way. There would be a last resort, using WinBox MAC mode, but you told us that you disables all IP services except SSH. :/

But I assume you rather have changed the default static IP address from 192.168.88.1 to 192.168.188.1. So you could be in luck, when configuring static IP address on your computers ethernet adapter. 192.168.188.2/24 maybe. See if connection is possible.

marsbeetle · Fri Sep 27, 2024 11:59 am

Check the leases on your DHCP server and see if you can spot the ATL, make sure you have "Active Host Names" selected.

After I upgraded to 7.16 I lost all my static addresses but noticed they had been assigned other ip addresses.

utiker · Fri Sep 27, 2024 12:41 pm

The POE controller on your HAP may unintentionally powercycled or something.

Is that Mikrotik's fault? (e.g. bug)

See changelog of 7.16

Yeah, I have seen that. However, I have no idea how I am supposed to use the information provided. OK - "will cause brief power interruption" - Why? If such interruption is bad - how come this is a "stable" version resulting in such troubles? This is quite confusing.

The difference between connecting your ATL to computer or HAP is pretty easy to understand: your HAP may act as an DHCP server, whereas your computer most probably not.

Alright. But why should I need the computer to be a DHCP server to connect to the ATL? I have always been able to connect to the HAP using either DHCP or a static address (e.g. 192.168.88.*) and then access the ATL (which is 192.168.188.1) both from the HAP and from the LAN computers.

Not being a network expert, my understanding is that the ATL itself acts as a DHCP server, the HAP receives an IP address from it, then the HAP itself (running a DHCP server too) gives an IP address to the computers on the LAN. In that sense, I have never had any problem connecting to the ATL (network 192.168.188.1) while being assigned either a static IP address on the HAP network (192.168.88.*) or DHCP (also 192.168.88.*). In that sense, I really don't understand what may have changed, assuming that updates don't touch configuration.

But I assume you rather have changed the default static IP address from 192.168.88.1 to 192.168.188.1.

The default IP address of the ATL is 192.168.188.1. Also mentioned in the manual. I have not changed it.

@marsbeetle

Check the leases on your DHCP server and see if you can spot the ATL, make sure you have "Active Host Names" selected.

What are "leases", which DHCP server do you mean and how do I do what you suggest, please?

utiker · Fri Sep 27, 2024 1:03 pm

So you could be in luck, when configuring static IP address on your computers ethernet adapter. 192.168.188.2/24 maybe. See if connection is possible.

Yes, I am able to create such connection in NetworkManager:

ipv4.addresses: 192.168.188.22/24
ipv4.gateway: 192.168.188.1

and I can connect to that.
The problem is that it results in nothing useful - I still can't SSH to the ATL, and trying to ping it gives:

$ ping 192.168.188.1
PING 192.168.188.1 (192.168.188.1) 56(84) bytes of data.
From 10.138.15.200 icmp_seq=1 Packet filtered
From 10.138.15.200 icmp_seq=2 Packet filtered
...

infabo · Fri Sep 27, 2024 1:08 pm

What host is 10.x? can you please unplug any other network devices. just connect your PC and ATL.

utiker · Fri Sep 27, 2024 1:18 pm

What host is 10.x?

According to https://ipinfo.io/10.138.15.200 it is a bogon IP address. I have absolutely no idea why it shows.

can you please unplug any other network devices. just connect your PC and ATL.

Previously you said that I should connect PC -> HAP --(PoE injector)--> APC.
Are you suggesting now that I should connect PC --(PoE injector)--> APC? Or something else?
To avoid further confusion, please clarify. Thanks.

utiker · Fri Sep 27, 2024 1:23 pm

I have absolutely no idea why it shows.

It might be due to the inter-VM network which is 10.0.0.0. Testing outside that network, i.e. directly from the physical Ethernet interface of the PC:

$ ping 192.168.188.1
PING 192.168.188.1 (192.168.188.1) 56(84) bytes of data.
From 192.168.188.22 icmp_seq=1 Destination Host Unreachable
From 192.168.188.22 icmp_seq=2 Destination Host Unreachable
From 192.168.188.22 icmp_seq=3 Destination Host Unreachable
...

infabo · Fri Sep 27, 2024 2:08 pm

Your ATL may need to be "Netinstalled". Tick "Keep configuration". This should bring it back to life. https://help.mikrotik.com/docs/display/ROS/Netinstall. I know, its hard to swallow, as you need to press the reset-button 10+ seconds to get into Etherboot mode....

utiker · Fri Sep 27, 2024 2:29 pm

Isn't there really anything else to try?

This is a disaster. If Mikrotik can cause such troubles to a customer through "stable" updates - how can one possibly trust any update going forward?

Or is anyone using remote/difficult-to-reach-physically devices doomed to such issues?

I wonder if there is any safe way to update in such cases.

Amm0 · Fri Sep 27, 2024 3:38 pm

Are you using winbox? Assuming you have the defaults, you should be able to get in via its MAC, not IP, address in the WinBox app from the LAN side of the router. If you can get in, look at the Logs & do an :export at Terminal and paste those here if you'd like. If Winbox with MAC address does NOT work, then you'd be down to re-flashing it with netinstall tool as suggested above (but that is a more involved process...so do try using the MAC+winbox first)

utiker · Fri Sep 27, 2024 4:22 pm

Are you using winbox?

No, only SSH from Linux.

As for Netinstall - this seems quite complicated. I wonder if I will be able to do it right.

Amm0 · Fri Sep 27, 2024 5:23 pm

I guess I'm confused. Are you not able to get in after upgrade? Or does it just not work for LTE after upgrade?

To clarify my earlier answer:

Or is anyone using remote/difficult-to-reach-physically devices doomed to such issues?

As noted, the "winbox" client app using ethernet(layer2) so even if the config is FUBAR, if RouterOS boots you should be able to get using the MAC address. The defaults should have it enabled, but control via:
/tool mac-server set allowed-interface-list=all
/tool mac-server mac-winbox set allowed-interface-list=all
/tool mac-server ping set enabled=yes

If those were enabled, and you cannot get into the router...

There is an intermediate step before netinstall. You can reset it the default configuration by de-powering it, THEN plug it back in WHILE holding the reset button for ~7 seconds (>5 sec, <10 sec) - the reset button needs to be press when power is applied.

If you need to save then config, and cannot get in...then netinstall is needed. But I cannot imagine your config is very far from default, so the "reset to defaults" may be easier than netinstall.

But if you CAN get into the router and something is not right, just post the config here by using ":export" in the Terminal/ssh.

Amm0 · Fri Sep 27, 2024 5:29 pm

Also, since I think you have a hAP... this won't help now... but if you enable RoMON & the hAP was on same network as ATL, then RoMON be able to get into ATL via the hAP. It does require using winbox, where you connect to romon on the hAP, and assuming romon was enabled on ATL, winbox then show the ATL as an option connect (via the hAP "proxying" the winbox protocol).

And in the pantheon of ways to set it up BEFOREHAND for remote access, there is also "back-to-home"... so if you enabled that directly on the ATL, if LTE was up but LAN had issues... you could use VPN via BTH app/WG to get in too... Same with zerotier...

Anyway RouterOS has lots of options to do avoid a netinstall. Some do require setup before

utiker · Fri Sep 27, 2024 5:40 pm

Are you not able to get in after upgrade? Or does it just not work for LTE after upgrade?

What I can do:

Set up a network connection in NetworkManager manually with a static gateway 192.168.188.1 (the ATL) and client IP address 192.168.188.<anything>/24. Then I can connect to that connection (i.e. have the link up) but that's all - there is nothing I can actually use it for: I can't ping anything (even the gateway), I can't SSH to the gateway. So, this "possible connection" is very low layer. I am not proficient enough to explain it better, sorry.

As noted, the "winbox" client app using ethernet(layer2) so even if the config is FUBAR, if RouterOS boots you should be able to get using the MAC address. The defaults should have it enabled, but control via:
/tool mac-server set allowed-interface-list=all
/tool mac-server mac-winbox set allowed-interface-list=all
/tool mac-server ping set enabled=yes

If those were enabled, and you cannot get into the router...

Considering I have no access to the RouterOS of the ATL, I have no way to check. One thing is sure: after buying and configuring the ATL, I explicitly disabled all possibilities for connection to it and left SSH only. For security reasons.

There is an intermediate step before netinstall. You can reset it the default configuration by de-powering it, THEN plug it back in WHILE holding the reset button for ~7 seconds - the reset button needs to be press when power is applied.

That's my initial assumption. If I could get hands on the ATL, I can try this "factory reset" which you explain, then work my way up. The big question remains though - what about updating? (now, with the obviously buggy version, and in future) I surely don't want to engage into construction/deconstruction work just because the new software version obviously does not work.

My biggest concern with Netinstall is the security of the process. I don't quite understand how one should have security by downloading some proprietary piece of software, running it as root, without any firewall protection whatsoever and allowing external device to communicate directly with the Ethernet port of the PC. IOW, to restore the functionality of a device, one should expose even working systems to who knows what. (Yes, I distrust network infrastructure by default).

But if you CAN get into the router and something is not right, just post the config here by using ":export" in the Terminal/ssh.

I can get only to the hAP ac3. If you think we can see something from there - please let me know.

Amm0 · Fri Sep 27, 2024 5:46 pm

Anyway RouterOS has lots of options to do avoid a netinstall.

Just to complete the thread...

If you do get to needing a netinstall... you can run it as a container on the hAP. See https://hub.docker.com/r/ammo74/netinstall - this avoid all the setup required on Windows for netinstall Netinstall on Windows is just error prone since Windows security scheme really does not like the low-level networking things & one thing wrong, netinstall will not work. If you have a Linux box somewhere, that's better than windows to run netinstall too.

Now the fact it's already up on mast & reset button is far way, either the current "reset-to-default" or "netinstall" options are not going help solve getting it down part.

So I get your problem here... and RouterOS is complex so not easy the first time on any of this - but they do have a lot tools to deal with remote devices.

utiker · Fri Sep 27, 2024 5:56 pm

Amm0 · Fri Sep 27, 2024 5:58 pm

Sorry I was finishing my thread since I like to keep the options together

.

Reading your response. That seems like good news — If you can get into the ATL via ssh and 192.168.188.1 - there is no need for going to mast.

Next question be is the LTE connection working, since something there go wrong during upgrade in that part for sure...

And "/interface/lte [find] monitor" show that via ssh. If that's connected... then do an ":export" and cut-and-paste the configuration here. If you also wanted to run "/ip address print" and "/ip route print" and "/log print" that help too.

You may just consider a downgrade to 7.15.3, which involve using scp to copy the packages download from Mikrotik to router. But let's see what's going on with LTE first.

utiker · Fri Sep 27, 2024 6:34 pm

That seems like good news — If you can get into the ATL via ssh and 192.168.188.1 - there is no need for going to mast.

No, it's the opposite. Quoting myself:

I can't ping anything (even the gateway), I can't SSH to the gateway.

Amm0 · Fri Sep 27, 2024 7:14 pm

That seems like good news — If you can get into the ATL via ssh and 192.168.188.1 - there is no need for going to mast.
No, it's the opposite. Quoting myself:
I can't ping anything (even the gateway), I can't SSH to the gateway.

And you tried winbox to see if shows up as "Neighbor" with MAC address?

utiker · Fri Sep 27, 2024 8:16 pm

As I said, I use only SSH to connect to any of the routers and all other methods were intentionally disabled for security reasons.

Amm0 · Fri Sep 27, 2024 8:30 pm

As I said, I use only SSH to connect to any of the routers and all other methods were intentionally disabled for security reasons.

Gotcha. Well, then it's getting it off the roof/tower to reset it one way or another.

I'll note at some level, you can get too crazy locking these down...

infabo · Fri Sep 27, 2024 8:46 pm

netinstall-cli is for Linux. Give it a try.

utiker · Sat Sep 28, 2024 1:35 pm

Great news!
I can't believe it myself but we were able to unmount the ATL and I was able to Netinstall it. Now I can login, I have Internet etc. Still about to re-configure it.

Something strange though:

I assumed that I was supposed to turn off the ATL, then hold the Reset while powering it on, then I expected in about 15 seconds to see it in the netinstall tool. Something else happened though:

1. I turn off the ATL
2. I run the netinstall tool:

user@NETINSTALL:~/netinstall > sudo ./netinstall-cli -r -a 192.168.188.1 routeros-7.5-arm64.npk 
Version: 7.5(2022-08-30 09:34:59)
Will reset config
connect: Network unreachable
Using server IP: 0.0.0.0
Starting PXE server
Waiting for RouterBOARD...

3. I turn on the ATL while holding the Reset for 15 seconds, NetworkManager connects to it and I wait to see it in the console but it does not show up:

client: <MAC hidden for privacy reasons>
client: <MAC hidden for privacy reasons>
client: <MAC hidden for privacy reasons>
client: <MAC hidden for privacy reasons>
client: <MAC hidden for privacy reasons>
client: <MAC hidden for privacy reasons>
client: <MAC hidden for privacy reasons>
client: <MAC hidden for privacy reasons>
client: <MAC hidden for privacy reasons>
^C

4. So, I interrupt with CTRL+C and assuming that the ATL is already in Etherboot mode, I re-run the netinstall:

 
user@NETINSTALL:~/netinstall > sudo ./netinstall-cli -r -a 192.168.188.1 routeros-7.5-arm64.npk 
Version: 7.5(2022-08-30 09:34:59)
Will reset config
Using server IP: 192.168.188.2
Use Netmask: 255.255.255.0
Starting PXE server
Waiting for RouterBOARD...
client: <MAC hidden for privacy reasons>
Sending image: arm64
sendFile 9046272
Discovered RouterBOARD...
Formatting...
Sending package routeros-7.5-arm64.npk ...
Ready for reboot...
Sent reboot command
user@NETINSTALL:~/netinstall >

5. The ATL reboots and I am able to ping it, login to it with factory credentials, I have Internet too.

Questions:

1. Did I do everything correctly?

As you notice, I am using version 7.5. The reason for that is something important that I read in the ATL manual today (and which might have saved me all the troubles?):

The device supports RouterOS software version 7.5. The specific factory-installed version number is indicated in the RouterOS menu /system resource. Other operating systems have not been tested.

So, my second question is:
2. Considering this, should I even attempt to update now (thus potentially risking to end up in a situation when even Netinstall might not be able to help)?

utiker · Sat Sep 28, 2024 6:11 pm

Update:

I still tried to update to see what happens:

[admin@MikroTik] > /system/routerboard/print 
       routerboard: yes
             model: ATLGM
     serial-number: <hidden for privacy reasons>
     firmware-type: a3700
  factory-firmware: 7.8
  current-firmware: 7.11.2
  upgrade-firmware: 7.5
[admin@MikroTik] > /system/package/update/check-for-updates 
            channel: stable
  installed-version: 7.5
     latest-version: 7.12.1
             status: New version is available
[admin@MikroTik] > /system/package/update/install 
            channel: stable
  installed-version: 7.5
     latest-version: 7.12.1
             status: Downloaded 99% (13.3MiB)
Received disconnect from 192.168.188.1 port 22:11: shutdown/reboot
Disconnected from 192.168.188.1 port 22

$ ssh -l admin 192.168.188.1
ssh: connect to host 192.168.188.1 port 22: Network is unreachable

IOW, that brought me back to the state of not being able to connect to ATL. Fortunately, I was able to netinstall 7.5 again and the ATL works again.

New questions arise from that:

3. Why does system packages update disconnect and break everything at 99%?
4. How come current firmware is 7.11.2 and not 7.5?
5. How come factory firmware is newer than 7.5? Does it mean netinstall does not actually touch the firmware?
6. Why upgrade firmware is older? What is supposed to happen if one runs /system/routerboard/upgrade? How does one check for and get latest stable firmware? What if that breaks something again?
7. What would happen if I netinstall a version newer than 7.5? How to choose the actual version? Is there a risk to break everything in an unrecoverable way (considering the info in the manual)?

This is deeply confusing.

Amm0 · Sat Sep 28, 2024 6:47 pm

I'd netinstall 7.16, and try that. If there was some security patch, you'd be force to upgrade form 7.5 anyway & have these same troubles (except it could be potentially be back on roof) . If you troubleshoot 7.16 while it NOT on the pole, you'd be better set for future updates.

And, importantly, there are a gazzon updates in LTE from 7.5, so you'd being running to problems with LTE that are likely now fixed.

Mikrotik is not always the best at updating the docs... so someone just forgot to taking out the note about "only testing on 7.5" reference. Since the "factory-firmware" is 7.8, clearly they bump the version in manufacturing. You may have followed the docs advice off a cliff (or roof, I guess).

utiker · Sat Sep 28, 2024 7:04 pm

I'd netinstall 7.16, and try that.

Based on what? The manual is clear - 7.5. What is there to try? I mean - I already tried a regular update... and you see what happens. What is the guarantee that if 7.16 doesn't work I will be able to netinstall 7.5? Yes, it is possible that the doc is outdated. But it is also possible that it is not. So far, considering the facts, it kind of seems the latter.

If you troubleshoot 7.16 while it NOT on the pole, you'd be better set for future updates.

That's exactly what I thought too. I do care about security updates. The thing is, I am quite concerned not to end up with a blocked device in unrecoverable mode. With dysfunctional hardware the concept of security is meaningless.

I really wonder how to approach this safely. Support still doesn't even reply.

Amm0 · Sat Sep 28, 2024 7:58 pm

I'd netinstall 7.16, and try that.
Based on what? The manual is clear - 7.5. What is there to try?

Well... a decade of knowing Mikrotik is not great at updating documentation.

And, you have a pretty locked down router, so if you have security needs... there no security patches/hotfixes/etc in older version. Security fixes are applied by going to latest version in the "stable" channel. You're WAY over reading the docs.

And yes you'd be able to downgrade to 7.5 from 7.16 using netinstall. Or any from/to version, it reformats the "disk".

You're really worried about the wrong things if you want this working+stable, and secure. 7.5 is not the way to go. If 7.16 didn't work, try 7.15.3 first (and report a bug that 7.16 did not work). If you report a problem with 7.5, the first thing support will tell you is to upgrade (and when you point out the docs, they'd likely just delete that part form wiki. Nothing stops you from opening a case at help.mikroitk.com and report your problem with 7.5 & see what they say

.

utiker · Sat Sep 28, 2024 8:21 pm

Well... a decade of knowing Mikrotik is not great at updating documentation.

I agree. It is not great as a whole but that is off-topic.

And yes you'd be able to downgrade to 7.5 from 7.16 using netinstall. Or any from/to version, it reformats the "disk".

I understand that it reformats and so on. The question is can some ROS version break that reformat-and-so-on functionality. I don't see that documented anywhere.

You're really worried about the wrong things if you want this working+stable, and secure.

Well, that worry is based on current experience. The ATL is not cheap either.

Nothing stops you from opening a case at help.mikroitk.com ...

As mentioned, I have already emailed support, which I believe is the same, as I got an autoreply from Jira with a case ID.

infabo · Sat Sep 28, 2024 8:31 pm

/system/resources/print

What is shown as you factory version?

utiker · Sat Sep 28, 2024 8:36 pm

@infabo

factory-software: 7.4.1

I have no idea what to use this information for though.

infabo · Sat Sep 28, 2024 8:50 pm

If you don't believe Ammo: visit https://mikrotik.com/product/atl18#fndtn-downloads Product page, downloads section and there is Router OS package linked.

I would update to 7.16 as well, as long the device is unmounted. In case of a defacto factory configuration I would go even further: netinstall 7.16 with reset. Tabula rasa is sometimes a cure for acting weird devices. And dying after an update is weird and not the regular case. Default configuration may changed as well between 7.5 and 7.16. At least some minor firewall changes I am aware of.

/interface/lte/firmware-upgrade

may be a good idea as well.

Amm0 · Sat Sep 28, 2024 9:03 pm

Netinstall problems are rare. Well, other than not be able to run it because of OS security block DHCP/TFTP (aka windows), or doing the "reset button dance" incorrectly.

More relevant than "factory-*" version, at least to netinstall, is current-firmware= in /system/routerboard. That is the boot loader (RouterBOOT) and that's what interacts with netinstall. And so your report about is telling:

/system/routerboard/print 
       routerboard: yes
             model: ATLGM
     serial-number: <hidden for privacy reasons>
     firmware-type: a3700
  factory-firmware: 7.8
  current-firmware: 7.11.2
  upgrade-firmware: 7.5

So netinstall will actually be working with the 7.11.2 firmware, so that should be fine. But if something/somehow does go wrong with netinstall, and you really brick the router... Which again is very unlikely — that's when you can likely exchange it with Mikrotik since they do require someone try netinstall before they accept a return.

Where there might be a slighter greater potential for trouble with netinstall is if you're trying to use a very old 6.x.x firmware/RouterBOOT to go to a recent V& – and even you'd have to go back to a pretty old V6 for that to happen. But V6 was never supported on the ATL, so there is no chance of that happening.

utiker · Sat Sep 28, 2024 9:06 pm

@infabo

What did the factory version tell you in relation to the rest of the info you kindly provided?

It is not that I don't believe Ammo - I do. I am just extra careful.

I would update to 7.16 as well, as long the device is unmounted. In case of a defacto factory configuration I would go even further: netinstall 7.16 with reset.

Update (through /system/package/update) is obviously not possible, as I can't reach 7.16 through sequential step-by-step version updates and reboots. As for netinstall 7.16 - the question remains: Can this break things in a way resulting in impossible netinstall of older version afterwards.

Re. /interface/lte/firmware-upgrade - How does this differ from /system/routerboard/upgrade and should it be run:

- after downloading newer packages but before rebooting
- after downloading newer packages and after rebooting
- any other way?

FWIW, as I am typing this, I am trying download instead of install:

[admin@MikroTik] > /system/package/update/download 
            channel: stable
  installed-version: 7.5
     latest-version: 7.12.1
             status: Downloaded, please reboot router to upgrade it

[admin@MikroTik] >

I will report right after reboot (or after another netinstall). Fingers crossed.

Amm0 · Sat Sep 28, 2024 9:17 pm

This has gotten silly. I don't know what to tell you. You should run latest LTE firmware, RouterBOOT and RouterOS to start and troubleshoot any issues from there.

FWIW... now you've likely downgraded the RouterBOOT firmware to 7.5. And if there was somehow fix for netinstall in the firmware between 7.5 and 7.11 - you just opened yourself up to it. I don't think there is, so likely not problem.

It is extremely hard to actually brick the router so that a netinstall would not work. Could it happen, sure, but that's a different support case & Mikrotik would likely exchange it if truly bricked from even netinstall. Similar with RouterOS version, while there might be some bug 7.16 (it is new)... your odds are way higher there a lot of bug in 7.5 (that were fixed) that you run into.

utiker · Sat Sep 28, 2024 9:26 pm

Amazing. With "download" it worked! I updated and rebooted. Then I updated the firmware using /system/routerboard/upgrade.
Now I am getting:

[admin@MikroTik] > /system/package/update/check-for-updates 
            channel: stable
  installed-version: 7.12.1
     latest-version: 7.16
             status: New version is available

[admin@MikroTik] > /system/package/update/download          
            channel: stable
  installed-version: 7.12.1
     latest-version: 7.16
             status: Downloaded, please reboot router to upgrade it

Rebooted successfully, upgraded firmware to 7.16 too, rebooted again and everything works:

[admin@MikroTik] > /system/routerboard/print   
       routerboard: yes
             model: ATLGM
     serial-number: ...
     firmware-type: a3700
  factory-firmware: 7.8
  current-firmware: 7.16
  upgrade-firmware: 7.16
[admin@MikroTik] > /system/package/update/check-for-updates      
            channel: stable
  installed-version: 7.16
     latest-version: 7.16
             status: System is already up to date

I don't know what to say. Of course I thank everyone for your time!

The question is - is "download" vs "install" the key to safe updates? What did "install" do at 99% which "download" did not? Besides that, I have no explanation.

infabo · Sat Sep 28, 2024 10:16 pm

@infabo

What did the factory version tell you in relation to the rest of the info you kindly provided?

You insisted to use ROS 7.5. If your factory version would have been already a newer version, like e g. 7.6, then you wouldn't even be able to install 7.5. You can't downgrade below factory version.

Interesting is: your factory routerboard firmware is 7.8 but your factory ROS version is 7.4.6. don't know how this is even possible

And finally don't forget /system/routerboard/upgrade

Amm0 · Sat Sep 28, 2024 10:33 pm

Good news. Sorry if I sounded short, but 7.5 was just a bad idea. And totally get it was some Mikrotik that who somehow broke it, to require the roof + netinstall. And, clearly MT should remove that "7.5" reference (or update the page so highlight that someone re-confirmed it

) to avoid this problem in future. Since the docs are ~95+% right, someone might actually believe that part is also right

.

The question is - is "download" vs "install" the key to safe updates? What did "install" do at 99% which "download" did not? Besides that, I have no explanation.

Download does just that. It download the package files to the flash, but does not install them. The way packages work is if any are in the root file system when you reboot, they get applied. So you can download them yourself from mikrotik.com, and put the in the root to also upgrade a router. So the "download" essentially just automates that for the channel (i.e. "stable").

The install does the download, and also triggers the install, by rebooting. But as soon as it finish downloading, it also reboots the router to use same .NPK files in root. Since that reboot happens immediately after download, the 99% may show because it started the reboot before the UI update made it to you.

Finally, If LTE is working, you might have done enough upgrades for one day. But it might be worth it to upgrade the LTE modem firmware too. https://help.mikrotik.com/docs/display/ ... areupgrade - which ironically has the note:

Before attempting an LTE modem firmware upgrade - upgrade RouterOS version to the latest releases How To Upgrade RouterOS

... and the HOWTO upgrade link explain how packages work.

Basically the docs are reference manual, not a guide. So the information need to just setup is spread across a lot of pages.

Amm0 · Sat Sep 28, 2024 10:50 pm

And in this post, there are the CLI commands for upgrading LTE things:
viewtopic.php?t=199087&hilit=band+66#p1025119

utiker · Sun Sep 29, 2024 11:21 am

@infabo

Thanks for answering my question.

Just for clarity:

You insisted to use ROS 7.5.

I only pointed out that the manual said no other OS had been tested with this device. I definitely do not insist on using stale software - obviously, otherwise I wouldn't even end up in the current situation. The thing is I postponed updating the ATL for quite a few months, as I was busy with other things and if such unfortunate unmounting of the pole would have been necessary, I would have no physical possibility to do it (or to have the necessary help for it). IOW, the choice was between having no Internet access for a few months vs staying with somewhat outdated software. So, about a year later, it turned out that my postponing was quite well reasoned, after all.

Now, the bigger question is: what update strategy to develop after mounting on the pole again? In winter (which is quite a few months) it is absolutely impossible to unmount in case of failures. So, I suppose it may be safer not to touch during that time, i.e. stick to current strategy and postpone again. I wish Mikrotik's software was made more intelligent to e.g. auto-revert to older version in cases of such glitches.

@Amm0

Thanks a lot for the extra info.

I assumed (obviously wrongly) that /system/routerboard/upgrade updates all firmware, including that of the LTE modem. Now, after the info you provided, I learned that it is not so. So, I upgraded that one too and after rebooting everything seems to work fine.

I did not do the extra steps in regards to APN, as I don't understand why this is necessary and what it actually does.

Since that reboot happens immediately after download, the 99% may show because it started the reboot before the UI update made it to you.

It is logical to assume that "install" works that way, i.e. "install" = "download" + "reboot". The facts prove the result is not equivalent at all though. So, I am inclined to think there is something buggy in the "install' command. ROS being closed-source makes it impossible to investigate further.

utiker · Sun Sep 29, 2024 1:27 pm

I wish Mikrotik's software was made more intelligent to e.g. auto-revert to older version in cases of such glitches.

I just discovered this:
https://help.mikrotik.com/docs/display/ ... tetherboot

Now, the question is security: What would prevent an attacker with physical access to the cable to netinstall the device and do further mischief?

Potential solution:
1. Enable preboot-etherboot before updating.
2. Update and reboot
3a. If everything works after updating: disable preboot-etherboot. Done
3b. Else - netinstall. Go to 2.

What do you think?

mkx · Sun Sep 29, 2024 1:41 pm

Now, the question is security: What would prevent an attacker with physical access to the cable to netinstall the device and do further mischief?

Physical security. As soon as attacker gains physical access to your device, you've already lost the game. Guess how "denial of service" looks like in this case? Attacker simply steals your device.

Having possibility to do netinstall is an essential means of re-posessing device if remote attacker manages to lock yourself out (e.g. by removing admin permissions from your administrative user) ... and that's way more probable than somebody climbing on your pole in the middle of winter and do funny stuff to your ATL. Another probable thing as you discovered yourself is boot loop (and it's not always result of manual action) ... so no way of getting into ROS to toggle the netinstall allowed thingie before doing netinstall to recover from boot loop.

utiker · Sun Sep 29, 2024 2:48 pm

Physical security. As soon as attacker gains physical access to your device, you've already lost the game. Guess how "denial of service" looks like in this case? Attacker simply steals your device.

Perhaps only armed guards can prevent that.

Fortunately, my threat model does not need that.

Having possibility to do netinstall is an essential means of re-posessing device if remote attacker manages to lock yourself out (e.g. by removing admin permissions from your administrative user) ... and that's way more probable than somebody climbing on your pole in the middle of winter and do funny stuff to your ATL.

Hence the proposed solution. Perhaps I should add as a very first step:

0. Check there is no physical attacker near the cable

Another probable thing as you discovered yourself is boot loop (and it's not always result of manual action) ...

What do you mean I discovered? Are you saying that my trouble was the result of a boot loop? If that was so, I think I would be experiencing repeated connect/disconnect in NetworkManager and I did not notice anything like that. Or what do you mean?

so no way of getting into ROS to toggle the netinstall allowed thingie before doing netinstall to recover from boot loop.

The doc says it is a BIOS setting:

"Also, note that RouterOS reinstall does not affect BIOS settings, and preboot-etherboot would still be enabled and would try to enter etherboot."

Unless I am misreading this, it means that the BIOS will always try netinstall first before even attempting to boot the OS (and potentially boot loop). No?

utiker · Sun Sep 29, 2024 10:44 pm

viewtopic.php?t=211344

Who is online