Hi,

My HAP ac2 running on 7.13 is about to run out of space. Is Mikrotik aware of that? Maybe it would be a good idea to exclude ppp and routing from the main package bundle?

This was discovered long ago (as soon as 7.13 came out). I certainly hope MT reads the comments in release topic they created themselves ... and I hope will do something about it.

Beware that when space runs out, you won't be able to change configuration. And device might become unstable (rebooting for no actual reason). Or anything else.

It seems that the hAP ac2 is the most affected by this problem.
Maybe part of the issue is that the flash isn't 16MB to start with, it is only 15.3MB on that device.
(no idea why, it could be due to limitations of the processor, memory-mapped IO, boot code, whatever)

OF COURSE MikroTik know about this! But they are in a state of denial.
I have written my concerns about 16MB flash devices many times, but the only reply you can get is
like "we make sure it fits, we test every release with default config and it always is OK" etc.

The 7.13 version makes it worse because now the "old" wireless config has been made a separate package while
the "new" code is still a mandatory part of the main package. So now you have 2 config menus, one for
the old and one for the new, again taking up more space. Which already was nearly 0 in 7.12 and below.

Still, for those that have a hAP ac2 and do not want to migrate to the new WiFi, I would recommend staying
at 7.12.1 and not upgrade until that becomes absolutely necessary.

Is a downgrade possible from 7.13 to 7.12.1?

Maybe part of the issue is that the flash isn't 16MB to start with, it is only 15.3MB on that device.

I think it's the matter of Mebi vs Mega. 16*10⁶ Bytes=15.258789 MiB.

The thing is that storage producers use MB and MiB interchangable and don't tell which is it. And MT is to blame here as it reports capacity using wrong unit as well.

Is a downgrade possible from 7.13 to 7.12.1?

It is possible. But you have to take extra care of uploading appropriate packages. Also beware that some settings you may have done in 7.13 won't be backported to 7.12 (because config converter in 7.12 isn't aware of certain new features, such as renaming /interface/wifiwave2 to /interface/wifi).

Beware that when space runs out, you won't be able to change configuration. And device might become unstable (rebooting for no actual reason). Or anything else.

The last time it ran out of space it didn't come up after the reboot. So one needs to be careful not to run out of space because it effectively bricks the device.

Maybe part of the issue is that the flash isn't 16MB to start with, it is only 15.3MB on that device.

I think it's the matter of Mebi vs Mega. 16*10⁶ Bytes=15.258789 MiB.

It isn't. One does not even have to know what the exact unit is, just by looking at System->Resources in the field "Total HDD Size" it will show 16MB in almost all models but it shows 15.3MB in the hAP ac2.
So it does not matter if it is MiB or MB in reality, it is just less.

About downgrade: it depends on what kind of configuration you have.
When you are just using it as a basic home router with NAT and WiFi and no special config I would recommend to just do a netinstall without keeping configuration, and start completely fresh from defaults.
You can do a "/export show-sensitive file=filename" and download the filename.rsc file to your computer before doing that, and have the filename.rsc alongside in an editor while doing the new configuration (so you do not forget things).
If desired, you can also copy/paste some section of the export to your new setup, like a VPN setup or similar.

When the config is complicated, you could do a netinstall with "keep config" or you could download the 7.12.1 npk file and upload it to the router, and also in the packages section do a "remove" on the wireless or wlan package of 7.13, then click "downgrade".
But the abovementioned netinstall and fresh start method will give you the most free space.

Maybe part of the issue is that the flash isn't 16MB to start with, it is only 15.3MB on that device.

I think it's the matter of Mebi vs Mega. 16*10⁶ Bytes=15.258789 MiB.

The thing is that storage producers use MB and MiB interchangable and don't tell which is it. And MT is to blame here as it reports capacity using wrong unit as well.

Would that not constitute as false advertising?

eg. "Routerboard that has 16MB Flash storage" when in reality it's 15.2Mib.

D53G-5HacD2HnD is having the same issue - only 596 KiB reported free after upgrading to v7.13, total storage is 15.3MiB, on the MT's Product page on website its showing MB

And I have written several times my concerns about integrating the wifi module into the default package. why they opted for this I can't understand without insights into their architecture. extract the whole wifi part as extra package and all be good again.

Indeed!
Also I do not understand why they made everything one routeros package.
I can understand why they merged packages like security, dhcp, ppp and ipv6 from the v6 version in to the main package: there are just too many inter-dependencies between features all over the place and those specific packages.
But why "routing" (BGP/OSPF/etc), "hotspot", and "mpls" were put in the routeros package, I do not understand.
Probably 95% of the users is never going to use those, now that they focused so much on home routers.
These should have remained optional packages...

@t0mm13b https://www.gbmb.org/mb-to-mib Afaik any storage device is advertised in MB/GB/TB sizes, not in MiB/GiB/TiB.

That is wrong. Internal memory is in MB but "external" storage like flash is always in MiB.
When you buy a "16MB flash card" it will be 16000000000 bytes, not 17179869184.
That would make it 14.9MB instead of 16MB.
However, that is all that is going on here. As mentioned, the hAP ac2 reports less space than all the other 16MB models.

Someone marked this topic as SOLVED although it's NOT.

I think only the original poster (YOU) can mark it "solved".
Anyway, did you try a netinstall to 7.12.1 ?

I haven't tried netinstall option yet. I'm waiting for Mikrotik's response on how they're planning to address this problem. 7.14beta uses even more space than 7.13 so eventually, more and more users will be affected by this problem unless a sustainable solution is found.

Worst case scenario I will be forced to upgrade from ac2 to ax2. Maybe that's what business people from Mikrotik want.

I do not expect that they will provide you with a useful reply or solution.
As I wrote before, concern has been expressed about decisions and about the ac2 and the only response is denial.

Worst case scenario I will be forced to upgrade from ac2 to ax2. Maybe that's what business people from Mikrotik want.

Maybe MikroTik can start a sort of exchange program where they offer minimum 50% discount to entice people to upgrade from ac2 to ax2 if they swap their hAP ac2 for ax2

...wishful thinking !

...or offer to purchase ROS compatible flash memory to replace it on board

In case of the ac2 it could even have been an option to install a USB stick that can offload part of the storage requirement...

But why "routing" (BGP/OSPF/etc), "hotspot", and "mpls" were put in the routeros package, I do not understand.
Probably 95% of the users is never going to use those, now that they focused so much on home routers.
These should have remained optional packages...

That's the point. Many of MTs recent wifi AX products focus on SOHO.

Maybe basic routing stuff should remain like Filters, Nexthops, RouterID, Rules, Settings and Tables but other parts as part of extended routing optional package and also Dot1X and RADIUS can be optional.

Hi! I hope someone from MT is aware of this issue. I´d like to add, that my WAPac (ARM Version, RBwAPG-5HacD2HnD, r3) is also affected.
After upgrading to 7.14b4 516 KiB are left from 15.3MB.
Only routeros and wifi-qcom-ac are installed. Everything is working until now, but I don´t know for how long...

I had to remain on legacy wireless package to have container package because it not fit with wifi-qcom-ac and my config.

Maybe basic routing stuff should remain ...

None of routing stuff should remain in basic package, none are used by vast majority of SOHO users (who are primary users of infamous 16MB flash ARM wifi devices). And that includes dot1x, radius and any other (mostly) corporate stuff.

IMO running containers on 128MB RAM devices with wireless (hAP ac2, cAP ac, etc. nominally have 128MB RAM) is silly.

Such exercises look just like those photographs from 3rd world showing guys transporting mountains of goods on a moped. Yes, somehow some manage to fo it, but nobody in their right minds are requesting from moped producers to officially support that.

I'm running containers on Chateau LTE12 which has 256MB RAM size but just 16MB flash, have no problem with RAM while containers are running, constantly 2 are running as DNS service (Pi-hole and Unbound - recursive) and periodically starting some others, so far RAM is not issue but flash is.

Afaik for basic routing setup you don't need additional tools on linux so there is no extra space to gain by just removing options. Maybe better to separate VPNs into separate packages and then let users to decide which will install for their needs. Also some supporting drivers for devices that are not part of MT hardware can be also optional packages, like external LTE modems support as bundle package, etc...

I'm running containers on Chateau LTE12 which has 256MB RAM size but just 16MB flash, have no problem with RAM while containers are running, constantly 2 are running as DNS service (Pi-hole and Unbound - recursive) and periodically starting some others, so far RAM is not issue but flash is.

Afaik for basic routing setup you don't need additional tools on linux so there is no extra space to gain by just removing options. Maybe better to separate VPNs into separate packages and then let users to decide which will install for their needs. Also some supporting drivers for devices that are not part of MT hardware can be also optional packages, like external LTE modems support as bundle package, etc...

I do hav PIhole/unbound running off pi which is plugged directly into ethernet port.

what's your experience like with running containers on Chateau (owner of same device here... ) ?

what's your experience like with running containers on Chateau (owner of same device here... ) ?

Positive after some ROS version upgrades, at beginning was troubles due to some bugs related to containers.
I had to tweak configurations to have small RAM usage, on Pi-hole DNS cache is disabled since it's upstream is Unbound which owns cache and Unbound is tweaked for RAM and cores usage. Containers config has ram-high=80M and it's enough for me.
DNS performance mostly depends on network quality here (which over LTE isn't always best).

There is a solution without cutting features. I own hAP ac2 and out of curiosity I looked at drivers section of ARM RouterOS. There is for example prestera_dx_max.ko file taking 687 KB of space. Then quectel_mhi.ko of 148 KB. These drivers are not needed for hAP ac2. How about making a list of drivers (files?) not needed for each device. Then during install procedure, installer using this list, would not install unnecessary files onto "precious" 15.3 MB flash. I'm not sure how simple that would be as the npk file is compressed and it would need to be repacked in RAM before install. Also space savings would be less than 687+148 KB due to compression. But probably more than these two files are unnecessary for hAP ac2, so maybe it would do the job.
Another solution would be allowing to install optional packages onto (encrypted?) partition on USB drive. Moving wifi-qcom-ac there would save almost 3 MB.

The issue probably is that .npk packages are mostly a read-only compressed filesystem that is copied to the flash in its entirety and then are mounted at runtime (with the appropriate code to de-compress files on access).
So, it is not a simple matter of "during installation see which drivers are required and delete the remainder", as there is no support to delete files in the compressed filesystem.
If a .npk would be just like a .deb or .rpm (a compressed archive that is being unpacked during install) it would be easy.

Some drivers are shared by a lot of things, there is no separate driver for each chip or device. One big driver.

Firmware is per chipset.

It isn't

Whatever is the situation, something has to be done. Denying the problem no longer works.

Our team of experts is working on this issue at this very moment. As to "why", many products were designed years ago, but only are released now. When designing devices with 16MB SPI chips, at that moment there was no issue yet. Today we are designing devices with more storage, but they will reach customers after "some time" as you can imagine.

Don't tell us you were not aware, and were not warned, about possible space issues on 16MB flash devices!
It was discussed on the forum from the time those were released. Remember the original MikroTik products usually had 64MB or 128MB flash.

Our team of experts is working on this issue at this very moment.

Thanks for the update. I know it's not an easy problem to "just fix". But the 16MB problem has been on a slow simmer for "some time" .

It isn't

I do not understand. What else?

Quick question,

I recently noticed severe internet problems at home (using a hAP AC2).
It's very slow, connection drops often, and basically unusable.

I thought it's the cable modem, but a direct connection is fine.
I then found out that there is almost no disk space left:

free-hdd-space: 952.0KiB
total-hdd-space: 15.2MiB

Could this cause these kinds of problems?
I have not found any other problems in the config.
I've attached it if anyone wants to have a look.

If it's really caused by the lack of memory, I'll replace it...

Thanks for your help!

i am very happy with the performance of wifi-qcom-ac package on hap ac2 on 7.14.3

but dont have free memory anymore

If it's really caused by the lack of memory, I'll replace it...

Netinstall and applying exact same config might also help.
(NO restore, apply config from earlier made export)

i am very happy with the performance of wifi-qcom-ac package on hap ac2 on 7.14.3

but dont have free memory anymore

hapac2-mem.jpg

Hm weird, no performance issues?
I have no idea what else could cause it...

i am very happy with the performance of wifi-qcom-ac package on hap ac2 on 7.14.3

but dont have free memory anymore

hapac2-mem.jpg

Hm weird, no performance issues?
I have no idea what else could cause it...

no, but
i disabled graphing, no disk logging , i use a very simple config, i only use it as a wireless access-point, only fast-path bridging, no tracking, no rules etc

free-hdd-space: 952.0KiB

I have even less (440 KiB) and it's not affecting performance, create topic with issue description and full config, maybe someone will help.

free-hdd-space: 952.0KiB

I have even less (440 KiB) and it's not affecting performance, create topic with issue description and full config, maybe someone will help.

Thanks! Weird... I've created a thread.

i am very happy with the performance of wifi-qcom-ac package on hap ac2 on 7.14.3

but dont have free memory anymore

hapac2-mem.jpg

Wait for 7.15 (or install the RC). It frees up some disk memory. That's why my AC devices still run on 7.13.5 - because 7.14 consumed too much disk space.

What helped in my case is reinstalling RouterOS with netinstall. It somehow freed some space that was previously unavailable. Upgrade to 7.15 reduced space usage even further. So problem solved for now.

Yes, netinstall helps. Because the RouterOS configuration is in a kind of "database" which tends to expand over time.
E.g. when you add some config and then remove it, the space remains occupied in the flash. Also because you can "undo" the remove.
Besides that, when you have v6 on a device and upgrade to v7, a new database is created for v7 but the v6 one is kept.
(so when you downgrade to v6 it still has its original configuration)
This all gets cleaned up when you do netinstall. But you can also export the config, reset the configuration, and import it again.

Anyway, in version 7.15 apparently some optimization was done and again about 1MB of space was found.
Incredible that they still can find so much space, maybe a new compression method or so.
For now, the hAP ac2 (the most-used device that had real problems) again works well.
Of course the problem will come back in the future. Make sure you always read the forum before upgrading to a new major release.

What keeps me amazed is how much functionality they manage to squeeze into 15MB software package.

Hopefully this problem won't come back for at least few more years. Then I'm fine to upgrade to ax2 or whatever will be Mikrotik's latest-greatest device.

Besides that, when you have v6 on a device and upgrade to v7, a new database is created for v7 but the v6 one is kept.
(so when you downgrade to v6 it still has its original configuration)

For how long is the previous version's configuration kept? For example, if I decide to downgrade a year later, will it resurrect a year-old config from v6? Is there any way to clean out the old configuration remotely?

P.S. What about the autosupout.rif file? Could it be created after a crash and fill up the entire flash memory, leading to a boot loop?

Few notes here since this whole 16MB flash comes up in my world...

- The only thing that truly cleans things is doing netinstall. And, if want to run wifi-qcom-ac, IMO, you should go through the trouble of netinstall.

- The amount of free space even if clean can still vary because not everyone is going to have the same # of bad blocks. I've seen swings of 100+KBs between 16MBs units, so possible one unit runs same config+packages and another cannot. For example, the routeros+wifi-qcom-ac+zerotier package combo puts the device <1MB from being full. So it's down to how many many bad sectors there are on whether that will combo will work (and even nothing is bad, it's still close). So if you want zerotier+wifi-qcom-ac on hAPac2... you absolutely need to do a netinstall.

- I do get netinstall is PITA. Until wifi-qcom-ac, netinstall was very infrequent thing for me. I'll offer the Linux netinstall or a RouterOS container (on another router) is often easier than Windows, but still an elongated process - hopeful once to purge V6-config/past-upgrades/graphs/etc. But since netinstall re-formats the disk - it is the ONLY way to be sure all files are removed.

- RouterOS goes to memory, so even if you have no disk space, it can still generally run. Failed upgrade (& often because of being close on disk space) is how you end up in generally bootloop, and again only fix is netinstall if that happens. But in most recent V7, it is not as common as when this thread started. If you do run out of disk space after upgrade, most likely problem is save the config is lost. In fact, I have a 16MB wAPacR that's sitting on my disk awaiting a netinstall because it has 0 disk space – device works fine — but since I cannot save anything, I cannot command a downgrade (since that requires a config write) thus the familar refrain: netinstall.

- Most people want downgrading, in general, to work - so they leave the needed bit... But RouterOS does not know when/what you might want to downgrade. Since we're talking about KBs, that make sense - except on 16MB flash things... But again a netinstall will clean up all of the actual+theoritical files that could exist and start fresh - so that the recommend way of removing the option+files needed for downgrade.

- Since you generally only run into disk space issues if you want to run wifi-qcom-ac – while that's better – it's just a strain with the 16MB flash. So even if everything clean/netinstall'ed, there still not a lot of breathing room. While still quite a good device... the hAPac2 are getting older... so another option is to get some newer AX wifi AP & remove ANY wifi package from hAPac2 so it end it's life as just a router. Or, since it has USB, it be make a fine Dude server for some new hAPax3/2. Or, even container fit, if there is no Wi-Fi, so it run HomeAssistant/Pi-Hole or any of RPi-kinda things on hAPac2 containers. Just something to consider here. These 16MB units are still pretty close to be maxed on disk with the recent V7s & that ain't going to get better over time...