Very high sector writes

arielgrin · Thu Jul 04, 2019 2:48 pm

Hi All: I'm running RouterOS 6.43.16 (long-term) on RB751g-2HnD and I'm seeing more than 7000 sector writes per day, I've opened webfig and checked the System->Resources menu and I can see how it writes 32 sectors every 6 minutes, approximately. Not only that, it almost reached 400000 sector writes after only 30 days last month, which I think it might be excessive, as nothing is supposed to be writing to disk at all. Just in case, I reset the router to default configuration and disabled DHCP writes to disk, there are no graphs running and logging is to memory only, but anyway the 32 writes every 6 minutes continue to happen, with over 7000 write after just 24 hours. This is my configuration in case anyone sees what could be causing this, unless it might be a bug.

/interface bridge
add admin-mac=D4:CA:6D:51:FB:51 auto-mac=no comment=defconf name=bridge
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-XX distance=indoors frequency=auto ht-supported-mcs=\
    mcs-0,mcs-1,mcs-2,mcs-3,mcs-4,mcs-5,mcs-6,mcs-7,mcs-8,mcs-9,mcs-10,mcs-11,mcs-12,mcs-13,mcs-14,mcs-15 mode=ap-bridge ssid=MikroTik-51FB55 wireless-protocol=802.11
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp ranges=192.168.8.21-192.168.8.100
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge lease-time=1d name=defconf
/tool user-manager customer
set admin access=own-routers,own-users,own-profiles,own-limits,config-payment-gw
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=wlan1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/ip address
add address=192.168.8.1/24 comment=defconf interface=ether2 network=192.168.8.0
/ip cloud
set update-time=no
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=ether1
/ip dhcp-server config
set store-leases-disk=never
/ip dhcp-server lease
add address=192.168.8.4 client-id=1:40:8d:5c:96:c1:24 mac-address=40:8D:5C:96:C1:24 server=defconf
/ip dhcp-server network
add address=192.168.8.0/24 comment=defconf gateway=192.168.8.1 netmask=24
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.8.1 name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
/system clock
set time-zone-name=America/Argentina/Salta
/system lcd
set contrast=0 enabled=no port=parallel type=24x4
/system lcd page
set time disabled=yes display-time=5s
set resources disabled=yes display-time=5s
set uptime disabled=yes display-time=5s
set packets disabled=yes display-time=5s
set bits disabled=yes display-time=5s
set version disabled=yes display-time=5s
set identity disabled=yes display-time=5s
set bridge disabled=yes display-time=5s
set wlan1 disabled=yes display-time=5s
set ether1 disabled=yes display-time=5s
set ether2 disabled=yes display-time=5s
set ether3 disabled=yes display-time=5s
set ether4 disabled=yes display-time=5s
set ether5 disabled=yes display-time=5s
/system logging
add topics=debug
/system package update
set channel=long-term
/system watchdog
set watchdog-timer=no
/tool graphing
set store-every=24hours
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool user-manager database
set db-path=user-manager

mkx · Thu Jul 04, 2019 2:54 pm

My guess: user manager ... try to attach an USB flash disk and configure user manager to store its database to USB stick.

arielgrin · Thu Jul 04, 2019 9:21 pm

Thanks @mkx for your reply.

I have disabled user-manager package, as I don't need it. I rebooted the router and will check again after a couple of hours to see if the writes have stopped.

Do you happen to see anything else that could be a cause for writes? User manager has been always enabled, but never used, and this write frequency just increased in the last couple of months, it was not like this before.

arielgrin · Thu Jul 04, 2019 10:41 pm

Indeed it seems it is not user-manager, or at least not only that. The router has been running for less than an hour and there are already 650 writes logged.

Pea · Thu Jul 04, 2019 10:42 pm

Many DHCP leases? Try:

/ip dhcp-server config set store-leases-disk=never

Edit: I see you have this already, so it must be something else...

mkx · Thu Jul 04, 2019 10:45 pm

I wonder if this setting

/ip dhcp-server config
set store-leases-disk=never

really disables writing to flash. Try to set it to something like 6h and see if it makes any difference.

arielgrin · Fri Jul 05, 2019 1:13 am

I will try changing the setting. I thought that store-leases-disk=never was the way to go to avoid leases being written to disk, but I will try other setting. In any case, there are only 5 leases and the lease time is 1 day, so the refresh frequency is really low, once every 12 hours for just 5 leases. The router has been running for 4 hours and it already has more than 2000 sector writes.

mkx · Fri Jul 05, 2019 9:16 am

I thought that store-leases-disk=never was the way to go to avoid leases being written to disk

It should ... but there's always room for some bugs

arielgrin · Fri Jul 05, 2019 1:20 pm

Well I changed the setting to store the leases every 24 hours, but the writes continue to raise. The router has been running for 12 hours and there are more than 4000 writes already. I don't know what else to check or do, in less than 2 months there have been more than 1 million writes. I think that is definitely excessive.

arielgrin · Fri Jul 05, 2019 10:09 pm

Router has been running for 36 hours and there are more than 20000 sector writes. I don't know what else to do. Should I contact support? I don't have a support contract or anything like that.

sindy · Fri Jul 05, 2019 10:28 pm

As you have already tried a reset to default configuration, I would try to export (not backup) the configuration into a file, download the file to PC, netinstall the router, and check again with default configuration. If the ghost writes stop appearing, I'd re-import the configuration, otherwise I'd use some other RouterOS release (e.g. 6.43.15, as 6.43.16 only fixes some 60 GHz related issue).

support@mikrotik.com does accept information about well-documented real bugs even without a support contract, but they cannot deal with "how do I change the default IP address" class of questions, that's why they suggest this forum as the primary support channel and refer to the support from your reseller.

2frogs · Sat Jul 06, 2019 3:25 am

Code: Select all
/system logging
add topics=debug

Have tried disabling this?

mn62 · Sat Jul 06, 2019 6:52 pm

All versions of RouterOS above 6.34.6 in the old line of devices from MikroTik cause an increased amount of write to flash memory.

arielgrin · Sun Jul 07, 2019 4:25 pm

That setting is no longer active, the router has been reset to factory defaults, but the writes keep on raising no matter what.

sid5632 · Sun Jul 07, 2019 9:10 pm

Netinstall it then. I expect it's been compromised.

arielgrin · Mon Jul 08, 2019 3:37 pm

Will try netinstall and report. Just curious, compromised how? Do you mean hacked? Or something else?

sid5632 · Mon Jul 08, 2019 7:08 pm

Yes, hacked.

arielgrin · Wed Jul 10, 2019 10:49 pm

I don't see how could it get hacked. There are no open ports nor forwarded ports on the wan interface. None of the admin tools, like webfig or ssh are open to the exterior. The only one using the router is me.

sid5632 · Wed Jul 10, 2019 11:06 pm

It was only a guess based on almost zero information.
You will only find out if it's cured by Netinstalling it.

2frogs · Wed Jul 10, 2019 11:18 pm

Most likely a partially failed update or some corruption in OS.

sindy · Wed Jul 10, 2019 11:31 pm

There are no open ports nor forwarded ports on the wan interface. None of the admin tools, like webfig or ssh are open to the exterior. The only one using the router is me.

The times when this used to be enough to feel reasonably safe are unfortunately gone. I don't say it is the case here, but cross-platform malware does exist, which squats on a PC in your LAN to which it gets via connections permitted by the firewall of the router (such as browsing on infected web sites), and from there it starts malicious activity towards other machines on the LAN and also towards the router, as even security-aware users usually only do what you've described above, i.e. prevent attacks from WAN side but keep all managament access open for any source in LAN.

mn62 · Thu Jul 11, 2019 3:54 pm

Netinstall will not help you.
All versions above 6.34.6 have nand refresh.

What's new in 6.35 (2016-Apr-14 12:55):
*) nand - implemented once a week nand refresh to improve stored data integrity (will increase sector writes);

sindy · Thu Jul 11, 2019 4:36 pm

*) nand - implemented once a week nand refresh to improve stored data integrity (will increase sector writes);

20000 sector writes in 36 hours give something like 93000 sector writes per week. What's the NAND Flash size in your device?

arielgrin · Thu Jul 11, 2019 6:33 pm

I have netinstalled 6.45.1 and the issue persists. But the writes are not done once a week, as in the "refresh" update, they are done every couple of minutes. After 1 hour there is already more than 300 writes. So the netinstall procedure hasn't fixed it. I noticed that even though leases are set to never write to disk, any time a lease is granted, renewed or removed, write count raises by 16. No matter what interval I use, be it never, 1 hour, 5 hours, 1 day, etc, any lease change causes sector writes immediately.
I have already contacted support, they are investigating, so I will keep this thread updated with any info that might help.

mn62 · Fri Jul 12, 2019 10:54 pm

What's the NAND Flash size in your device?

128.0 MiB, RB2011UAS-2HnD

My device is old

Anastasia · Thu Jan 16, 2020 3:37 pm

Hi All: I'm running RouterOS 6.43.16 (long-term) on RB751g-2HnD and I'm seeing more than 7000 sector writes per day

did you solve the problem? What have you done for this?

shiyiqiang08 · Sat Jan 18, 2020 8:11 am

do you enable the vpn service?
or you use vpn?
i find vpn will cause sector writting....

DiamondBA · Fri Apr 30, 2021 9:31 am

I have similar behaviour with my CAP ac since last couple releases (6.48.x). Used only as ap station with 3-5 clients, no DHCP, no graphing, no logging to disk.

It looks like SNTP client is causing writes to flash - at the moment installed 6.48.2, SNTP client on - 6-8K wrtes per 24h, SNTP client off - 1-1.5K writes per 24h.
I have 3 CAP ac (different configurations), but the ones with h/w revision v2 are not making high sector writes with SNTP on/off, only older model without revision has high sector writes.

rextended · Fri Apr 30, 2021 10:38 am

From 4 July 2019 to now no one has guessed...

For now only this comes to my mind:

1) Last logged in on users are stored on NAND, cannot be disabled,
Just opening winbox or terminal
For example, just this are 2 writes

2) RouterBOARD do not have battery for RTC :((((
SNTP and NTP store time on NAND, it's happen often.
It's for set time at bootup as closest right time.
it cannot be disabled, also if SNTP and NTP is disabled

3) On DEFAULT, critical logs, like failed login access, are stored on NAND till 100 events, the newer clear the older, etc.
These logs are stored on NAND and displayed on terminal on open and removed just after displayed on terminal
for disble the DEFAULT:

/system logging action set echo remember=no

4) Terminal history:
Everytime you write something on terminal, that go to "history" of commands.
Is persistent and keeped on NAND after reboots
it cannot be disabled
hint: for clear history on terminal

/console clear-history

5) DHCP Leases are stored on default every 5 minutes on "disk"
For change the update interval:

/ip dhcp-server config set interim-update=00:05:00

For disable saving

/ip dhcp-server config set store-leases-disk=never

Sector write history count?
Yes... it's writed somewhere... ;))

EDIT: added point 5)

KayBur · Fri Apr 30, 2021 4:12 pm

I have similar behaviour with my CAP ac since last couple releases (6.48.x). Used only as ap station with 3-5 clients, no DHCP, no graphing, no logging to disk.

It looks like SNTP client is causing writes to flash - at the moment installed 6.48.2, SNTP client on - 6-8K wrtes per 24h, SNTP client off - 1-1.5K writes per 24h.
I have 3 CAP ac (different configurations), but the ones with h/w revision v2 are not making high sector writes with SNTP on/off, only older model without revision has high sector writes.

This might be a silly idea, but what if there simply isn't enough power to commit that much data? Or just an update bug.

DiamondBA · Tue May 04, 2021 2:21 pm

2) RouterBOARD do not have battery for RTC :((((
NTP store time on NAND, it's happen often.
It's for set time at bootup as closest right time.
it cannot be disabled, also if NTP is disabled

I can definitely confirm that my high sector writes occur when SNTP client enabled. With SNTP client disabled I have 600-800 writes per 24h now and nothing else is chagned in configuration to get writes dropped from 8K to 800 per day.

Whether time adjustments are causing other modules to create more NAND writes (kind of chain reaction) or SNTP client itself stores a lot of info on disk - that is question to MikroTik.

mkx · Tue May 04, 2021 2:49 pm

If this indeed has anything to do with SNTP client, then it's NTP client from stand-alone ntp package guilty as well.

DiamondBA · Thu May 06, 2021 1:58 pm

If this indeed has anything to do with SNTP client, then it's NTP client from stand-alone ntp package guilty as well.

If I'm not mistaken, separate NTP package should be installed additionally (I did that recently to get NTP server running on my hex S), SNTP client is part of default packaging.

mkx · Fri May 07, 2021 12:14 am

What I'm saying is that I also see enormous number of sector writes, but my devices all have the separate ntp package installed. AFAIK separate ntp package provides different ntp client than system package. So if it's ntp client that causes high sector writes, it's ntp client from separate package doing it as well.

rextended · Fri May 07, 2021 6:54 pm

viewtopic.php?f=2&t=149907&p=855525#p855525

Added point 5)

ckl88 · Sat May 08, 2021 3:27 am

Hi. Interesting topic.

What number is considered to be high sector writes within a 24 hour period? 1000, 5000?

mkx · Sat May 08, 2021 10:24 am

My hAP ac2 recorded 1.5M sector writes since boot ... which was 90 days ago, so it's averaging more than 15k sector writes per day.

DiamondBA · Sat May 08, 2021 12:16 pm

I have 1 cap ac running for 2 years 24x7, it collected 600K (~800 per 24h). After update to 6.48.x this year it got another 90K in 2 weeks, so I started worrying and testing different settings on/off.

Stats for last 7 days with SNTP off (including other 3 MTs at home):

2021-05-08 10_01_18-MikroTik Resources _ Splunk 8.1.3 — Mozilla Firefox.png

PS I wouldn't be worrying much if I don't have HP server (gen 8) which NAND flash (soldered) died after 5 years, which made internal SD card slot unworkable + iLO became extremely slow and only way to resolve problem - replace motherboard. Don't want to replace router in next 3 years because somthing started using NAND not for purpose (I hardly can imagine what should be written to disk when time is adjusted, my very wild theory - it tries to amend everything which is already timestamped and stored on disk with new timings).

mkx · Sat May 08, 2021 1:50 pm

My hAP ac2 collected 5.5M sector writes so far, bad blocks is still at 0.0%. This device is my main home router.
My RB951G collected 96k sector writes in 12 weeks, 16.3M in total, bad blocks are at 0.5% since long time ago (pretty sure predates the massive sector write feature). This device is used as switch/AP (stand-alone).
My other RB951G collected 95k writes in 12 weeks, 21.4M in total, bad blocks are at 0.1% (also since long time ago). This one is used as CAPsMAN-controlled AP and some time ago I used it to try to play with ROS v7beta (I partitioned NAND for that).
All devices currently run ROS 6.47.9.

I'm comforting myself with the fact that these devices cost quite less than a HPE ProLiant main board. The RB951Gs have done their tasks "magna cum laude" and I'll reture tgem light-heartedly if their NAND happen to die. I won't be so hapoy if RBD52G dies on me (it's not done with its duties not yet).

But in any case MT should either reduce the number of writes (if it really increased from what it used to be) or explain the situation clearly if actual sector writes did not increase, only counting started to show real numbers. I seem to remember a vague note by one of MT staffers indicating that actual sector writes did not actually increase when counter started to gallop but rather that they fixed counting ...

rextended · Sat May 08, 2021 3:35 pm

One simple consideration, from 2007 now are 14 years than I work as ISP.
For now, no one single device depleted his memory write times.
Considering I do not use anymore RB1xx, RB5xx, RB4xx, and RB7xx, but I still have many RB1100 working as user-manager from 2010, and all still perfectly working after 11 years of read and write.

Paternot · Mon May 10, 2021 2:20 pm

AFAIK, all Mikrotik - except the hAP AC3 - use flash certified to 100k writes. Each sector is 512 bytes. 16 MB would be 31250 sectors. Now, since it uses wear leveling, we can say it supports 100k times 31250 writes.

Let's be conservative, and say 10k times 31250 writes.

That's 3.125.000.000 writes. Yes, it's a lot of writes. But, You say, I'm wrong - each sector is 4kB, not 512B!

Ok. That's 3.125.000.000/8 = 390.625.000 writes. If we do 100k writes/day, it would be 390.625.000/100.000 = 3906,25 days. About 10,7 years.
My hEX is doing about 8700 writes/day. I have one CRS328 doing about 12k writes/day. My 3 hAP ac lite (used basically as AP) are doing about 9k - 10k writes/day.

No, I'm not really worried about it.

sindy · Mon May 10, 2021 5:09 pm

Does wear leveling include moving static data, thus causing more writes? If not, all your calculations have to be adjusted to take into account that all that happnes only in the part of the flash which doesn't hold the RouterOS image itself. So from your 100 years life expectancy for full 16 MB with 10 k writes/day, you may easily get to 20 years if only 20 % of the 16 MB are changing whereas the RouterOS image and the bootloaders sit where they are all the time between upgrades. And this is still true even if part of the writes are the refresh ones used to renew the data, which is necessary with some flash types, but not every day.

rextended · Mon May 10, 2021 6:59 pm

It is a problem that we will face in 20 years.
Who cares now?

Probably in 20 years 10Gbps are obsolete and all devices are replaced with 1Tbit/s per port,
Wi-Fi 17 are out (but MikroTik devices finally have WiFi 6)
And finally RouterOS 7.12 long-term will come out.

Paternot · Mon May 10, 2021 10:44 pm

Does wear leveling include moving static data, thus causing more writes? If not, all your calculations have to be adjusted to take into account that all that happnes only in the part of the flash which doesn't hold the RouterOS image itself. So from your 100 years life expectancy for full 16 MB with 10 k writes/day, you may easily get to 20 years if only 20 % of the 16 MB are changing whereas the RouterOS image and the bootloaders sit where they are all the time between upgrades. And this is still true even if part of the writes are the refresh ones used to renew the data, which is necessary with some flash types, but not every day.

I can't say for sure, since it depends on the implementation. I'd say yes, since this is something quite old - and these flashes are of the high quality type, hence the 100k writes cycles. Even low cost SSDs are capable of wear leveling used areas. It's one of the things that lead to the feared "write amplification": the need to move the data from one less used cell to another more used.

Yes, we need to refresh what was written. But it's not that frequent. SSDs (again, lower quality flash, smaller cells, lower retention period) usually quote a certified retention time of 5 years powered off.

But, even with your worst case scenario: are we worried about a projected life of 20 years? I don't think a single router will be used 20 years from now. By then it would be unfit to route even IOT traffic...

Who is online