MikroTik

hi all,

having some issues of late with the Dude, we keep getting this message pop up in the terminal,
Setup
CCR1036 8Gb ram, with SSD 200GB HDD plugged into USB port.
Currently testing 6.37rc12

my thoughts are the USB->HDD may be too slow?
we have well over 1200 devices, and I have turned off link monitoring and graphing.

any thoughts? thanks

CCR usb performance should be pretty decent overall. Is disk still responding normally?

yes the disk is responding fine.
i considered getting a second SSD drive to test.

Its happening quite often too, every few hours especially if a few things go up and down.

MikroTik RouterOS 6.37rc40
still having same issue.

as you can see it gets worse and worse over time,

Check if disk path has not changed. By data amount that has been queued it almost seems that no data has been written into db.

Same problem on CCR1036, ros6.37, more then ~2000 devices

So, I had this issue when I deployed a CHR and use VMware to grow the disk. RouterOS detected the disk change, but the performance on it was dismal with SQL Lite. Ended up having to mount a 2nd volume to the VM, format it with Router OS and have the Dude use the new volume's path for it's DB and files. Much better now. I never see the queued writes error.

Check if disk path has not changed. By data amount that has been queued it almost seems that no data has been written into db.

disk path of dude store or anything else?
i get this error to, when i disable enable dude or create backup file dude database, it always show some queued bytes, and it take long time for dude to be able to enabled again

enabled: yes
data-directory: dude
status: disabling before export: waiting for config changes to flush (54514930 bytes in queue)

same situation with 6.37.1 on x86 vm in esxi, tried mounting a second disk as suggested but problem remains. Also tried moving the vm to an empty RAID10 datastore but the issue still persists.

it will show a increasing number in queued bytes until ROS goes on kernel failure and reboot.

Already submitted supout to support, but no luck yet :c

Check if disk path has not changed. By data amount that has been queued it almost seems that no data has been written into db.

disk path of dude store or anything else?
i get this error to, when i disable enable dude or create backup file dude database, it always show some queued bytes, and it take long time for dude to be able to enabled again

enabled: yes
data-directory: dude
status: disabling before export: waiting for config changes to flush (54514930 bytes in queue)

Every 13s or so the db-wal file flushes data into main db. like scubyx said, it takes a long time because actually if you keep printing in terminal the status of dude, it only "flushes" around 200KB every 13s

even for a usb device that's real slow imo.

Same problem on CCR1036 - Dude 6.39rc19

CCR1036 v6.39rc25
Storage: USB Flash 80/80Mb write/read speed
Running only dude - the same problem + loss of information

x86 v6.39rc37 - running on SSD

The same problem


x86 - 2 hours of work
11:03:32 echo: dude,critical queued 33576606 bytes for write to db. Please conside
r faster storage otherwise out of memory condition or data loss can occur
12:06:52 echo: dude,critical queued 67180517 bytes for write to db. Please conside
r faster storage otherwise out of memory condition or data loss can occur

CCR1036
00:41:54 dude,critical queued 33615689 bytes for write to db. Please consider faster storage otherwise out of memory condition or data loss can occur
01:52:15 dude,critical queued 67337353 bytes for write to db. Please consider faster storage otherwise out of memory condition or data loss can occur
05:45:42 dude,critical queued 134754953 bytes for write to db. Please consider faster storage otherwise out of memory condition or data loss can occur
...
13:48:48 dude,critical queued 269573459 bytes for write to db. Please consider faster storage otherwise out of memory condition or data loss can occur

is really scared to add some new devices or maps
After 1-2 days dude is back to default-backup settings

I write many times to support and every time i get answer get better storage...

x86 on vmware
2Gb ram
SSD - 50Gb space
CPU 2x4 2.6Ghz
When disable dude >
applying settings: disabling: waiting for config changes to flush (77351175 bytes in queue)
Every 37 Seconds he write around 120 000 bytes
What is very low

Checked on CHR 6.38.1
Works great, no problem anymore with write speed.

So i am running Dude 6.38.3 I spend 12 hours building out our entire network to wake up the next morning and having it all Gone. So I rebuilt it again. about 200 Devices running on a CCR1016. after I got the network completely done I went to back it up so that if i lost the data again i could restore. I started the backup 16 hours ago. 16 Hours ago. again for dramatic effect 16 HOURS AGO.

currently it is stating this:

/dude print
enabled: yes
data-directory: SSD
status: disabling before export: waiting for config changes to flush (3910678 bytes in queue)

After 16 Hours it has yet to backup a 45 Meg file. WTH. I am more then pleased that the DUDE was resurrected. However, I do not understand why a simple backup takes this long. It should take minute. I have backup my entire cloud platform 1.5 TBs twice during this simple database backup.

Can someone please give me and idea what the problem is.

I am running a CCR1016-12s-1S+ with an SSD dedicated for just the DUDE.

Disk Details

# NAME LABEL TYPE DISK FREE SIZE
0 SSD SSD ext3 SD 58.9GiB 59.5GiB

I am with the same problem in CCR1016.
Any solution? I am lost a lot of changes in the dude.

I am having the same issue on a CHR. I has drives that should easily be able to write to the database. Running 6.39.1. It starts compounding. it started complaining about not being able to write about 20 megs. Now it is complaining about 131 megs it can not write. Any disk should be able to handle a few megs of data.

Is the disk writable?

Really looking for some help like everyone else,

I got a CHR on AWS that is giving the same error. Yesterday I had almost a gig of data that hadn't been written. Eventually had to just reboot it and redo everything I had done.

I am running 6.39.1 on a t2.medium (2 cores, 4gb ram) using general SSD storage. This should be plenty capable of running the dude services when I look at resources I am using at most 10%.

Any suggestions on how to fix this? Also have a issue with outages not showing up also...

Thanks
Jarred

The disk is writable.

I am also having this issue. What is going on?

I have tried a vertical machine, an x86 with two separate hard drives, a 3011 with a usb 3.0 flash drive, and a cloud core with an external hard drive.

All receive the error.

What do I need to do to get this working? I cant imaging it has all been the hard drives.

I tried rolling back to 6.38 and 6.36 and it is still doing it.

I have reported the same problem to MT(Ticket#2017040722000952) and they working on a FIX. Problem exists on sd slot and usb sticks. CHR is working good.

We've been having the same issue as well. We're running CHR on proxmox. It was working great for a few weeks until our database started to get larger (1.5GB.) We have around 1900 devices currently.

Right now our last critical alert was that it had 268mb queued to write to db. Like others have said this will just keep increasing until routeros gets upset and kernal panics and reboots.

We've stopped modifying or adding items to the dude because we know it will get lost.

This is our current disk usage on the CHR VM:



We've emailed support about the issue but haven't heard anything back yet.

We have confirmed that CHR works fine.

Those of you running Dude on ESXi (CHR) try changing Virtual Device Node from IDE to SCSI in your virtual machine settings.

So i am running Dude 6.38.3 I spend 12 hours building out our entire network to wake up the next morning and having it all Gone. So I rebuilt it again. about 200 Devices running on a CCR1016. after I got the network completely done I went to back it up so that if i lost the data again i could restore. I started the backup 16 hours ago. 16 Hours ago. again for dramatic effect 16 HOURS AGO.

currently it is stating this:

/dude print
enabled: yes
data-directory: SSD
status: disabling before export: waiting for config changes to flush (3910678 bytes in queue)

After 16 Hours it has yet to backup a 45 Meg file. WTH. I am more then pleased that the DUDE was resurrected. However, I do not understand why a simple backup takes this long. It should take minute. I have backup my entire cloud platform 1.5 TBs twice during this simple database backup.

Can someone please give me and idea what the problem is.

I am running a CCR1016-12s-1S+ with an SSD dedicated for just the DUDE.

Disk Details

# NAME LABEL TYPE DISK FREE SIZE
0 SSD SSD ext3 SD 58.9GiB 59.5GiB

I'm facing the same issue you do and laughing about it. If 45 MB took you more than 16 hours, then my backup should be done in approx. 1 week. Ahhh.. this is not fun!
Thinking about scheduling a backup more frequently to keep the queue size less.

So i am running Dude 6.38.3 I spend 12 hours building out our entire network to wake up the next morning and having it all Gone. So I rebuilt it again. about 200 Devices running on a CCR1016. after I got the network completely done I went to back it up so that if i lost the data again i could restore. I started the backup 16 hours ago. 16 Hours ago. again for dramatic effect 16 HOURS AGO.

currently it is stating this:

/dude print
enabled: yes
data-directory: SSD
status: disabling before export: waiting for config changes to flush (3910678 bytes in queue)

After 16 Hours it has yet to backup a 45 Meg file. WTH. I am more then pleased that the DUDE was resurrected. However, I do not understand why a simple backup takes this long. It should take minute. I have backup my entire cloud platform 1.5 TBs twice during this simple database backup.

Can someone please give me and idea what the problem is.

I am running a CCR1016-12s-1S+ with an SSD dedicated for just the DUDE.

Disk Details

# NAME LABEL TYPE DISK FREE SIZE
0 SSD SSD ext3 SD 58.9GiB 59.5GiB

I'm facing the same issue you do and laughing about it. If 45 MB took you more than 16 hours, then my backup should be done in approx. 1 week. Ahhh.. this is not fun!
Thinking about scheduling a backup more frequently to keep the queue size less.

It took approx. 2 days as I miscalculated the queued size. I set a scheduled task where I take a local DB export every day, so to avoid to large queues again. Will try changing to SATA virtual adapter in ESXi as advised here.

I have the same problem on 6.38.5. It started about two days ago on RB750r3 hEX with microSD. Now I can't add anything to the map because changes will be lost after each reboot because I don't have time to wait 4 days for backup to finish. Is there any solution to this problem yet?

So i am running Dude 6.38.3 I spend 12 hours building out our entire network to wake up the next morning and having it all Gone. So I rebuilt it again. about 200 Devices running on a CCR1016. after I got the network completely done I went to back it up so that if i lost the data again i could restore. I started the backup 16 hours ago. 16 Hours ago. again for dramatic effect 16 HOURS AGO.

currently it is stating this:

/dude print
enabled: yes
data-directory: SSD
status: disabling before export: waiting for config changes to flush (3910678 bytes in queue)

After 16 Hours it has yet to backup a 45 Meg file. WTH. I am more then pleased that the DUDE was resurrected. However, I do not understand why a simple backup takes this long. It should take minute. I have backup my entire cloud platform 1.5 TBs twice during this simple database backup.

Can someone please give me and idea what the problem is.

I am running a CCR1016-12s-1S+ with an SSD dedicated for just the DUDE.

Disk Details

# NAME LABEL TYPE DISK FREE SIZE
0 SSD SSD ext3 SD 58.9GiB 59.5GiB

I'm facing the same issue you do and laughing about it. If 45 MB took you more than 16 hours, then my backup should be done in approx. 1 week. Ahhh.. this is not fun!
Thinking about scheduling a backup more frequently to keep the queue size less.

It took approx. 2 days as I miscalculated the queued size. I set a scheduled task where I take a local DB export every day, so to avoid to large queues again. Will try changing to SATA virtual adapter in ESXi as advised here.

Funny. SCSI adapters seem to not be supported by x86 RouterOS.. I had to use the CHR RouterOS and create a new virtual hard disk with SCSI adapter. I will monitor if this fixes the issue and update you.

I am sitting here with 137 meg of data to flush out. At the rate its going my dude will be down for days while I wait for a backup. I have a class 10 sd card in this machine.
Is there any solution or is it time to look at other NMS systems.

bobney,

We use Dude v6 on ESXi without any issue.
Some bugs presents. We reported them and waiting to be fixed.

The Dude Alternatives

For us using a Linux server is not an option. We are running in a cloud core in a closet. If Mikrotik puts out a package for a cloud core to run dude,
then it should do so without what I would consider a critical bug. What I find depressing is that there has been no constructive response from Mikrotik
devs regarding this issue. Most software companies consider data loss the highest level of critical bug.

So we moved to an x86 the a Samsung solid state drive and have not had any issues.

I believe i have a little egg on my face because it seems to have been the drive speeds.

So i am running Dude 6.38.3 I spend 12 hours building out our entire network to wake up the next morning and having it all Gone. So I rebuilt it again. about 200 Devices running on a CCR1016. after I got the network completely done I went to back it up so that if i lost the data again i could restore. I started the backup 16 hours ago. 16 Hours ago. again for dramatic effect 16 HOURS AGO.

currently it is stating this:

/dude print
enabled: yes
data-directory: SSD
status: disabling before export: waiting for config changes to flush (3910678 bytes in queue)

After 16 Hours it has yet to backup a 45 Meg file. WTH. I am more then pleased that the DUDE was resurrected. However, I do not understand why a simple backup takes this long. It should take minute. I have backup my entire cloud platform 1.5 TBs twice during this simple database backup.

Can someone please give me and idea what the problem is.

I am running a CCR1016-12s-1S+ with an SSD dedicated for just the DUDE.

Disk Details

# NAME LABEL TYPE DISK FREE SIZE
0 SSD SSD ext3 SD 58.9GiB 59.5GiB

I'm facing the same issue you do and laughing about it. If 45 MB took you more than 16 hours, then my backup should be done in approx. 1 week. Ahhh.. this is not fun!
Thinking about scheduling a backup more frequently to keep the queue size less.

It took approx. 2 days as I miscalculated the queued size. I set a scheduled task where I take a local DB export every day, so to avoid to large queues again. Will try changing to SATA virtual adapter in ESXi as advised here.

Funny. SCSI adapters seem to not be supported by x86 RouterOS.. I had to use the CHR RouterOS and create a new virtual hard disk with SCSI adapter. I will monitor if this fixes the issue and update you.

Changing from x86 with IDE to CHR with SCSI adapter in VMware has apparently resolved the issue. No queued Bytes errors so far and the DB export went quick and smooth.

So i am running Dude 6.38.3 I spend 12 hours building out our entire network to wake up the next morning and having it all Gone. So I rebuilt it again. about 200 Devices running on a CCR1016. after I got the network completely done I went to back it up so that if i lost the data again i could restore. I started the backup 16 hours ago. 16 Hours ago. again for dramatic effect 16 HOURS AGO.

currently it is stating this:

/dude print
enabled: yes
data-directory: SSD
status: disabling before export: waiting for config changes to flush (3910678 bytes in queue)

After 16 Hours it has yet to backup a 45 Meg file. WTH. I am more then pleased that the DUDE was resurrected. However, I do not understand why a simple backup takes this long. It should take minute. I have backup my entire cloud platform 1.5 TBs twice during this simple database backup.

Can someone please give me and idea what the problem is.

I am running a CCR1016-12s-1S+ with an SSD dedicated for just the DUDE.

Disk Details

# NAME LABEL TYPE DISK FREE SIZE
0 SSD SSD ext3 SD 58.9GiB 59.5GiB

I'm facing the same issue you do and laughing about it. If 45 MB took you more than 16 hours, then my backup should be done in approx. 1 week. Ahhh.. this is not fun!
Thinking about scheduling a backup more frequently to keep the queue size less.

It took approx. 2 days as I miscalculated the queued size. I set a scheduled task where I take a local DB export every day, so to avoid to large queues again. Will try changing to SATA virtual adapter in ESXi as advised here.

Funny. SCSI adapters seem to not be supported by x86 RouterOS.. I had to use the CHR RouterOS and create a new virtual hard disk with SCSI adapter. I will monitor if this fixes the issue and update you.

Changing from x86 with IDE to CHR with SCSI adapter in VMware has apparently resolved the issue. No queued Bytes errors so far and the DB export went quick and smooth.

i still have problem with that. i'm running on chr 6.42.6 on vmware. 2nd harddisk in vmware already scsi adapter and formated it fat32 use for dude files but still "queued 6xxxxxxxx bytes....".

when ram usage is almost full only the dude restart itself (not hte routeros) that i saw on dude log and the ram usage is back to normal.

Same problem on CCR 1016 & CHR (disks, NAND are fast)
Anyone have SOLUTION?

i am also facing disk i/o error issue in my dude database file. while i want to connect to my dude server i am getting error db failure disk i/o error : stopped.

i have added more than 500 devices for monitoring and while today morning i wake up i get the above error.

kindly help me how to get out of it and solve it.

hi all,

i can not logged in into my dude box this morning; from winbox, telnet and dude, so i decide to do power-cycle
than i found this on the log..

nov/17/2019 22:08:41 dude,critical queued 537276952 bytes for write to db. Please consider faster storage otherwise out of memory condition or data loss can occur
nov/22/2019 02:09:01 dude,critical queued 33563940 bytes for write to db. Please consider faster storage otherwise out of memory condition or data loss can occur
nov/22/2019 08:08:31 dude,critical queued 67137154 bytes for write to db. Please consider faster storage otherwise out of memory condition or data loss can occur
nov/22/2019 20:18:31 dude,critical queued 134283731 bytes for write to db. Please consider faster storage otherwise out of memory condition or data loss can occur
nov/24/2019 01:40:32 dude,critical queued 268579297 bytes for write to db. Please consider faster storage otherwise out of memory condition or data loss can occur
nov/26/2019 15:49:44 dude,critical queued 537162926 bytes for write to db. Please consider faster storage otherwise out of memory condition or data loss can occur

on average the cpu usages still low, the traffic also low (less than 1Mbps)
RoS 6.42.12

the dude backup take more than 12 hours with number of devices monitored less than 200

any clue ?

thank you

Paul

Hello,

I am experiencing the same issue now.
disabling before export: waiting for config changes to flush (286035502 bytes in queue)
6.45.6 on CCR1036-8G-2S+
Writing files to the disk is quick when moving items to the flash chip. Why is it taking so long to flush changes from the DB?
Anyone get to the root cause as to why this happens?

I was going to update to the latest 6.46 so I went to export the DB and here we have the problem.

I am experiencing the same issue now.
disabling before export: waiting for config changes to flush (286035502 bytes in queue)

Same here. Running 6.46.6 on a 3.16GHz x86, currently flushing 27MB at an average rate of 128KB per minute. Let that sink in: at this rate, it will take 3.6 hours to commit a mere 27MB of data. On a 3.16GHz x86 machine. What kind of database delivers such dismal performance--and can we expect it to be better in ROS 7??

Same Problem.
ROS 6.48.3 Stable, CCR1016-12S-1S+ and MicroSD card.
After reboot I lost all changes in The Dude (devices, links ...).

Now I stop The Dude and see "applying settings: disabling: waiting for config changes to flush (920639 bytes in queue).
Process ia very longб and:

=(