The Dude 6.40.8 - db failure: database disk image is malformed

locodog · Sun Sep 02, 2018 2:25 pm

This morning I was greeted with this error message and my Dude is not running. Database was around 30MB.
I've found on Wiki manual for db repair, but there is a warning: Works only on db versions up to v6.37 software builds.
So since i have 6.40.8, repair from wiki won't work for me.
I did try vacuum-db and it does the same as restarting Mikrotik, Dude starts working for minute or so and error appears again.
Any help on how to proceed?

Thu Sep 20, 2018 10:28 am

Delete the dude storage and import older dude backup.

sid5632 · Thu Sep 20, 2018 11:44 am

Have you got enough storage space for the database?
When this happens to me, it usually means I've forgotten to increase the disk space on the CHR VM.

locodog · Thu Sep 20, 2018 1:22 pm

I imported older backup of dude but I'm concerned that this will happen again.

I have enough storage available.

Thu Sep 20, 2018 3:00 pm

Yes. It will happen again. Sooner or later it will for sure.
No. Mikrotik does not do anything in order to correct it. Schedule often backups and restore it when necessary. There is no other solution for now.

locodog · Thu Sep 20, 2018 3:42 pm

Do you (or anyone else) know what is causing it?

Fri Sep 21, 2018 10:45 am

None knows. Even mikrotik didn't say anything more than that they noted that for the further correction while they resurrect the dude development in the future. The dude is on hold again, like usually.

eddieb · Fri Sep 21, 2018 11:07 am

FWIW

I updated Dude with every release, running 6.43.2 now.
Dude runs on a CHR vm over here.
Running for more than a year without problems.
Creating daily backups but never needed to restore

Fri Sep 21, 2018 1:54 pm

Lucky you. May happen that problems are linked with usb / mSD storage type on routerboards while on CHR this does not apply.

locodog · Mon Sep 24, 2018 8:06 pm

My Dude dB also runs from sdcard. I'm considering to swap micro SD card for USB to sata connector and connect small ssd for database.
I've tried connecting hard disk over USB-to-SATA connector, hard disk is detected, I got it formatted but when I try to write data to it Mikrotik restarts, I'm guessing it's because it doesn't have enough power to run hdd over USB, so I may try to find powered USB to sata adapter or maybe ssd because I'm hoping router will have enough power to run it.

Mon Sep 24, 2018 10:28 pm

My rb750gr3 started to report disk i/o errors on microSD card. I pulled it out, formatted in pc, tested few times with no errors, put back to rb750gr3, reformat to ext3, let the dude run on it and guess what? Again dtb corruption reported, followed by i/o error again. Made tests once more - no problems at all. I moved the dude back to chr but ir has its own problems too (switching off the cpu so the hypervisor decides to stop the virtual machine). At least it means that the rb750gr3 is not able to run dude on microSD card reliably.

Tue Sep 25, 2018 12:19 pm

Dude in CHR 6.43.2 does the same too. Mikrotik was officially informed.

There is opened ticket for this problem [Ticket#2017090222000354] by me and as you can see, it celebrated 1st year anniversary few days ago. Stll no positive answer, no hope...

Tue Sep 25, 2018 12:29 pm

vmware workstation 12. Maybe it is not linked to storage or to architecture, it may be common dude server application programming error which is there since the era of dude 4b3, maybe longer, according to the old forum topics.

Last error was overcome by stopping and starting dude again. Question how long it will last.

Thu Sep 27, 2018 12:17 pm

My latest observations leads me to the actual conclusion that in case of this error it is enough just to restart the dude. As soon as possible in order to gat the smallest gap in recorded data. Finally I have created and regularly scheduled this script that checks the status "running" an in any other case it logs the situation and restarts the dude. It also maintains the counter of restarts, last date and time of restart and the last unwanted dude status. There is no need to make any exclusion, even when the database backup is in progress.

Fell free to use/modify according to your needs and in case of modification / improvement, share here for others.

For ease of use just paste it into the terminal:

/system scheduler
add interval=5m name=DudeCheck on-event=DudeCheck policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-time=startup
/system script
add dont-require-permissions=no name=DudeCheck policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":global DudeA [/ dude get status];\r\
    \n:global DudeW running;\r\
    \n:global DudeLR;\r\
    \n:global DudeR;\r\
    \n:global DudeT;\r\
    \n\r\
    \n:if (\$DudeA != \$DudeW)  do={\r\
    \n:log info \"Dude status: \$dDudeA.\";\r\
    \n:set DudeT [/system clock get date];\r\
    \n:set DudeT (\$DudeT . \" \".  [/system clock get time] );\r\
    \n:set DudeLR [/ dude get status];\r\
    \n:set DudeR (\$DudeR + 1);\r\
    \n:log info \"Dude is not running, Make automatic restart.\";\r\
    \n/dude set enabled=no;\r\
    \n:delay 5s;\r\
    \n/dude set enabled=yes\r\
    \n}"

locodog · Thu Sep 27, 2018 2:39 pm

My latest observations leads me to the actual conclusion that in case of this error it is enough just to restart the dude. As soon as possible in order to gat the smallest gap in recorded data. Finally I have created and regularly scheduled this script that checks the status "running" an in any other case it logs the situation and restarts the dude. It also maintains the counter of restarts, last date and time of restart and the last unwanted dude status. There is no need to make any exclusion, even when the database backup is in progress.

Fell free to use/modify according to your needs and in case of modification / improvement, share here for others.

For ease of use just paste it into the terminal:
Code: Select all
/system scheduler
add interval=5m name=DudeCheck on-event=DudeCheck policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-time=startup
/system script
add dont-require-permissions=no name=DudeCheck policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":global DudeA [/ dude get status];\r\
    \n:global DudeW running;\r\
    \n:global DudeLR;\r\
    \n:global DudeR;\r\
    \n:global DudeT;\r\
    \n\r\
    \n:if (\$DudeA != \$DudeW)  do={\r\
    \n:log info \"Dude status: \$dDudeA.\";\r\
    \n:set DudeT [/system clock get date];\r\
    \n:set DudeT (\$DudeT . \" \".  [/system clock get time] );\r\
    \n:set DudeLR [/ dude get status];\r\
    \n:set DudeR (\$DudeR + 1);\r\
    \n:log info \"Dude is not running, Make automatic restart.\";\r\
    \n/dude set enabled=no;\r\
    \n:delay 5s;\r\
    \n/dude set enabled=yes\r\
    \n}"

Are you talking about my problem with disk image malformed? I've tried restarting dude and it would start working for minute or two and after that same error happens.

Thu Sep 27, 2018 3:08 pm

That was similar in my case or rb750gr3 /microSD. Now when I gave up running the dude on it and moved it to CHR, the simple restart solves the situation for days without any other noticeable side effects. So far... I suggest you to follow. The client behavior is also much smoother with server in CHR than in rb750gr3. My DB file size is 300MB, backup tgz file has 115MB.

locodog · Wed Oct 03, 2018 9:46 pm

I've found SSD and connected it over USB-to-SATA connector and for now dude is working over it. I will leave it for the night and see how it behaves.

billjellis · Mon Oct 15, 2018 10:54 pm

Running 6.42.7 and this started (Oct 14 2018) dude,critical db failure: database disk image is malformed.

It is one hosted on a 750G r3 Hex with a 30gb sd

Going to move to a VM

Thanks for suggestions.
Bill

Thu Oct 18, 2018 11:14 am

I have not seen this problem in CHR during last two weeks. Really strange.

grappy77 · Sat Nov 03, 2018 6:46 am

Hello,

Start mongod with --repair . To repair the data files, start the mongod instance with the --repair option. If the mongod instance uses MMAPv1 Storage Engine, you can include the --repairpath option to specify a temporary directory used during repair. For details, see --repairpath .
Thanks...

Sat Nov 03, 2018 5:37 pm

Recent Dude is the same regardless the subversion number changes. The latest "older" version could be 4.0b3.

I suspect the time changes during the device startup to be the reason for this error. It is something that does not happen in virtual environment as the running vm has still more or less correct time. I have no evidence of this, it is just an idea or feeling.

dmitryfromchel · Wed Jun 26, 2019 11:13 am

This article helps me. Thanks author! CHR 6.43.4, 240 objects in DUDE.
http://www.abit.ie/how_to_solve_dude_is ... _database/

cdemers · Thu Jun 27, 2019 12:35 pm

If you are on Linux you can also use these instructions:

https://r.tapatalk.com/shareLink?url=ht ... are_type=t

Sent from my SM-A520W using Tapatalk

Tue Jul 02, 2019 11:42 pm

This article helps me. Thanks author! CHR 6.43.4, 240 objects in DUDE.
http://www.abit.ie/how_to_solve_dude_is ... _database/

Thanks for hint, I will need to try this procedure also. Is there any long-term experience with this "solution"? Did it help for good or the error comes back after a while again?

dmitryfromchel · Wed Jul 03, 2019 7:18 am

Thanks for hint, I will need to try this procedure also. Is there any long-term experience with this "solution"? Did it help for good or the error comes back after a while again?

Today is 7days16hours online without crashes. In this time was added and deleted some objects and links - dude works stable.

Wed Jul 03, 2019 8:35 am

Not so long, though. That was the period the dude worked without an interrution just after switching it off and on until it has started to work just for 1 or two minutes only.

I was thinking of manual reconstruction in new installation, but this looks like way lower amount of workload. The other thing is that the database started to grow recently (during last half of year or so), I suspect it does not delete the old raw measurements

Wed Jul 03, 2019 9:06 am

OK, looks I have some problems. When I do vacuum command, I get error message and subsequent dump creates zero sized file.

sqlite> vacuum;
Error: UNIQUE constraint failed: objs.id
sqlite> .output dude.sql
sqlite> .dump
sqlite>

Edit: looks like I moved forward, dump successfully ended, but reading the dump file into new database takes ages. There si no progress counter but roughly it does 10% each hour (1GB original database size, 0,5GB SQL dump file size).

dmitryfromchel · Wed Jul 03, 2019 1:40 pm

dump file into new database takes ages...

Yes it goes very low speed, and I have some errors during this operation too. I have start dump in the end of day. Next morning new db was created - I don't know how much it longs. db before "repair" 50Mb, after - 33 mb (i3-3.3Ghz, SSD). Errors was in some strings in db, I try to review this strings in .sql file and I think it is something old imported/migrated data, maybe from 4.0bX versions. Maybe really "true repair" is create new dude map?
P.S. All of this like a magic, i think.

Wed Jul 03, 2019 8:01 pm

Finally the dude works again. Just for one hour so far, but even this is much longer than usual two minutes before...

The dump import took around 10 hours. It was running on i7/SSD, not taking more than 2% of single cpu core. Really "effective"

. Final db file is 0,25GB comparing to 1GB of original. The graphs are displaying much faster and the client seems not to be locking itself like during last several years.

Hope it will keep working for some time now.

Mikrotik should include the sqlite3 core and provide the db corrective application or involve the corrective function in the dude package so everyone should be able to run the corrective procedure by single command from CLI. At least. Despite they resigned to develop the dude furthermore.

bda · Wed Jul 10, 2019 11:19 am

Finally the dude works again. Just for one hour so far, but even this is much longer than usual two minutes before...

The dump import took around 10 hours. It was running on i7/SSD, not taking more than 2% of single cpu core. Really "effective" . Final db file is 0,25GB comparing to 1GB of original. The graphs are displaying much faster and the client seems not to be locking itself like during last several years.

Hope it will keep working for some time now.

Mikrotik should include the sqlite3 core and provide the db corrective application or involve the corrective function in the dude package so everyone should be able to run the corrective procedure by single command from CLI. At least. Despite they resigned to develop the dude furthermore.

Does this method applicable for >6.41 dude CHR-installations ?

dmitryfromchel · Wed Jul 10, 2019 12:52 pm

Does this method applicable for >6.41 dude CHR-installations ?

viewtopic.php?f=8&t=138716&p=737790#p736644

Wed Jul 10, 2019 4:32 pm

Yes. The dude was not subject to change for few years and it seems mikrotik is not going to change anything on it in near future.

bda · Wed Jul 10, 2019 8:18 pm

Yes. The dude was not subject to change for few years and it seems mikrotik is not going to change anything on it in near future.

Thanks, I will try.
One more question. Do I need transfer all dude files or only the .db file?

Wed Jul 10, 2019 10:30 pm

Only db file.

Fri Sep 20, 2019 9:20 am

It looks like I have found some long-term working solution to keep the dude run healthy.

I sheduled this script to be run once a day:

dude vacuum-db ; dude export-db disk2/dude-backup-9999-99-99.tgz

After this the database is kept at roughly constant size and no coruption occured during last weeks.
On the other side it makes small hole in the data, because during these operations no data is recorded.

Any other hints or observations from anybody?

floaty · Sat Oct 12, 2019 12:01 am

Lucky you. May happen that problems are linked with usb / mSD storage type on routerboards while on CHR this does not apply.

.
just found, that well aged statement ... sounds like Dude on 'usb / mSD' is not one of the brightest ideas ?!
... should I keep the hands off it ? ... planned to test such, on a production site.
.
not much to monitor, not much to see for the customer ... how long would a USB-flash live ?

Mon Oct 21, 2019 3:35 pm

Well,
to use external flash memory is much wise than to use internal flash storage of a routerboard to store Dude database and files. At least you can simply replace it in case it breaks.
If you will use good usb/mSD memory, heavily oversized with some wear leveling feature, it could be working for really long time. If you just want to see how the Dude works and what is it able to do for you, just use whatever storage you have without any hesitation.
You can test the Dude for free using the CHR running virtually on any of computers you have around (1CPU, 300MB of RAM and 1GB of disk is enough). With some small limitations/discomfort you can keep it running for free how long you need.

floaty · Thu Oct 24, 2019 2:28 am

At least you can simply replace it in case it breaks

.

mmh ... doesn't saw that - really good - point

Just migrated my busiest-helper-files to the m2 of my "RB1100AHx4 Dude"
... to bad I didn't got attentive of this potential issue before I began to script.
Is there a tool to check the health of the internal flash ?
... would be a reproach to ruin the hardware in cause of such a numbo.

psurmont · Thu Oct 14, 2021 4:26 am

hello everyone,

You can download sqlite3.exe here : https://sqlite.org/download.html
put your dude.db in the same directory
Run it
type .open dude.db
type .recover
type .quit

et Voila ... ur dude.db should be ok

rextended · Thu Oct 14, 2021 6:25 pm

do not forget VACUUM

type VACUUM;

between recover and quit ( ; must be typed!!! )