Page 1 of 1
The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Sun Sep 02, 2018 2:25 pm
by locodog
This morning I was greeted with this error message and my Dude is not running. Database was around 30MB.
I've found on Wiki manual for db repair, but there is a warning: Works only on db versions up to v6.37 software builds.
So since i have 6.40.8, repair from wiki won't work for me.
I did try vacuum-db and it does the same as restarting Mikrotik, Dude starts working for minute or so and error appears again.
Any help on how to proceed?
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Thu Sep 20, 2018 10:28 am
by jarda
Delete the dude storage and import older dude backup.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Thu Sep 20, 2018 11:44 am
by sid5632
Have you got enough storage space for the database?
When this happens to me, it usually means I've forgotten to increase the disk space on the CHR VM.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Thu Sep 20, 2018 1:22 pm
by locodog
I imported older backup of dude but I'm concerned that this will happen again.
I have enough storage available.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Thu Sep 20, 2018 3:00 pm
by jarda
Yes. It will happen again. Sooner or later it will for sure.
No. Mikrotik does not do anything in order to correct it. Schedule often backups and restore it when necessary. There is no other solution for now.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Thu Sep 20, 2018 3:42 pm
by locodog
Do you (or anyone else) know what is causing it?
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Fri Sep 21, 2018 10:45 am
by jarda
None knows. Even mikrotik didn't say anything more than that they noted that for the further correction while they resurrect the dude development in the future. The dude is on hold again, like usually.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Fri Sep 21, 2018 11:07 am
by eddieb
FWIW
I updated Dude with every release, running 6.43.2 now.
Dude runs on a CHR vm over here.
Running for more than a year without problems.
Creating daily backups but never needed to restore
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Fri Sep 21, 2018 1:54 pm
by jarda
Lucky you. May happen that problems are linked with usb / mSD storage type on routerboards while on CHR this does not apply.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Mon Sep 24, 2018 8:06 pm
by locodog
My Dude dB also runs from sdcard. I'm considering to swap micro SD card for USB to sata connector and connect small ssd for database.
I've tried connecting hard disk over USB-to-SATA connector, hard disk is detected, I got it formatted but when I try to write data to it Mikrotik restarts, I'm guessing it's because it doesn't have enough power to run hdd over USB, so I may try to find powered USB to sata adapter or maybe ssd because I'm hoping router will have enough power to run it.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Mon Sep 24, 2018 10:28 pm
by jarda
My rb750gr3 started to report disk i/o errors on microSD card. I pulled it out, formatted in pc, tested few times with no errors, put back to rb750gr3, reformat to ext3, let the dude run on it and guess what? Again dtb corruption reported, followed by i/o error again. Made tests once more - no problems at all. I moved the dude back to chr but ir has its own problems too (switching off the cpu so the hypervisor decides to stop the virtual machine). At least it means that the rb750gr3 is not able to run dude on microSD card reliably.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Tue Sep 25, 2018 12:19 pm
by jarda
Dude in CHR 6.43.2 does the same too. Mikrotik was officially informed.
There is opened ticket for this problem [Ticket#2017090222000354] by me and as you can see, it celebrated 1st year anniversary few days ago. Stll no positive answer, no hope...
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Tue Sep 25, 2018 12:29 pm
by jarda
vmware workstation 12. Maybe it is not linked to storage or to architecture, it may be common dude server application programming error which is there since the era of dude 4b3, maybe longer, according to the old forum topics.
Last error was overcome by stopping and starting dude again. Question how long it will last.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Thu Sep 27, 2018 12:17 pm
by jarda
My latest observations leads me to the actual conclusion that in case of this error it is enough just to restart the dude. As soon as possible in order to gat the smallest gap in recorded data. Finally I have created and regularly scheduled this script that checks the status "running" an in any other case it logs the situation and restarts the dude. It also maintains the counter of restarts, last date and time of restart and the last unwanted dude status. There is no need to make any exclusion, even when the database backup is in progress.
Fell free to use/modify according to your needs and in case of modification / improvement, share here for others.
For ease of use just paste it into the terminal:
/system scheduler
add interval=5m name=DudeCheck on-event=DudeCheck policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-time=startup
/system script
add dont-require-permissions=no name=DudeCheck policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":global DudeA [/ dude get status];\r\
\n:global DudeW running;\r\
\n:global DudeLR;\r\
\n:global DudeR;\r\
\n:global DudeT;\r\
\n\r\
\n:if (\$DudeA != \$DudeW) do={\r\
\n:log info \"Dude status: \$dDudeA.\";\r\
\n:set DudeT [/system clock get date];\r\
\n:set DudeT (\$DudeT . \" \". [/system clock get time] );\r\
\n:set DudeLR [/ dude get status];\r\
\n:set DudeR (\$DudeR + 1);\r\
\n:log info \"Dude is not running, Make automatic restart.\";\r\
\n/dude set enabled=no;\r\
\n:delay 5s;\r\
\n/dude set enabled=yes\r\
\n}"
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Thu Sep 27, 2018 2:39 pm
by locodog
My latest observations leads me to the actual conclusion that in case of this error it is enough just to restart the dude. As soon as possible in order to gat the smallest gap in recorded data. Finally I have created and regularly scheduled this script that checks the status "running" an in any other case it logs the situation and restarts the dude. It also maintains the counter of restarts, last date and time of restart and the last unwanted dude status. There is no need to make any exclusion, even when the database backup is in progress.
Fell free to use/modify according to your needs and in case of modification / improvement, share here for others.
For ease of use just paste it into the terminal:
/system scheduler
add interval=5m name=DudeCheck on-event=DudeCheck policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-time=startup
/system script
add dont-require-permissions=no name=DudeCheck policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":global DudeA [/ dude get status];\r\
\n:global DudeW running;\r\
\n:global DudeLR;\r\
\n:global DudeR;\r\
\n:global DudeT;\r\
\n\r\
\n:if (\$DudeA != \$DudeW) do={\r\
\n:log info \"Dude status: \$dDudeA.\";\r\
\n:set DudeT [/system clock get date];\r\
\n:set DudeT (\$DudeT . \" \". [/system clock get time] );\r\
\n:set DudeLR [/ dude get status];\r\
\n:set DudeR (\$DudeR + 1);\r\
\n:log info \"Dude is not running, Make automatic restart.\";\r\
\n/dude set enabled=no;\r\
\n:delay 5s;\r\
\n/dude set enabled=yes\r\
\n}"
Are you talking about my problem with disk image malformed? I've tried restarting dude and it would start working for minute or two and after that same error happens.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Thu Sep 27, 2018 3:08 pm
by jarda
That was similar in my case or rb750gr3 /microSD. Now when I gave up running the dude on it and moved it to CHR, the simple restart solves the situation for days without any other noticeable side effects. So far... I suggest you to follow. The client behavior is also much smoother with server in CHR than in rb750gr3. My DB file size is 300MB, backup tgz file has 115MB.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Wed Oct 03, 2018 9:46 pm
by locodog
I've found SSD and connected it over USB-to-SATA connector and for now dude is working over it. I will leave it for the night and see how it behaves.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Mon Oct 15, 2018 10:54 pm
by billjellis
Running 6.42.7 and this started (Oct 14 2018) dude,critical db failure: database disk image is malformed.
It is one hosted on a 750G r3 Hex with a 30gb sd
Going to move to a VM
Thanks for suggestions.
Bill
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Thu Oct 18, 2018 11:14 am
by jarda
I have not seen this problem in CHR during last two weeks. Really strange.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Sat Nov 03, 2018 6:46 am
by grappy77
Hello,
Start mongod with --repair . To repair the data files, start the mongod instance with the --repair option. If the mongod instance uses MMAPv1 Storage Engine, you can include the --repairpath option to specify a temporary directory used during repair. For details, see --repairpath .
Thanks...
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Sat Nov 03, 2018 5:37 pm
by jarda
Recent Dude is the same regardless the subversion number changes. The latest "older" version could be 4.0b3.
I suspect the time changes during the device startup to be the reason for this error. It is something that does not happen in virtual environment as the running vm has still more or less correct time. I have no evidence of this, it is just an idea or feeling.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Wed Jun 26, 2019 11:13 am
by dmitryfromchel
This article helps me. Thanks author! CHR 6.43.4, 240 objects in DUDE.
http://www.abit.ie/how_to_solve_dude_is ... _database/
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Thu Jun 27, 2019 12:35 pm
by cdemers
If you are on Linux you can also use these instructions:
https://r.tapatalk.com/shareLink?url=ht ... are_type=t
Sent from my SM-A520W using Tapatalk
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Tue Jul 02, 2019 11:42 pm
by jarda
Thanks for hint, I will need to try this procedure also. Is there any long-term experience with this "solution"? Did it help for good or the error comes back after a while again?
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Wed Jul 03, 2019 7:18 am
by dmitryfromchel
Thanks for hint, I will need to try this procedure also. Is there any long-term experience with this "solution"? Did it help for good or the error comes back after a while again?
Today is 7days16hours online without crashes. In this time was added and deleted some objects and links - dude works stable.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Wed Jul 03, 2019 8:35 am
by jarda
Not so long, though. That was the period the dude worked without an interrution just after switching it off and on until it has started to work just for 1 or two minutes only.
I was thinking of manual reconstruction in new installation, but this looks like way lower amount of workload. The other thing is that the database started to grow recently (during last half of year or so), I suspect it does not delete the old raw measurements
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Wed Jul 03, 2019 9:06 am
by jarda
OK, looks I have some problems. When I do vacuum command, I get error message and subsequent dump creates zero sized file.
sqlite> vacuum;
Error: UNIQUE constraint failed: objs.id
sqlite> .output dude.sql
sqlite> .dump
sqlite>
Edit: looks like I moved forward, dump successfully ended, but reading the dump file into new database takes ages. There si no progress counter but roughly it does 10% each hour (1GB original database size, 0,5GB SQL dump file size).
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Wed Jul 03, 2019 1:40 pm
by dmitryfromchel
dump file into new database takes ages...
Yes it goes very low speed, and I have some errors during this operation too. I have start dump in the end of day. Next morning new db was created - I don't know how much it longs. db before "repair" 50Mb, after - 33 mb (i3-3.3Ghz, SSD). Errors was in some strings in db, I try to review this strings in .sql file and I think it is something old imported/migrated data, maybe from 4.0bX versions. Maybe really "true repair" is create new dude map?
P.S. All of this like a magic, i think.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Wed Jul 03, 2019 8:01 pm
by jarda
Finally the dude works again. Just for one hour so far, but even this is much longer than usual two minutes before...
The dump import took around 10 hours. It was running on i7/SSD, not taking more than 2% of single cpu core. Really "effective"
. Final db file is 0,25GB comparing to 1GB of original. The graphs are displaying much faster and the client seems not to be locking itself like during last several years.
Hope it will keep working for some time now.
Mikrotik should include the sqlite3 core and provide the db corrective application or involve the corrective function in the dude package so everyone should be able to run the corrective procedure by single command from CLI. At least. Despite they resigned to develop the dude furthermore.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Wed Jul 10, 2019 11:19 am
by bda
Finally the dude works again. Just for one hour so far, but even this is much longer than usual two minutes before...
The dump import took around 10 hours. It was running on i7/SSD, not taking more than 2% of single cpu core. Really "effective"
. Final db file is 0,25GB comparing to 1GB of original. The graphs are displaying much faster and the client seems not to be locking itself like during last several years.
Hope it will keep working for some time now.
Mikrotik should include the sqlite3 core and provide the db corrective application or involve the corrective function in the dude package so everyone should be able to run the corrective procedure by single command from CLI. At least. Despite they resigned to develop the dude furthermore.
Does this method applicable for >6.41 dude CHR-installations ?
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Wed Jul 10, 2019 12:52 pm
by dmitryfromchel
Does this method applicable for >6.41 dude CHR-installations ?
viewtopic.php?f=8&t=138716&p=737790#p736644
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Wed Jul 10, 2019 4:32 pm
by jarda
Yes. The dude was not subject to change for few years and it seems mikrotik is not going to change anything on it in near future.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Wed Jul 10, 2019 8:18 pm
by bda
Yes. The dude was not subject to change for few years and it seems mikrotik is not going to change anything on it in near future.
Thanks, I will try.
One more question. Do I need transfer all dude files or only the .db file?
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Wed Jul 10, 2019 10:30 pm
by jarda
Only db file.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Fri Sep 20, 2019 9:20 am
by jarda
It looks like I have found some long-term working solution to keep the dude run healthy.
I sheduled this script to be run once a day:
dude vacuum-db ; dude export-db disk2/dude-backup-9999-99-99.tgz
After this the database is kept at roughly constant size and no coruption occured during last weeks.
On the other side it makes small hole in the data, because during these operations no data is recorded.
Any other hints or observations from anybody?
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Sat Oct 12, 2019 12:01 am
by floaty
Lucky you. May happen that problems are linked with usb / mSD storage type on routerboards while on CHR this does not apply.
.
just found, that well aged statement ... sounds like Dude on 'usb / mSD' is not one of the brightest ideas ?!
... should I keep the hands off it ? ... planned to test such, on a production site.
.
not much to monitor, not much to see for the customer ... how long would a USB-flash live ?
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Mon Oct 21, 2019 3:35 pm
by jarda
Well,
to use external flash memory is much wise than to use internal flash storage of a routerboard to store Dude database and files. At least you can simply replace it in case it breaks.
If you will use good usb/mSD memory, heavily oversized with some wear leveling feature, it could be working for really long time. If you just want to see how the Dude works and what is it able to do for you, just use whatever storage you have without any hesitation.
You can test the Dude for free using the CHR running virtually on any of computers you have around (1CPU, 300MB of RAM and 1GB of disk is enough). With some small limitations/discomfort you can keep it running for free how long you need.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Thu Oct 24, 2019 2:28 am
by floaty
At least you can simply replace it in case it breaks
.
mmh ... doesn't saw that - really good - point
Just migrated my busiest-helper-files to the m2 of my "RB1100AHx4 Dude"
... to bad I didn't got attentive of this potential issue before I began to script.
Is there a tool to check the health of the internal flash ?
... would be a reproach to ruin the hardware in cause of such a numbo.
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Thu Oct 14, 2021 4:26 am
by psurmont
hello everyone,
You can download sqlite3.exe here :
https://sqlite.org/download.html
put your dude.db in the same directory
Run it
type .open dude.db
type .recover
type .quit
et Voila ... ur dude.db should be ok
Re: The Dude 6.40.8 - db failure: database disk image is malformed
Posted: Thu Oct 14, 2021 6:25 pm
by rextended
do not forget VACUUM
type VACUUM;
between recover and quit ( ; must be typed!!! )