NGFilesystem file corruption

AmigaOne X5000 platform specific issues.
xenic
Posts: 1165
Joined: Sun Jun 19, 2011 12:06 am

NGFilesystem file corruption

Post by xenic »

While fixing some problems like USB Wilreless mouse failure and sound card issues, I've had to reboot a lot. Whenever I had to reboot with Odyssey still open, my Odyssey cookies got corrupted and I had to sign in again at all the sites where I have elected "remember me" (i.e. stay logged in). I've never had a problem like this on my X1000. I finally moved Odyssey to an SFS2 partition and the cookie file corruption when rebooting stopped. Presumably, Odyessy is keeping the cookies file open due to the constant updating of the file. I'm guessing that NGFilesystem reacts differently to a file not being closed before a reboot. Rebooting with Odyssey open never created problems for me with FastFilesystem or SFS1/2 on my X1000. Moving Odyssey to an SFS2 partition seems to have fixed the problem for me.

To reproduce:
Have Odyssey installed on an NGFilesystem.
Make sure the Odyssey prefs are set to save cookies.
Start Odyssey and open the Cookies window.
Unless your cookies file is already corrupt you should see all your cookies in the window.
Check your Conf/cookies file and note the size and date.
Visit some WEB sites and check the cookies file to be sure cookies have been written.
Wait until you're sure nothing is being written to a disk and perform a soft reboot.
After the system reboots, start Odyssey again and open the cookies window.
If the cookies file has been corrupted the window will be empty. No cookies shown.
If you check the cookies file, the date and filesize may be different that before you rebooted.

Can ayone else reproduce this problem??
AmigaOne X1000 with 2GB memory - OS4.1 FE
User avatar
tonyw
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 1444
Joined: Wed Mar 09, 2011 1:36 pm
Location: Sydney, Australia

Re: NGFilesystem file corruption

Post by tonyw »

Thanks for that report. I'll look into it (with the current version of NGFS).

As you can imagine, several advances have been made since that version was released (bug fixes, improvements).
cheers
tony
Spectre660
Posts: 1494
Joined: Sat Jun 18, 2011 2:16 pm
Location: Montserrat

Re: NGFilesystem file corruption

Post by Spectre660 »

xenic
Posts: 1165
Joined: Sun Jun 19, 2011 12:06 am

Re: NGFilesystem file corruption

Post by xenic »

That topic seems to confirm the problem. I investigated it for a day and at one point the cookies file showed up as one size with the list command but when I copied it to ram: it was actually a smaller size. AmigaDOS 'copy' command wouldn't copy it at all but returned an error about an invalid lock.
AmigaOne X1000 with 2GB memory - OS4.1 FE
User avatar
daveyw
Posts: 237
Joined: Mon Sep 12, 2011 7:44 pm

Re: NGFilesystem file corruption

Post by daveyw »

I haven't had many reboots, but I have noticed that Odyessy isn't always caching my logins. Couldn't remember if it ever did.... 8-)
xenic
Posts: 1165
Joined: Sun Jun 19, 2011 12:06 am

Re: NGFilesystem file corruption

Post by xenic »

daveyw wrote:I haven't had many reboots, but I have noticed that Odyessy isn't always caching my logins. Couldn't remember if it ever did.... 8-)
That's how I discovered this problem. I was needing to sign in to this site and others every day which never happened when using an X1000 or SAM440.

Your logins are remembered (cached) with cookies If you select 'remember me' or 'stay logged in' or similar, The WEB site will set a cookie in your browser which the browser will save in it's cookie database. You need to have accept and save cookies set in the Odyssey Settings/Privacy window. Sites can't remember you if the cookies aren't saved.

It's easy to tell if your cookies have been corrupted. If you open the Odyssey Cookies window and it's empty after visiting a number of WEB sites, then the cookies file is probably corrupt. The only way to fix it is to delete the Odyssey/Conf/cookies.db file.
AmigaOne X1000 with 2GB memory - OS4.1 FE
xenic
Posts: 1165
Joined: Sun Jun 19, 2011 12:06 am

Re: NGFilesystem file corruption

Post by xenic »

tonyw wrote:Thanks for that report. I'll look into it (with the current version of NGFS).
If you read the 'NGFilesystem' topic in this forum you will see that this problem might go beyond Odyssey and affect other programs that use SQL files.
AmigaOne X1000 with 2GB memory - OS4.1 FE
User avatar
tonyw
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 1444
Joined: Wed Mar 09, 2011 1:36 pm
Location: Sydney, Australia

Re: NGFilesystem file corruption

Post by tonyw »

Thanks to xenic's help, I can now confirm the behaviour. It is caused by the fact that NGFS "writes behind" its meta data. File data is written immediately, but meta data (eg the file header, containing the file size) is not updated until the next write or the file is closed. So, if the system is rebooted before closing the file, some data will be lost.

I am still not sure whether to treat this as a bug or not.
cheers
tony
User avatar
tonyw
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 1444
Joined: Wed Mar 09, 2011 1:36 pm
Location: Sydney, Australia

Re: NGFilesystem file corruption

Post by tonyw »

This is the sort of behaviour that you would expect - reboot the system with files open for Write and you are likely to lose data. The journalling feature ensures that the disk contents remain consistent.

Since it isn't explicitly defined by the DOS API, it is not surprising that one file system behaves differently from another, especially when one of them is a journalling FS and the others are not.

If I were to suggest any changes here, they would go along these lines:
1) Operator should not reboot system and expect data in open files to be preserved.
2) Odyssey could flush data to its cookie file after making changes, in case of crashes or reboots, but it should not be blamed for (1).

I shall not be "fixing" this.
cheers
tony
xenic
Posts: 1165
Joined: Sun Jun 19, 2011 12:06 am

Re: NGFilesystem file corruption

Post by xenic »

tonyw wrote:This is the sort of behaviour that you would expect - reboot the system with files open for Write and you are likely to lose data. The journalling feature ensures that the disk contents remain consistent.

Since it isn't explicitly defined by the DOS API, it is not surprising that one file system behaves differently from another, especially when one of them is a journalling FS and the others are not.

If I were to suggest any changes here, they would go along these lines:
1) Operator should not reboot system and expect data in open files to be preserved.
2) Odyssey could flush data to its cookie file after making changes, in case of crashes or reboots, but it should not be blamed for (1).

I shall not be "fixing" this.
In that case I shall be using SFS2 instead :-)
I already expressed my opinion in a private email.

EDIT: With reference to item 2 in your post: I used Snoopy to check what IO functions Odyssey is using. It's using unbuffered AmigaDOS 'Read' and 'Write' commands for cookies so I don't think there's any flushing to be done.
AmigaOne X1000 with 2GB memory - OS4.1 FE
Post Reply