Page 1 of 3

NGFilesystem file corruption

Posted: Tue Aug 15, 2017 11:22 pm
by xenic
While fixing some problems like USB Wilreless mouse failure and sound card issues, I've had to reboot a lot. Whenever I had to reboot with Odyssey still open, my Odyssey cookies got corrupted and I had to sign in again at all the sites where I have elected "remember me" (i.e. stay logged in). I've never had a problem like this on my X1000. I finally moved Odyssey to an SFS2 partition and the cookie file corruption when rebooting stopped. Presumably, Odyessy is keeping the cookies file open due to the constant updating of the file. I'm guessing that NGFilesystem reacts differently to a file not being closed before a reboot. Rebooting with Odyssey open never created problems for me with FastFilesystem or SFS1/2 on my X1000. Moving Odyssey to an SFS2 partition seems to have fixed the problem for me.

To reproduce:
Have Odyssey installed on an NGFilesystem.
Make sure the Odyssey prefs are set to save cookies.
Start Odyssey and open the Cookies window.
Unless your cookies file is already corrupt you should see all your cookies in the window.
Check your Conf/cookies file and note the size and date.
Visit some WEB sites and check the cookies file to be sure cookies have been written.
Wait until you're sure nothing is being written to a disk and perform a soft reboot.
After the system reboots, start Odyssey again and open the cookies window.
If the cookies file has been corrupted the window will be empty. No cookies shown.
If you check the cookies file, the date and filesize may be different that before you rebooted.

Can ayone else reproduce this problem??

Re: NGFilesystem file corruption

Posted: Tue Aug 15, 2017 11:38 pm
by tonyw
Thanks for that report. I'll look into it (with the current version of NGFS).

As you can imagine, several advances have been made since that version was released (bug fixes, improvements).

Re: NGFilesystem file corruption

Posted: Tue Aug 15, 2017 11:40 pm
by Spectre660

Re: NGFilesystem file corruption

Posted: Wed Aug 16, 2017 1:24 am
by xenic
That topic seems to confirm the problem. I investigated it for a day and at one point the cookies file showed up as one size with the list command but when I copied it to ram: it was actually a smaller size. AmigaDOS 'copy' command wouldn't copy it at all but returned an error about an invalid lock.

Re: NGFilesystem file corruption

Posted: Wed Aug 16, 2017 6:14 am
by daveyw
I haven't had many reboots, but I have noticed that Odyessy isn't always caching my logins. Couldn't remember if it ever did.... 8-)

Re: NGFilesystem file corruption

Posted: Wed Aug 16, 2017 6:01 pm
by xenic
daveyw wrote:I haven't had many reboots, but I have noticed that Odyessy isn't always caching my logins. Couldn't remember if it ever did.... 8-)
That's how I discovered this problem. I was needing to sign in to this site and others every day which never happened when using an X1000 or SAM440.

Your logins are remembered (cached) with cookies If you select 'remember me' or 'stay logged in' or similar, The WEB site will set a cookie in your browser which the browser will save in it's cookie database. You need to have accept and save cookies set in the Odyssey Settings/Privacy window. Sites can't remember you if the cookies aren't saved.

It's easy to tell if your cookies have been corrupted. If you open the Odyssey Cookies window and it's empty after visiting a number of WEB sites, then the cookies file is probably corrupt. The only way to fix it is to delete the Odyssey/Conf/cookies.db file.

Re: NGFilesystem file corruption

Posted: Wed Aug 16, 2017 8:05 pm
by xenic
tonyw wrote:Thanks for that report. I'll look into it (with the current version of NGFS).
If you read the 'NGFilesystem' topic in this forum you will see that this problem might go beyond Odyssey and affect other programs that use SQL files.

Re: NGFilesystem file corruption

Posted: Thu Aug 17, 2017 1:05 am
by tonyw
Thanks to xenic's help, I can now confirm the behaviour. It is caused by the fact that NGFS "writes behind" its meta data. File data is written immediately, but meta data (eg the file header, containing the file size) is not updated until the next write or the file is closed. So, if the system is rebooted before closing the file, some data will be lost.

I am still not sure whether to treat this as a bug or not.

Re: NGFilesystem file corruption

Posted: Thu Aug 17, 2017 6:29 am
by tonyw
This is the sort of behaviour that you would expect - reboot the system with files open for Write and you are likely to lose data. The journalling feature ensures that the disk contents remain consistent.

Since it isn't explicitly defined by the DOS API, it is not surprising that one file system behaves differently from another, especially when one of them is a journalling FS and the others are not.

If I were to suggest any changes here, they would go along these lines:
1) Operator should not reboot system and expect data in open files to be preserved.
2) Odyssey could flush data to its cookie file after making changes, in case of crashes or reboots, but it should not be blamed for (1).

I shall not be "fixing" this.

Re: NGFilesystem file corruption

Posted: Thu Aug 17, 2017 2:52 pm
by xenic
tonyw wrote:This is the sort of behaviour that you would expect - reboot the system with files open for Write and you are likely to lose data. The journalling feature ensures that the disk contents remain consistent.

Since it isn't explicitly defined by the DOS API, it is not surprising that one file system behaves differently from another, especially when one of them is a journalling FS and the others are not.

If I were to suggest any changes here, they would go along these lines:
1) Operator should not reboot system and expect data in open files to be preserved.
2) Odyssey could flush data to its cookie file after making changes, in case of crashes or reboots, but it should not be blamed for (1).

I shall not be "fixing" this.
In that case I shall be using SFS2 instead :-)
I already expressed my opinion in a private email.

EDIT: With reference to item 2 in your post: I used Snoopy to check what IO functions Odyssey is using. It's using unbuffered AmigaDOS 'Read' and 'Write' commands for cookies so I don't think there's any flushing to be done.