Hyperion Entertainment Message Boards

Hi,
I'm looking for the best solution to copy files and tested FOpen, FRead, FWrite, FClose DOS method but it is very slower than fopen, fread, fwrite, fclose.

Of course, I tested on the same file with the same BUFSIZ (1024).

1) Is the speed difference normal ?
See below for samples/results.

2) Is fopen() callable in a Process ?
I tried but always encounter a DSI in fopen(). For the same Process call, FOpen() works well.


Samples :
In the following samples, I test to write a file (SYS:Internet/OWB/OWB) in RAM: and measures the elapsed time thanks to "timer" os4depot program.

Binaries and sources :
http://zzd10h.amiga-ng.org/Divers/FOpen_test.lha


FOpen DOS :

timer FOpen_DOS SYS:Internet/OWB/OWB RAM:
source = (SYS:Internet/OWB/OWB) dest = (RAM:)
starting copying
Copy complete
Command duration: 1.1491 sec.

fopen stdio :

timer FOpen_stdio SYS:Internet/OWB/OWB RAM:
source = (SYS:Internet/OWB/OWB) dest = (RAM:)
nom_fichier_source=SYS:Internet/OWB/OWB RAM:OWB
starting copying
Copy complete
Command duration: 0.1138 sec.

I obtain the same difference (stdio fopen is far more fast than DOS FOpen method) for all files than I tested and for all buffer size.

3) Is the FOpen() method that I use not efficient ?

Thank you very much for your advices and replies on these 3 points.
Guillaume

The buffered I/O functions use BYTE operations into an internal buffer and are fast at handling very small read/writes/seeks.
These kind of small operations are very slow when doing I/O directly with the filesystem/handler.

If you are going to dump anything larger than about 16K, it will be much faster just writing it without buffering, in one go.
Use; Open(), Read(), Write(), Close() directly for large unbuffered data moves and use the buffered functions for small data operations.

Thank you Colin for your explanations.

Therefore I will have to determine the size of the file before and use
-Open/Read... for >16k files unbuffered
-FOpen/FRead... for <16k files buffered

What is strange is that fopen/fread seems very fast for all size of files, around as fast as CLI copy command.

And according to the AutoDocs, using a blocksize of 1 for FWrite() FRead() emulates a unbuffered Write() Read().
It's what I use in the FOpen_DOS example linked in my first post, therefore it should be the most efficient for OWB copy test (far larger than 16k) , no ?

Thx again for your explanations, it's kind

I think rather you should be doing something more like:
having opened with Open() not FOpen()

Note the test fo there for ch > 0 is because -1 == error, (thus ch != 0 is not sufficient )in working code you would have some test for the error condition too.

Thank you for the bug fix !

I just tested Open()/Read/Write and you are right, it's just a little bit slower than fopen() for small and big files.

I will no more use fopen and overall FOpen().

Thank you very much.

Test with a 359MB file

fopen stdio
timer fopen_stdio medias:Web_Junkie_13032015_H.mp4 RAM:
Command duration: 6.1313 sec.

Open DOS
timer Open_DOS medias:Web_Junkie_13032015_H.mp4 RAM:
Command duration: 6.9481 sec.

FOpen DOS
timer FOpen_DOS medias:Web_Junkie_13032015_H.mp4 RAM:
Command duration: 26.3812 sec.

zzd10h wrote: Therefore I will have to determine the size of the file before and use.
[...]
What is strange is that fopen/fread seems very fast for all size of files, around as fast as CLI copy command.

That's probably because it's somewhat smarter than the DOS buffered functions, it's likely the clib calls do some
usage checks to determine what you are wanting to do and it uses the best algorithm to suit the operation.

DOS is pretty dumb, if you use buffered functions, then there's no deterministic code to try and second guess
what you want, it just does it, whether it be the slowest way or not.
So, it's up to you to know what you are doing and what works best for your particular application or whether you
want the additional byte buffering code sitting on top of your Read() Write() operations.

So, once again, if you want to do lots of seeks and reads or writes of a few bytes, like reading and parsing a file
on the fly with FGetC() and such like, or anything involving small data moves, then the buffering code will be faster,
especially to DOSPacket based handlers.

But, if you simply want to suck in a file in one go, or copy a file somewhere with minimum read/writes,
then a large data moves will always be faster doing I/O directly with the handler, without the additional byte buffering
code sitting on top.

Thank you Colinw for your detailed explanations.

i just added a size check before to use Open/Read... or FOpen/FRead...

zzd10h wrote: i just added a size check before to use Open/Read... or FOpen/FRead...

There is no reason to do such a check as for very small files your copy code will only do one Read() and one Write() call and the extra buffering of FOpen()/FRead() will be wasted.

What you can do if you want to improve your file copy code is query the source file size with GetFileSize() and then set the destination file size immediately with ChangeFileSize() before you start copying. This way the filesystem will be able to allocate data blocks more efficiently and possibly the disk will become less fragmented. Do note that JXFileSystem has a bugged implementation of this operation but this filesystem is deprecated and not to be used any more anyway so I guess it shouldn't matter...