SWAP and low memory handling

[xenic]

You are being overly dramatic. We are working on a workaround for this realloc() problem. That buys us time to find a root cause. The memory system is also changing for multicore support. Nobody is ignoring any problems.

Sorry. I was just pointing out that some people respond to problems (with proof) before actually testing to see if it is a valid problem. I hope you can find a workaround and subsequent fix that isn't as slow as the CLIB2 method. The tests I performed (and included in the chart in my last post) indicate that there are big speed benefits to using newlib.

I hope you noticed the problem I found with DOS copy when ram: can't allocate any more memory. As soon as I get a chance I'm going to test a copy to a near-full disk to see if it's a DOS problem or a ramdisk issue.

EDIT: I ran some copy tests to a near-full disk and it works as it should; DOS reports a disk full error and removes the partially copied file. Apparently the Ram: disk isn't returning an error to DOS when you try to copy a file that is larger than the remaining memory.

[ssolie]

I hope you noticed the problem I found with DOS copy when ram: can't allocate any more memory. As soon as I get a chance I'm going to test a copy to a near-full disk to see if it's a DOS problem or a ramdisk issue...

I wouldn't bother. The ram-handler has been completely rewritten so any bugs you report likely won't apply.

[kas1e]

@xenic, steven
From another side, if Xenic can made a test case or explain what i need to do exactly (i.e. steps #1/2/3), i can check if it happens on new ram-handler or not, and if yes then can make a BZ for, but if not, then at least can say that its have no place with new handler anymore.

[xenic]

@xenic, steven
From another side, if Xenic can made a test case or explain what i need to do exactly (i.e. steps #1/2/3), i can check if it happens on new ram-handler or not, and if yes then can make a BZ for, but if not, then at least can say that its have no place with new handler anymore.

You can simply copy big files (music, video or pictures) to ram: until the copy command stops copying. In my case I used the allocmem command from OS4Depot to tie up most of my available memory so I didn't need to copy as many files to ram: to see the result.
1. Use allocmem to remove most of your memory. With 2GB installed memory, I was able to use "allocmem 1200 M". If you have less installed memory, you will need to use a smaller number.
2. Copy more files than will fit in ram: with something like: " copy dh1:mybigdir/#? TO ram: ".
3. Use the list command to list the last file printed in the console window by the copy command. The filesize will be smaller than the original filesize; indicating that the file was only partially copied before Ram: ran out of memory.

There is no indication that the file was not completely copied. When you copy to a disk partition or USB flash drive that runs out of space, the copy command doesn't copy just part of a file; it reports disk full.

[kas1e]

@xenic
I just get allocmem , and for sake of test just alloc whole memory: holy, os4 just freezes right after memory ends. Its tryint to slowdowns a bit for first few seconds, and then total freeze, even 3 buttons didn't works. Even if developers say to us (i remember that kind of threads before, like this one) that there is no way to check the end of memory, os should never freezes anyway. Return NULL when no memory, bring a window "you have no memory, ouch!" , anything like that, but not simply freeze.

Anyway, i then just hard-reboot, and alloc a bit less memory (so 43mb free after alloc), and then just spawn a shell, and do "copy SDK-53.20.lha" (80mb) to RAM => once i have 0.5 mb free, OS4 just freezes. And i can't check what happens with RAM:, and what happens at all. What mean its the same "low-mem" problem, and even ram-handler involved again. And that or point out us that RAM handler still use MEMF_SHARED, or, it is use MEMF_PRIVATE, but there is just a bug in MEMF_PRIVATE handler itself (see bellow).

@ssolie
Even if freeze "should happens when locking all shared memory" (which is pretty bad for sure and need all kind of workorounds, even if they will slow everything down), RAM handler should't use that kind of memory, and os should't freeze when i have 43mb of free memory, and copy 80mb file to the RAM: , right ? But if it, then it can mean only that: or our ram-handler still use MEMF_SHARED , but not MEMF_PRIVATE, or there is bug in MEMF_PRIVATE handler in whole, or both.

@all
I just see that "allocmem" tool have sources, and have ability to do alloc with MEMF_PRIVATE. So, i tryed to alloc more memory than i have on 10mb with MEMF_PRIVATE ("allocmem 900 M PRIVATE") => os4 freezes. What mean, that we still have bug there , and even MEMF_PRIVATE can't be swapped out by some reassons now. And thats the reassons why SWAPS stop working since some update imho.

[xenic]

@kas1e
In one of my previous posts I speculated that the lack of a SWAP partition on my X1000 makes it react differently to low memory situations. If you have a SWAP partition you may not be able to run the test I suggested. You may need to disable your SWAP partition by unchecking the "Automount" gadget for the SWAP partition in the "MediaToolbox/Edit Partitions and filesystems" window. A reset or reboot will probably be required for the change to take effect. Do so at your own risk if you feel the need to run the test.

[kas1e]

@xenic

If you have a SWAP partition you may not be able to run the test I suggested.

I for now just disable my swap parition as you suggest. And run few tests with disabled swap.

1). "Allocmem 900 M SHARED". AllocMem says:
Allocating 943718400 bytes...
Error: Memory allocation failed!

So far all seems ok. Not have enough memory, no swap : exit with NULL (kind of).

2). "Allocmem 900 M PRIVATE". AllocMem says:
Allocating 943718400 bytes...
Error: Memory allocation failed!

So far all seems ok as well. Not have enough memory, no swap : exit with NULL (kind of).

3). "Allocmem 800 M PRIVATE". Allocmem alloc it fine , and i have left about 30mb free, then "copy work:distrib/SDK_53.20.lha ram:" , and after few secs of copy i have a dos window:

AmigaDOS
RAM Disk
is full
proces: 220 "copy"

retry/chancel.

Press "retry" by some reassons close window, and exits (while, i assume it should all the time try to retry without exit, right ?). But still, file is never half-copied and always nothing if there is no place in RAM: anymore.

But what i notice,is that warning window happens not all the time. Sometime "copy" silency exit, and no warning window from amigados (but what is important, is that i never have "half-copied" file happens). I.e. if there is no free ram in RAM: , and files is bigger, then them never copies and never half-copies. Some time its just by some reassons didn't bring amigados warnings, but in most cases are.

Now, what is mean is:

1). Beta ram-disk seems handle situation, just by some reassons not everytime bring a warning windowses (let's say in 10 times 8 warnings, 2 silence, but at least no-half-copy files). So i will make a BZ about it as well, and will try to find out 100% reproducable test case. But as it all work in progress, its all understanable.

2). os4 didn't freezes without swap if there is not enough memory. Its didn't alloc memory, with or without RAM: involved (so, no BZ for that, all is fine, and seems NULL returns as it should and so on. No problems not with RAM, not without).

3). swap broken and once it tryed to do anything with, os4 just freezes hard. It happens in all the cases. with MEMF_SHARED allocations (intended, but bad), with MEMF_PRIVATE (not intended, should not, so bug). And do not know what about RAM: when Swap is involved and what kind of memory it uses, but as it not freezes without SWAP, then it mean problem with SWAP.

So far, 2 BZ can be done : one for "why sometime RAM: didn't bring warnings, but still handle operation ok" and second one "swap is broken at all with all kind of memory, and nothing swapped at all does not matter what.

Imho sounds right and logical, what you think ?

[xenic]

@kas1e

So far, 2 BZ can be done : one for "why sometime RAM: didn't bring warnings, but still handle operation ok" and second one "swap is broken at all with all kind of memory, and nothing swapped at all does not matter what.

Imho sounds right and logical, what you think ?

I think the new ram-handler might be O.K. I think the inconsistancy in warnings (somtimes silence) is due to a lack of enough memory to open a window. When I try to open a Workbench window when there isn't enough memory, Workbench places an indication (warning) in the screen dragbar. If AmigaDOS can't allocate enough memory to open a requester window it has no way to notify the user. If I were you, I would raise a BZ for the SWAP partition crash but not about the infrequent lack of warning that a file can't be copied because of a lack of memory.

In the process of testing low memory situations I discovered that about 480MB of X1000 memory is inaccessable. You can get the allocmem program to allocate about 800MB of your 1GB installed memory. On my X1000 I can allocmem only 1.25 GB of the installed 2GB memory. I have also tested in other ways and there is always about 480GB of memory that can't be used. I have opened a topic in the X1000 forum to report the problem but nobody has responded yet. I guess fewer people are monitoring that forum than this one.

[kas1e]

@xenic

If I were you, I would raise a BZ for the SWAP partition crash but not about the infrequent lack of warning that a file can't be copied because of a lack of memory.

Yep, right now i trying to make a proper BZ and collecting all the info

@all
Anyone who in interest to fix SWAP freeze problems can do a tests with a different kernels to catch out the one from which freezes starts ? For that just dl http://os4depot.net/share/utility/shell/allocmem.lha, and do:

Code: Select all

RAM:> Allocmem 1000 M SHARED

I.e. to allocate the more memory than you have with MEMF_SHARED: as MEMF_SHARED can't be swapped, you should just have usuall "not enough memory" in the shell, and no freeze should be (of course, try that all with enabled SWAP partition, that the key of tests). Need to find out on what kernel exactly that stop to works and start to freeze. I currently just do debug outputs with different debug levels to see what going on on my peg2, so will be not bad is some of you will do the same on another HWs (i.e. enable swap parition, allocate with MEMF_SHARED whole memory, and catch all the debug ouput of level 3,5 and 7).

[kas1e]

@all
To make you all known, its in end leads to be 2 (or more) different problems, one of which i report few days ago as major one (BZ #8338). Once it will be fixed, the remaining ones should be trivial.