SWAP broken

[Raziel]

AmigaOneXE:
System: PPCMotorola MPC 7447/7457 Apollo/AltiVec(tm) (emulated MC68020/FPU) (INST: Cache) (External Cache)
Level 1 Cache size: 32768, Level 2 Cache size: 524288
L2CR: 80000000, enabled, partity check disabled, instruction & data, replacement algorithm: pseudo-random
MSSCR0: 00000000, L2 prefetch disabled (0 engines), 8 maximum outstanding data-bus transactions.
512MB RAM
Radeon9200 - 128MB RAM

I'll come up with a bold statement and would love if one of the devs can prove me wrong :-/

SWAP is definitely broken on OS4upd3 on at least an AmigaOneXE (my machine) and a Sam440ep

It doesn't matter what settings i use for SWAP...it simply don't work as it should or i understand, but maybe i'm expecting too much from a new feature?

I tried the settings described in the release notes (see also here for more info on my SWAP).
Just for fun i played around with SWAP and used 512, 1024, 2048, 4096 blocks...it simply doesn't matter, whenever SWAP is called it's bombing my system...ideally first bringing down USB (with the cursed "Unsolvable errors in UHCI..." requester) and then locking up Workbench

The first linked thread is originally about a game (Aquaria) and together with this game i have found a reproduceable way to crash my Workbench...again :-/

I have 512MB RAM, the game is ~222MB in size.
I copy the game into RAM Disk, leaving me with ~120MB free space available.

Normally the game would lock up randomly while playing, but now if one starts the game from RAM (or HD, doesn't matter) it will try to load in it's data and hopelessly lockup as soon as SWAP should kick in (for me it's about 50 to 75% of the games progress bar.

Please someone explain to me what do i have to change to make SWAP work and/or confirm this misbehavour.

I'm gladly sharing any more needed info

Thank you very much

EDIT: Tried from FFS, SFS and JXFS partitions

[Slayer]

To be honest I'm not too sure why people insist on testing SWAP by forcing a low memory situation (eg filling ram up with files so to reduce free memory - perhaps storage allocation is different from program utilisation?)

That out of the way, to clarify, you have setup SWAP as described in the manual (anything else has just been experimentation on your part trying to find a solution to your predictament) when you run into a low memory situation (as far as I can tell its forced by filling your RAM disk up?) and the software you are trying to actually run doesn't but crashes or causes problems in your OS?

Maybe SWAP doesn't work this way, perhaps you need enough memory to execute the main file first THEN any further memory is allocated properly... so if there isn't enough memory to setup the initial program it just fails...

For example is this game WB friendly? What if you didn't fill the ram disk first, ran the game then switched back to Workbench and filled the ram disk up, maybe the game would run then because the paging etc is allocating correctly... a game probably isn't a good example anyway since it could be doing funny stuff in the background...

Of course I could be completely wrong but I always try and look at the logical overview

But like Rigo commented in the other thread, something is coming out soon in relation to this?

p.s. also, when's the last time you checked your memory? and sorry if I my answer isn't exactly specific to everything you're saying - I'm just throwing a few ideas around...

[Raziel]

Yes, it was pure exeperimentation on my part to probably find differences...there weren't any, sadly

All of which you described matches that game (WB friendliness)...as said in the other thread at amigans i was able to play quite a way into the game, but now it's only partly possible to play the game due to random crashes in read/write situations *when RAM is already filled to the top* (i checked by switching to WB) so it's qiute possibly paged memory acting up

But as you said there is something in the works...maybe we even get a short notice on what went wrong?

I do not want to stir things up any further as a bug has been confirmed as i understand it

But thanks for taking part in my thread

[Deniil]

I just made a very simple CLI tool to test memory allocations using MEMF_VIRTUAL.
http://hem.bredband.net/deniil/files/swaptest.lha

It allows you to allocate 20MB chunks and fill them with data and free them.

Some initial tests shows that swapping simply doesn't work. First I had my system up normally with 277 or 512MB free and run the program and allocated until I had 5MB left. I then made one more allocation (of 20MB). That went fine. It took a while, like 2 seconds. It swapped something out and gave me those 20 extra MB. Free memory was now about 500k. Then I tried yet another allocation and the system locked up. No HD activity.

Ok then, I though. Let's boot WITHOUT startup-sequence! So I did and run the program. Now I had 454/512MB free. I allocated up until I had 7MB left. Then I made one more allocation and it failed to get the memory, i.e. AllocMem returned NULL, the program said. Then the system lockup up! WTF??

Looks like even a plain and simple out-of-memory is not even handled properly but causes a problems.

NOTE: I call AvailMem(MEMF_ANY) after any action in the program. WB also calls that periodically. Could it be that it is really AvailMem that has got some bitrot in it causing all these problems with update 3, not the actual swapping??!? In the test without s-s it locked up after returning from AllocMem and after a printf line, but before the AvailMem and a second printf had returned.

My SWAP partition uses 4096 blocksize and has 87382 cylinders which seems to add up to a couple of kilobytes below 2GB. Would a too large SWAP partition cause instant death?

[Thomas Frieden]

I just made a very simple CLI tool to test memory allocations using MEMF_VIRTUAL.
http://hem.bredband.net/deniil/files/swaptest.lha

Why MEMF_VIRTUAL ? Have you actually checked the autodocs ? If yes, where does it even mention MEMF_VIRTUAL ?

MEMF_VIRTUAL is NOT for memory allocations. Use either MEMF_SHARED or MEMF_PRIVATE. Using MEMF_VIRTUAL will make use of the default memory type MEMF_SHARED.

And there's your problem. MEMF_SHARED allocates memory that is automatically locked in memory. This can NOT be paged out. In other words, you basically fill the whole memory and lock it in place. Yes, this will lead to a lockup at one point because there simply is no way to page anything in or out if all memory ends up being locked. Linux "solves" this by killing off random processes and re-using their memory. AmigaOS does not have memory ownership, so there is NOTHING that can be done once this situation arises.

Some initial tests shows that swapping simply doesn't work.

See above. Your tests are flawed because you are making assumptions that are actually not true.

Looks like even a plain and simple out-of-memory is not even handled properly but causes a problems.

Do you honestly think we would have put this out without tests ? I can assure you that it works, but your test only showed the one single case (all memory is locked) that can not be handled, and normally doesn't even occur in normal use.

Would a too large SWAP partition cause instant death?

No

[nexus]

Linux "solves" this by killing off random processes and re-using their memory.

... which is really a pain and does not work very well. More than once, I had to witness that it kills important services (like the sshd on a server) such that there is no way but pressing the reset button (instead of killing the task which consumes the biggest memory chunk)

So, AmigaOS4 isn't worse in this regard

nexus

[sundown]

The SWAP problem hase been fixed. My settings follow;

SWAP
Blocksize=4096
Buffers=10
Maxtransfer=7FFFFFFF (7 Fs)
Mask=FFFFFC

Not sure why the buffer setting is low, guessing default, but it does work. I can copy a 1.4GB file to RAM without problems now, crashed the system before the fix.

[Raziel]

Yay

Thanks to all

[Deniil]

Why MEMF_VIRTUAL ? Have you actually checked the autodocs ? If yes, where does it even mention MEMF_VIRTUAL ?

MEMF_VIRTUAL is NOT for memory allocations. Use either MEMF_SHARED or MEMF_PRIVATE. Using MEMF_VIRTUAL will make use of the default memory type MEMF_SHARED.

I did read somewhere (apparently not in the latest autodocs. May have been the migration guide) that VIRTUAL is required to make it swappable, and also that VIRTUAL automatically implies PRIVATE (not the other way around). That's why.

But as a side note: Why would SHARED be locked when it can't be used in interrupts anyway? I don't see the point of that since the memory subsystem should be below the tasks anyway and access' and page faults be handled by the MMU irregardless of what task accesses the address? Just asking.

And there's your problem. MEMF_SHARED allocates memory that is automatically locked in memory. This can NOT be paged out. In other words, you basically fill the whole memory and lock it in place. Yes, this will lead to a lockup at one point because there simply is no way to page anything in or out if all memory ends up being locked.

Well that explains why it failed completely when running without startup-sequence and my test program was basically the only thing in RAM. However, running out of memory should NEVER EVER cause a lockup of the machine (even if the caller ignores the return code, which my test program obviously doesn't, it should then just DSI with a NULL page exception).

But I read elsewhere that update 3 does have a bug causing this.

[ChrisH]

@Deniil
I checked the OS4 migration guide, and the only really relevant things it has to say are:

On AmigaOS 3.9 and earlier you had to allocate e.g. Messages and Message Ports in public memory; this is no longer needed under AmigaOS 4.0 instead use the MEMF_SHARED flag. The MEMF_SHARED flag shares a lot of the properties that its public counterpart has, but is much more friendly to the system. Most notably, MEMF_SHARED memory
1. is unlimited in size as far as address space and swap space is sufficient
2. can be read by all tasks in the system, but doesn't have too many strings attached
3. can be locked by a special API call to keep in from being swapped out

The only place where MEMF_PUBLIC should be used if when your memory needs to be accessible by an interrupt. In all other cases use MEMF_SHARED. If you are allocating private data, like storage for strings or other data, use MEMF_ANY or MEMF_PRIVATE. This makes your memory inaccessible from other tasks, which is currently not enforced but will be so in the future.

No mention of MEMF_VIRTUAL !

But the bolded part seems to suggest that it can be swapped out?!? This appears to contradict what Thomas said:

MEMF_SHARED allocates memory that is automatically locked in memory. This can NOT be paged out.