Hyperion Entertainment Message Boards

Posted: **Thu Jul 19, 2012 11:13 am**

Hi,
I am a bit confused regarding the different memory types. Looking at the developer wiki this can be read:

MEMF_PRIVATE: This memory is private and only accessible within the context of the Task which allocated it. Private memory should always be preferred. Private memory is also swappable by default (i.e. not locked). This memory will not be visible to any other address space.

MEMF_SHARED The memory is shared and accessible by any Task in the system without restriction. This memory can be shared between all address spaces and will always appear at the same address in any address space. Shared memory is locked by default and must be explicitly unlocked with UnlockMem() prior to freeing.

MEMF_EXECUTABLE The memory is used to store executable PowerPC code. This is used two-fold in AmigaOS. First, it allows the system to determine if a function pointer points to real native PowerPC code as opposed to 68k code which needs to be emulated. Second, it prevents common exploits that use stack overflows to execute malicious code. Executable memory is locked by default and must be explicitly unlocked with UnlockMem() prior to freeing.

In the "Swap issue also on Update 4 ?" thread it is said:

Apps should avoid using shared memory because it is locked and eats up available address space.

How I am supposed to allocate memory I want to share between my own private Processes (or Tasks), created using CreateNewProc? Imagine I send a message to a private process with data to process, does this memory need to be MEMF_SHARED? If I want this to be swappable, and consequently not locked, could I simply set AVT_Lock to FALSE, and override the default for MEMF_SHARED?
When my subprocess uses the shared data, must it be locked in some way?

I am guessing the reason for AVT_Lock defaulting to TRUE for MEMF_SHARED is that each process possibly tinkering with the memory must know where the memory is at. In the case of MEMF_PRIVATE, the OS knows exactly which process uses the unlocked memory and can change its pointers accordinlgy to how the memory is moved.

I'm asking this partly in the context of my Java project, where the JVM handles memory allocation. I haven't yet focused on optimizing the memory handling for AmigaOS, so now I just allocate a big chunk of MEMF_SHARED memory for the Java heap. And here I would wan't to take advantage of all the new AmigaOS 4 memory stuff.

Posted: **Thu Jul 19, 2012 7:04 pm**

jaokim wrote:In the "Swap issue also on Update 4 ?" thread it is said:
Apps should avoid using shared memory because it is locked and eats up available address space.

The wiki contains the most up to date information and should be your sole reference from now on. That is the goal.

Use IExec->AllocVecTags() with MEMF_SHARED and specify AVT_Lock, FALSE to allocate shared memory. The default for MEMF_SHARED memory is locked for backwards compatibility. There should always be a way to allocate shared memory without locking throughout the API.

If anything is unclear on the wiki please let us know and we'll try and fix it. If you'd like to contribute to the wiki let me know as well.

Posted: **Thu Jul 19, 2012 7:57 pm**

The wiki could do with a few more code examples, and perhaps explanations on not only what the functions do, but also why. For instance when and why I would need to lock memory using LockMem(). The wiki says "If there is no good reason to lock memory then do not do it". But in the meantime, the shared memory is by default locked for backwards compatibility.
For what reason do I need locked memory, and in what way can unlocked memory, i.e. unmapped or swapped out memory break my code?

http://wiki.amigaos.net/index.php/Exec_ ... Allocation:

MEMF_SHARED and MEMF_EXECUTABLE memory is locked when allocated and must be explicitly unlocked before freeing. [...] Any memory that is locked must be explicitly unlocked with UnlockMem() prior to freeing it.

So, this would be considered wrong:

Code: Select all

void * ptr = IExec->AllocVecTags(1, 
           AVT_Type, MEMF_SHARED, 
           TAG_DONE);
IExec->FreeVec(ptr);

and this correct:

Code: Select all

void * ptr = IExec->AllocVecTags(1, 
           AVT_Type, MEMF_SHARED, 
           TAG_DONE);
IExec->UnlockMem(ptr);
IExec->FreeVec(ptr);

Posted: **Thu Jul 19, 2012 9:07 pm**

jaokim wrote:The wiki could do with a few more code examples, and perhaps explanations on not only what the functions do, but also why...

I've added another section to the wiki:
http://wiki.amigaos.net/index.php/Exec_ ... by_Default

I hope that is enough.

Posted: **Thu Jul 19, 2012 10:26 pm**

In the words of your avatar: "excellent".

In essence, when using memory in a multi threaded application, i.e. not in interrupts or device drivers, the preferred way of allocating, and freeing your memory would be:

Code: Select all

APTR ptr = IExec->AllocVecTags(1, 
           AVT_Type, MEMF_SHARED, 
           AVT_Lock, FALSE, 
           TAG_DONE);
IExec->FreeVec(ptr);

Furthermore, to clarify things, the reason for wanting locked memory when creating device drivers; is it because they are interrupt driven, and it is actually the interrupt that poses the restriction, or is it for some other reason? I would think that it doesn't matter whether the data I store in memory is always in the same place or not, since the OS will translate my pointer address to wherever the data really is. However, since device drivers might execute in an interrupt context, I assume the OS memory handling is unavailable there.

Posted: **Thu Jul 19, 2012 11:02 pm**

jaokim wrote:In essence, when using memory in a multi threaded application, i.e. not in interrupts or device drivers, the preferred way of allocating, and freeing your memory would be:

The section on allocating and freeing memory has been updated and should be clearer now:
http://wiki.amigaos.net/index.php/Exec_ ... tem_Memory

jaokim wrote:However, since device drivers might execute in an interrupt context, I assume the OS memory handling is unavailable there.

The section on Locking should have already explained this:
http://wiki.amigaos.net/index.php/Exec_ ... tem_Memory

When you are in the middle of an interrupt the last thing you want is for the system to page fault, swap memory from disk into RAM and then continue. Interrupt routines must always be lean and mean.

Posted: **Fri Jul 20, 2012 12:00 am**

Yes, now I understand. Great clarification!

One more thing: (http://wiki.amigaos.net/index.php/Exec_ ... tem_Memory)

This function will make sure your memory block is not unmapped, swapped out or somehow made inaccessible.

This sounds a bit scary. It sounds as though unlocked memory can, at random, be made inaccessible for my program. Can the memory really be made "inaccessible"? Isn't it more a question of the data being swapped out and accessible after some delay, possible disk I/O or other stuff -- which you typically don't want to have in an interrupt.

Perhaps just strike the last part; "or somehow made inaccessible", to remove any signs of danger in having your memory unlocked. (I know the sentence after tells us to "Use this function wisely", but "wisely" is really only meaningful for the wise, who probably don't need the info in the first place.

)

Posted: **Fri Jul 20, 2012 11:24 pm**

jaokim wrote:
This function will make sure your memory block is not unmapped, swapped out or somehow made inaccessible.
This sounds a bit scary. It sounds as though unlocked memory can, at random, be made inaccessible for my program. Can the memory really be made "inaccessible"? Isn't it more a question of the data being swapped out and accessible after some delay, possible disk I/O or other stuff -- which you typically don't want to have in an interrupt.

Well, it might be that your system is so memory-starved that there isn't even room left to perform a page swap. If nothing can be swapped out, your page can't be swapped in, so you're lost. Hopefully this happens rarely - unless you're a beta tester who deliberately puts his system in such a tight spot with an evil grin on his face ...

Best regards,

Niels

Posted: **Sat Jul 21, 2012 9:49 am**

@ssolie

ssolie wrote:I've added another section to the wiki /.../ I hope that is enough.

Good. So if I got it right, the private type is preferred for most "generic" memory allocations done by a program that doesn't share any data/resources with other programs.

One more question: in AmigaOS4's newlib.library, is malloc() implemented using MEMF_PRIVATE or MEMF_SHARED? I think this information could be useful for people porting code from other platforms.

Posted: **Sat Jul 21, 2012 11:04 am**

nbache wrote:Well, it might be that your system is so memory-starved that there isn't even room left to perform a page swap. If nothing can be swapped out, your page can't be swapped in, so you're lost. Hopefully this happens rarely - unless you're a beta tester who deliberately puts his system in such a tight spot with an evil grin on his face ...

It might also be due to application developer locking their memory in order to avoid it "somehow being made inaccessible".

Hyperion Entertainment Message Boards

Memory allocation

Memory allocation

Re: Memory allocation

Re: Memory allocation

Re: Memory allocation

Re: Memory allocation

Re: Memory allocation

Re: Memory allocation

Re: Memory allocation

Re: Memory allocation

Re: Memory allocation