Memory allocation

[ChrisH]

One more question: in AmigaOS4's newlib.library, is malloc() implemented using MEMF_PRIVATE or MEMF_SHARED? I think this information could be useful for people porting code from other platforms.

I'd also like to know this. Quoted for emphasis

P.S. It was very interesting to read that "In a future version of AmigaOS, it is planned to have Task specific address spaces. This means each task could potentially address up to 4 GB of private memory each." Although I have no idea how you can (efficiently) achieve that given AmigaOS's current reliance on pointers for system structures & message passing...

[trixie]

I have one more question concerning memory allocations.

I'm implementing a text.datatype subclass. The subclass is responsible for creating a sequence of struct Line's and for passing it to the superclass in an Exec list. Memory for each struct Line needs to be allocated - what memory type is recommended? Could MEMF_PRIVATE be used at all if the line list is used by the superclass (in a different process I believe)? Or should I use MEMF_SHARED? Locked or unlocked?

Thanks for clarification, this is all rather new to me.

[Hans]

I have one more question concerning memory allocations.

I'm implementing a text.datatype subclass. The subclass is responsible for creating a sequence of struct Line's and for passing it to the superclass in an Exec list. Memory for each struct Line needs to be allocated - what memory type is recommended? Could MEMF_PRIVATE be used at all if the line list is used by the superclass (in a different process I believe)? Or should I use MEMF_SHARED? Locked or unlocked?

Whether it's called by the superclass or not is irrelevant, as this happens within the same task/process. What matters is if that memory needs to be shared between multiple tasks/processes, which you believe is the case. Since Intuition runs separately from applications, the chance is high that a datatype's internal data structures will have to be accessed from multiple tasks, unless you create a special worker task and use message passing for everything (might not be worth the overhead though).

IMHO, you should be using unlocked memory in a datatype, or in any application. The only situation that I can think of in which locked memory is needed, is memory used by a driver. For example, how would the pager be able to move a page to disk, if the disk driver's internal data structures were paged out? Likewise, a driver that needs to perform certain tasks in real-time wouldn't be able to wait for memory to be paged back in. By contrast, waiting for paging in a datatype or application might be slightly annoying, but it won't lock up the whole machine.

Hans

[ssolie]

Locked or unlocked?

I was hoping this was fully explained by the wiki. If not, please point out where in the wiki it is not clear.

All memory should be unlocked unless there is a good reason not to. The default is locked for MEMF_SHARED only for backward compatibility with the broken (and deprecated) MEMF_PUBLIC attribute.

[ssolie]

This sounds a bit scary. It sounds as though unlocked memory can, at random, be made inaccessible for my program....

Don't worry about it. Just use unlocked memory unless you are toying with DMA drivers or other situations which obviously require locked memory.

[jaokim]

This sounds a bit scary. It sounds as though unlocked memory can, at random, be made inaccessible for my program....

Don't worry about it. Just use unlocked memory unless you are toying with DMA drivers or other situations which obviously require locked memory.

Yes, I get it now. Sorry for nagging about it, but what I mean is that the term "somehow made inaccesible" is very unclear.

Imagine an unexperienced Amiga developer trying to find out why some memory access fails (for whatever reason, unrelated to unlocked memory). Without knowing much of AmigaOS's internals, and only from the wiki understanding that 1) some memory is locked by default, and 2) unlocked memory can "somehow be made inaccessible". It wouldn't be very unlikely, for the unexperienced developer to lock the memory "just to be sure that it isn't somehow made inaccessible -- its done default sometimes, how bad can it be?".

Both the "somehow made inaccessible" and "Use this function wisely" sentences tells the unexperienced developer nothing. Developing a device driver requires experience and anyone doing this will probably read the docs/wiki thoroughly and have some idea of what they're doing, whereas the casual developer might only read certain parts.

I'd suggest changing it to:

The LockMem() function is used to explicitly lock a memory block. This function will make sure your memory block is not unmapped or swapped out. If there is no good reason to lock memory then do not do it. It will prevent the memory system from optimizing memory layout and may lead to poor performance. This function is only of use when writing device drivers or other situations where locked memory is required.

This will give the LockMem-function a sense of more danger, rather than to be used as a safety precaution.

[trixie]

@ssolie

All memory should be unlocked unless there is a good reason not to. The default is locked for MEMF_SHARED only for backward compatibility with the broken (and deprecated) MEMF_PUBLIC attribute.

I understand, and tell my poor memory to take down a note in red pencil But locking was just part of the question; I'm more curious to hear a clear answer about using private/shared memory for various class/datatype resources (see above). And also about the malloc() implementation (see above, plus ChrisH's comment).

[ssolie]

@ssolie

All memory should be unlocked unless there is a good reason not to. The default is locked for MEMF_SHARED only for backward compatibility with the broken (and deprecated) MEMF_PUBLIC attribute.

I understand, and tell my poor memory to take down a note in red pencil But locking was just part of the question; I'm more curious to hear a clear answer about using private/shared memory for various class/datatype resources (see above). And also about the malloc() implementation (see above, plus ChrisH's comment).

It is all really very simple. I think you guys are reading way too much into things.

Use MEMF_SHARED to share memory between Tasks/Processes. If you don't know if the memory will be shared then we need more work on the wiki. You must know what context your code is running on. If this is unclear we have work to do.

As for the C library, you cannot assume the memory you allocate is shared. This is true for any platform. That is why specific APIs exist for each operating system (see shmget() man page) to share memory which are beyond the scope of ISO.

[xenic]

Locked or unlocked?

I was hoping this was fully explained by the wiki. If not, please point out where in the wiki it is not clear.

All memory should be unlocked unless there is a good reason not to. The default is locked for MEMF_SHARED only for backward compatibility with the broken (and deprecated) MEMF_PUBLIC attribute.

If the memory is locked by default, shouldn't it be automatically unlocked by default when calling Freevec(). I say that for 2 reasons:

1. I collect all the OS4 sources I can obtain and found several programs that use AllocVecTags() without any UnLockMem() calls. To me that indicates that there may be a lot of OS4 programs for (which I don't have the sources) that are using AllocVecTags() without UnLockMem(). There is no way of knowing how much locked memory is accumulating on our systems.

2. One of the reasons behind the implementation of the original AllocVec() was automatic memory resource tracking (i.e. you don't need to track memory allocation sizes in your program). What sense does it make to have a memory allocation function ( AllocVecTags() ) that tracks memory allocation sizes and then require a program to track all the memory sizes in order to unlock the memory before freeing it??

Maybe I've overlooked something in this discussion or in the WIKI & autodocs but having a function that tracks allocation sizes automatically and reguiring programs to track the allocation sizes seems very counterintuitive.

[ssolie]

Maybe I've overlooked something in this discussion or in the WIKI & autodocs but having a function that tracks allocation sizes automatically and reguiring programs to track the allocation sizes seems very counterintuitive.

Yes it is a bit strange but that is the way the API is right now.

This is a good example of why autodocs and includes are not enough. For many years people have been confused about the memory system which is why it was one of the areas I wanted to focus on first. If anything on the wiki is still confusing let us know.