newlib's realloc() is using vast quantities of RAM

A forum for general AmigaOS 4.x support questions that are not platform-specific
chris
Posts: 562
Joined: Sat Jun 18, 2011 11:05 am
Contact:

newlib's realloc() is using vast quantities of RAM

Post by chris »

When using realloc() from newlib, memory gets used up very quickly (it is allocating far more memory than it needs and/or not freeing it correctly)

This test code demonstrates the problem:

Code: Select all

#include <stdio.h>
#include <stdlib.h>

#define ALLOC_MAX 100 * 1024 *1024
#define ALLOC_STEP 64

int main(void)
{
	int i = ALLOC_STEP;
	void *memptr = NULL;
	void *temp = NULL;
	
	memptr = malloc(i);
	
	while(i < ALLOC_MAX) {
		i += ALLOC_STEP;
		temp = realloc(memptr, i);
		
		if(temp != NULL) memptr = temp;
	}
	
	printf("freeing %ld bytes\n", i);
	free(memptr);
	return 0;
}
It allocates 100MB by continually realloc()ing in 64 byte chunks (yes, it's an extreme example).
When built against newlib, running the above program will quickly use up all memory (ie. long before the program completes and far more memory than ultimately requested)
When built against clib2 it works correctly (but takes forever).

If I change it to allocate only 10MB, around 400MB of RAM disappears. I appreciate this might be an intentional design decision to speed up further realloc()s, but the amount that gets taken from the free pool seems completely disproportionate to the amounts being requested.

I hate to think how much software this is affecting.
User avatar
ssolie
Beta Tester
Beta Tester
Posts: 1010
Joined: Mon Dec 20, 2010 8:51 pm
Location: Canada
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by ssolie »

How do you know memory is being freed or not? I ask because the memory subsystem will not free memory until necessary and there is no system supported way to force a flush. How you measure "free" memory is extremely important to understanding what is going on.

Note also that newlib also does its own memory caching which is why it is superior to clib2 in some scenarios.

See http://wiki.amigaos.net/index.php/Exec_ ... Allocation for more information on the memory subsystem.
ExecSG Team Lead
chris
Posts: 562
Joined: Sat Jun 18, 2011 11:05 am
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by chris »

ssolie wrote:How do you know memory is being freed or not?
Well, OK, I don't know that for sure, but it's not particulary relevant here. The allocation above (which should only amount to 100MB) is running into virtual (ie. paged) RAM even though I have a reported 400MB+ free (according to Avail) at the time I start the program running.

I'm not particulary bothered whether the memory is getting freed properly or not at program exit, but during run-time it should be re-using real memory, not necessitating memory being paged out, when four times the amount requested was unused according to OS4.
Note also that newlib also does its own memory caching which is why it is superior to clib2 in some scenarios.
It is this which I believe to be causing the problem.
User avatar
tonyw
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 1479
Joined: Wed Mar 09, 2011 1:36 pm
Location: Sydney, Australia

Re: newlib's realloc() is using vast quantities of RAM

Post by tonyw »

Why do you think there is a problem?

There is no reliable way of measuring or reporting the amount of available memory. To put it very simply, Avail only lists the memory that has never been used or has been used, freed and cleaned, ready for reuse. It does not count the memory that has been used and freed, but not yet cleaned and made ready for reuse. The Exec does not waste time cleaning and returning freed memory to the heap (where Avail can see it), until it needs to.

There is only a problem if you can't load something that you should be able to load. For instance, if you could load TimberWolf before your test, but not load it afterwards, then there could be a problem. Don't take the output of Avail as gospel. Most likely all the "missing" memory will reappear as soon as there is a need for it.
cheers
tony
whose
Posts: 92
Joined: Sun Sep 04, 2011 3:11 pm

Re: newlib's realloc() is using vast quantities of RAM

Post by whose »

I dont think that it is the right approach to render chris arguments void by stating that there is "an own memory subsystem that makes newlib superior in certain scenarios" or "there is no reliable way to measure free hardware memory" and so on.

We heard all this many times before, but if "an own memory subsystem" runs into paging without obvious need, there is something definetly going wrong. Furthermore, a memory system that isnt able to report exact amounts of free and useable memory (be it hardware or virtual/paged memory) is weird and should be reworked as soon as possible.

I could imagine certain scenarios, were exact reporting of used/free memory of any kind is needed (besides the effects chris discovered).

Nonetheless, I tried chris example, too, and it is running into paging, even on a clean minimalistic system. The memory system and C libraries memory subsystems shouldnt do this for re-allocations, as long as there is physical memory left (btw., try it yourself: Dont boot into Workbench, start the example from the boot shell. Have fun! ;-) ). But even if its less memory thats left, the system should start with paging out unneeded memory (and there is some for sure), but not with re-allocating memory directly from the pager.

At least its worth some investigation, why newlib behaves different to clib2 regarding the pager, I think.
User avatar
trixie
Posts: 409
Joined: Thu Jun 30, 2011 2:54 pm
Location: Czech Republic

Re: newlib's realloc() is using vast quantities of RAM

Post by trixie »

I am with Whose on this.
The Rear Window blog

AmigaOne X5000 @ 2GHz / 4GB RAM / Radeon RX 560 / ESI Juli@ / AmigaOS 4.1 Final Edition
SAM440ep-flex @ 667MHz / 1GB RAM / Radeon 9250 / AmigaOS 4.1 Final Edition
chris
Posts: 562
Joined: Sat Jun 18, 2011 11:05 am
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by chris »

@whose

Thank you for confirming I'm not imagining things.

Whilst the example code is a bit contrived, this bug/feature is causing me real world problems, as paging is occuring when it absolutely shouldn't need to, causing everything to grind to a halt with stop-start mouse pointer action. You can see it for yourself in the current NetSurf 3.0 development builds if you fill in enough textareas, eg. forum replies such as this.
User avatar
ssolie
Beta Tester
Beta Tester
Posts: 1010
Joined: Mon Dec 20, 2010 8:51 pm
Location: Canada
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by ssolie »

chris wrote:Whilst the example code is a bit contrived, this bug/feature is causing me real world problems, as paging is occuring when it absolutely shouldn't need to, causing everything to grind to a halt with stop-start mouse pointer action. You can see it for yourself in the current NetSurf 3.0 development builds if you fill in enough textareas, eg. forum replies such as this.
I would try replacing newlib's realloc() function with your own equivalent which just does a malloc()/memcpy()/free() sequence to see what happens.

If it works out it may be a viable workaround for the time being.
ExecSG Team Lead
whose
Posts: 92
Joined: Sun Sep 04, 2011 3:11 pm

Re: newlib's realloc() is using vast quantities of RAM

Post by whose »

@ssolie:

Good suggestion :-) This way chris could investigate the problem a bit and make his code running more snappy.

@chris:

You should try what ssolie suggested. If it works for you, there is some work needed on newlib/system memory management. Maybe you can guide the developers to the spot then.
chris
Posts: 562
Joined: Sat Jun 18, 2011 11:05 am
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by chris »

I was trying that already as a workaround, however my free() is just crashing - I think because newlib has allocated the memory somewhere else (in a shared object?) and the freeing is using my code. I can't test further until I get that sorted (which requires some jiggling of my buildsystem to get the static libs updated)
Post Reply