Page 1 of 4

newlib's realloc() is using vast quantities of RAM

Posted: Mon Feb 18, 2013 6:27 pm
by chris
When using realloc() from newlib, memory gets used up very quickly (it is allocating far more memory than it needs and/or not freeing it correctly)

This test code demonstrates the problem:

Code: Select all

#include <stdio.h>
#include <stdlib.h>

#define ALLOC_MAX 100 * 1024 *1024
#define ALLOC_STEP 64

int main(void)
{
	int i = ALLOC_STEP;
	void *memptr = NULL;
	void *temp = NULL;
	
	memptr = malloc(i);
	
	while(i < ALLOC_MAX) {
		i += ALLOC_STEP;
		temp = realloc(memptr, i);
		
		if(temp != NULL) memptr = temp;
	}
	
	printf("freeing %ld bytes\n", i);
	free(memptr);
	return 0;
}
It allocates 100MB by continually realloc()ing in 64 byte chunks (yes, it's an extreme example).
When built against newlib, running the above program will quickly use up all memory (ie. long before the program completes and far more memory than ultimately requested)
When built against clib2 it works correctly (but takes forever).

If I change it to allocate only 10MB, around 400MB of RAM disappears. I appreciate this might be an intentional design decision to speed up further realloc()s, but the amount that gets taken from the free pool seems completely disproportionate to the amounts being requested.

I hate to think how much software this is affecting.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Tue Feb 19, 2013 5:35 pm
by ssolie
How do you know memory is being freed or not? I ask because the memory subsystem will not free memory until necessary and there is no system supported way to force a flush. How you measure "free" memory is extremely important to understanding what is going on.

Note also that newlib also does its own memory caching which is why it is superior to clib2 in some scenarios.

See http://wiki.amigaos.net/index.php/Exec_ ... Allocation for more information on the memory subsystem.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Tue Feb 19, 2013 8:17 pm
by chris
ssolie wrote:How do you know memory is being freed or not?
Well, OK, I don't know that for sure, but it's not particulary relevant here. The allocation above (which should only amount to 100MB) is running into virtual (ie. paged) RAM even though I have a reported 400MB+ free (according to Avail) at the time I start the program running.

I'm not particulary bothered whether the memory is getting freed properly or not at program exit, but during run-time it should be re-using real memory, not necessitating memory being paged out, when four times the amount requested was unused according to OS4.
Note also that newlib also does its own memory caching which is why it is superior to clib2 in some scenarios.
It is this which I believe to be causing the problem.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Tue Feb 19, 2013 10:30 pm
by tonyw
Why do you think there is a problem?

There is no reliable way of measuring or reporting the amount of available memory. To put it very simply, Avail only lists the memory that has never been used or has been used, freed and cleaned, ready for reuse. It does not count the memory that has been used and freed, but not yet cleaned and made ready for reuse. The Exec does not waste time cleaning and returning freed memory to the heap (where Avail can see it), until it needs to.

There is only a problem if you can't load something that you should be able to load. For instance, if you could load TimberWolf before your test, but not load it afterwards, then there could be a problem. Don't take the output of Avail as gospel. Most likely all the "missing" memory will reappear as soon as there is a need for it.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Wed Feb 20, 2013 12:42 am
by whose
I dont think that it is the right approach to render chris arguments void by stating that there is "an own memory subsystem that makes newlib superior in certain scenarios" or "there is no reliable way to measure free hardware memory" and so on.

We heard all this many times before, but if "an own memory subsystem" runs into paging without obvious need, there is something definetly going wrong. Furthermore, a memory system that isnt able to report exact amounts of free and useable memory (be it hardware or virtual/paged memory) is weird and should be reworked as soon as possible.

I could imagine certain scenarios, were exact reporting of used/free memory of any kind is needed (besides the effects chris discovered).

Nonetheless, I tried chris example, too, and it is running into paging, even on a clean minimalistic system. The memory system and C libraries memory subsystems shouldnt do this for re-allocations, as long as there is physical memory left (btw., try it yourself: Dont boot into Workbench, start the example from the boot shell. Have fun! ;-) ). But even if its less memory thats left, the system should start with paging out unneeded memory (and there is some for sure), but not with re-allocating memory directly from the pager.

At least its worth some investigation, why newlib behaves different to clib2 regarding the pager, I think.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Wed Feb 20, 2013 8:57 am
by trixie
I am with Whose on this.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Wed Feb 20, 2013 7:42 pm
by chris
@whose

Thank you for confirming I'm not imagining things.

Whilst the example code is a bit contrived, this bug/feature is causing me real world problems, as paging is occuring when it absolutely shouldn't need to, causing everything to grind to a halt with stop-start mouse pointer action. You can see it for yourself in the current NetSurf 3.0 development builds if you fill in enough textareas, eg. forum replies such as this.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Wed Feb 20, 2013 8:30 pm
by ssolie
chris wrote:Whilst the example code is a bit contrived, this bug/feature is causing me real world problems, as paging is occuring when it absolutely shouldn't need to, causing everything to grind to a halt with stop-start mouse pointer action. You can see it for yourself in the current NetSurf 3.0 development builds if you fill in enough textareas, eg. forum replies such as this.
I would try replacing newlib's realloc() function with your own equivalent which just does a malloc()/memcpy()/free() sequence to see what happens.

If it works out it may be a viable workaround for the time being.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Wed Feb 20, 2013 11:45 pm
by whose
@ssolie:

Good suggestion :-) This way chris could investigate the problem a bit and make his code running more snappy.

@chris:

You should try what ssolie suggested. If it works for you, there is some work needed on newlib/system memory management. Maybe you can guide the developers to the spot then.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Thu Feb 21, 2013 12:40 am
by chris
I was trying that already as a workaround, however my free() is just crashing - I think because newlib has allocated the memory somewhere else (in a shared object?) and the freeing is using my code. I can't test further until I get that sorted (which requires some jiggling of my buildsystem to get the static libs updated)