Page 2 of 4

Re: newlib's realloc() is using vast quantities of RAM

Posted: Thu Feb 21, 2013 9:54 pm
by chris
Hmm, ReallocVec() doesn't seem to work at all.

Code: Select all

#include <stdio.h>
#include <stdlib.h>

#include <proto/exec.h>

#define ALLOC_MAX 100 * 1024 *1024
#define ALLOC_STEP 64

int main(void)
{
	int i = ALLOC_STEP;
	void *memptr = NULL;
	int temp = 0;
	
	memptr = IExec->AllocVec(i, MEMF_PRIVATE);
	
	while(i < ALLOC_MAX) {
		i += ALLOC_STEP;
		temp = IExec->ReallocVec(memptr, i, 0);
		if(temp != i) break;
	}
	
	printf("freeing %ld bytes\n", i);
	IExec->FreeVec(memptr);
	return 0;
}
The first call to ReallocVec() always returns 0. Unless the documentation is wrong, that means it can't even realloc a 64 byte area to a 128 byte area.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Thu Feb 21, 2013 11:10 pm
by abalaban
I think ReAllocVec stopped working after the introduction of the new memory managing system. I don't remember if it was supposed to be only temporary and get forgotten or if it was definitive and the autodocs were not updated accordingly...

Re: newlib's realloc() is using vast quantities of RAM

Posted: Thu Feb 21, 2013 11:36 pm
by ssolie
chris wrote:Hmm, ReallocVec() doesn't seem to work at all.
Don't use it. It was never implemented.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Fri Feb 22, 2013 12:49 am
by chris
ssolie wrote:
chris wrote:Hmm, ReallocVec() doesn't seem to work at all.
Don't use it. It was never implemented.
OK, I won't. Can we get newlib's realloc fixed then please? Or at least somebody admitting it has a problem will be a start.

Any attempts to link in my own replacement functions just results in a crash-fest, either directly in those functions (if I try to replace globally) or subtly in something that is poking around in that memory (probably because my versions don't conform exactly to C standards). Something as critical as memory allocation needs to work properly and I'm not willing to waste any more of my time trying to work around a bug in newlib that I've already isolated and produced a testcase for.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Fri Feb 22, 2013 9:49 am
by salass00
chris wrote:I was trying that already as a workaround, however my free() is just crashing - I think because newlib has allocated the memory somewhere else (in a shared object?) and the freeing is using my code. I can't test further until I get that sorted (which requires some jiggling of my buildsystem to get the static libs updated)
Have you tried replacing realloc() like this:

Code: Select all

#include <stdlib.h>
#include <string.h>

void *my_malloc(size_t size) {
    void *ptr = malloc(sizeof(size_t) + size);
    if (ptr) {
        *(size_t *)ptr = size;
        ptr = (size_t *)ptr + 1;
    }
    return ptr;
}

void my_free(void *ptr) {
    if (ptr) {
        ptr = (size_t *)ptr - 1;
        free(ptr);
    }
}

#ifndef MIN
#define MIN(a,b) ((a)<(b)?(a):(b))
#endif

void *my_realloc(void *optr, size_t nsize) {
	size_t osize = optr ? *((size_t *)optr - 1) : 0;
    void *nptr = my_malloc(nsize);
    if (nptr) memcpy(nptr, optr, MIN(osize,nsize));
    if (optr) my_free(optr);
    return nptr;
}

#define malloc(size) my_malloc(size)
#define realloc(ptr,size) my_realloc(ptr,size)
#define free(ptr) my_free(ptr)

Re: newlib's realloc() is using vast quantities of RAM

Posted: Fri Feb 22, 2013 4:50 pm
by ssolie
Any attempts to link in my own replacement functions just results in a crash-fest...
You should read the man pages for realloc() and make sure to implement it correctly. For example, if the pointer is null then realloc() acts like malloc().

I would like to see if a simple realloc() replacement fixes the problem before going any further.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Sun Feb 24, 2013 5:42 pm
by chris
ssolie wrote:
Any attempts to link in my own replacement functions just results in a crash-fest...
You should read the man pages for realloc() and make sure to implement it correctly. For example, if the pointer is null then realloc() acts like malloc().

I would like to see if a simple realloc() replacement fixes the problem before going any further.
I added salass00's realloc replacement to my original test code, and it seems to be behaving itself.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Mon Feb 25, 2013 12:24 am
by ssolie
chris wrote:I added salass00's realloc replacement to my original test code, and it seems to be behaving itself.
Does that mean you have a valid workaround while we work on a fix?

Re: newlib's realloc() is using vast quantities of RAM

Posted: Mon Feb 25, 2013 9:30 am
by salass00
A somewhat simpler replacement realloc implementation (not tested):

Code: Select all

#include <stdlib.h>
#include <string.h>

#ifndef MIN
#define MIN(a,b) ((a)<(b)?(a):(b))
#endif

void *my_realloc(void *optr, size_t nsize) {
    size_t osize = optr ? malloc_usable_size(optr) : 0;
    void *nptr = malloc(nsize);
    if (nptr) memcpy(nptr, optr, MIN(osize,nsize));
    if (optr) free(optr);
    return nptr;
}

#define realloc(ptr,size) my_realloc(ptr,size)

Re: newlib's realloc() is using vast quantities of RAM

Posted: Mon Feb 25, 2013 9:24 pm
by chris
ssolie wrote:
chris wrote:I added salass00's realloc replacement to my original test code, and it seems to be behaving itself.
Does that mean you have a valid workaround while we work on a fix?
I haven't tried it as part of a bigger project, but I suspect I'll get the same (crashing) problems as before - given that malloc, realloc and free are only three of the available memory allocation related functions in the c library, there's no guarantee that these are the only ones being used.