newlib's realloc() is using vast quantities of RAM

A forum for general AmigaOS 4.x support questions that are not platform-specific
chris
Posts: 562
Joined: Sat Jun 18, 2011 11:05 am
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by chris »

Hmm, ReallocVec() doesn't seem to work at all.

Code: Select all

#include <stdio.h>
#include <stdlib.h>

#include <proto/exec.h>

#define ALLOC_MAX 100 * 1024 *1024
#define ALLOC_STEP 64

int main(void)
{
	int i = ALLOC_STEP;
	void *memptr = NULL;
	int temp = 0;
	
	memptr = IExec->AllocVec(i, MEMF_PRIVATE);
	
	while(i < ALLOC_MAX) {
		i += ALLOC_STEP;
		temp = IExec->ReallocVec(memptr, i, 0);
		if(temp != i) break;
	}
	
	printf("freeing %ld bytes\n", i);
	IExec->FreeVec(memptr);
	return 0;
}
The first call to ReallocVec() always returns 0. Unless the documentation is wrong, that means it can't even realloc a 64 byte area to a 128 byte area.
User avatar
abalaban
Beta Tester
Beta Tester
Posts: 456
Joined: Mon Dec 20, 2010 2:09 pm
Location: France
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by abalaban »

I think ReAllocVec stopped working after the introduction of the new memory managing system. I don't remember if it was supposed to be only temporary and get forgotten or if it was definitive and the autodocs were not updated accordingly...
AmigaOne X1000 running AOS 4 beta
AmigaOne XE/G4
Amiga 1200/PPC 603e + BVision PPC
User avatar
ssolie
Beta Tester
Beta Tester
Posts: 1010
Joined: Mon Dec 20, 2010 8:51 pm
Location: Canada
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by ssolie »

chris wrote:Hmm, ReallocVec() doesn't seem to work at all.
Don't use it. It was never implemented.
ExecSG Team Lead
chris
Posts: 562
Joined: Sat Jun 18, 2011 11:05 am
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by chris »

ssolie wrote:
chris wrote:Hmm, ReallocVec() doesn't seem to work at all.
Don't use it. It was never implemented.
OK, I won't. Can we get newlib's realloc fixed then please? Or at least somebody admitting it has a problem will be a start.

Any attempts to link in my own replacement functions just results in a crash-fest, either directly in those functions (if I try to replace globally) or subtly in something that is poking around in that memory (probably because my versions don't conform exactly to C standards). Something as critical as memory allocation needs to work properly and I'm not willing to waste any more of my time trying to work around a bug in newlib that I've already isolated and produced a testcase for.
User avatar
salass00
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 530
Joined: Sat Jun 18, 2011 3:12 pm
Location: Finland
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by salass00 »

chris wrote:I was trying that already as a workaround, however my free() is just crashing - I think because newlib has allocated the memory somewhere else (in a shared object?) and the freeing is using my code. I can't test further until I get that sorted (which requires some jiggling of my buildsystem to get the static libs updated)
Have you tried replacing realloc() like this:

Code: Select all

#include <stdlib.h>
#include <string.h>

void *my_malloc(size_t size) {
    void *ptr = malloc(sizeof(size_t) + size);
    if (ptr) {
        *(size_t *)ptr = size;
        ptr = (size_t *)ptr + 1;
    }
    return ptr;
}

void my_free(void *ptr) {
    if (ptr) {
        ptr = (size_t *)ptr - 1;
        free(ptr);
    }
}

#ifndef MIN
#define MIN(a,b) ((a)<(b)?(a):(b))
#endif

void *my_realloc(void *optr, size_t nsize) {
	size_t osize = optr ? *((size_t *)optr - 1) : 0;
    void *nptr = my_malloc(nsize);
    if (nptr) memcpy(nptr, optr, MIN(osize,nsize));
    if (optr) my_free(optr);
    return nptr;
}

#define malloc(size) my_malloc(size)
#define realloc(ptr,size) my_realloc(ptr,size)
#define free(ptr) my_free(ptr)
Last edited by salass00 on Fri Feb 22, 2013 5:06 pm, edited 1 time in total.
User avatar
ssolie
Beta Tester
Beta Tester
Posts: 1010
Joined: Mon Dec 20, 2010 8:51 pm
Location: Canada
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by ssolie »

Any attempts to link in my own replacement functions just results in a crash-fest...
You should read the man pages for realloc() and make sure to implement it correctly. For example, if the pointer is null then realloc() acts like malloc().

I would like to see if a simple realloc() replacement fixes the problem before going any further.
ExecSG Team Lead
chris
Posts: 562
Joined: Sat Jun 18, 2011 11:05 am
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by chris »

ssolie wrote:
Any attempts to link in my own replacement functions just results in a crash-fest...
You should read the man pages for realloc() and make sure to implement it correctly. For example, if the pointer is null then realloc() acts like malloc().

I would like to see if a simple realloc() replacement fixes the problem before going any further.
I added salass00's realloc replacement to my original test code, and it seems to be behaving itself.
User avatar
ssolie
Beta Tester
Beta Tester
Posts: 1010
Joined: Mon Dec 20, 2010 8:51 pm
Location: Canada
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by ssolie »

chris wrote:I added salass00's realloc replacement to my original test code, and it seems to be behaving itself.
Does that mean you have a valid workaround while we work on a fix?
ExecSG Team Lead
User avatar
salass00
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 530
Joined: Sat Jun 18, 2011 3:12 pm
Location: Finland
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by salass00 »

A somewhat simpler replacement realloc implementation (not tested):

Code: Select all

#include <stdlib.h>
#include <string.h>

#ifndef MIN
#define MIN(a,b) ((a)<(b)?(a):(b))
#endif

void *my_realloc(void *optr, size_t nsize) {
    size_t osize = optr ? malloc_usable_size(optr) : 0;
    void *nptr = malloc(nsize);
    if (nptr) memcpy(nptr, optr, MIN(osize,nsize));
    if (optr) free(optr);
    return nptr;
}

#define realloc(ptr,size) my_realloc(ptr,size)
chris
Posts: 562
Joined: Sat Jun 18, 2011 11:05 am
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by chris »

ssolie wrote:
chris wrote:I added salass00's realloc replacement to my original test code, and it seems to be behaving itself.
Does that mean you have a valid workaround while we work on a fix?
I haven't tried it as part of a bigger project, but I suspect I'll get the same (crashing) problems as before - given that malloc, realloc and free are only three of the available memory allocation related functions in the c library, there's no guarantee that these are the only ones being used.
Post Reply