newlib's realloc() is using vast quantities of RAM

A forum for general AmigaOS 4.x support questions that are not platform-specific

Re: newlib's realloc() is using vast quantities of RAM

Postby chris » Thu Feb 21, 2013 10:54 pm

Hmm, ReallocVec() doesn't seem to work at all.

Code: Select all
#include <stdio.h>
#include <stdlib.h>

#include <proto/exec.h>

#define ALLOC_MAX 100 * 1024 *1024
#define ALLOC_STEP 64

int main(void)
{
   int i = ALLOC_STEP;
   void *memptr = NULL;
   int temp = 0;
   
   memptr = IExec->AllocVec(i, MEMF_PRIVATE);
   
   while(i < ALLOC_MAX) {
      i += ALLOC_STEP;
      temp = IExec->ReallocVec(memptr, i, 0);
      if(temp != i) break;
   }
   
   printf("freeing %ld bytes\n", i);
   IExec->FreeVec(memptr);
   return 0;
}


The first call to ReallocVec() always returns 0. Unless the documentation is wrong, that means it can't even realloc a 64 byte area to a 128 byte area.
chris
 
Posts: 552
Joined: Sat Jun 18, 2011 12:05 pm

Re: newlib's realloc() is using vast quantities of RAM

Postby abalaban » Fri Feb 22, 2013 12:10 am

I think ReAllocVec stopped working after the introduction of the new memory managing system. I don't remember if it was supposed to be only temporary and get forgotten or if it was definitive and the autodocs were not updated accordingly...
AmigaOne X1000 running AOS 4 beta
AmigaOne XE/G4
Amiga 1200/PPC 603e + BVision PPC
User avatar
abalaban
Beta Tester
Beta Tester
 
Posts: 456
Joined: Mon Dec 20, 2010 3:09 pm
Location: France

Re: newlib's realloc() is using vast quantities of RAM

Postby ssolie » Fri Feb 22, 2013 12:36 am

chris wrote:Hmm, ReallocVec() doesn't seem to work at all.

Don't use it. It was never implemented.
ExecSG Team Lead
User avatar
ssolie
Beta Tester
Beta Tester
 
Posts: 1010
Joined: Mon Dec 20, 2010 9:51 pm
Location: Canada

Re: newlib's realloc() is using vast quantities of RAM

Postby chris » Fri Feb 22, 2013 1:49 am

ssolie wrote:
chris wrote:Hmm, ReallocVec() doesn't seem to work at all.

Don't use it. It was never implemented.


OK, I won't. Can we get newlib's realloc fixed then please? Or at least somebody admitting it has a problem will be a start.

Any attempts to link in my own replacement functions just results in a crash-fest, either directly in those functions (if I try to replace globally) or subtly in something that is poking around in that memory (probably because my versions don't conform exactly to C standards). Something as critical as memory allocation needs to work properly and I'm not willing to waste any more of my time trying to work around a bug in newlib that I've already isolated and produced a testcase for.
chris
 
Posts: 552
Joined: Sat Jun 18, 2011 12:05 pm

Re: newlib's realloc() is using vast quantities of RAM

Postby salass00 » Fri Feb 22, 2013 10:49 am

chris wrote:I was trying that already as a workaround, however my free() is just crashing - I think because newlib has allocated the memory somewhere else (in a shared object?) and the freeing is using my code. I can't test further until I get that sorted (which requires some jiggling of my buildsystem to get the static libs updated)


Have you tried replacing realloc() like this:
Code: Select all
#include <stdlib.h>
#include <string.h>

void *my_malloc(size_t size) {
    void *ptr = malloc(sizeof(size_t) + size);
    if (ptr) {
        *(size_t *)ptr = size;
        ptr = (size_t *)ptr + 1;
    }
    return ptr;
}

void my_free(void *ptr) {
    if (ptr) {
        ptr = (size_t *)ptr - 1;
        free(ptr);
    }
}

#ifndef MIN
#define MIN(a,b) ((a)<(b)?(a):(b))
#endif

void *my_realloc(void *optr, size_t nsize) {
   size_t osize = optr ? *((size_t *)optr - 1) : 0;
    void *nptr = my_malloc(nsize);
    if (nptr) memcpy(nptr, optr, MIN(osize,nsize));
    if (optr) my_free(optr);
    return nptr;
}

#define malloc(size) my_malloc(size)
#define realloc(ptr,size) my_realloc(ptr,size)
#define free(ptr) my_free(ptr)
Last edited by salass00 on Fri Feb 22, 2013 6:06 pm, edited 1 time in total.
User avatar
salass00
AmigaOS Core Developer
AmigaOS Core Developer
 
Posts: 516
Joined: Sat Jun 18, 2011 4:12 pm
Location: Finland

Re: newlib's realloc() is using vast quantities of RAM

Postby ssolie » Fri Feb 22, 2013 5:50 pm

Any attempts to link in my own replacement functions just results in a crash-fest...

You should read the man pages for realloc() and make sure to implement it correctly. For example, if the pointer is null then realloc() acts like malloc().

I would like to see if a simple realloc() replacement fixes the problem before going any further.
ExecSG Team Lead
User avatar
ssolie
Beta Tester
Beta Tester
 
Posts: 1010
Joined: Mon Dec 20, 2010 9:51 pm
Location: Canada

Re: newlib's realloc() is using vast quantities of RAM

Postby chris » Sun Feb 24, 2013 6:42 pm

ssolie wrote:
Any attempts to link in my own replacement functions just results in a crash-fest...

You should read the man pages for realloc() and make sure to implement it correctly. For example, if the pointer is null then realloc() acts like malloc().

I would like to see if a simple realloc() replacement fixes the problem before going any further.


I added salass00's realloc replacement to my original test code, and it seems to be behaving itself.
chris
 
Posts: 552
Joined: Sat Jun 18, 2011 12:05 pm

Re: newlib's realloc() is using vast quantities of RAM

Postby ssolie » Mon Feb 25, 2013 1:24 am

chris wrote:I added salass00's realloc replacement to my original test code, and it seems to be behaving itself.

Does that mean you have a valid workaround while we work on a fix?
ExecSG Team Lead
User avatar
ssolie
Beta Tester
Beta Tester
 
Posts: 1010
Joined: Mon Dec 20, 2010 9:51 pm
Location: Canada

Re: newlib's realloc() is using vast quantities of RAM

Postby salass00 » Mon Feb 25, 2013 10:30 am

A somewhat simpler replacement realloc implementation (not tested):
Code: Select all
#include <stdlib.h>
#include <string.h>

#ifndef MIN
#define MIN(a,b) ((a)<(b)?(a):(b))
#endif

void *my_realloc(void *optr, size_t nsize) {
    size_t osize = optr ? malloc_usable_size(optr) : 0;
    void *nptr = malloc(nsize);
    if (nptr) memcpy(nptr, optr, MIN(osize,nsize));
    if (optr) free(optr);
    return nptr;
}

#define realloc(ptr,size) my_realloc(ptr,size)
User avatar
salass00
AmigaOS Core Developer
AmigaOS Core Developer
 
Posts: 516
Joined: Sat Jun 18, 2011 4:12 pm
Location: Finland

Re: newlib's realloc() is using vast quantities of RAM

Postby chris » Mon Feb 25, 2013 10:24 pm

ssolie wrote:
chris wrote:I added salass00's realloc replacement to my original test code, and it seems to be behaving itself.

Does that mean you have a valid workaround while we work on a fix?


I haven't tried it as part of a bigger project, but I suspect I'll get the same (crashing) problems as before - given that malloc, realloc and free are only three of the available memory allocation related functions in the c library, there's no guarantee that these are the only ones being used.
chris
 
Posts: 552
Joined: Sat Jun 18, 2011 12:05 pm

PreviousNext

Return to General AmigaOS

Who is online

Users browsing this forum: No registered users and 4 guests