Page 4 of 4

Re: newlib's realloc() is using vast quantities of RAM

Posted: Wed Mar 05, 2014 6:31 am
by ssolie
trixie wrote:None other can make crap answers as charming as you can, Steve :-D
I don't see how knowing the details will help you in any way whatsoever. The people that need to know already do know. The only question is how to schedule the work which is something I'm helping with. So really, you don't need to know.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Fri Jan 11, 2019 12:28 pm
by kas1e
Just for record and future reference if it ever will be deal with:

I meet with the same issue when trying to port some game. Game has some function which decode OGG files to the memory, with usage of realloc() , all done over c++ and compiled with gcc 8.2.0 and -O3 optimisation, over latest beta of newlib.library (53.30). Code of that function looks like this:

Code: Select all

void OpenALSound::CreateSampleFromOGG(const IDataStream& Stream, bool Looping) 
  int Length = Stream.Size(); 
  printf ("DEBUG AUDIO: Length = %dn", Length); 

  auto  pBuffer = new byte[Length]; 
  OggVorbis_File vf; 
  ov_callbacks ov_call = {nullptr, nullptr, nullptr, nullptr}; 
  Stream.Read(Length, pBuffer); 

  ov_open_callbacks(nullptr, &vf, (const char *)pBuffer, Length, ov_call); 
  int bitstream = 0; 
  vorbis_info *info = ov_info(&vf, -1); 
  long size = 0; 
  ALenum format = (info->channels == 1) ? AL_FORMAT_MONO16 : AL_FORMAT_STEREO16; 
  ALuint freq = info->rate; 
  printf("DEBUG AUDIO: info->channels = %dn", info->channels); 
  printf("DEBUG AUDIO: info->rate = %dn", info->rate); 

  long rc = 0; 
  long allocated = 4*1024*16; 
  ALubyte *retval = static_cast<ALubyte*>(malloc(allocated)); 
  char *buff = static_cast<char*>(malloc(allocated)); 

  #ifdef __amigaos4__ 
  while ( (rc = ov_read(&vf, buff, allocated, 1, 2, 1, &bitstream)) != 0 )  
  while ( (rc = ov_read(&vf, buff, allocated, 0, 2, 1, &bitstream)) != 0 )  
    printf("AUDIO DEBUG : rc = %dn",rc); 
    printf("AUDIO DEBUG : size = %dn",rc); 
    if (rc <= 0) { 
    size += rc; 

    ALubyte *tmp = static_cast<ALubyte*>(realloc(retval, size));  
    if (tmp == nullptr) 
        retval = nullptr; 
    retval = tmp; 
    memmove(retval+size-rc, buff, rc); 
  alGenBuffers(1, &buffer); 
  alBufferData(buffer, format, retval, size, freq); 
That code working fine over linux and win32. But on amigaos4 i have some strange issue named : buffer expanding (reallocation) is eat so MUCH memory for little chunks, that it easyly ends up with eating up whole 2GB of memory and then just crashes.

Once i change "long allocated = 4*1024*16;" on "long allocated = 1024*1024*16;", so to avoid many reallocs by enlarge the buffer , it dodn't crashes. It just eat about 100mb of memory for my decompressed ogg file, and continue futher (and when game starts , memory of reallocations freed).

If i change on "long allocated = 2048*1024*16;", then it just eat up only 30-40mb while do reallocaitons and all fine.

What all of this mean : that the more realloc() we call, the more memory it eat up just for their own needs. If you will realloc many small chunks, even for 30-40-100mb of data, you will end up with loosing 1-2GB (!!) of memory and then crash.

As result: don't use newlib's realloc() ever.

Re: newlib's realloc() is using vast quantities of RAM

Posted: Sun Feb 03, 2019 2:53 pm
by TSK
I just realized this is a 5-6 years old thread and nothing have happened during all those years !