newlib's realloc() is using vast quantities of RAM

A forum for general AmigaOS 4.x support questions that are not platform-specific
User avatar
ssolie
Beta Tester
Beta Tester
Posts: 1010
Joined: Mon Dec 20, 2010 8:51 pm
Location: Canada
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by ssolie »

trixie wrote:None other can make crap answers as charming as you can, Steve :-D
I don't see how knowing the details will help you in any way whatsoever. The people that need to know already do know. The only question is how to schedule the work which is something I'm helping with. So really, you don't need to know.
ExecSG Team Lead
kas1e
Beta Tester
Beta Tester
Posts: 542
Joined: Sat Jun 18, 2011 7:56 am
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by kas1e »

Just for record and future reference if it ever will be deal with:

I meet with the same issue when trying to port some game. Game has some function which decode OGG files to the memory, with usage of realloc() , all done over c++ and compiled with gcc 8.2.0 and -O3 optimisation, over latest beta of newlib.library (53.30). Code of that function looks like this:

Code: Select all

void OpenALSound::CreateSampleFromOGG(const IDataStream& Stream, bool Looping) 
{ 
  int Length = Stream.Size(); 
  printf ("DEBUG AUDIO: Length = %dn", Length); 

  auto  pBuffer = new byte[Length]; 
  OggVorbis_File vf; 
  ov_callbacks ov_call = {nullptr, nullptr, nullptr, nullptr}; 
  Stream.Read(Length, pBuffer); 

  ov_open_callbacks(nullptr, &vf, (const char *)pBuffer, Length, ov_call); 
  int bitstream = 0; 
  vorbis_info *info = ov_info(&vf, -1); 
  long size = 0; 
  ALenum format = (info->channels == 1) ? AL_FORMAT_MONO16 : AL_FORMAT_STEREO16; 
  ALuint freq = info->rate; 
  printf("DEBUG AUDIO: info->channels = %dn", info->channels); 
  printf("DEBUG AUDIO: info->rate = %dn", info->rate); 

  long rc = 0; 
  long allocated = 4*1024*16; 
  ALubyte *retval = static_cast<ALubyte*>(malloc(allocated)); 
  char *buff = static_cast<char*>(malloc(allocated)); 

  #ifdef __amigaos4__ 
  while ( (rc = ov_read(&vf, buff, allocated, 1, 2, 1, &bitstream)) != 0 )  
  #else     
  while ( (rc = ov_read(&vf, buff, allocated, 0, 2, 1, &bitstream)) != 0 )  
  #endif 
  { 
    printf("AUDIO DEBUG : rc = %dn",rc); 
    printf("AUDIO DEBUG : size = %dn",rc); 
    if (rc <= 0) { 
      break; 
    } 
    size += rc; 

    ALubyte *tmp = static_cast<ALubyte*>(realloc(retval, size));  
    if (tmp == nullptr) 
      { 
        free(retval); 
        retval = nullptr; 
        break; 
      } 
    retval = tmp; 
    memmove(retval+size-rc, buff, rc); 
  } 
  alGenBuffers(1, &buffer); 
  alBufferData(buffer, format, retval, size, freq); 
  free(retval); 
  free(buff); 
  ov_clear(&vf); 
}
That code working fine over linux and win32. But on amigaos4 i have some strange issue named : buffer expanding (reallocation) is eat so MUCH memory for little chunks, that it easyly ends up with eating up whole 2GB of memory and then just crashes.

Once i change "long allocated = 4*1024*16;" on "long allocated = 1024*1024*16;", so to avoid many reallocs by enlarge the buffer , it dodn't crashes. It just eat about 100mb of memory for my decompressed ogg file, and continue futher (and when game starts , memory of reallocations freed).

If i change on "long allocated = 2048*1024*16;", then it just eat up only 30-40mb while do reallocaitons and all fine.

What all of this mean : that the more realloc() we call, the more memory it eat up just for their own needs. If you will realloc many small chunks, even for 30-40-100mb of data, you will end up with loosing 1-2GB (!!) of memory and then crash.

As result: don't use newlib's realloc() ever.
User avatar
TSK
Beta Tester
Beta Tester
Posts: 225
Joined: Mon Dec 20, 2010 1:15 pm
Location: Home land of Santa C., sauna, sisu and salmiakki

Re: newlib's realloc() is using vast quantities of RAM

Post by TSK »

I just realized this is a 5-6 years old thread and nothing have happened during all those years !
Keep the party going !
Post Reply