newlib's realloc() is using vast quantities of RAM

A forum for general AmigaOS 4.x support questions that are not platform-specific

Re: newlib's realloc() is using vast quantities of RAM

Postby ssolie » Wed Mar 05, 2014 7:31 am

trixie wrote:None other can make crap answers as charming as you can, Steve :-D

I don't see how knowing the details will help you in any way whatsoever. The people that need to know already do know. The only question is how to schedule the work which is something I'm helping with. So really, you don't need to know.
Seatec Astronomy
User avatar
ssolie
Beta Tester
Beta Tester
 
Posts: 1010
Joined: Mon Dec 20, 2010 9:51 pm
Location: Canada

Re: newlib's realloc() is using vast quantities of RAM

Postby kas1e » Fri Jan 11, 2019 1:28 pm

Just for record and future reference if it ever will be deal with:

I meet with the same issue when trying to port some game. Game has some function which decode OGG files to the memory, with usage of realloc() , all done over c++ and compiled with gcc 8.2.0 and -O3 optimisation, over latest beta of newlib.library (53.30). Code of that function looks like this:

Code: Select all
void OpenALSound::CreateSampleFromOGG(const IDataStream& Stream, bool Looping)
{
  int Length = Stream.Size();
  printf ("DEBUG AUDIO: Length = %dn", Length);

  auto  pBuffer = new byte[Length];
  OggVorbis_File vf;
  ov_callbacks ov_call = {nullptr, nullptr, nullptr, nullptr};
  Stream.Read(Length, pBuffer);

  ov_open_callbacks(nullptr, &vf, (const char *)pBuffer, Length, ov_call);
  int bitstream = 0;
  vorbis_info *info = ov_info(&vf, -1);
  long size = 0;
  ALenum format = (info->channels == 1) ? AL_FORMAT_MONO16 : AL_FORMAT_STEREO16;
  ALuint freq = info->rate;
  printf("DEBUG AUDIO: info->channels = %dn", info->channels);
  printf("DEBUG AUDIO: info->rate = %dn", info->rate);

  long rc = 0;
  long allocated = 4*1024*16;
  ALubyte *retval = static_cast<ALubyte*>(malloc(allocated));
  char *buff = static_cast<char*>(malloc(allocated));

  #ifdef __amigaos4__
  while ( (rc = ov_read(&vf, buff, allocated, 1, 2, 1, &bitstream)) != 0 ) 
  #else     
  while ( (rc = ov_read(&vf, buff, allocated, 0, 2, 1, &bitstream)) != 0 ) 
  #endif
  {
    printf("AUDIO DEBUG : rc = %dn",rc);
    printf("AUDIO DEBUG : size = %dn",rc);
    if (rc <= 0) {
      break;
    }
    size += rc;

    ALubyte *tmp = static_cast<ALubyte*>(realloc(retval, size)); 
    if (tmp == nullptr)
      {
        free(retval);
        retval = nullptr;
        break;
      }
    retval = tmp;
    memmove(retval+size-rc, buff, rc);
  }
  alGenBuffers(1, &buffer);
  alBufferData(buffer, format, retval, size, freq);
  free(retval);
  free(buff);
  ov_clear(&vf);
}


That code working fine over linux and win32. But on amigaos4 i have some strange issue named : buffer expanding (reallocation) is eat so MUCH memory for little chunks, that it easyly ends up with eating up whole 2GB of memory and then just crashes.

Once i change "long allocated = 4*1024*16;" on "long allocated = 1024*1024*16;", so to avoid many reallocs by enlarge the buffer , it dodn't crashes. It just eat about 100mb of memory for my decompressed ogg file, and continue futher (and when game starts , memory of reallocations freed).

If i change on "long allocated = 2048*1024*16;", then it just eat up only 30-40mb while do reallocaitons and all fine.

What all of this mean : that the more realloc() we call, the more memory it eat up just for their own needs. If you will realloc many small chunks, even for 30-40-100mb of data, you will end up with loosing 1-2GB (!!) of memory and then crash.

As result: don't use newlib's realloc() ever.
kas1e
Beta Tester
Beta Tester
 
Posts: 481
Joined: Sat Jun 18, 2011 8:56 am

Re: newlib's realloc() is using vast quantities of RAM

Postby TSK » Sun Feb 03, 2019 3:53 pm

I just realized this is a 5-6 years old thread and nothing have happened during all those years !
Keep the party going !
User avatar
TSK
Beta Tester
Beta Tester
 
Posts: 220
Joined: Mon Dec 20, 2010 2:15 pm
Location: Home land of Santa C., sauna, sisu and salmiakki

Previous

Return to General AmigaOS

Who is online

Users browsing this forum: No registered users and 4 guests