newlib's realloc() is using vast quantities of RAM

A forum for general AmigaOS 4.x support questions that are not platform-specific
xenic
Posts: 1185
Joined: Sun Jun 19, 2011 12:06 am

Re: newlib's realloc() is using vast quantities of RAM

Post by xenic »

tonyw wrote:Why do you think there is a problem?

There is no reliable way of measuring or reporting the amount of available memory. To put it very simply, Avail only lists the memory that has never been used or has been used, freed and cleaned, ready for reuse. It does not count the memory that has been used and freed, but not yet cleaned and made ready for reuse. The Exec does not waste time cleaning and returning freed memory to the heap (where Avail can see it), until it needs to.

There is only a problem if you can't load something that you should be able to load. For instance, if you could load TimberWolf before your test, but not load it afterwards, then there could be a problem. Don't take the output of Avail as gospel. Most likely all the "missing" memory will reappear as soon as there is a need for it.
We keep hearing that claim every time we discuss memory allocation. I tested with Chris's program. SInce my X1000 has 2GB memory I don't have a SWAP partion and Chris's program runs to completion. After the program runs for about 5 seconds my available memory as displayed by WorkBench drops from 1.8GB to 470MB. While the program was running and WorkBench showed 470MB, I tried to copy a 500MB file to ram: and it failed. When I tried the copy with Dopus4, it failed and reported DOS error code 0. When I used the copy command, only 9MB of the 500MB file was copied and the Copy command just returned as though nothing was wrong.

My test indicates that in this case the claim that the memory will reappear when needed is wrong. Why is it that the response to reporting problems (with code proof) is frequently the same as the 5 stages of grief (the first step being denial). If people would actually try some testing with the provided code, it would be obvious if something is wrong (or not).

I think this test reveals a bug in the AmigaDOS copy command. If there is not enough memory or disk space available for a complete copy, an error should be returned; not an incomplete copy with no indication that anything went wrong. Regardless of the memory issues, that bug needs to be added to the list of things needing fixed.

EDIT: I tried your test after Chris's test program was running and the memory displayed in WorkBench dropped to 470MB. Timberwolf crashes when I try to start it. If I try to start OWB instead, it opens an empty window and stops. After OWB stops, all programs that require memory stop operating until Chris's program exits and returns all the memory to the system.
AmigaOne X1000 with 2GB memory - OS4.1 FE
xenic
Posts: 1185
Joined: Sun Jun 19, 2011 12:06 am

Re: newlib's realloc() is using vast quantities of RAM

Post by xenic »

chris wrote: I haven't tried it as part of a bigger project, but I suspect I'll get the same (crashing) problems as before - given that malloc, realloc and free are only three of the available memory allocation related functions in the c library, there's no guarantee that these are the only ones being used.
As I mentioned previously, because I don't have a SWAP partition your test code runs to completion. I'd say that your crash problem with the test code is SWAP related. I also took your test to a different level by comparing the results of a CLIB compiled version to a Newlib compiled version. The Newlib version seems to grab (and keep) most of the system memory after about 5 seconds. and doesn't free the memory until it exits. The CLIB version seems to work as though it's performing internal memory management like most C programming manuals describe. I can see larger chunks of memory allocated; with smaller chunks of memory released shortly afterward. However, there seems to be a huge time penalty imposed by the CLIB memory management. The newlib version of the program completed in 109 seconds while the CLIB version was still running after 12 hours!

I lowered the maximum amount of memory allocated in order to compare the run times of the newlib and CLIB versions of your test program. I tested with the timer program from OS4Depot and the results were amazing. Here is a brief chart of the results:

Max Memory Allocation------Clib time------------Newlib time
1 x 1024 x 1024------------------13.7146 sec-------0.0224 sec
2 x 1024 x 1024------------------52.2557 sec-------0.0659 sec
3 x 1024 x 1024----------------115.5391 sec-------0.1339 sec
4 x 1024 x 1024----------------206.6238 sec-------0.2289 sec
AmigaOne X1000 with 2GB memory - OS4.1 FE
User avatar
ssolie
Beta Tester
Beta Tester
Posts: 1010
Joined: Mon Dec 20, 2010 8:51 pm
Location: Canada
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by ssolie »

xenic wrote:My test indicates that in this case the claim that the memory will reappear when needed is wrong. Why is it that the response to reporting problems (with code proof) is frequently the same as the 5 stages of grief (the first step being denial). If people would actually try some testing with the provided code, it would be obvious if something is wrong (or not).
You are being overly dramatic. We are working on a workaround for this realloc() problem. That buys us time to find a root cause. The memory system is also changing for multicore support. Nobody is ignoring any problems.
ExecSG Team Lead
User avatar
trixie
Posts: 409
Joined: Thu Jun 30, 2011 2:54 pm
Location: Czech Republic

Re: newlib's realloc() is using vast quantities of RAM

Post by trixie »

@ssolie
We are working on a workaround for this realloc() problem.
Just wondering is this has been sorted out yet?
The Rear Window blog

AmigaOne X5000 @ 2GHz / 4GB RAM / Radeon RX 560 / ESI Juli@ / AmigaOS 4.1 Final Edition
SAM440ep-flex @ 667MHz / 1GB RAM / Radeon 9250 / AmigaOS 4.1 Final Edition
User avatar
salass00
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 530
Joined: Sat Jun 18, 2011 3:12 pm
Location: Finland
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by salass00 »

chris wrote: I haven't tried it as part of a bigger project, but I suspect I'll get the same (crashing) problems as before - given that malloc, realloc and free are only three of the available memory allocation related functions in the c library, there's no guarantee that these are the only ones being used.
Of course as newlib is an AmigaOS shared library if your project uses other newlib functions which indirectly call malloc() or any of the other memory functions then you need to replace those as well, like f.e. strdup(), strndup(), asprintf(), etc. similar to how I do for NTFS3G.

NTFS3G uses a pool based reimplementation of the memory functions. See the src/ntfs3g-startup_amigaos.c file in the source code:
https://dl.dropboxusercontent.com/u/265 ... .24-src.7z
User avatar
ssolie
Beta Tester
Beta Tester
Posts: 1010
Joined: Mon Dec 20, 2010 8:51 pm
Location: Canada
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by ssolie »

trixie wrote:Just wondering is this has been sorted out yet?
No and it isn't a high priority given it is a special case and a workaround is very easy to do.
ExecSG Team Lead
joerg
Posts: 371
Joined: Sat Mar 01, 2014 5:42 am

Re: newlib's realloc() is using vast quantities of RAM

Post by joerg »

ssolie wrote:If it works out it may be a viable workaround for the time being.
I don't think it will work, and the memory allocator in newlib can't be fixed as long as #4458 isn't added to the kernel.
Chris' example is just the worst case for an incomplete (because of the missing kernel functions) buddy allocator like the one newlib uses.
User avatar
trixie
Posts: 409
Joined: Thu Jun 30, 2011 2:54 pm
Location: Czech Republic

Re: newlib's realloc() is using vast quantities of RAM

Post by trixie »

@joerg

What's #4458?
The Rear Window blog

AmigaOne X5000 @ 2GHz / 4GB RAM / Radeon RX 560 / ESI Juli@ / AmigaOS 4.1 Final Edition
SAM440ep-flex @ 667MHz / 1GB RAM / Radeon 9250 / AmigaOS 4.1 Final Edition
User avatar
ssolie
Beta Tester
Beta Tester
Posts: 1010
Joined: Mon Dec 20, 2010 8:51 pm
Location: Canada
Contact:

Re: newlib's realloc() is using vast quantities of RAM

Post by ssolie »

trixie wrote:@joerg
What's #4458?
It is a bug number. You don't need to know the internals.
ExecSG Team Lead
User avatar
trixie
Posts: 409
Joined: Thu Jun 30, 2011 2:54 pm
Location: Czech Republic

Re: newlib's realloc() is using vast quantities of RAM

Post by trixie »

@ssolie
It is a bug number.
None other can make crap answers as charming as you can, Steve :-D
The Rear Window blog

AmigaOne X5000 @ 2GHz / 4GB RAM / Radeon RX 560 / ESI Juli@ / AmigaOS 4.1 Final Edition
SAM440ep-flex @ 667MHz / 1GB RAM / Radeon 9250 / AmigaOS 4.1 Final Edition
Post Reply