On 2020-05-29 14:28:59 +0900, Inada Naoki wrote: > pymalloc manages only small blocks of memory. > Large (more than 512 byte) memory blocks are managed by malloc/free. > > glibc malloc doesn't return much freed memory to OS.
That depends on what "much" means.
Glibc does return blocks to the OS which it allocated via mmap, By
default, these are allocations larger than 128 kB.
So that means that
* Blocks smaller than 512 bytes are returned to the OS if the arena they
are in is completely empty.
* Blocks between 512 bytes and 128 kB are not returned to the OS (unless
they happen to be at the end of the heap).
* Blocks larger than 128 kB are returned to the OS:
Most Python objects are probably smaller than 512 bytes, so whether
"much" is returned to the OS mostly depends on whether arenas ever get
completely empty.
This stupid little test program returns memory to the OS quite nicely,
because it allocates and frees lots of objects together:
---8<------8<------8<------8<------8<------8<------8<------8<---
#!/usr/bin/python3
import time
a = []
for i in range(10):
print('a', i)
x = []
for j in range(1000000):
x.append(j)
a.append(x)
time.sleep(1)
if i >= 5:
print('d', i - 5)
a[i - 5] = None
time.sleep(1)
print('f')
---8<------8<------8<------8<------8<------8<------8<------8<---
(run it with strace to watch the mmap/munmap system calls)
A program with more random allocation patterns may suffer severe
internal fragmentation and end up with many mostly empty arenas.
> You can try jemalloc instead of glibc.
I think that would only make a difference if you have lots of objects
between 512 B and 128 kB. And only if those of similar size are
allocated and freed together.
hp
--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | h...@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"
signature.asc
Description: PGP signature
-- https://mail.python.org/mailman/listinfo/python-list
