Croepha <croe...@gmail.com> writes: > Does python in general terms (apart from extensions or gc manipulation), > exhibit a "high water" type leak of allocated memory in recent python > versions (2.7+)?
Likely because it is very difficult to avoid. Would you need to definitely prevent it is "memory compaction": not only is garbage collected and freed but in addition, the used memory is also relocated to form contigous blocks of used and free memory. Without memory compaction, long running processes tend to suffer from "memory fragmentation": while sufficient free memory is available, it is available only in small blocks, not large enough for some memory requests and those requests then call for more memory from the operating system. When this memory is later released, it may become split up in smaller blocks and when another large memory request arrives, new memory may be requested from the operating system (even though the original block would have been large enough). Python tries hard to limit the effect of fragmentation (maintaining the free blocks in bins of given size) but cannot eliminate it completely. In order to support memory compaction, all C extensions must adhere to a strict memory access protocol: they cannot freely use C pointers to access the memory (as the compaction may invalidate those pointers at any time) but must beforehand announce "I will now be using this pointer" (such that the compaction does not move the memory) and afterwards announce "I am no longer using this pointer" (such that memory compaction becomes again possible for this memory). As you see from the description, memory compaction presents a heavy burden for all extension writers. -- https://mail.python.org/mailman/listinfo/python-list
