Sébastien Sablé <sa...@users.sourceforge.net> added the comment: Another reason why you should not force dlmalloc for all applications linked with libpython is because dlmalloc is (by default) not thread safe, while the system malloc is (generally) thread-safe. It is possible to define a constant in dlmalloc to make it thread-safe (using locks) but it will be slower and it is not needed in Python since the GIL must be held when using PyMem_ functions.
If a thread-safe implementation was needed, it would be better to switch to ptmalloc2. Also that addresses the issue of "two threads inside different malloc implementations at the same time": it is currently not allowed with PyMem_Malloc. > Most of the allocations come from the heap - through sbrk Most python objects will be allocated in pymalloc arenas (if they are smaller than 256 bytes) which (if compiled with --with-pymalloc-mmap) will be directly allocated by calling mmap, or (without --with-pymalloc-mmap) will be allocated in dlmalloc by calling mmap (because arenas are 256KB). So most of the python objects will end up in mmap segments separate from the heap. The only allocations that will end up in the heap are for the medium python objects (>256 bytes and <256KB) or for allocations directly by calling PyMem_Malloc (and for a size <256KB). Also dlmalloc will not call sbrk for each of those allocations: dlmalloc allocates some large memory pools and manage the smaller allocations within those pools in a very efficient way. So the heap fragmentation should be indeed reduced by using dlmalloc. Most modern malloc implementations are also using pools/arenas anyway, so the heap will mostly contain a mix of native malloc arenas and dlmalloc pools. So the fragmentation should not be too much of a concern if you mix 2 malloc implementations. Here is OpenSolaris malloc implementation for example: http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libmalloc/common/malloc.c#514 Concerning trimming: the reason why I am proposing to use dlmalloc on AIX and Solaris is that the native malloc/free do not correctly trim the heap in the first place on those platforms! If malloc/free correctly worked on those platforms and the heap was trimmed when possible, I would not have taken the trouble of proposing this patch and using dlmalloc, I would happily use the native malloc/free. So mixing 2 malloc implementations should not be a problem as long as you keep track of the right 'free' implementation to use for each pointer (which should already be the case when you call PyMem_Malloc/PyMem_Free instead of malloc/free). If you are really concerned about mixing 2 malloc implementations in the heap, you can define "HAVE_MORECORE 0" in dlmalloc and that way dlmalloc will always use mmap and not use the heap at all. My application uses the provided patch so that dlmalloc is used for Python objects and the native malloc for all the rest (much less consuming than the Python part) on AIX and SunOS. It has been in production for years and we never experienced any crash related to memory problems. ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue3526> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
