Charles-François Natali <neolo...@free.fr> added the comment:
> Also that addresses the issue of "two threads inside different malloc
> implementations at the same time": it is currently not allowed with
> PyMem_Malloc.
>
That's not true.
You can perfectly have one thread inside PyMem_Malloc while another
one is inside libc's malloc.
For example, posix_listdir does:
Py_BEGIN_ALLOW_THREADS
dirp = opendir(name);
Py_END_ALLOW_THREADS
Where opendir calls malloc internally. Since the GIL is released, you
can have another thread inside PyMem_Malloc at the same time. This is
perfectly safe, as long as the libc's malloc version is thread-safe.
But with your patch, such code wouldn't be thread-safe anymore. This
patch implies that a thread can't call malloc directly or indirectly
(printf, opendir, and many others) while it doesn't hold the GIL. This
is going to break a lot of existing code.
This thread-safety issue is not theoretical: I wrote up a small
program with two threads, one allocating/freeing memory in loop with
glibc's malloc and the other one with dlmalloc: it crashes immediately
on a Linux box.
> Most python objects will be allocated in pymalloc arenas (if they are smaller
> than 256 bytes) which (if compiled with --with-pymalloc-mmap) will be
> directly allocated by calling mmap, or (without --with-pymalloc-mmap) will be
> allocated in dlmalloc by calling mmap (because arenas are 256KB).
> So most of the python objects will end up in mmap segments separate from the
> heap.
>
> The only allocations that will end up in the heap are for the medium python
> objects (>256 bytes and <256KB) or for allocations directly by calling
> PyMem_Malloc (and for a size <256KB).
Note that there are actually many objects falling into this category:
for example, on 64-bit, a dictionary exceeds 256B, and is thus
allocated directly from the heap (well, it changed really recently
actually), the same holds for medium-sized lists and strings. So,
depending on your workload, the heap can extend and shrink quite a
bit.
> If you are really concerned about mixing 2 malloc implementations in the
> heap, you can define "HAVE_MORECORE 0" in dlmalloc and that way dlmalloc will
> always use mmap and not use the heap at all.
>
It will also be slower, and consume more memory.
----------
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue3526>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
