[issue18203] Replace calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc()

Changes by STINNER Victor : -- title: Replace calls to malloc() with PyMem_Malloc() -> Replace calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc() ___ Python tracker ___

[issue18203] Replace calls to malloc() with PyMem_Malloc()

Nick Coghlan added the comment: Note that CPython's main function accesses the C API before calling Py_Initialize(). This is insane, but fixing it is hard (and one of the main goals of PEP 432). I suggest using Py_IsInitialized() to exclude those from your debug checks for now. -- _

[issue18203] Replace calls to malloc() with PyMem_Malloc()

STINNER Victor added the comment: Ok, initial patches are attached. Let describe them a little bit. - pymem_debugcheckgil-2.patch: I don't think that this patch can be commited before the "bootstrap" issue is solved (Python doesn't start in debug mode with this patch when -W or -X command line

[issue18203] Replace calls to malloc() with PyMem_Malloc()

STINNER Victor added the comment: malloc_modules.patch: Patch for different Python modules. -- Added file: http://bugs.python.org/file30595/malloc_modules.patch ___ Python tracker __

[issue18203] Replace calls to malloc() with PyMem_Malloc()

STINNER Victor added the comment: malloc_init.patch: Patch for functions called at Python initialization. This patch is not complete: to parse "-X" option (PySys_AddXOption) and "-W" (PySys_AddWarnOption), PyMem_Malloc() and PyObject_Malloc() are still called indirectly. Fixing this issue may

[issue18203] Replace calls to malloc() with PyMem_Malloc()

STINNER Victor added the comment: See also issue #16742: "PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe". -- ___ Python tracker ___ _

[issue18203] Replace calls to malloc() with PyMem_Malloc()

STINNER Victor added the comment: pymem_debugcheckgil.patch: oops, a forgot a change in pgenmain.c. New fixed patch attached. -- Added file: http://bugs.python.org/file30592/pymem_debugcheckgil-2.patch ___ Python tracker

[issue18203] Replace calls to malloc() with PyMem_Malloc()

STINNER Victor added the comment: py_finalize.patch: modify Py_Finalize() to destroy the GIL after the last call to PyMem_Malloc() or PyObject_Malloc(). For example, PyCFunction_Fini() calls PyObject_GC_Del() which calls PyObject_FREE(). -- Added file: http://bugs.python.org/file30591/

[issue18203] Replace calls to malloc() with PyMem_Malloc()

STINNER Victor added the comment: pymem_debugcheckgil.patch: Patch adding check in debug mode to ensure that PyMem_Malloc() and PyObject_Malloc() are called with the GIL held. -- keywords: +patch Added file: http://bugs.python.org/file30590/pymem_debugcheckgil.patch ___

[issue18203] Replace calls to malloc() with PyMem_Malloc()

STINNER Victor added the comment: I commited my new API to customize memory allocators: New changeset 6661a8154eb3 by Victor Stinner in branch 'default': Issue #3329: Add new APIs to customize memory allocators http://hg.python.org/cpython/rev/6661a8154eb3 I added PyMem_RawMalloc(), PyMem_RawRe

[issue18203] Replace calls to malloc() with PyMem_Malloc()

Kristján Valur Jónsson added the comment: Keeping the GIL requirement is _very_ useful for PyMem_MALLOC et al. It allows applications to hook in their own monitoring code, accessible from python itself, without having to worry about conflicts with python. even if it were not for the GIL itself

[issue18203] Replace calls to malloc() with PyMem_Malloc()

STINNER Victor added the comment: >> We have to be careful with the GIL: PyMem_*() functions can only be >> called when holding the GIL. > (...) > I think there's a potential problem here :) I didn't understand the motivation to require the GIL held for PyMem_Malloc(). I searched in the source

[issue18203] Replace calls to malloc() with PyMem_Malloc()

Georg Brandl added the comment: > We have to be careful with the GIL: PyMem_*() functions can only be > called when holding the GIL. > Some libraries can be configured to use a custom memory allocators: > [...] > We should probably uses these functions to reuse Python allocators > (PyMem_Malloc

[issue18203] Replace calls to malloc() with PyMem_Malloc()

Christian Heimes added the comment: expat has a XML_Memory_Handling_Suite. You just have to replace XML_ParserCreate() and XML_ParserCreateNS with XML_ParserCreate_MM(). -- nosy: +christian.heimes ___ Python tracker

[issue18203] Replace calls to malloc() with PyMem_Malloc()

STINNER Victor added the comment: > Be aware about external code which allocate memory itself (i.e. expat). Well, I know that it will hard to cover 100% of the stdlib. I just want to replace the most obvious calls. Some libraries can be configured to use a custom memory allocators: - zlib: z

[issue18203] Replace calls to malloc() with PyMem_Malloc()

Serhiy Storchaka added the comment: Be aware about external code which allocate memory itself (i.e. expat). -- nosy: +serhiy.storchaka stage: -> needs patch ___ Python tracker _

[issue18203] Replace calls to malloc() with PyMem_Malloc()

New submission from STINNER Victor: The issue #3329 proposes an API to replace memory allocator functions. But Python calls directly malloc(), realloc() and free() in some functions, so custom allocators would not be used there. Examples of functions calling malloc/realloc/free directly: _PyS