[issue3329] API for setting the memory allocator used by Python

miss-islington added the comment: New changeset ea46c7bc503d1103d60c0ec7023bb52c5defa11d by Miss Islington (bot) in branch '3.9': bpo-3329: Fix typo in PyObjectArenaAllocator doc (GH-24795) https://github.com/python/cpython/commit/ea46c7bc503d1103d60c0ec7023bb52c5defa11d -- ___

[issue3329] API for setting the memory allocator used by Python

miss-islington added the comment: New changeset 5ca02c4799c07dba5192f87f9f2378dc24097166 by Miss Islington (bot) in branch '3.8': bpo-3329: Fix typo in PyObjectArenaAllocator doc (GH-24795) https://github.com/python/cpython/commit/5ca02c4799c07dba5192f87f9f2378dc24097166 -- ___

[issue3329] API for setting the memory allocator used by Python

Change by miss-islington : -- pull_requests: +23568 pull_request: https://github.com/python/cpython/pull/24800 ___ Python tracker ___ ___

[issue3329] API for setting the memory allocator used by Python

Change by miss-islington : -- nosy: +miss-islington nosy_count: 17.0 -> 18.0 pull_requests: +23567 pull_request: https://github.com/python/cpython/pull/24799 ___ Python tracker

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: New changeset 0d6bd1ca7c683137d52041194f3a2b02219f225a by Victor Stinner in branch 'master': bpo-3329: Fix typo in PyObjectArenaAllocator doc (GH-24795) https://github.com/python/cpython/commit/0d6bd1ca7c683137d52041194f3a2b02219f225a -- __

[issue3329] API for setting the memory allocator used by Python

Change by STINNER Victor : -- pull_requests: +23563 pull_request: https://github.com/python/cpython/pull/24795 ___ Python tracker ___ ___

[issue3329] API for setting the memory allocator used by Python

Kristján Valur Jónsson added the comment: Well done. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://m

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: Buildbots are happy, changeset 51ed51d10e60 fixed the memory leak on Windows XP. Let's close this issue, 5 years after its creation! -- resolution: -> fixed status: open -> closed ___ Python tracker

[issue3329] API for setting the memory allocator used by Python

Roundup Robot added the comment: New changeset 51ed51d10e60 by Victor Stinner in branch 'default': Issue #3329: Fix _PyObject_ArenaVirtualFree() http://hg.python.org/cpython/rev/51ed51d10e60 -- ___ Python tracker __

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: It looks like the changeset ca78c974e938 broke the "x86 XP-4 3.x" buildbot: buildbot.python.org/all/builders/x86 XP-4 3.x/builds/8795/ Traceback (most recent call last): File "../lib/test/regrtest.py", line 1305, in runtest_inner test_runner() File "D:\

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: Ok, let see if buildbots like the PEP 445 (keep this issue open until we have the result of all 3.4 buildbots). I created the issue #18392 to document PyObject_Malloc(). -- ___ Python tracker

[issue3329] API for setting the memory allocator used by Python

Roundup Robot added the comment: New changeset ca78c974e938 by Victor Stinner in branch 'default': Issue #3329: Implement the PEP 445 http://hg.python.org/cpython/rev/ca78c974e938 -- ___ Python tracker _

[issue3329] API for setting the memory allocator used by Python

Changes by STINNER Victor : Removed file: http://bugs.python.org/file30629/py_setallocators-6.patch ___ Python tracker ___ ___ Python-bugs-list

[issue3329] API for setting the memory allocator used by Python

Changes by STINNER Victor : Removed file: http://bugs.python.org/file30653/py_setallocators-8.patch ___ Python tracker ___ ___ Python-bugs-list

[issue3329] API for setting the memory allocator used by Python

Changes by STINNER Victor : Removed file: http://bugs.python.org/file30645/py_setallocators-7.patch ___ Python tracker ___ ___ Python-bugs-list

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: Updated patch (version 9): - update API to the last version of the PEP - PYMEM_DOMAIN_RAW now also have a well defined behaviour when requesting an allocation of zero bytes: PyMem_RawMalloc(0) now calls malloc(1) - enhance the documentation (ex: mention default

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: Update patch according to the last version of the PEP. -- Added file: http://bugs.python.org/file30653/py_setallocators-8.patch ___ Python tracker __

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: Update the patch to follow the API described in the PEP 445 (2013-06-18 22:33:41 +0200). -- Added file: http://bugs.python.org/file30645/py_setallocators-7.patch ___ Python tracker

[issue3329] API for setting the memory allocator used by Python

Changes by STINNER Victor : Removed file: http://bugs.python.org/file30453/py_setallocators-2.patch ___ Python tracker ___ ___ Python-bugs-list

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: Convert changeset 6661a8154eb3 into a patch: py_setallocators-6.patch. -- Added file: http://bugs.python.org/file30629/py_setallocators-6.patch ___ Python tracker __

[issue3329] API for setting the memory allocator used by Python

Changes by STINNER Victor : Removed file: http://bugs.python.org/file29324/py_setallocators.patch ___ Python tracker ___ ___ Python-bugs-list m

[issue3329] API for setting the memory allocator used by Python

Changes by STINNER Victor : Removed file: http://bugs.python.org/file30575/py_setallocators-5.patch ___ Python tracker ___ ___ Python-bugs-list

[issue3329] API for setting the memory allocator used by Python

Changes by STINNER Victor : Removed file: http://bugs.python.org/file30538/py_setallocators-3.patch ___ Python tracker ___ ___ Python-bugs-list

[issue3329] API for setting the memory allocator used by Python

Changes by STINNER Victor : Removed file: http://bugs.python.org/file30564/py_setallocators-4.patch ___ Python tracker ___ ___ Python-bugs-list

[issue3329] API for setting the memory allocator used by Python

Changes by Trent Nelson : -- nosy: +trent ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org

[issue3329] API for setting the memory allocator used by Python

Roundup Robot added the comment: New changeset b1455dd08000 by Victor Stinner in branch 'default': Revert changeset 6661a8154eb3: Issue #3329: Add new APIs to customize memory allocators http://hg.python.org/cpython/rev/b1455dd08000 -- ___ Python tra

[issue3329] API for setting the memory allocator used by Python

Roundup Robot added the comment: New changeset 6661a8154eb3 by Victor Stinner in branch 'default': Issue #3329: Add new APIs to customize memory allocators http://hg.python.org/cpython/rev/6661a8154eb3 -- nosy: +python-dev ___ Python tracker

[issue3329] API for setting the memory allocator used by Python

Changes by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: > To be exhaustive, another patch should be developed to replace > all calls for malloc/realloc/free by > PyMem_Malloc/PyMem_Realloc/PyMem_Free. I created issue #18203 for this point. > PyObject_Malloc() is still using mmap() or malloc() internally > for exampl

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: > This patch does not propose a simple API to reuse internal > debug hooks when replacing system (PyMem) allocators. Ok, this is now fixed with new patch (version 5). Nick does not want a new environment variable, so I added instead a new function PyMem_SetupDe

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: py_setallocators-4.patch: - Oh, I forgot another change: Py*_Get/SetAllocators() cannot fail anymore (because of an unknown API identifier), so the return type is now void I just saw that I forgot ".. versionadded:: 3.4" in the doc. -- ___

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: New version (4) of the patch: - move the opaque pointer (now called "void *ctx", "context") as the first parameter instead of the last parameter, as done in zlib, lzma and Oracle's OCI APIs; ctx is also the first parameter of Py*_GetFunctions() and Py*_SetFu

[issue3329] API for setting the memory allocator used by Python

Changes by STINNER Victor : Added file: http://bugs.python.org/file30559/pybench.txt ___ Python tracker ___ ___ Python-bugs-list mailing list U

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: > If I understood correctly, the overhead is really really low (near zero). See attached output pybench.txt and benchmarks.txt. -- ___ Python tracker __

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: Benchmark of py_setallocators-3.patch: - benchmarks suite (-b 2n3): some tests are 1.04x faster, some tests are 1.04 slower, significant is between 115 and -191. I don't understand these output, but I guess that the overhead cannot be seen with such test. -

[issue3329] API for setting the memory allocator used by Python

Changes by STINNER Victor : Added file: http://bugs.python.org/file30560/benchmarks.txt ___ Python tracker ___ ___ Python-bugs-list mailing lis

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: Amaury Forgeot d'Arc added the comment: > I prefer the new version without PYMEM_TRACE_MALLOC :-) Well, py_setallocators-filename.patch is more a proof-of-concept showing how to use my Py_SetAllocators() API to pass the C trace (filename/line number), than a rea

[issue3329] API for setting the memory allocator used by Python

Amaury Forgeot d'Arc added the comment: I prefer the new version without PYMEM_TRACE_MALLOC :-) Can we rename "API" and "api_id" to something more specific? maybe DOMAIN and domain_id? -- ___ Python tracker __

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: New version of the patch, py_setallocators-3.patch: - _PyMem_DebugMalloc(), _PyMem_DebugFree() and _PyMem_DebugRealloc() are now setup as hooks to the system allocator and are hook on PyMem API *and* on PyObject API - move "if (size > PY_SSIZE_T_MAX)" check i

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: py_setallocators-filename.patch: Here is a try to define an API providing the filename and line number of the C code. The Py_SetAllocators() API is unchanged: PyAPI_FUNC(int) Py_SetAllocators( char api, void* (*malloc) (size_t size, void *user_data),

[issue3329] API for setting the memory allocator used by Python

Kristján Valur Jónsson added the comment: I'd like to add some argument to providing a "file" and "line number" to the allocation api. I know that currently this is not provided e.g. by the PyMem_Allocate() functions, but I think it would be wise to provide a "debug" version of these function

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: New patch (version 2), more complete: * add "void *data" argument to all allocator functions * add "block" API used for pymalloc allocator to allocate arenas. Use mmap or malloc, but may use VirtualAlloc in a near future (see #13483). Callbacks prototype:

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: """ I'm happy with the api you provide, with a small addition: PyAPI_FUNC(int) Py_SetAllocators( char api, void* (*malloc) (size_t size, void *data), void* (*realloc) (void* ptr, size_t size, void *data), void (*free) (void* ptr, void *data),

[issue3329] API for setting the memory allocator used by Python

Kristján Valur Jónsson added the comment: Also, our ccpmem.h, the interface to the ccpmem.cpp, internal flexible memory allocator framework. Again, just FYI. There are no trade secrets here, so please ask me for more details, if interested. One particular trick we have been using, which might

[issue3329] API for setting the memory allocator used by Python

Kristján Valur Jónsson added the comment: Hi. the file and line arguments are for expanding from macros such as PyMem_MALLOC. I had them added because they provide the features of a comprehensive debugging API. Of course, I'm not showing you the entire set of modifications that we have made

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: > typedef void *(*PyCCP_Malloc_t)(size_t size, void *arg, const char *file, int > line, const char *msg); I don't understand the purpose of the filename and line number. Python does not have such information. Is it just to have the API expected by Unreal engin

[issue3329] API for setting the memory allocator used by Python

Kristján Valur Jónsson added the comment: Absolutely. Although there is a very useful scenario where this could be consided a run-time setting: # turboprofiler.py # Load up the memory hooker which will supply us with all the info import _turboprofiler _turboprofiler.hookup() Perhaps p

[issue3329] API for setting the memory allocator used by Python

Nick Coghlan added the comment: Note that I'm definitely open to including extra settings to set up custom allocators as part of Py_CoreConfig in PEP 432 (http://www.python.org/dev/peps/pep-0432/#pre-initialization-phase). I don't really want to continue the tradition of additional PySet_* API

[issue3329] API for setting the memory allocator used by Python

Kristján Valur Jónsson added the comment: At ccp we have something similar. We are embedding python in the UnrealEngine on the PS3 and need to get everything through their allocators. For the purpose of flexibility, we added an api similar to the OPs, but more flexible: /* Support for custom

[issue3329] API for setting the memory allocator used by Python

Changes by Kristján Valur Jónsson : -- nosy: +kristjan.jonsson ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: h

[issue3329] API for setting the memory allocator used by Python

Changes by Gregory P. Smith : -- nosy: +gregory.p.smith ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://m

[issue3329] API for setting the memory allocator used by Python

Amaury Forgeot d'Arc added the comment: Some customizable memory allocators I know have an extra parameter "void *opaque" that is passed to all functions: - in zlib: zalloc and zfree: http://www.zlib.net/manual.html#Usage - same thing for bz2. - lzma's ISzAlloc: http://www.asawicki.info/news_13

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: To be exhaustive, another patch should be developed to replace all calls for malloc/realloc/free by PyMem_Malloc/PyMem_Realloc/PyMem_Free. PyObject_Malloc() is still using mmap() or malloc() internally for example. Other examples of functions calling malloc/re

[issue3329] API for setting the memory allocator used by Python

Changes by STINNER Victor : -- versions: +Python 3.4 -Python 3.3 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue3329] API for setting the memory allocator used by Python

STINNER Victor added the comment: I attached a patch that I wrote for Wyplay: py_setallocators.patch. The patch adds two functions: PyAPI_FUNC(int) Py_GetAllocators( char api, void* (**malloc_p) (size_t), void* (**realloc_p) (void*, size_t), void (**free_p) (void*) ); PyAPI

[issue3329] API for setting the memory allocator used by Python

Changes by Brett Cannon : -- nosy: -brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.py

[issue3329] API for setting the memory allocator used by Python

Antoine Pitrou added the comment: All this needs is a patch. Note that there are some places where we call malloc()/free() without going through our abstraction API. This is not in allocation-heavy paths, though. -- nosy: +pitrou versions: +Python 3.3 -Python 3.2 _

[issue3329] API for setting the memory allocator used by Python

Changes by Terry J. Reedy : -- versions: -Python 2.7, Python 3.1 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue3329] API for setting the memory allocator used by Python

Changes by Tim Lesher : -- nosy: +tlesher ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org

[issue3329] API for setting the memory allocator used by Python

PJ McNerney added the comment: Has the ability to set the memory allocator been added to Python 2.7/3.1? Thanks, PJ -- nosy: +pjmcnerney versions: +Python 3.2 -Python 2.5, Python 2.6, Python 3.0 ___ Python tracker

[issue3329] API for setting the memory allocator used by Python

Changes by John Szakmeister : -- nosy: +jszakmeister ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail

[issue3329] API for setting the memory allocator used by Python

Jukka Laurila added the comment: Brett is right. Macroing the memory allocator is a better choice than forcing indirection on all platforms. We did this on Python for S60, using the macros PyCore_{MALLOC,REALLOC,FREE}_FUNC for interpreter's allocations, and then redirected those to a mechanism t

[issue3329] API for setting the memory allocator used by Python

Neil Richardson added the comment: I'll be in agreement here. I integrated Python into a game engine not too long ago, and had to a do a fair chunk of work to isolate Python into it's own heap - given that fragmentation on low memory systems can be a bit of a killer. Would also make future up

[issue3329] API for setting the memory allocator used by Python

Adam Olsen <[EMAIL PROTECTED]> added the comment: Basically you just want to kick the malloc implementation into doing some housekeeping, freeing its caches? I'm kinda surprised you don't add the hook directly to your libc's malloc. IMO, there's no use-case for this until Py_Finalize can comple

[issue3329] API for setting the memory allocator used by Python

Raymond Hettinger <[EMAIL PROTECTED]> added the comment: I think it is reasonable to get a macro definition change into 2.6. The OP's request is essential for his application (running Python on Nokia phones) and it would be a loss to wait two years for this. Also, his request for a macro will ena

[issue3329] API for setting the memory allocator used by Python

Nick Coghlan <[EMAIL PROTECTED]> added the comment: Given where we are in the release cycle, I've bumped the target releases to 2.7/3.1. So Symbian are probably going to have to do something port-specific anyway in order to get 2.6/3.0 up and running. And in terms of hooking into this kind of th

[issue3329] API for setting the memory allocator used by Python

Jukka Laurila <[EMAIL PROTECTED]> added the comment: Brett, the ability to define the allocator dynamically at runtime could be a compile time option, turned on by default only on small memory platforms. On most platforms you can live with plain old malloc and may want to avoid the indirection. I

[issue3329] API for setting the memory allocator used by Python

Adam Olsen <[EMAIL PROTECTED]> added the comment: How would this allow you to free all memory? The interpreter will still reference it, so you'd have to have called Py_Finalize already, and promise not to call Py_Initialize afterwords. This further supposes the process will live a long time aft

[issue3329] API for setting the memory allocator used by Python

Brett Cannon <[EMAIL PROTECTED]> added the comment: Is registering pointers to functions really necessary, or would defining macros work as well? From a performance perspective I would like to avoid having a pointer indirection step every time malloc/realloc/free is called. I guess my question b

[issue3329] API for setting the memory allocator used by Python

New submission from Jukka Laurila <[EMAIL PROTECTED]>: Currently Python always uses the C library malloc/realloc/free as the underlying mechanism for requesting memory from the OS, but especially on memory-limited platforms it is often desirable to be able to override the allocator and to redirec