[issue13390] Hunt memory allocations in addition to reference leaks

Roundup Robot added the comment: New changeset a85673b55177 by Antoine Pitrou in branch 'default': Following issue #13390, fix compilation --without-pymalloc, and make sys.getallocatedblocks() return 0 in that situation. http://hg.python.org/cpython/rev/a85673b55177 --

[issue13390] Hunt memory allocations in addition to reference leaks

Stefan Krah added the comment: Antoine Pitrou wrote: > Hmm, interesting. When built --without-pymalloc, we could make > sys.getallocatedblocks() always return 0, or we could not define it all > (which would make like a bit harder for regrtest). What do you think? Given the name getallocatedbloc

[issue13390] Hunt memory allocations in addition to reference leaks

Antoine Pitrou added the comment: > sys.gettotalrefcount() is only defined when Python is compiled in > debug mode. sys.getallocatedblocks() should only be available when > the > right debug option is present. This function is specific to CPython > anyway, Python module should not rely on this (t

[issue13390] Hunt memory allocations in addition to reference leaks

anatoly techtonik added the comment: Memory control over the stuff that Python creates is a practical feature that compensates OS disability for tracking memory usage. If all Python scripts could measure their memory usage, we could see more memory effective and adaptive programs around.

[issue13390] Hunt memory allocations in addition to reference leaks

STINNER Victor added the comment: sys.gettotalrefcount() is only defined when Python is compiled in debug mode. sys.getallocatedblocks() should only be available when the right debug option is present. This function is specific to CPython anyway, Python module should not rely on this (too much) ;

[issue13390] Hunt memory allocations in addition to reference leaks

Antoine Pitrou added the comment: > ``./configure --without-pymalloc'' fails here: > > gcc -pthread -Xlinker -export-dynamic -o python Modules/python.o > libpython3.4.a -lpthread -ldl -lutil -lm > libpython3.4.a(sysmodule.o): In function `sys_getallocatedblocks': > /home/stefan/hg/cpytho

[issue13390] Hunt memory allocations in addition to reference leaks

Stefan Krah added the comment: ``./configure --without-pymalloc'' fails here: gcc -pthread -Xlinker -export-dynamic -o python Modules/python.o libpython3.4.a -lpthread -ldl -lutil -lm libpython3.4.a(sysmodule.o): In function `sys_getallocatedblocks': /home/stefan/hg/cpython/./Python/sysm

[issue13390] Hunt memory allocations in addition to reference leaks

Antoine Pitrou added the comment: Committed and pushed! -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker ___ _

[issue13390] Hunt memory allocations in addition to reference leaks

Roundup Robot added the comment: New changeset c40f4c19d20b by Antoine Pitrou in branch 'default': Issue #13390: New function :func:`sys.getallocatedblocks()` returns the number of memory blocks currently allocated. http://hg.python.org/cpython/rev/c40f4c19d20b -- nosy: +python-dev ___

[issue13390] Hunt memory allocations in addition to reference leaks

Antoine Pitrou added the comment: > Minor bikeshedding/spelling nit: should > "_Py_AllocedBlocks" > be changed to > "_Py_AllocatedBlocks" > (and s/_Py_GetAllocedBlocks/_Py_GetAllocatedBlocks/)? Right indeed. I'll do the change. -- ___ Python trac

[issue13390] Hunt memory allocations in addition to reference leaks

Dave Malcolm added the comment: Minor bikeshedding/spelling nit: should "_Py_AllocedBlocks" be changed to "_Py_AllocatedBlocks" (and s/_Py_GetAllocedBlocks/_Py_GetAllocatedBlocks/)? -- ___ Python tracker _

[issue13390] Hunt memory allocations in addition to reference leaks

Antoine Pitrou added the comment: Updated patch with doc. If noone objects, I will commit soon. -- Added file: http://bugs.python.org/file28227/debugblocks6.patch ___ Python tracker

[issue13390] Hunt memory allocations in addition to reference leaks

Antoine Pitrou added the comment: As it turns out, ctypes does leak memory: see issue 16628. -- ___ Python tracker ___ ___ Python-bugs

[issue13390] Hunt memory allocations in addition to reference leaks

Antoine Pitrou added the comment: Here is an updated patch. test_ctypes still leaks memory blocks with it: $ ./python -m test -R 3:8 test_ctypes [1/1] test_ctypes beginning 11 repetitions 12345678901 ... test_ctypes leaked [2, 2, 1, 1, 1, 1, 1, 1] memory blocks, sum=10 -- nosy:

[issue13390] Hunt memory allocations in addition to reference leaks

Changes by Antoine Pitrou : -- versions: +Python 3.4 -Python 3.3 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue13390] Hunt memory allocations in addition to reference leaks

anatoly techtonik added the comment: On Mon, Dec 12, 2011 at 6:11 PM, STINNER Victor wrote: > > STINNER Victor added the comment: > > > How different is the performance cost of this solution compared > > to inserting DTrace probe for the same purpose? > > DTrace is only available on some platf

[issue13390] Hunt memory allocations in addition to reference leaks

Changes by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue13390] Hunt memory allocations in addition to reference leaks

STINNER Victor added the comment: > How different is the performance cost of this solution compared > to inserting DTrace probe for the same purpose? DTrace is only available on some platforms (Solaris and maybe FreeBSD?). -- ___ Python tracker

[issue13390] Hunt memory allocations in addition to reference leaks

anatoly techtonik added the comment: How different is the performance cost of this solution compared to inserting DTrace probe for the same purpose? -- nosy: +techtonik ___ Python tracker

[issue13390] Hunt memory allocations in addition to reference leaks

Antoine Pitrou added the comment: > So, I am not sure whether there really is a leak, if this is just > a very senstive area of 'regrtest.py', or something else I am missing. Well, test_ctypes seems to be the only test exhibiting that behaviour. And since your script reproduces it, it's probabl

[issue13390] Hunt memory allocations in addition to reference leaks

Meador Inge added the comment: I looked at the 'ctypes' "leak" a bit. I haven't determined exactly what is going on, but the leak has something to do with a change in the patch that runs 'dash_R_cleanup' twice instead of once. The new behavior can be reduced to something like: import sys,

[issue13390] Hunt memory allocations in addition to reference leaks

Meador Inge added the comment: On Fri, Dec 2, 2011 at 12:12 PM, Antoine Pitrou wrote: > yes, c6dafa2e2594 was found with this patch. It's the only one, though > (there's also a leak in test_ctypes but I don't want to investigate :-)). I'll take a look at the ctypes leak. -- nosy: +m

[issue13390] Hunt memory allocations in addition to reference leaks

Antoine Pitrou added the comment: > The feature is interesting, but I'm not convinced that a very simple > counter is enough to track memory leaks. It may help the CPython test > suite, but what about real world application? Good question. A simple counter is the only thing we can enable by def

[issue13390] Hunt memory allocations in addition to reference leaks

Changes by Dave Malcolm : -- nosy: +dmalcolm ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python

[issue13390] Hunt memory allocations in addition to reference leaks

STINNER Victor added the comment: The overhead on PyObject_Malloc() is just an increment on an integer, so it is very low (or null). The feature is interesting, but I'm not convinced that a very simple counter is enough to track memory leaks. It may help the CPython test suite, but what abou

[issue13390] Hunt memory allocations in addition to reference leaks

Antoine Pitrou added the comment: > > If someone finds a way to sanitize Valgrind output, > > Did you use the valgrind suppression file (Misc/valgrind-python.supp)? Ah, I hadn't. But using it doesn't seem to make much of a difference. Also, the suppressions file seems quite inflexible (it has

[issue13390] Hunt memory allocations in addition to reference leaks

Charles-François Natali added the comment: > If someone finds a way to sanitize Valgrind output, Did you use the valgrind suppression file (Misc/valgrind-python.supp)? If yes, then one could simply use --gen-suppressions=yes to add those spurious warning to the suppression file. > a daily Val

[issue13390] Hunt memory allocations in addition to reference leaks

Antoine Pitrou added the comment: > Valgrind does a much better job at this: it will also show you where > the leaked blocks were allocated. > OTOH, Valgrind is Linux-only and slow, but since I haven't used the > '-R' option much, I don't know how usable this will be in practice > (detecting mem

[issue13390] Hunt memory allocations in addition to reference leaks

Nick Coghlan added the comment: This will likely be a decent "you have a problem" indicator, but you may still need tools like Valgrind to actually track down the *cause* of that problem. -- ___ Python tracker __

[issue13390] Hunt memory allocations in addition to reference leaks

Charles-François Natali added the comment: > Together with -R, it can help chase those memory leaks which aren't > reference leaks (see c6dafa2e2594). Valgrind does a much better job at this: it will also show you where the leaked blocks were allocated. OTOH, Valgrind is Linux-only and slow,

[issue13390] Hunt memory allocations in addition to reference leaks

Antoine Pitrou added the comment: And the latest patch (debugblocks3.patch) adds said heuristic. -- ___ Python tracker ___ ___ Python

[issue13390] Hunt memory allocations in addition to reference leaks

Changes by Antoine Pitrou : Added file: http://bugs.python.org/file23681/debugblocks3.patch ___ Python tracker ___ ___ Python-bugs-list mailin

[issue13390] Hunt memory allocations in addition to reference leaks

Changes by Antoine Pitrou : Added file: http://bugs.python.org/file23680/debugblocks2.patch ___ Python tracker ___ ___ Python-bugs-list mailin

[issue13390] Hunt memory allocations in addition to reference leaks

Antoine Pitrou added the comment: Thanks for the comments, here is a new patch addressing them. I've kept the C API available in all builds (since it's private), but sys.getallocatedblocks() is only available in debug builds. As for the memory leak run results, I think we may have to devise a

[issue13390] Hunt memory allocations in addition to reference leaks

Nick Coghlan added the comment: I added some review comments to the patch, but I'm not sure how usable this is going to be in practice. References generally stay fairly stable while using the interactive interpreter, but the new block accounting jumps around all over the place due to the inte

[issue13390] Hunt memory allocations in addition to reference leaks

Changes by Andreas Stührk : -- nosy: +Trundle ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pytho

[issue13390] Hunt memory allocations in addition to reference leaks

New submission from Antoine Pitrou : This patch adds a counting of the number of allocated memory blocks (through the PyObject_Malloc API). Together with -R, it can help chase those memory leaks which aren't reference leaks (see c6dafa2e2594). The sys.getallocedblocks() function is also availa