[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: "I don't see any reason why not to.": Ok, I created the issue #30054. -- ___ Python tracker ___ ___

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Julian Taylor added the comment: I don't see any reason why not to. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubs

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: Julian Taylor: "The api looks good to me. Works fine in numpy." Cool! Should it be made public in that case? (Remove _ prefix and document it.) -- ___ Python tracker _

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Julian Taylor added the comment: The api looks good to me. Works fine in numpy. -- nosy: +jtaylor ___ Python tracker ___ ___ Python-bu

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: So, the API is implemented, but I leave it as private because nobody tried it whereas I was waiting for a feedback from numpy at least. If you want a public API in Python 3.7, please tell if the API fits your use case and if the implementation works.

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Antoine Pitrou added the comment: Uh... ok, I thought you wanted some feedback on the patches posted, since I didn't know you had committed a new version of them. I haven't tried to use the new API (for various reasons it's a bit cumbersome to use a self-compiled Python for Numba), but I'm sur

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: > Which patch are we talking about? I'm talking about this change which was already merged into the default branch of Python in March 2016: New changeset 60655e543d8a by Victor Stinner in branch 'default': Add C functions _PyTraceMalloc_Track() https://hg.pyth

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Antoine Pitrou added the comment: Which patch are we talking about? In tracemalloc_track-3.patch, I see: + If memory block is already tracked, update the existing trace. */ +PyAPI_FUNC(int) _PyTraceMalloc_Track(void *ptr, size_t size); + +/* Untrack an allocated memory block in the tracemalloc

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: > I may be a bit confused, but the "domain" integer you added in issue 26588 > doesn't seem to be part of this API... Is it deliberate? They are part of this API. msg262180: """Ok, I added the following C functions: int _PyTraceMalloc_Track(_PyTraceMalloc_d

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Antoine Pitrou added the comment: I may be a bit confused, but the "domain" integer you added in issue 26588 doesn't seem to be part of this API... Is it deliberate? -- ___ Python tracker _

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: > Is the timeout part of the API? :-) I don't want to make the API public before you validated that it is usable for your use case or to track numpy memory usage. I guess that the first beta release of Python 3.6 is the deadline for this issue. Otherwise, we

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Antoine Pitrou added the comment: Is the timeout part of the API? :-) -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsu

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: @Nathaniel, Antoine: last ping before timeout :-p -- ___ Python tracker ___ ___ Python-bugs-list mai

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: Nathaniel, Antoine: ping again? No reaction to the new shiny C API of tracemalloc that *you* requested? :-p -- ___ Python tracker ___ _

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: Nathaniel, Antoine: ping? -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: tracemalloc_track_fd.py: Proof-of-concept to track file descriptors using monkey-patching on os.open() and os.close(). It's just to show to the API can be used in Python, I don't think that it's very useful to track where file descriptors are allocated. -

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: Ok, I added the following C functions: int _PyTraceMalloc_Track(_PyTraceMalloc_domain_t domain, Py_uintptr_t ptr, size_t size); int _PyTraceMalloc_Untrack(_PyTraceMalloc_domain_t domain, Py_uintptr_t ptr); Antoine, Nathaniel: Please play with it, I'm waiti

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Roundup Robot added the comment: New changeset 60655e543d8a by Victor Stinner in branch 'default': Add C functions _PyTraceMalloc_Track() https://hg.python.org/cpython/rev/60655e543d8a -- nosy: +python-dev ___ Python tracker

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: Instead of having to change the API for tracking GPU memory, I suggest to try a hack. Memory allocations use an alignement. For pymalloc, it's 8 bytes for example. On a GPU I expect at least 2 bytes or more likey something much bigger like 16 bytes. For malloc, I

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: > So you guarantee tracemalloc itself won't use such a hack for other purposes? ;-) Hum, maybe we can make the "hack" "official": add two C macro to store and retrieve the domain from a pointer. The macro can validate that the domain is smaller or equal to 8 u

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Antoine Pitrou added the comment: Le 18/03/2016 04:57, Nathaniel Smith a écrit : > > What I'm talking about is different: I think it should be possible > to re-use the tracemalloc infrastructure to track other resources besides "heap allocations". So for my use case, it's crucial that we index b

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Nathaniel Smith added the comment: There are other leakable resources besides heap and GPU memory -- shmem segments and file descriptors are two that I thought of earlier, but there are probably others too. (Note that not all file descriptors are associated with a python file object.) I guess

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Antoine Pitrou added the comment: A n-GPU system will have n+1 address spaces. Such systems exist in high-performance computing. You can also probably find weird systems (embedded, mostly, I'd say) with more than two address spaces. -- ___ Python t

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: Nathaniel Smith: > So PyMem_Malloc would just call PyMem_RecordAlloc("heap", ptr, size) (or act > equivalently to something that called that, etc.), but something like PyCuda > might do PyMem_RecordAlloc("gpu", ptr, size) to track allocations in GPU > memory.

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: > So you guarantee tracemalloc itself won't use such a hack for other purposes? ;-) As the defacto maintainer of the module, yes, I can guarantee that. > FWIW, LLVM calls the "domain" "address space", and simply uses an > integer for it (0 being the default CPU

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Antoine Pitrou added the comment: Le 18/03/2016 08:09, STINNER Victor a écrit : > > If it's the minimum is 2 bytes, good. Use the lowest bit as a "GPU" flag! > pointer|1. So you guarantee tracemalloc itself won't use such a hack for other purposes? ;-) -- _

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Nathaniel Smith added the comment: I think we're talking past each other :-). > If I change tracemalloc, it's not to fullfit numpy requirements, it must > remain very generic Nothing about what I'm saying is relevant to numpy -- the patch attached to this bug report is already plenty for what

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: Nathaniel Smith: > There are other leakable resources besides heap and GPU memory -- shmem > segments and file descriptors are two that I thought of earlier, but there > are probably others too. (Note that not all file descriptors are associated > with a pytho

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: If you consider that using least signifiant bits to store an identifier of the address space is a bad idea, I'm open to discuss how tracemalloc should be extended to support this use case. -- malloc in POSIX standard: http://pubs.opengroup.org/onlinepubs/009695

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Nathaniel Smith added the comment: > Note: I don't understand what are domain, tag (address of the memory block?) > or quantity (size of the memory block?). Yes, the idea is basically the same as now, except with an extra const char * specifying the "domain", so that one could keep separate tr

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Nathaniel Smith added the comment: Hi Victor, This is really great, thanks for working on it! What do you think about the richer api I proposed here? http://bugs.python.org/issue18835#msg232221 -- ___ Python tracker

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: Hum, there is an issue in the patch version 2: tracemalloc_add_trace() rely on the assumption that the pointer is not already tracked. Since we give control to the traces to the user, this assumption may become wrong (if the API is badly used). This issue is

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: Antoine Pitrou: > This may indeed be useful but: > - those functions should be no-ops when tracing isn't enabled (so as to be as > fast as possible) Done > - you should take the GIL if necessary (but only if tracing is enabled, of > course) Ok, done. The ne

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

Antoine Pitrou added the comment: This may indeed be useful but: - those functions should be no-ops when tracing isn't enabled (so as to be as fast as possible) - you should take the GIL if necessary (but only if tracing is enabled, of course) -- nosy: +pitrou

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

STINNER Victor added the comment: Patch without test. I chose to require the GIL to be held for efficiency. IMHO the common case is more than the GIL is held. Example to call the function if the GIL is not held: int res; PyGILState_STATE gil_state; gil_state = PyGILState_Ensure();

[issue26530] tracemalloc: add C API to manually track/untrack memory allocations

New submission from STINNER Victor: The API of Python memory allocators was extended for numpy: calloc() was added (issue #21233). It looks like it's not enough because allocations with an alignment is also required for vector operations (SIMD). For Python, there is the issue #18835 but the be