So, here's some info about how to see what's going on with Python's memory allocation: https://docs.python.org/3/library/tracemalloc.html . I haven't looked into this in a long time, but it used to be the case that you needed to compile native modules (and probably Python itself?) so that instrumentation is possible (I think incref / decref macros should give you a hint, because they would have to naturally report some of that info).
Anyways. The problem of tracing memory allocation / deallocation in Python can be roughly split into these categories: 1. Memory legitimately claimed by objects created through Python runtime, but not reclaimed due to programmer error. I.e. the programmer wrote a program that keeps references to objects which it will never use again. 2. Memory claimed through native objects obtained by means of interacting with Python's allocator. When working with Python C API it's best to interface with Python allocator to deal with dynamic memory allocation and release. However, it's somewhat cumbersome, and some module authors simply might not know about it, or wouldn't want to use it because they prefer a different allocator. Sometimes library authors don't implement memory deallocation well. Which brings us to: 3. Memory claimed by any user-space code that is associated with the Python process. This can be for example shared libraries loaded by means of Python bindings, that is on top of the situation described above. 4. System memory associated with the process. Some system calls need to allocate memory on the system side. Typical examples are opening files, creating sockets etc. Typically, the system will limit the number of such objects, and the user program will hit the numerical limit before it hits the memory limit, but it can also happen that this will manifest as a memory problem (one example I ran into was trying to run conda-build and it would fail due to enormous amounts of memory it requested, but the specifics of the failure were due to it trying to create new sub-processes -- another system resource that requires memory allocation). There isn't a universal strategy to cover all these cases. But, if you have reasons to suspect (4), for example, you'd probably start by using strace utility (on Linux) to see what system calls are executed. For something like the (3), you could try to utilize Valgrind (but it's a lot of work to set it up). It's also possible to use jemalloc to profile a program, but you would have to build Python with its allocator modified to use jemalloc (I've seen an issue in the Python bug tracker where someone wrote a script to do that, so it should be possible). Both of these are quite labor intensive and not trivial to set up. (2) could be often diagnosed with tracemalloc Python module and (1) is something that can be helped with Python's gc module. It's always better though to have an actual error and work from there. Or, at least, have some monitoring data that suggests that your application memory use increases over time. Otherwise you could be spending a lot of time chasing problems you don't have. -- https://mail.python.org/mailman/listinfo/python-list
