I know how to check the refcounts, but I don't know how to check the memory usage, since it's not a program, it's a simple library. Is there not a way to check inside Python the memory usage? I have to use a bash script (I'm on Linux)?
On Sat, 20 Nov 2021 at 19:00, MRAB <pyt...@mrabarnett.plus.com> wrote: > > On 2021-11-20 17:40, Marco Sulla wrote: > > Indeed I have introduced a command line parameter in my bench.py > > script that simply specifies the number of times the benchmarks are > > performed. This way I have a sort of segfault checker. > > > > But I don't bench any part of the library. I suppose I have to create > > a separate script that does a simple loop for all the cases, and > > remove the optional parameter from bench. How boring. > > PS: is there a way to monitor the Python consumed memory inside Python > > itself? In this way I could also trap memory leaks. > > > I'm on Windows 10, so I debug in Microsoft Visual Studio. I also have a > look at the memory usage in Task Manager. If the program uses more > memory when there are more iterations, then that's a sign of a memory > leak. For some objects I'd look at the reference count to see if it's > increasing or decreasing for each iteration when it should be constant > over time. > > > On Sat, 20 Nov 2021 at 01:46, MRAB <pyt...@mrabarnett.plus.com> wrote: > >> > >> On 2021-11-19 23:44, Marco Sulla wrote: > >> > On Fri, 19 Nov 2021 at 20:38, MRAB <pyt...@mrabarnett.plus.com> wrote: > >> >> > >> >> On 2021-11-19 17:48, Marco Sulla wrote: > >> >> > I have a battery of tests done with pytest. My tests break with a > >> >> > segfault if I run them normally. If I run them using pytest -v, the > >> >> > segfault does not happen. > >> >> > > >> >> > What could cause this quantical phenomenon? > >> >> > > >> >> Are you testing an extension that you're compiling? That kind of problem > >> >> can occur if there's an uninitialised variable or incorrect reference > >> >> counting (Py_INCREF/Py_DECREF). > >> > > >> > Ok, I know. But why can't it be reproduced if I do pytest -v? This way > >> > I don't know which test fails. > >> > Furthermore I noticed that if I remove the __pycache__ dir of tests, > >> > pytest does not crash, until I re-ran it with the __pycache__ dir > >> > present. > >> > This way is very hard for me to understand what caused the segfault. > >> > I'm starting to think pytest is not good for testing C extensions. > >> > > >> If there are too few Py_INCREF or too many Py_DECREF, it'll free the > >> object too soon, and whether or when that will cause a segfault will > >> depend on whatever other code is running. That's the nature of the > >> beast: it's unpredictable! > >> > >> You could try running each of the tests in a loop to see which one > >> causes a segfault. (Trying several in a loop will let you narrow it down > >> more quickly.) > >> > >> pytest et al. are good for testing behaviour, but not for narrowing down > >> segfaults. > > > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
