Hi! A user recently reported a memory leak in python bindings (C extension module) to a C library[1] I wrote. I've been trying to fix it since but so far without success. Since I'm probably dealing with a space leak rather than actual memory leak, valgrind didn't help much even when using malloc as allocator. I'm now trying to use PYTHONMALLOCSTATS but need some help on how to interpret the output emitted it's enabled.
I'm setting PYTHONMALLOCSTATS=1 & PYTHONMALLOC=pymalloc_debug and then
running the script that triggers the leak. The last debug message is
as follows:
class size num pools blocks in use avail blocks
----- ---- --------- ------------- ------------
3 32 1 2 124
4 40 2 9 193
5 48 1 3 81
6 56 1 4 68
7 64 11 295 398
8 72 9 260 244
9 80 36 831 969
10 88 79 1542 2092
11 96 131 3262 2240
12 104 70 1903 757
13 112 19 289 395
14 120 11 139 224
15 128 7 88 129
16 136 6 70 104
17 144 5 44 96
18 152 4 47 57
19 160 24 342 258
20 168 4 17 79
21 176 24 360 192
22 184 2 8 36
23 192 2 11 31
24 200 22 227 213
25 208 3 13 44
26 216 3 7 47
27 224 2 13 23
28 232 2 6 28
29 240 3 8 40
30 248 2 10 22
31 256 3 10 35
32 264 2 9 21
33 272 3 11 31
34 280 2 10 18
35 288 1 3 11
36 296 2 9 17
37 304 2 9 17
38 312 2 5 19
39 320 2 5 19
40 328 14 105 63
41 336 2 3 21
42 344 1 3 8
43 352 1 3 8
44 360 2 3 19
45 368 1 3 8
46 376 1 1 9
47 384 2 4 16
48 392 2 6 14
49 400 2 3 17
50 408 1 1 8
51 416 1 3 6
52 424 2 4 14
53 432 50967 458680 23
54 440 3 9 18
55 448 4 15 21
56 456 4 12 20
57 464 3 8 16
58 472 2 5 11
59 480 1 4 4
60 488 1 3 5
61 496 4 11 21
62 504 4 13 19
63 512 2 7 7
# times object malloc called = 2,811,245
# arenas allocated total = 810
# arenas reclaimed = 0
# arenas highwater mark = 810
# arenas allocated current = 810
810 arenas * 262144 bytes/arena = 212,336,640
# bytes in allocated blocks = 199,277,432
# bytes in available blocks = 1,138,472
308 unused pools * 4096 bytes = 1,261,568
# bytes lost to pool headers = 2,473,536
# bytes lost to quantization = 8,185,632
# bytes lost to arena alignment = 0
Total = 212,336,640
The number of pools in arena 53 continuously grows. Its size column
says: 432. I couldn't find any documentation on what it means but I
assume it's an allocation of 432 bytes. I launched gdb and set the
following breakpoint: 'b PyMem_Malloc if size == 432' but the
breakpoint was never triggered. I tried the same for PyObject_Malloc
and still nothing.
How do I use the info produced by PYTHONMALLOCSTATS do get to the
culprit of the leak? Is there anything wrong in my reasoning here?
Best regards,
Bartosz Golaszewski
[1] https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/
--
https://mail.python.org/mailman/listinfo/python-list
