[issue11849] ElementTree memory leak

Charles-Francois Natali added the comment: This is definitely a malloc bug. Test with default malloc on a Debian box: cf@neobox:~/cpython$ ./python ../issue11849_test.py *** Python 3.3.0 alpha --- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 0 3778 pts/2S+ 0:00

[issue11849] ElementTree memory leak

Charles-Francois Natali added the comment: > BTW, after utilize lxml instead of ElementTree, such phenomenon of increasing > memory usage disappeared. If you looked at the link I posted, you'll see that lxml had some similar issues and solved it by calling malloc_trim systematically when free

[issue11849] ElementTree memory leak

kaifeng added the comment: I applied your patch to Python 3.2, also I added a function call to 'malloc_trim' via ctypes, as you can see in issue11849_test2.py. In fact I have a daemon written in Python 2.5, parsing an XML of size 10+ MB every 5 minutes, after 16+ hours running, the program fi

[issue11849] ElementTree memory leak

Charles-Francois Natali added the comment: > kaifeng added the comment: > > I added 'malloc_trim' to the test code and rerun the test with Python 2.5 / > 3.2 on CentOS 5.3.  The problem still exists. > Well, malloc_trim can fail, but how did you "add" it ? Did you use patch to apply the diff

[issue11849] ElementTree memory leak

kaifeng added the comment: Found a minor defect of Python 3.2 / 3.3: line 1676 of xml/etree/ElementTree.py was: del self.target, self._parser # get rid of circular references should be: del self.target, self._target, self.parser, self._parser # get rid of circular references While it d

[issue11849] ElementTree memory leak

kaifeng added the comment: I added 'malloc_trim' to the test code and rerun the test with Python 2.5 / 3.2 on CentOS 5.3. The problem still exists. *** Python 2.5.5 final --- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 0 2567 pts/0S+ 0:00 0 1 8206 4

[issue11849] ElementTree memory leak

Antoine Pitrou added the comment: > To sum up, python is returning memory, but your libc is not. > You can force it using malloc_trim, see the attached patch (I'm not at > all suggesting its inclusion, it's just an illustration). That's an interesting thing, perhaps you want to open a feature

[issue11849] ElementTree memory leak

Changes by Charles-Francois Natali : -- keywords: +patch Added file: http://bugs.python.org/file21696/gc_trim.diff ___ Python tracker ___

[issue11849] ElementTree memory leak

Charles-Francois Natali added the comment: The "problem" is not with Python, but with your libc. When a program - such as Python - returns memory, it uses the free(3) library call. But the libc is free to either return the memory immediately to the kernel using the relevant syscall (brk, munma

[issue11849] ElementTree memory leak

kaifeng added the comment: Python 3.2 On Linux (CentOS 5.3) *** Python 3.2.0 final --- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 0 15116 pts/0S+ 0:00 1 1316 11055 6452 0.6 python3.2 issue11849_test.py 1 15116 pts/0S+ 0:02 1 1316 53155 47

[issue11849] ElementTree memory leak

Florent Xicluna added the comment: this is the output for 2.7.1: $ python2.7 issue11849_test.py *** Python 2.7.1 final --- PID STAT TIME SL RE PAGEIN VSZRSS LIM TSIZ %CPU %MEM COMMAND 0 2754 S+ 0:00.07 0 0 0 2441472 5372 -0 11,7 0,1

[issue11849] ElementTree memory leak

Florent Xicluna added the comment: I've tested a small variant of your script, on OSX. It seems to behave correctly (with 2.5, 2.6, 2.7 and 3.1). You can force Python to release memory immediately by calling "gc.collect()". -- Added file: http://bugs.python.org/file21670/issue11849_tes

[issue11849] ElementTree memory leak

Kaifeng Zhu added the comment: Yes. Just tested with Python 2.7 and 3.2 in Windows 7, the memory usage is still unexpected high after 'Done' is printed. -- ___ Python tracker _

[issue11849] ElementTree memory leak

Florent Xicluna added the comment: Do you experience same issue with current versions of Python? (3.2 or 2.7) The package was upgraded in latest versions. -- nosy: +flox ___ Python tracker

[issue11849] ElementTree memory leak

New submission from Kaifeng Zhu : I'm using xml.etree.ElementTree to parse large XML file, while the memory keep increasing consistently. You can run attached test script to reproduce it. From 'top' in Linux or 'Task Manager' in Windows, the memory usage of python is not decreased as expected