On 31 mai, 16:22, Paul Melis <[EMAIL PROTECTED]> wrote:
> Hello,
>
> [EMAIL PROTECTED] wrote:
> > I've some troubles getting my memory freed by python, how can I force
> > it to release the memory ?
> > I've tried del and gc.collect() with no success.
>
> [...]
>
>
>
> > The same problem here with a simple file.readlines()
> > #Python interpreter memory usage : 1.1 Mb private, 1.4 Mb shared
> > import gc #no memory change
> > f=open('primary.xml') #no memory change
> > data=f.readlines() #meminfo: 12 Mb private, 1.4 Mb shared
> > del data #meminfo: 11.5 Mb private, 1.4 Mb shared
> > gc.collect() # no memory change
>
> > But works great with file.read() :
> > #Python interpreter memory usage : 1.1 Mb private, 1.4 Mb shared
> > import gc #no memory change
> > f=open('primary.xml') #no memory change
> > data=f.read() #meminfo: 7.3Mb private, 1.4 Mb shared
> > del data #meminfo: 1.1 Mb private, 1.4 Mb shared
> > gc.collect() # no memory change
>
> > So as I can see, python maintain a memory pool for lists.
> > In my first example, if I reparse the xml file, the memory doesn't
> > grow very much (0.1 Mb precisely)
> > So I think I'm right with the memory pool.
>
> > But is there a way to force python to release this memory ?!
>
> This is from the 2.5 series release notes
> (http://www.python.org/download/releases/2.5.1/NEWS.txt):
>
> "[...]
>
> - Patch #1123430: Python's small-object allocator now returns an arena to
> the system ``free()`` when all memory within an arena becomes unused
> again. Prior to Python 2.5, arenas (256KB chunks of memory) were never
> freed. Some applications will see a drop in virtual memory size now,
> especially long-running applications that, from time to time, temporarily
> use a large number of small objects. Note that when Python returns an
> arena to the platform C's ``free()``, there's no guarantee that the
> platform C library will in turn return that memory to the operating
> system.
> The effect of the patch is to stop making that impossible, and in
> tests it
> appears to be effective at least on Microsoft C and gcc-based systems.
> Thanks to Evan Jones for hard work and patience.
>
> [...]"
>
> So with 2.4 under linux (as you tested) you will indeed not always get
> the used memory back, with respect to lots of small objects being
> collected.
>
> The difference therefore (I think) you see between doing an f.read() and
> an f.readlines() is that the former reads in the whole file as one large
> string object (i.e. not a small object), while the latter returns a list
> of lines where each line is a python object.
>
> I wonder how 2.5 would work out on linux in this situation for you.
>
> Paul
Hello,
I will try later with python 2.5 under linux, but as far as I can see,
it's the same problem under my windows python 2.5
After reading this document :
http://evanjones.ca/memoryallocator/python-memory.pdf
I think it's because list or dictionnaries are used by the parser, and
python use an internal memory pool (not pymalloc) for them...
Regards,
FP
--
http://mail.python.org/mailman/listinfo/python-list
