Wierd behavior of gc.collect
I have a python process that does some operations and is supposed to release memory after those. The issue is that memory is not released (as seen through top). So I do a gc.collect() to see if there is any cycle etc. Immediately after doing the collect memory usage drops as expected, but strangely gc.collect() returns 0. This means I cannot find out what the problem is by setting the debug option on gc which is what I usually do to figure out issues like this. Maybe its that my understanding about it is incorrect, but if gc.collect returned 0, how come some memory was freed? -- http://mail.python.org/mailman/listinfo/python-list
Re: Wierd behavior of gc.collect
I know this, but my question is what does gc.collect do which results in the c library to free memory? Usually it is because of unreferenced objects in a cycle or something, but here that doesn't seem to be the case. On Tuesday, March 19, 2013 9:32:27 PM UTC+5:30, Dave Angel wrote: > On 03/19/2013 11:47 AM, Bodhi wrote: > > > I have a python process that does some operations and is supposed to > > release memory after those. The issue is that memory is not released (as > > seen through top). So I do a gc.collect() to see if there is any cycle etc. > > Immediately after doing the collect memory usage drops as expected, but > > strangely gc.collect() returns 0. > > > This means I cannot find out what the problem is by setting the debug > > option on gc which is what I usually do to figure out issues like this. > > > > > > Maybe its that my understanding about it is incorrect, but if gc.collect > > returned 0, how come some memory was freed? > > > > > > > To put it simply, top won't in general show you that things are freed. > > The C libraries for malloc and free will reuse the memory, but not > > usually release it to the operating system. So it's not usually going > > to show up in 'top.' > > > > There was a long thread on this quite recently, but I can't seem to find > > it right now. > > > > > > -- > > DaveA -- http://mail.python.org/mailman/listinfo/python-list
Re: Wierd behavior of gc.collect
Thanks for the info. I now suspect that the free lists are taking up the memory which won't be released unless we do a collect. I'm verifying that. On Tuesday, March 19, 2013 10:43:11 PM UTC+5:30, Dave Angel wrote: > On 03/19/2013 12:36 PM, Bodhi wrote: > > > I know this, but my question is what does gc.collect do which results in > > the c library to free memory? Usually it is because of unreferenced objects > > in a cycle or something, but here that doesn't seem to be the case. > > > > > > > As I said, python calls the C free() function, whether it's when an > > object's ref-count goes to zero, or whether it's during a gc call, where > > circular refs are freed. > > > > But free() does not necessarily release the memory to the OS. And the > > times it does depends on which C library is being used, and what OS it's > > running on. > > > > If the freed memory affects top in some situations, it's a C library > > detail. I've written a replacement C allocator in the past for Windows > > that used a different scheme for blocks over a certain threshold, and > > when those blocks were freed, it gave them back to the OS. But such > > blocks were multiples of 64k, which was the increment for VirtualAlloc. > > > > > > > > > > -- > > DaveA -- http://mail.python.org/mailman/listinfo/python-list
