Re: python gc performance in large apps

2005-11-06 Thread Tim Peters
[Robby Dermody] > ... > -len(gc.get_objects()) will show linearly increasing counts over time > with the director (that match the curve of the director's memory usage > exactly), but with the harvester, the object count doesn't increase over > time (although memory usage does). This might mean that

Re: python gc performance in large apps

2005-11-05 Thread Jay Parlar
You may also want to look at the following Summer of Code project: http://pysizer.8325.org/ Their SVN repo is at http://codespeak.net/svn/user/nick8325/sizer/ I haven't had a chance to try it yet, but it might be exactly what you need. The project is described as "PySizer is a library for mea

Re: python gc performance in large apps

2005-11-05 Thread Stephen Kellett
In message <[EMAIL PROTECTED]>, Robby Dermody <[EMAIL PROTECTED]> writes >An update (along with a request for paid help at the end): Over the >past week and a half I've improved the memory usage situation by quite >a bit. Going to python 2.4, linux kernel 2.6, twisted 2.0 and altering >some cod

Re: python gc performance in large apps

2005-11-04 Thread Bryan Olson
Robby Dermody wrote: > [...] However, on a > simulated call center environment (constant 50 conversations, switching > every 300 seconds) the director component still consumes an average of > 1MB more per hour and the harvester is taking an average of 4MB more per > hour. With the director, 2/3

Re: python gc performance in large apps

2005-11-04 Thread Paul Rubin
Robby Dermody <[EMAIL PROTECTED]> writes: > t = 120 seconds (1st run after being fully initialized): > list alloced: 2394620, freed: 17565, max in use: 2377056 > dict alloced: 2447968, freed: 67999, max in use: 2379969 This looks like a garden variety memory leak. I think the next thi

Re: python gc performance in large apps

2005-11-04 Thread David Rushby
Robby Dermody wrote: > I have not yet run the director with COUNT_ALLOCS yet, due to a > problem dynamically loading in the kinterbasdb (Firebird SQL > interface) module. What're the specifics of the loading problem with kinterbasdb? AFAIK, kinterbasdb itself runs fine in a debug build of Python.

Re: python gc performance in large apps

2005-11-04 Thread Robby Dermody
Hey guys, An update (along with a request for paid help at the end): Over the past week and a half I've improved the memory usage situation by quite a bit. Going to python 2.4, linux kernel 2.6, twisted 2.0 and altering some code have reduced the severity by an order of a magnitude. However, on

Re: python gc performance in large apps

2005-10-25 Thread Fredrik Lundh
Robby Dermody wrote: > In the diagrams above, one can see the night-day separation clearly. At > night, the memory usage growth seemed to all but stop, but with the > increased call volume of the day, it started shooting off again. When I > first started gathering this data, I was hoping for a log

Re: python gc performance in large apps

2005-10-24 Thread Chris Lambacher
A couple of strategic gc.collect() calls can be useful. You can also tweak how the garbage collector gets run by changing settings in the gc module. -Chris On Fri, Oct 21, 2005 at 04:13:09PM -0400, Robby Dermody wrote: > > Hey guys (thus begins a book of a post :), > > I'm in the process of wri

Re: python gc performance in large apps

2005-10-22 Thread Neal Norwitz
Jp Calderone wrote: > On Fri, 21 Oct 2005 16:13:09 -0400, Robby Dermody <[EMAIL PROTECTED]> wrote: > > > > [snip - it leaks memory] > > One thing to consider is that the process may be growing in size, not because > garbage objects are not being freed, but because objects which should be > garbag

Re: python gc performance in large apps

2005-10-22 Thread Jp Calderone
On Fri, 21 Oct 2005 16:13:09 -0400, Robby Dermody <[EMAIL PROTECTED]> wrote: > >Hey guys (thus begins a book of a post :), > >I'm in the process of writing a commercial VoIP call monitoring and >recording application suite in python and pyrex. Basically, this >software sits in a VoIP callcenter-typ

python gc performance in large apps

2005-10-21 Thread Robby Dermody
Hey guys (thus begins a book of a post :), I'm in the process of writing a commercial VoIP call monitoring and recording application suite in python and pyrex. Basically, this software sits in a VoIP callcenter-type environment (complete with agent phones and VoIP servers), sniffs voice data o