Improper creating of logger instances or a Memory Leak?
I've run across a memory leak in a long running process which I can't
determine if its my issue or if its the logger.
The long and short is I'm doing load testing on an application server
which spawns handlers threads which in turn each spawn a single
application thread. A graphic representation would be One Server ->
(to many pairs of) [ Handler <-> Application ].
Each application thread gets a logger instance in it's init() method
via:
self.logger = logging.getLogger('ivr-'+str(self.rand))
where self.rand is a suitably large random number to avoid collisions
of the log file's name. Until the log file gets created I attach am
memory handler
self.memhandler =
logging.handlers.MemoryHandler(1000)
self.memhandler.setLevel(10)
formatter = logging.Formatter('%
(levelname)s %(message)s')
self.memhandler.setFormatter(formatter)
self.logger.addHandler(self.memhandler)
when the application thread formally starts with the run() method I
create the log file and terminate the memory handler
filehandler = logging.FileHandler(logfilename)
filehandler.setLevel(10)
formatter = logging.Formatter('%(levelname)s %(message)s')
filehandler.setFormatter(formatter)
self.memhandler.setTarget(filehandler)
self.memhandler.close()
self.logger.removeHandler(self.memhandler)
self.logger.addHandler(filehandler)
finally the last statements in the run() method are:
filehandler.close()
self.logger.removeHandler(filehandler)
del self.logger #this was added to try and force a clean up of
the logger instances.
Using the objgraph to look at the objects in memory I find the number
of logger instances equal to the total number of threads to have lived
despite the fact that either a) there are only the standard load
testing number of threads alive, 35 or b) that there no threads
running nor are there any stale waiting for the GC.
>From objgraph a selection of the most prevalent objects in memory are
(this is with the system idle post-run):
list 256730
dict128933
Logger 128164# total
application threads executed running load testing.
function 2356
wrapper_descriptor 1028
builtin_function_or_method702
method_descriptor 648
tuple 643
weakref 629
getset_descriptor 304
type252
set 224
member_descriptor209
module 128
WeakSet102
The only references to self.logger other than those listed are wrapper
methods defined in the application thread to wrap up the log / debug
methods. Any help or direction would be much appreciated.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Improper creating of logger instances or a Memory Leak?
Yes, I asked it on stack overflow first and didn't see an quick reply. I'm trying to tighten up this code as much as possible in a final pre-production push; I apologize for being overly antsy about this. This is my pet project to upgrade our core systems from an ancient IBM language that Moses might have used. Currently I'm using python 3.1.2 (sorry for the obvious omission). Regarding adding a new logger for each thread - each thread represents a telephone call in a data collection system. I need to be able to cleanly provided call-logging for debugging to my programmers as well as data logging and verification; having a single log file is somewhat impractical. To use the logging filtering then I would have to be dynamically adding to the filtering hierarchy continuously, no? Thanks! Bill On Jun 19, 10:42 am, Vinay Sajip wrote: > foobar gmail.com> writes: > > > > > I've run across a memory leak in a long running process which I can't > > determine if its my issue or if its the logger. > > BTW did you also ask this question on Stack Overflow? I've answered there, > too. > > http://stackoverflow.com/questions/6388514/ > > Regards, > > Vinay Sajip -- http://mail.python.org/mailman/listinfo/python-list
Re: wxPython vs. pyQt
tc wrote: > Has anyone compiled binaries for qt/pyqt/eric3. i'd really like to try > it. at the moment i work with wxWindows and BoaConstructor which i'm > actually not so happy with. design of gui's with wx is not very > efficient... > > so is there already a binary for qt/pyqt/eric3 available or when can i > excpect qt4 to be released? > > > tc Hi, If you're on a linux fedora3/suse 9.2 platform you can give a try to: pyvm.sourceforge.net regards, antonio -- http://mail.python.org/mailman/listinfo/python-list
