You might find the logging infrastructure in Foolscap interesting:
http://foolscap.lothar.com/docs/logging.html
The basic idea is to record log events only in memory, in separate
circular buffers for each priority level. When something "weird"
happens, all the events are bundled into an "Inciden
Just wanted to respond to the input everyone's given. Thanks a lot; I'm
glad I asked.
I'll probably look into the syslog route. I thought about it before,
but I wasn't sure how much flexibility it gives. Given the suggestions
here, I've since looked into it a little more and it seems to be the
Definitely use syslog. To see how much logging is costing you without
changing code, you can simply route to /dev/null via syslog. Btw, w/ local
logging, you may not see it in the cpu usage - more likely in the i/o wait.
In my experience, logging to a remote syslog-ng server over udp has very
li
Hi Paul,
Is is sporadic? (the CPU burst) If so, you may be seeing the log
rotation code kicking...and it's probably blocking your app too until
it completes.
I've got a couple patches to fix the blocking issue:
http://twistedmatrix.com/trac/ticket/4372
The patch which eliminates old log rotatio
On Fri, 2010-04-09 at 15:23 +0200, Itamar Turner-Trauring wrote:
> On Fri, 2010-04-09 at 11:49 +0200, Kees Bos wrote:
>
> > I think you could try to use a custom light weight log function that
> > just queues the log messages (FIFO) and bursts them, say every second,
> > to the log file (e.g. in a
On Fri, 2010-04-09 at 11:49 +0200, Kees Bos wrote:
> I think you could try to use a custom light weight log function that
> just queues the log messages (FIFO) and bursts them, say every second,
> to the log file (e.g. in a thread to use a multicore cpu).
Or:
0. Profile logging system - what exa
On Fri, Apr 09, 2010 at 05:35:51PM +0900, Paul Goins wrote:
> We use DailyLogFiles for file rotation along with the
> PythonLoggingObserver to allow for logLevel support. (Of course, this
> gives us significant overhead from Python's complex logging
> architecture; maybe we should drop this...)
I
On Fri, 2010-04-09 at 10:35 +0200, Paul Goins wrote:
> There may not be a clear answer to this problem, and the answer may or
> may not be Twisted-specific. I'm just trying to fish for ideas here.
>
> The server I'm working on suffers from extremely high CPU time spent in
> the logging functions.
