Definitely use syslog. To see how much logging is costing you without changing code, you can simply route to /dev/null via syslog. Btw, w/ local logging, you may not see it in the cpu usage - more likely in the i/o wait. In my experience, logging to a remote syslog-ng server over udp has very little impact on the app, almost like logging to /dev/null (which in turn is like not logging at all).
Are you by chance on AWS/EC2 ? If so, please contact me to discuss possible design partnership. m On Fri, Apr 9, 2010 at 11:57 AM, Jason J. W. Williams < jasonjwwilli...@gmail.com> wrote: > Hi Paul, > > Is is sporadic? (the CPU burst) If so, you may be seeing the log > rotation code kicking...and it's probably blocking your app too until > it completes. > > I've got a couple patches to fix the blocking issue: > > http://twistedmatrix.com/trac/ticket/4372 > > The patch which eliminates old log rotation will probably get rid of > your CPU problem too. > > -J > > On Fri, Apr 9, 2010 at 2:35 AM, Paul Goins <gene...@vultaire.net> wrote: > > There may not be a clear answer to this problem, and the answer may or > > may not be Twisted-specific. I'm just trying to fish for ideas here. > > > > The server I'm working on suffers from extremely high CPU time spent in > > the logging functions. I'm spending between 30 and 40% of all CPU time > > just in twisted.python.log.msg. > > > > Describing the environment: > > > > We use DailyLogFiles for file rotation along with the > > PythonLoggingObserver to allow for logLevel support. (Of course, this > > gives us significant overhead from Python's complex logging > > architecture; maybe we should drop this...) > > > > On a 1000 request test of this server, we generate 58,274 calls to > > t.p.log.msg. There's room for cleanup, but there's lots of pressure to > > keep many of the log calls we have for forensic purposes, so we can only > > cut this back so far. > > > > Anyway, does anyone have any ideas for speedups? Any stories from > > experience especially? Shots in the dark are welcome. ;-) > > > > - > > Paul Goins > > > > _______________________________________________ > > Twisted-Python mailing list > > Twisted-Python@twistedmatrix.com > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
