Re: [Twisted-Python] Seeking advice: Coping with heavy log file loads

[Paul Goins]

Just wanted to respond to the input everyone's given.  Thanks a lot; I'm
glad I asked.

I'll probably look into the syslog route.  I thought about it before,
but I wasn't sure how much flexibility it gives.  Given the suggestions
here, I've since looked into it a little more and it seems to be the
smartest route, especially due to the size of the load.

Thanks everyone!

- Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Seeking advice: Coping with heavy log file loads

[Brian Warner]

You might find the logging infrastructure in Foolscap interesting:

 http://foolscap.lothar.com/docs/logging.html

The basic idea is to record log events only in memory, in separate
circular buffers for each priority level. When something "weird"
happens, all the events are bundled into an "Incident File", and then
the incident file is written out (either to disk, or to a gatherer
process over a network). Other tools are then used to reassemble the
pieces of the incident into chronological order and point out where the
gaps are.

I haven't done any specific performance measurements, but I've taken a
busy server and stubbed out the logging calls and didn't see a huge
change in throughput rate. The Foolscap approach gets you useful (and
detailed) information when bad things happen, but throws out most of the
other stuff, so it doesn't touch the disk or make system calls very often.

cheers,
 -Brian (author of Foolscap)



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Weekly Bug Summary

[exarkun]

Bug summary
__
Summary for 2010-04-04 through 2010-04-11
Bugs opened: 4Bugs closed: 2  Total open bugs: 1210 (+2)

|== Type Changes   |== Priority Changes   |== Component Changes   
|Defect:   +2  |Normal:  +2   |Core:  +3  
|Enhancement:  -1  |Low: -1   |Mail:  +0  
|Task: +1  |Lowest:  +1   |Web:   -1  



Total Tickets
Open Tickets



New / Reopened Bugs
__
= Normal =
[#4392] Improper quoting in IMAP KEYWORD search (opened by jothan)
defect  mail   http://twistedmatrix.com/trac/ticket/4392

[#4394] fix DeprecationWarning: IPv4Address.__getitem__ in protocols.ident (opened by forrestv)
defect  core   http://twistedmatrix.com/trac/ticket/4394

[#4300] Write some unit tests for Deferreds (opened by jml)
taskcore   http://twistedmatrix.com/trac/ticket/4300

= Lowest =
[#4391] Clean up twistd termination messages. (opened by kench)
defect  core   http://twistedmatrix.com/trac/ticket/4391



Closed Bugs
__
= Normal =
[#4385] IMAP4 server loops indefinitely if a search query yields more than 4 items (opened by wulczer, closed by exarkun, fixed)
defect  mail   http://twistedmatrix.com/trac/ticket/4385

= Low =
[#4377] Make the Deferred returned by twisted.web.xmlrpc.Proxy.callRemote cancellable (opened by davep, closed by exarkun, fixed)
enhancement webhttp://twistedmatrix.com/trac/ticket/4377



Ticket Lifetime Stats
__
Oldest open ticket - [#50] conch command-line client doesn't work in win32 (since 2003-07-12 16:41:06).
Newest open ticket - [#4394] fix DeprecationWarning: IPv4Address.__getitem__ in protocols.ident (since 2010-04-10 16:34:30).

Mean open ticket age: 883 days, 3:13:54.462324.
Median: 799 days, 10:58:59.213564.
Standard deviation: 636 days, 2:08:13.150541.
Interquartile range: 967 days, 17:07:51.

Mean time between ticket creation and ticket resolution: 235 days, 7:58:34.487921.
Median: 28 days, 22:07:55.
Standard deviation is 403 days, 13:21:21.098477.
The interquartile range is 280 days, 18:50:16.

Mean time spent in review: 77 days, 12:28:09.087373.
Median: 4 days, 0:51:40.
Standard deviation: 279 days, 9:20:03.514434.
Interquartile range: 16 days, 10:43:34.

Mean number of times a ticket is reviewed: 2.03037037037.
Median: 1
Standard deviation: 1.67060752197.
Interquartile range: 1.


Contributor Stats
__
In the last 4 weeks,
20 unique ticket reporters
9 unique ticket reviewers
8 unique ticket resolvers
In the last 24 weeks,
109 unique ticket reporters
33 unique ticket reviewers
21 unique ticket resolvers
In the last 48 weeks,
191 unique ticket reporters
37 unique ticket reviewers
26 unique ticket resolvers





___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python