I recently encountered the problem of underlying IO issues triggering gigs of repeated logs that were compounded by the extra logging. To help improve the signal to noise ratio of the logs and prevent the downward spiral of IO issues causing logging that cause more IO issues, I created a logging filter to help reduce the repeated messages.
The basic design of the filter is to classify each message based upon a few configurable criteria and then only allow a configurable number of repeated messages in a given interval. The cache is used to track the frequency. There were three specific logging cases I addressed with my approach. System wide issues, a single point of failure, and user specific issues. The user specific issues are mostly to address our GSA and other internal crawlers. To catch system wide issues, logging messages are grouped by exception type and repr(), if type is Exception. Single point of failure really means a single logging call in the code. These are detected by logger name, function name and line number. The filter can be configured (in settings) to ignore specific logging statements to allow uncaught or common exception handlers from being incorrectly grouped together. Specific user logging messages are grouped if the filter can access the request. Code and example LOGGING configuration can be found at https://gist.github.com/1264432 Regards, Michael Manfre -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
