On Friday, September 11, 2015 at 9:22:42 AM UTC+2, Thomas Güttler wrote: > Am Donnerstag, 10. September 2015 08:42:47 UTC+2 schrieb dieter: > > Thomas Güttler writes: > > > ... > > > Why we are unhappy with logging to files: > > > > > > - filtering: We don't want to get INFO messages over the VPN. > > > > You can quite easily control at what level messages are logged with > > the standard Python logging framework. Each handler has a level > > and will ignore messages at a lower level. > > > I want INFO to be logged and stored on the remote host. > Therefore I must not filter INFO messages. > > I don't want to pull INFO messages over the VPN. > > Ergo, the filtering at Python level does not help in my use case. > Or I am missing something. > > And now I have an ugly soup. > > The ugly soup is a text file with not well defined syntax. It looks line > based. But sometimes there are log messages which span multiple lines .... > > Remember: "Life is too short to (re)write parsers" > > Yes, there are tools to parse that soup. But why create this soup in > the first place? > > That's why I want to move from file based logging to a different solution. > > Unfortunately there too many choices (graylog, logstash, sentry, ...) :-( > > > > > > - Rotating: Rotating files is possible, but somehow cumbersome. > > > > There are standard tools to rotate logfiles. > > > > > - Support structured logging of values (json) in the future. > > > > Again, the Python logging framework is quite flexible with > > respect to the format of logged messages. > > > > > ... > > > Which solution could fit for our environment? > > > > I work for a customer with a similar environment (he uses "Zope" instead > > of "Django") - and he uses logfiles. The logfiles are automatically > > rotated and there are in the order of half a dozen to a dozen logfiles > > per day. > > > > When I have to analyse a problem with the help of the logfiles, > > I do not copy them via VPN but do the filtering remotely and only > > copy the filtered portion, if necessary. > > Good to know that I am not the only one running servers in remote intranets. > > Regards, > Thomas Güttler
So, if logging to json is on the horizon anyway, why don't you create something like a MongoDb handler in the standard Python logging framework and run a MongoDb server in the client intranet? You could then connect over VPN to MongoDb, filter the warning/error messages there on the server side and fetch them to your local systems for analysis. Marco -- https://mail.python.org/mailman/listinfo/python-list
