On May 9, 12:37 am, Vinay Sajip <[EMAIL PROTECTED]> wrote: > On May 9, 12:52 am, [EMAIL PROTECTED] wrote:> The infrastructure in which I > am work needs the ability to have log > > files written to from multiple instances of the same script and > > potentially from hundreds or more different machines. > > > I know that the documentation suggests using a networkloggingserver > > but I wanted to know if anyone had any other solutions to allow us to > > build off of the current pythonloggingpackages. > > Dennis is right - the logging system is threadsafe but not safe > against multiple processes (separate Python instances) writing to the > samefile. It certainly sounds like you need a scalable solution - and > having each script send the events to a network logging server seems a > good way of handling the scalability requirement. The logger name used > can include the script instance and machine name, e.g. by starting > with hostname.scriptname.scriptpid... The socket server which receives > the events can demultiplex them based on this information and write > them to a central repository in any arrangement you care to implement > (e.g. into onefileor several). > > Given that the example in the docs is a (basic) working example, is > there any particular reason why you don't want to follow the suggested > approach? > > Regards, > > Vinay Sajip
Our biggest concerns with the network solution is having a single point of failure and the need for scalability. We could have potentially thousands of machines doing logging across multiple geographic sites. Our fear with the network solution is overwhelming the server or group of servers as well as having a single point of failure for the logging interface. In addition using a server or servers would require added support for the logigng server machines. Our NFS infrastructure is very well supported and can handle the load generated by these machines already (A load which would be many times more than what the logging would generate) which is why we would like to log directly to file system without going through a separate server. Also adding in a logging server introduces one more level where we could potentially have failure. We would like to keep the infrastructure for our logging as simple as possible as we rely on log files to give us critical information when troubleshooting issues. It sounds like my only option may be using a server in order to handle the logging from different hosts. That or possibly having individual log files for each host. Thanks for your input. It is much appreciated. -Nick -- http://mail.python.org/mailman/listinfo/python-list
