Hi, I'm looking for ideas on building a simple architecture that allows a bunch of independent Python processes to exchange data using files and perform calculations.
One Python program would be collecting data from boat instruments on a serial port, then writing that info out to a file, and also transmitting data to instruments as needed (again through the serial port). It would be the most complex program in terms of having to poll file descriptors, or something similar, but I want to limit its job to pure communication handling. Then, I would have a bunch of programs doing various calculations on the input stream. It's totally acceptable if they just sleep a bit, wake up, see if there is any new data, and if so, do some calculations, so I will probably just use this: http://www.dabeaz.com/generators/follow.py Some of the "calculator" programs might be looking for trends in the data, and when a certain threshold is reached, they will need to notify the comm program of the new message. To avoid collisions between all the different calculator programs, I am thinking the simplest thing is just have them each write to their own output file. So then the comm program has multiple data sources. It wants to track to the serial port, but it also wants to monitor the output files from the various calculator processes. I want to do this in a mostly nonblocking way, so I would probably poll on the file descriptors. The only problems are that select "...cannot be used on regular files to determine whether a file has grown since it was last read," and it only works on Unix for files. (I'm using Unix myself, but one of the other programmers insists on Windows, and is resisting Python as well.) So a variation would be that the calculator programs talk to the comm program via sockets, which seems slightly on the heavy side, but it is certainly feasible. I know that something like twisted could probably solve my problem, but I'm also wondering if there is some solution that is a little more lightweight and batteries-included. At the heart of the matter, I just want the programs to be able to do the following tasks. 1) All programs will occasionally need to write a line of text to some data stream without having to lock it. (I don't care if the write blocks.) 2) All programs will need to be able to do a non-blocking read of a line of text from a data stream. (And there may be multiple consumers.) 3) The comm program will need to be able to poll the serial port and input data streams to see which ones are ready. Any thoughts? -- http://mail.python.org/mailman/listinfo/python-list
