On Mar 1, 2:28 pm, Nigel Rantor <wig...@wiggly.org> wrote: > bruce wrote: > > Hi. > > > Got a bit of a question/issue that I'm trying to resolve. I'm asking > > this of a few groups so bear with me. > > > I'm considering a situation where I have multiple processes running, > > and each process is going to access a number of files in a dir. Each > > process accesses a unique group of files, and then writes the group > > of files to another dir. I can easily handle this by using a form of > > locking, where I have the processes lock/read a file and only access > > the group of files in the dir based on the open/free status of the > > lockfile. > > > However, the issue with the approach is that it's somewhat > > synchronous. I'm looking for something that might be more > > asynchronous/parallel, in that I'd like to have multiple processes > > each access a unique group of files from the given dir as fast as > > possible. > > I don't see how this is synchronous if you have a lock per file. Perhaps > you've missed something out of your description of your problem. > > > So.. Any thoughts/pointers/comments would be greatly appreciated. Any > > pointers to academic research, etc.. would be useful. > > I'm not sure you need academic papers here. > > One trivial solution to this problem is to have a single process > determine the complete set of files that require processing then fork > off children, each with a different set of files to process. > > The parent then just waits for them to finish and does any > post-processing required. > > A more concrete problem statement may of course change the solution... > > n
Using twisted might also be helpful. Then you can avoid the problems associated with threading too. -- http://mail.python.org/mailman/listinfo/python-list
