Hi everyone, for a project I'm working on I wanted to create a helper program which can spawn an instance of itself as a separate process, slaving itself to the spawner, and speaking PB to it.
The idea is that the master can spawn processes which will run code that might segfault or otherwise behave badly (like consume too much memory). I'm planning to use this as a job server for a django site, where the spawned jobs then do all kind of media analysis (acoustical fingerprinting, tag reading, ...) The jobs themselves however I simply want to write in python, defining them as methods on a slave-side pb object that the master process can call on. I prefer to go with spawnProcess and ProcessProtocol because that makes cleanup easier - no sockets or anything to deal with. I made a first implementation here: https://thomas.apestaart.org/thomas/trac/browser/tests/twisted/daemon/daemon.py Basically, I tried to respect Twisted's setup for protocols and factories and such, providing a listenProcess and connectProcess method much like the usual ones. That would mean, I think, that this master/slave abstraction and the ProcessProtocol could be used by any kind of server/client factory, not just PB. Currently, I have the processprotocol creates process pipes instead of using stdin/stdout. I'm guessing that would make it windows-incompatible, correct ? I might have to fall back to using stdin/stdout, which would make the debugging harder. Questions: a) anyone want to take a look at the code and comment ? Does it make sense to do it like this ? Any obvious problems ? b) as the comment at the bottom states, uncommenting that line gives me a ProtocolError that I cannot figure out. It is repeatable; however, shuffling code around can make it go away (in fact, I had it in some other location as well at some point). I'm not sure what would be triggering that, any ideas ? I am probably doing something dirty in some part of the code. c) on hitting ctrl-c, tracebacks are generated, due to this block of code: # At least on OS X 10.4, exiting while stdout is non-blocking can # result in data loss. For some reason putting the file descriptor # back into blocking mode seems to resolve this issue. fdesc.setBlocking(self.fd) The exception is EBADF. That code looks suspicious to me - shouldn't it be at least checking for exceptions there ? the fd could already be closed IMO. Comments welcome! Thomas -- I love the way you love but I hate the way I'm supposed to love you back -- URGent, best radio on the net - 24/7 ! http://urgent.fm/ _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
