On Wed, Apr 27, 2011 at 5:55 AM, Hans Georg Schaathun <ge...@schaathun.net> wrote: > It is, of course, possible for the master thread upon processing the > results, to requeue the tasks for any missing results, but it seems > to me to be a cleaner solution if I could detect disconnects and > requeue the tasks from the networking threads. Is that possible > using python sockets? > > Somebody will probably ask why I am not using one of the multiprocessing > libraries. I have tried at least two, and got trapped by the overhead > of passing complex pickled objects across.
If I were doing this, I would devise my own socket-layer protocol and not bother with pickling objects at all. The two ends would read and write the byte stream and interpret it as data. But question: Why are you doing major number crunching in Python? On your quad-core machine, recode in C and see if you can do the whole job without bothering the unreliable boxen at all. Chris Angelico -- http://mail.python.org/mailman/listinfo/python-list
