Noam Aigerman wrote:
Hi,
Suppose I have an array of functions which I execute in threads (each
thread get a slice of the array, iterates over it and executes each
function in it’s slice one after the other). Now I want to distribute
these tasks between two machines, i.e give each machine half of the
slices and let it run them in threads as described above. Is there an
easy way, or an article on this matter you can point me to?
Here is a standard approach that is slightly more general than you
specified. It will work with more than two cores or machines.
Write master/server program that reads data, listens to socket for calls
from worker/client programs, splits work into multiple tasks, sends them
out to workers, and collects and re-assembles the results. After a
timeout, it should be able to reassign tasks not returned.
Write worker/client program than calls master, gets task, completes it,
and returns result.
Assume all data is on machine A.
Start master and worker on machine A.
Start worker(s) on other cores or machines.
You can either use socket module directly or use networking modules
built on top of it. Twisted and pyro are two that come to mind
immediately. I have seen code examples but do not remember where.
Note: I do not see any advantage to having multiple compute-bound
threads, as you seem to describe.
Terry Jan Reedy
--
http://mail.python.org/mailman/listinfo/python-list
