On 2011-01-16 20:57:41 -0800, Adam Skutt said:
On Jan 16, 11:39 pm, TomF <tomf.sess...@gmail.com> wrote:
One difficulty is that there is a queue of work to be done and a queue
of results to be incorporated back into the parent; there is no
one-to-one correspondence between the two. It's not obvious to me how
to coordinate the queues in a natural way to avoid deadlock or
starvation.
Depends on what you are doing. If you can enqueue all the jobs before
waiting for your results, then two queues are adequate. The first
queue is jobs to be accomplished, the second queue is the results.
The items you put on the result queue have both the result and some
sort of id so the results can be ordered after the fact. Your parent
thread of execution (thread hereafter) then:
1. Adds jobs to the queue
2. Blocks until all the results are returned. Given that you
suggested that there isn't a 1:1 correspondence between jobs and
results, have the queue support a message saying, 'Job X is done'.
You're finished when all jobs send such a message.
3. Sorts the results into the desired ordered.
4. Acts on them.
If you cannot enqueue all the jobs before waiting for the results, I
suggest turning the problem into a pipeline, such that the thread
submitting the jobs and the thread acting on the results are
different: submitter -> job processor -> results processor.
Adam
Thanks for your reply. I can enqueue all the jobs before waiting for
the results, it's just that I want the parent to process the results as
they come back. I don't want the parent to block until all results are
returned. I was hoping the Pool module had a test for whether all
processes were done, but I guess it isn't hard to keep track of that
myself.
-Tom
--
http://mail.python.org/mailman/listinfo/python-list
