Re: multiprocessing: pool with blocking queue

2009-07-02 Thread ryles
On Jul 2, 11:09 am, masher wrote: > My questions, then, is: Is there a more elegant/pythonic way of doing > what I am trying to do with the current Pool class? Another thing you might try is to subclass Pool and add an apply_async () wrapper which would wait for _taskqueue.qsize() to reach the de

Re: multiprocessing: pool with blocking queue

2009-07-02 Thread J Kenneth King
masher writes: > On Jul 2, 12:06 pm, J Kenneth King wrote: >> masher writes: >> > My questions, then, is: Is there a more elegant/pythonic way of doing >> > what I am trying to do with the current Pool class? >> >> Forgive me, I may not fully understand what you are trying to do here >> (I've n

Re: multiprocessing: pool with blocking queue

2009-07-02 Thread masher
On Jul 2, 12:06 pm, J Kenneth King wrote: > masher writes: > > My questions, then, is: Is there a more elegant/pythonic way of doing > > what I am trying to do with the current Pool class? > > Forgive me, I may not fully understand what you are trying to do here > (I've never really used multipro

Re: multiprocessing: pool with blocking queue

2009-07-02 Thread J Kenneth King
masher writes: > My questions, then, is: Is there a more elegant/pythonic way of doing > what I am trying to do with the current Pool class? Forgive me, I may not fully understand what you are trying to do here (I've never really used multiprocessing all that much)... But couldn't you just assi

multiprocessing: pool with blocking queue

2009-07-02 Thread masher
Hi, I am trying to implement a multiprocessing pool that assigns tasks from a blocking queue. My situation is a pretty classic producer/ consumer conundrum, where the producer can produce much faster than the consumers can consume. The wrinkle in the story is that the producer produces objects tha