Re: Ending data exchange through multiprocessing pipe

2009-04-23 Thread Paul Boddie
On 22 Apr, 17:43, Michal Chruszcz wrote: > > I am adding support for parallel processing to an existing program > which fetches some data and then performs some computation with > results saved to a database. Everything went just fine until I wanted > to gather all of the results from the subproce

Re: Ending data exchange through multiprocessing pipe

2009-04-23 Thread MRAB
Michal Chruszcz wrote: On Apr 22, 6:33 pm, MRAB wrote: You could do this: from multiprocessing.queues import Empty queue = Queue() Process(target=f, args=(queue,)).start() while active_children(): try: print queue.get(timeout=1) except Empty:

Re: Ending data exchange through multiprocessing pipe

2009-04-23 Thread Jesse Noller
On Thu, Apr 23, 2009 at 5:15 AM, Michal Chruszcz wrote: > On Apr 22, 10:30 pm, Scott David Daniels > wrote: >> Michal Chruszcz wrote: >> > ... First idea, which came to my mind, was using a queue. I've got many >> > producers (all of the workers) and one consumer. Seams quite simple, >> > but it

Re: Ending data exchange through multiprocessing pipe

2009-04-23 Thread Michal Chruszcz
On Apr 22, 10:30 pm, Scott David Daniels wrote: > Michal Chruszcz wrote: > > ... First idea, which came to my mind, was using a queue. I've got many > > producers (all of the workers) and one consumer. Seams quite simple, > > but it isn't, at least for me. I presumed that each worker will put() >

Re: Ending data exchange through multiprocessing pipe

2009-04-23 Thread Michal Chruszcz
On Apr 22, 6:33 pm, MRAB wrote: > You could do this: > >      from multiprocessing.queues import Empty > >      queue = Queue() >      Process(target=f, args=(queue,)).start() >      while active_children(): >          try: >              print queue.get(timeout=1) >          except Empty: >      

Re: Ending data exchange through multiprocessing pipe

2009-04-22 Thread Scott David Daniels
Michal Chruszcz wrote: ... First idea, which came to my mind, was using a queue. I've got many producers (all of the workers) and one consumer. Seams quite simple, but it isn't, at least for me. I presumed that each worker will put() its results to the queue, and finally will close() it, while th

Re: Ending data exchange through multiprocessing pipe

2009-04-22 Thread MRAB
Michal Chruszcz wrote: Hi, I am adding support for parallel processing to an existing program which fetches some data and then performs some computation with results saved to a database. Everything went just fine until I wanted to gather all of the results from the subprocesses. First idea, whi

Ending data exchange through multiprocessing pipe

2009-04-22 Thread Michal Chruszcz
Hi, I am adding support for parallel processing to an existing program which fetches some data and then performs some computation with results saved to a database. Everything went just fine until I wanted to gather all of the results from the subprocesses. First idea, which came to my mind, was u