Re: multiprocessing eats memory

2008-09-27 Thread redbaron
> When processing data in parallel you will use up as muchmemoryas > many datasets you are processing at any given time. Worker processes eats 2-4 times more than I pass to them. >If you need to > reducememoryuse then you need to start fewer processes and use some > mechanism to distribute the wo

Re: multiprocessing eats memory

2008-09-26 Thread Istvan Albert
On Sep 26, 4:52 am, redbaron <[EMAIL PROTECTED]> wrote: > How could I avoid of storing them? I need something to check does it > ready or not and retrieve results if ready. I couldn't see the way to > achieve same result without storing asyncs set. It all depends on what you are trying to do. The

Re: multiprocessing eats memory

2008-09-26 Thread redbaron
On 26 сент, 17:03, MRAB <[EMAIL PROTECTED]> wrote: > On Sep 26, 9:52 am, redbaron <[EMAIL PROTECTED]> wrote: > > > On 26 ÓÅÎÔ, 04:20, Istvan Albert <[EMAIL PROTECTED]> wrote: > > > > On Sep 25, 8:40šam, "Max Ivanov" <[EMAIL PROTECTED]> wrote: > > > > > At any time in main process there are shouldn'

Re: multiprocessing eats memory

2008-09-26 Thread MRAB
On Sep 26, 9:52 am, redbaron <[EMAIL PROTECTED]> wrote: > On 26 ÓÅÎÔ, 04:20, Istvan Albert <[EMAIL PROTECTED]> wrote: > > > On Sep 25, 8:40šam, "Max Ivanov" <[EMAIL PROTECTED]> wrote: > > > > At any time in main process there are shouldn't be no more than two > > > copies of data > > > (one origin

Re: multiprocessing eats memory

2008-09-26 Thread redbaron
On 26 сент, 04:20, Istvan Albert <[EMAIL PROTECTED]> wrote: > On Sep 25, 8:40 am, "Max Ivanov" <[EMAIL PROTECTED]> wrote: > > > At any time in main process there are shouldn't be no more than two copies > > of data > > (one original data and one result). > > From the looks of it you are storing a

Re: multiprocessing eats memory

2008-09-25 Thread Istvan Albert
On Sep 25, 8:40 am, "Max Ivanov" <[EMAIL PROTECTED]> wrote: > At any time in main process there are shouldn't be no more than two copies of > data > (one original data and one result). >From the looks of it you are storing a lots of references to various copies of your data via the async set. --