> When processing data in parallel you will use up as muchmemoryas
> many datasets you are processing at any given time.
Worker processes eats 2-4 times more than I pass to them.
>If you need to
> reducememoryuse then you need to start fewer processes and use some
> mechanism to distribute the wo
On Sep 26, 4:52 am, redbaron <[EMAIL PROTECTED]> wrote:
> How could I avoid of storing them? I need something to check does it
> ready or not and retrieve results if ready. I couldn't see the way to
> achieve same result without storing asyncs set.
It all depends on what you are trying to do. The
On 26 сент, 17:03, MRAB <[EMAIL PROTECTED]> wrote:
> On Sep 26, 9:52 am, redbaron <[EMAIL PROTECTED]> wrote:
>
> > On 26 ÓÅÎÔ, 04:20, Istvan Albert <[EMAIL PROTECTED]> wrote:
>
> > > On Sep 25, 8:40šam, "Max Ivanov" <[EMAIL PROTECTED]> wrote:
>
> > > > At any time in main process there are shouldn'
On Sep 26, 9:52 am, redbaron <[EMAIL PROTECTED]> wrote:
> On 26 ÓÅÎÔ, 04:20, Istvan Albert <[EMAIL PROTECTED]> wrote:
>
> > On Sep 25, 8:40šam, "Max Ivanov" <[EMAIL PROTECTED]> wrote:
>
> > > At any time in main process there are shouldn't be no more than two
> > > copies of data
> > > (one origin
On 26 сент, 04:20, Istvan Albert <[EMAIL PROTECTED]> wrote:
> On Sep 25, 8:40 am, "Max Ivanov" <[EMAIL PROTECTED]> wrote:
>
> > At any time in main process there are shouldn't be no more than two copies
> > of data
> > (one original data and one result).
>
> From the looks of it you are storing a
On Sep 25, 8:40 am, "Max Ivanov" <[EMAIL PROTECTED]> wrote:
> At any time in main process there are shouldn't be no more than two copies of
> data
> (one original data and one result).
>From the looks of it you are storing a lots of references to various
copies of your data via the async set.
--
I'm playing with pyprocessing module and found that it eats lot's of
memory. I've made small test case to show it. I pass ~45mb of data to
worker processes and than get it back slightly modified. At any time
in main process there are shouldn't be no more than two copies of data
(one original data a
