I'm using multiprocessing to spawn several subprocesses, each of which
uses a very large data structure (making it impractical to pass it via
pipes / pickling). I need to allocate this structure once when the
process is created and have it remain in memory for the duration of
the process. The way t
On Jul 16, 8:39 am, Piet van Oostrum wrote:
> >>>>> mheavner (m) wrote:
> >m> I'm using multiprocessing to spawn several subprocesses, each of which
> >m> uses a very large data structure (making it impractical to pass it via
> >m> pipes / pick
On Jul 16, 9:18 am, mheavner wrote:
> On Jul 16, 8:39 am, Piet van Oostrum wrote:
>
>
>
> > >>>>> mheavner (m) wrote:
> > >m> I'm using multiprocessing to spawn several subprocesses, each of which
> > >m> uses a very large data
:
> >>>>> mheavner (m) wrote:
> >m> 'The process' refers to the subprocess. I could do as you say, load
> >m> the data structure each time, but the problem is that takes a
> >m> considerable amount of time compared to the the actual computation
> >m&g
Piet,
The situation is 1a of your listed options, however my issue was
solved. I was stopping the subprocesses from consuming more data at
each iteration which led to the data being lost since the subprocess
worker function would then end - I now keep them alive across
iterations.
Thanks for your
