Piet, The situation is 1a of your listed options, however my issue was solved. I was stopping the subprocesses from consuming more data at each iteration which led to the data being lost since the subprocess worker function would then end - I now keep them alive across iterations.
Thanks for your help, I'm new to the multiprocessing module and this was very helpful! On Jul 17, 4:26 am, Piet van Oostrum <p...@cs.uu.nl> wrote: > There is stil something not clear in your description. > > >m> I'm using multiprocessing to spawn several subprocesses, each of which > >m> uses a very large data structure (making it impractical to pass it via > >m> pipes / pickling). I need to allocate this structure once when the > >m> process is created and have it remain in memory for the duration of > >m> the process. > > I have read this as that every subprocess has its own large > data structure and that there is no connection between these. > > But seeing where the discussion is going I guess there might be > different interpretations. So can you enlighten us how the situation is? > > 1. Each subprocess has a copy of a data structure that is prepared by the > master process. Therefore you want it to be passed by the fork > 1a. the data structure is constant i.e. the subprocess doesn't change it > 1b. the subprocess makes changes in its copy > 2. Each subprocess has a seperate data structure not equal to the others > 3. Something else. > -- > Piet van Oostrum <p...@cs.uu.nl> > URL:http://pietvanoostrum.com[PGP 8DAE142BE17999C4] > Private email: p...@vanoostrum.org -- http://mail.python.org/mailman/listinfo/python-list
