On 11/14/2010 11:08 AM, Artur Siekielski wrote:
Hi.
I'm using CPython 2.7 and Linux. In order to make parallel
computations on a large list of objects I want to use multiple
processes (by using multiprocessing module). In the first step I fill
the list with objects and then I fork() my worker processes that do
the job.
This should work optimally in the aspect of memory usage because Linux
implements copy-on-write in forked processes.
There used to be a memory leak when using Pickle to talk
to subprocesses. See what I wrote at
"http://groups.google.com/group/comp.lang.python/browse_thread/thread/3f8b999c25af263a%22";
The Pickle module has an optimization scheme and cache which
requires that both sender and receiver keep a reference to each
object transmitted. Cache clearing wasn't well worked out for
Pickle object reuse, and the receive end had a memory leak.
Did that get fixed?
John Nagle
--
http://mail.python.org/mailman/listinfo/python-list
