Hi all, I have a problem with the memory consumption of multiprocessing.Pool()'s worker processes. I have a parent process that has to handle big data structures and would like to use a pool of processes for computations.
The problem is, that all worker processes have the same memory
requirement as the parent one, although they do *not* use any the parent
processes data structures. The following snippet illustrates this
behaviour:
--- snip ---
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import multiprocessing
import time
def worker(el):
time.sleep(10)
def main():
print 'Init Pool 1'
p_pool = multiprocessing.Pool()
print 'Call pool.map()'
p_pool.map(worker, range(multiprocessing.cpu_count()))
print 'Allocate memory'
eat_memory = range(1000000)
print 'Call pool.map()'
p_pool.map(worker, range(multiprocessing.cpu_count()))
print 'Delete pool 1'
del p_pool
print 'Init Pool 2'
p_pool = multiprocessing.Pool()
print 'Call pool.map()'
p_pool.map(worker, range(multiprocessing.cpu_count()))
print 'Delete pool 2'
del p_pool
if __name__ == '__main__':
main()
--- snip ---
You will see that the memory consumption of the worker processes will be
roughly the same for the first two calls to p_pool.map(), but rise for
the third.
How can I make sure that 'eat_memory' does not use any memory in the
pool processes? This is important as I don't always know when a pool is
instanciated and the pool processes should *not* have the same memory
requirements as the parent process.
Am I missing something here?
--
.''`. Wolodja Wentland <wentl...@cl.uni-heidelberg.de>
: :' :
`. `'` 4096R/CAF14EFC
`- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC
signature.asc
Description: Digital signature
-- http://mail.python.org/mailman/listinfo/python-list
