New submission from Charles Cazabon <charlesc-pyt...@pyropus.ca>: Worker processes with multiprocessing.Pool live for the duration of the Pool. If the tasks they run happen to leak memory (from a C extension module, or from creating cycles of unreachable objects, etc) or open files or other resources, there's no easy way to clean them up.
Similarly, if one task passed to the pool allocates a large amount of memory, but further tasks are small, that additional memory isn't returned to the system because the process involved hasn't exited. A common approach to this problem (as used by Apache, mod_wsgi, and various other software) is to allow worker processes to exit (and be replaced with fresh processes) after completing a specified amount of work. The attached patch (against Python 2.6.2, but applies to various other versions with some fuzz) implements this as optional new behaviour in multiprocessing.Pool(). An additional optional argument is specified for the maximum number of tasks a worker process performs before it exits and is replaced with a fresh worker process. ---------- components: Library (Lib) files: worker-lifetime-python2.6.2.patch keywords: patch messages: 92971 nosy: charlesc severity: normal status: open title: Add worker process lifetime to multiprocessing.Pool - patch included type: feature request versions: Python 2.7, Python 3.1 Added file: http://bugs.python.org/file14946/worker-lifetime-python2.6.2.patch _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue6963> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
