New submission from Johannes Baiter:
While testing a module that uses multiprocessing.Pool to distribute load across
multiple processes, I noticed that my test suite was copmleting very quickly
(~0.15s) on Python 2.6, while Python 2.7 and above took around 10x as long
(~1.6s).
Upon debugging this, I pinned the slowdown down to the 'Pool.join()' method.
Removing it removed the slowdown almost completely.
I then checked the version history of the 'multiprocessing.pool' module between
2.6 and 2.7 and noticed that when the 'maxtasksperchild' parameter was
introduced, a thread to handle the workers was introduced, which was 'join()'ed
when the pool was joined.
This is the function that is executed in the thread (from latest CPython
checkout):
@staticmethod
def _handle_workers(pool):
thread = threading.current_thread()
# Keep maintaining workers until the cache gets drained, unless the pool
# is terminated.
while thread._state == RUN or (pool._cache and thread._state !=
TERMINATE):
pool._maintain_pool()
time.sleep(0.1) # <-- Cause of slow 'join()' after 2.6
# send sentinel to stop workers
pool._taskqueue.put(None)
util.debug('worker handler exiting')
I highlighted the portion that makes 'join()' take a rather long time with
short-lived processes in Python 2.7 and greater.
Replacing it with 'time.sleep(0)' (as is done in '_help_stuff_finish()' later
in the module) makes joining go as fast as in 2.6.
Is there a specific reason why this sleep period was chosen?
----------
components: Library (Lib)
messages: 217129
nosy: Johannes.Baiter
priority: normal
severity: normal
status: open
title: multiprocessing.Pool._handle_workers sleeps too long
type: performance
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue21345>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
