Dan O'Reilly added the comment: I'm seeing an even larger difference between multiprocessing.Pool and ProcessPoolExecutor on my machine, with Python 3.4:
Starting multiproc...done in 2.160644769668579 s. Starting futures...done in 67.953957319259644 s. Starting futures "fixed"...done in 2.134932041168213 s. I've updated the initial patch to address the comments Antoine made; the chunksize now defaults to 1, and itertools is used to chunk the input iterables, rather than building a list. Attached is an updated benchmark script: Starting multiproc...done in 2.2295100688934326 s. Starting futures...done in 68.5991039276123 s. Starting futures "fixed" (no chunking)...done in 69.18992304801941 s. Starting futures "fixed" (with chunking)...done in 2.352942705154419 s. The new implementation of map has essentially identical performance to the original with chunksize=1, but it performs much better with a larger chunksize provided. ---------- nosy: +dan.oreilly Added file: http://bugs.python.org/file36058/test_mult.py _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue11271> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
