New submission from Charles-François Natali: This is a spinoff of issue #19506: currently, subprocess.communicate() uses a 4K buffer when reading data from pipes. This was probably optimal a couple years ago, but nowadays most operating systems have larger pipes (e.g. Linux has 64K), so we might be able to gain some performance by increasing this buffer size.
For example, here's a benchmark reading from a subprocess spawning "dd if=/dev/zero bs=1M count=100": # before, 4K buffer $ ./python ~/test_sub_read.py 2.72450800300021 # after, 64K buffer $ ./python ~/test_sub_read.py 1.2509000449999803 The difference is impressive. I'm attaching the benchmark script so that others can experiment a bit (on multi-core machines and also different OSes). ---------- components: Library (Lib) files: test_sub_read.py messages: 205534 nosy: gregory.p.smith, haypo, neologix, pitrou, sbt, serhiy.storchaka priority: normal severity: normal stage: needs patch status: open title: subprocess: increase read buffer size type: performance versions: Python 3.4 Added file: http://bugs.python.org/file33041/test_sub_read.py _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue19929> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
