John O'Connor <tehj...@gmail.com> added the comment:
I experimented with a bunch of different options.
All benchmarks performed with:
$ for i in 1 4 128 256 1024 2048 4069 8192; do
echo -n "buffer_size=${i} ";
./python -m timeit -s "f=open('LICENSE','rb');b=bytearray(${i})" \
"f.seek(0)" "while f.readinto(b): pass";
done
with io.DEFAULT_BUFFER_SIZE = 8192
--------------------------------------------------------------------------------
Before patch
buffer_size=1 100 loops, best of 3: 10.4 msec per loop
buffer_size=4 100 loops, best of 3: 2.67 msec per loop
buffer_size=128 10000 loops, best of 3: 102 usec per loop
buffer_size=256 10000 loops, best of 3: 54.9 usec per loop
buffer_size=1024 10000 loops, best of 3: 26.9 usec per loop
buffer_size=2048 10000 loops, best of 3: 20.3 usec per loop
buffer_size=4069 100000 loops, best of 3: 16.3 usec per loop
buffer_size=8192 100000 loops, best of 3: 11.1 usec per loop
--------------------------------------------------------------------------------
Always read into caller's buffer
buffer_size=1 100 loops, best of 3: 14 msec per loop
buffer_size=4 100 loops, best of 3: 4.02 msec per loop
buffer_size=128 10000 loops, best of 3: 114 usec per loop
buffer_size=256 10000 loops, best of 3: 63.7 usec per loop
buffer_size=1024 100000 loops, best of 3: 19.4 usec per loop
buffer_size=2048 100000 loops, best of 3: 11.2 usec per loop
* buffer_size=4069 100000 loops, best of 3: 8.12 usec per loop
* buffer_size=8192 100000 loops, best of 3: 5.79 usec per loop
--------------------------------------------------------------------------------
Read into caller's buffer if java-like bound of internal buffer size is smaller
buffer_size=1 100 loops, best of 3: 5.01 msec per loop
buffer_size=4 1000 loops, best of 3: 1.27 msec per loop
buffer_size=128 10000 loops, best of 3: 46.9 usec per loop
buffer_size=256 10000 loops, best of 3: 29.5 usec per loop
buffer_size=1024 100000 loops, best of 3: 12.9 usec per loop
buffer_size=2048 100000 loops, best of 3: 10.8 usec per loop
buffer_size=4069 100000 loops, best of 3: 9.06 usec per loop
* buffer_size=8192 100000 loops, best of 3: 5.78 usec per loop
--------------------------------------------------------------------------------
Using bound = buffer_size / 2
buffer_size=1 100 loops, best of 3: 6.04 msec per loop
buffer_size=4 1000 loops, best of 3: 1.34 msec per loop
buffer_size=128 10000 loops, best of 3: 49.4 usec per loop
buffer_size=256 10000 loops, best of 3: 29.5 usec per loop
buffer_size=1024 100000 loops, best of 3: 13.2 usec per loop
buffer_size=2048 100000 loops, best of 3: 10.7 usec per loop
* buffer_size=4069 100000 loops, best of 3: 8.66 usec per loop
buffer_size=8192 100000 loops, best of 3: 6.1 usec per loop
--------------------------------------------------------------------------------
Using bound = buffer_size / 4
buffer_size=1 100 loops, best of 3: 5.45 msec per loop
buffer_size=4 1000 loops, best of 3: 1.34 msec per loop
buffer_size=128 10000 loops, best of 3: 49.6 usec per loop
buffer_size=256 10000 loops, best of 3: 28.8 usec per loop
buffer_size=1024 100000 loops, best of 3: 13.1 usec per loop
buffer_size=2048 100000 loops, best of 3: 12.8 usec per loop
buffer_size=4069 100000 loops, best of 3: 8.42 usec per loop
buffer_size=8192 100000 loops, best of 3: 5.93 usec per loop
--------------------------------------------------------------------------------
Always use internal buffer
* buffer_size=1 100 loops, best of 3: 4.53 msec per loop
* buffer_size=4 1000 loops, best of 3: 1.14 msec per loop
* buffer_size=128 10000 loops, best of 3: 45 usec per loop
* buffer_size=256 10000 loops, best of 3: 26.9 usec per loop
* buffer_size=1024 100000 loops, best of 3: 12.9 usec per loop
* buffer_size=2048 100000 loops, best of 3: 10.2 usec per loop
buffer_size=4069 100000 loops, best of 3: 9.15 usec per loop
buffer_size=8192 100000 loops, best of 3: 8.42 usec per loop
--------------------------------------------------------------------------------
Use read() instead
$ for i in 1 4 128 256 1024 2048 4069 8192; do
echo -n "size=${i} ";
./python -m timeit -s "f=open('LICENSE','rb');" "f.seek(0)" \
"while f.read(${i}): pass";
done
* size=1 100 loops, best of 3: 2.56 msec per loop
* size=4 1000 loops, best of 3: 709 usec per loop
* size=128 10000 loops, best of 3: 29.7 usec per loop
* size=256 100000 loops, best of 3: 18.6 usec per loop
size=1024 100000 loops, best of 3: 13.3 usec per loop
size=2048 100000 loops, best of 3: 10.8 usec per loop
size=4069 100000 loops, best of 3: 10.1 usec per loop
size=8192 100000 loops, best of 3: 6.41 usec per loop
--------------------------------------------------------------------------------
Based on the above I think you are right about using the internal buffer
regardless (revision attached). You pay a price with larger buffer sizes but on
balance it seems to be a much better general purpose solution. The java-like
solution is decent as well, it is only slightly slower for small reads and
optimal
for larger buffer sizes. Personally, I would opt for the performance with larger
buffer sizes but I can only speculate as to what the general user would do. We
could always note the trade-off in the docs.
Interestingly enough it seems like using read() is a better idea
if you use a smaller size (how is that right?).
I attached the afformentioned revisions.
----------
Added file: http://bugs.python.org/file21940/issue9971.patch
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue9971>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
