[issue18329] for line in socket.makefile() speed degradation

Марк Коренберг added the comment: mmarkk@mmwork:~$ python2.7 --version Python 2.7.12 mmarkk@mmwork:~$ python2.7 ./qwe.py TCP mode, makefile method. 292731.3 lines per second (279.2 MB/s). Delay is 3.42 seconds TCP mode, fdopen method. 2410875.9 lines per second (2299.2 MB/s). Delay is 0.41

[issue18329] for line in socket.makefile() speed degradation

Christian Heimes added the comment: Is this still a problem in 3.5 and newer? -- status: open -> pending versions: +Python 3.6, Python 3.7 -Python 3.2, Python 3.3, Python 3.4, Python 3.5 ___ Python tracker ___

[issue18329] for line in socket.makefile() speed degradation

Changes by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:

[issue18329] for line in socket.makefile() speed degradation

Richard Oudkerk added the comment: I find that by adding the lines fileobj.raw.readinto = ss.recv_into fileobj.raw.read = ss.recv the speed with makefile() is about 30% slower than with fdopen(). -- ___ Python tracker

[issue18329] for line in socket.makefile() speed degradation

STINNER Victor added the comment: My results of the benchmark on Linux 3.9, first with a loop calling fileobj.read(8192) ("read"), then with the "for line in fileobj: pass" ("readline"). $ python2.7 bench_socket_read.py TCP mode, makefile method. 1775085.8 lines per second (1692.9 MB/s). Dela

[issue18329] for line in socket.makefile() speed degradation

STINNER Victor added the comment: On Python 2, socket.makefile("rb") duplicates the file descriptor and creates a new file object: fileobj.read() calls the C function read(). On Python 3, socket.makefile("rb") creates a SocketIO wrapped in BufferedReader: fileobj.read() calls the C function re

[issue18329] for line in socket.makefile() speed degradation

Richard Oudkerk added the comment: Ah. I had not thought of socket timeouts. -- ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue18329] for line in socket.makefile() speed degradation

Christian Heimes added the comment: There is little bit more to it. A comment in the class' body describes the reason for SockteIO: # One might wonder why not let FileIO do the job instead. There are two # main reasons why FileIO is not adapted: # - it wouldn't work under Windows (

[issue18329] for line in socket.makefile() speed degradation

Richard Oudkerk added the comment: The only real reason for implementing SocketIO in pure Python is because read() and write() do not work on Windows with sockets. (I think there are also a few complications involving SSL sockets and the close() method.) On Windows I have implemented a file o

[issue18329] for line in socket.makefile() speed degradation

Christian Heimes added the comment: Err, you are right. It's buffered. I misread one function call and thought it's not buffered at all. >>> COUNT = 100 >>> LINELEN = 1000 >>> int(LINELEN * COUNT / 8192) 122070 The number looks familiar, doesn't it? :) I still stand by my analysis that th

[issue18329] for line in socket.makefile() speed degradation

Richard Oudkerk added the comment: Using while True: if not fileobj.read(8192): break instead of for line in fileobj: pass results in higher throughput, but a similar slowdown with makefile(). So this is not a problem specific to readl

[issue18329] for line in socket.makefile() speed degradation

Richard Oudkerk added the comment: > I think I know what's going on here. For socket IO readline() uses a > readahead buffer size of 1. Why is that? I think that makefile(mode='rb') and fdopen() both create BufferedReader objects with the same buffer size. It looks to me like there are the s

[issue18329] for line in socket.makefile() speed degradation

Christian Heimes added the comment: Summary: makefile: 611990 primitive calls fdopen: 1629 primitive calls -- ___ Python tracker ___

[issue18329] for line in socket.makefile() speed degradation

Christian Heimes added the comment: I first suspected that file based IO involves too many syscalls and context switches. However ``strace -c`` showed only minor differences in the amount of syscalls. The difference between read() and recvfrom() should not make any difference. perf revealed t

[issue18329] for line in socket.makefile() speed degradation

Changes by Christian Heimes : -- nosy: +christian.heimes ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:/

[issue18329] for line in socket.makefile() speed degradation

Serhiy Storchaka added the comment: 3.4 have same performance as 3.3. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsu

[issue18329] for line in socket.makefile() speed degradation

Марк Коренберг added the comment: Can anyone test in python 3.4 ? -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscr

[issue18329] for line in socket.makefile() speed degradation

Марк Коренберг added the comment: Eliminate unicode conversion for python3, but results still the same $ python2.7 qwe.py TCP mode, makefile method. 211416.5 lines per second (201.6 MB/s). Delay is 4.73 seconds TCP mode, fdopen method. 1041666.7 lines per second (993.4 MB/s). Delay is 0.9

[issue18329] for line in socket.makefile() speed degradation

Changes by Марк Коренберг : Removed file: http://bugs.python.org/file30731/qwe.py ___ Python tracker ___ ___ Python-bugs-list mailing list Uns

[issue18329] for line in socket.makefile() speed degradation

Richard Oudkerk added the comment: I think in Python 3 makefile() returns a TextIOWrapper object by default. To force the use of binary you need to specfiy the mode: fileobj = ss.makefile(mode='rb') -- nosy: +sbt ___ Python tracker

[issue18329] for line in socket.makefile() speed degradation

Serhiy Storchaka added the comment: Try to wrap socket.makefile() with io.BufferedReader(). -- nosy: +serhiy.storchaka ___ Python tracker ___

[issue18329] for line in socket.makefile() speed degradation

Марк Коренберг added the comment: Well, python 3.3 is slightly faster: $ python3.3 qwe.py TCP mode, makefile method. 380228.1 lines per second (362.6 MB/s). Delay is 2.63 seconds TCP mode, fdopen method. 877193.0 lines per second (836.6 MB/s). Delay is 1.14 seconds UNIX mode, makefile met

[issue18329] for line in socket.makefile() speed degradation

Марк Коренберг added the comment: Yes, results are repeatable, and for python 2.7 I have roughly same timings for UNIX socket. Also, I have straced all variants and see that in all 4 cases (and for both python versions) IO is done using 8192 blocks in size, so buffering is not cause of proble

[issue18329] for line in socket.makefile() speed degradation

New submission from Марк Коренберг: Results or running attached program: $ python2.7 qwe.py TCP mode, makefile method. 198807.2 lines per second (189.6 MB/s). Delay is 5.03 seconds TCP mode, fdopen method. 1041666.7 lines per second (993.4 MB/s). Delay is 0.96 seconds UNIX mode, makefile m