New submission from Aren Olson <[EMAIL PROTECTED]>: This is a reposting of issue 508157, as requested by gvanrossum.
The socket file object in httplib is opened without any buffering resulting in very slow performance of read(). The specific problem is in the httplib.HTTPResponse constructor. It calls sock.makefile() with a 0 for the buffer size. changing the buffer size to 4096 improved the time needed to download 10MB from 15.5s to 1.78s, almost 9x faster. Repeat downloads of the same file (meaning the server now has the file cached in memory), yield times of 15.5s and 0.03s, a 500x improvement. When fetching from a server on the local network, rather than from localhost, these times become 15.5s and 0.9s in both cases, a 17x speedup. Real-world situations will likely be a mix of these, however it is safe to say the speed improvement will be substantial. Adding an option to adjust the buffer size would be very welcome, though the default value should still be zero, to avoid the issues already mentioned in issue 508157. These speed results were obtained with python2.5 and apache2 under Ubuntu linux, using the code found here: http://pastebin.ca/973578 ---------- components: Library (Lib) messages: 65121 nosy: reacocard severity: normal status: open title: httplib read() very slow due to lack of socket buffer versions: Python 2.6 __________________________________ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2576> __________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
