On 07/26/2012 05:21 AM, Tim Chase wrote:
On 07/25/12 12:47, Simon Pirschel wrote:
I'm currently experimenting with IMAP using Python 2.7.3 and
IMAP4 from imaplib. I noticed the performance to be very bad. I
read 5000 files from a directory and append them to an IMAP
INBOX. The hole procedure of reading and appending is taking
about 210 seconds.

I set up the exact same code in Perl to check if there is a
general IMAP server configuration issue, since CPU and I/O isn't
the problem. The same amount of data on the same IMAP server is
done after 7.9 seconds using Perl.

The difference is huge and I tried to narrow the issue down by
profiling the Python code. The profile results are, 206 seconds
are spent in calling socket.recv.
While I don't know the ins and outs of the Perl code, it looks like
the Python imaplib.py is doing some pretty straight-forward
send/receive work.  However, IMAP4 does support both pipelining and
multi-append extensions, so if the Perl code takes advantage of the
features if the server offers them, it might be reaping significant
gains over the naïve Python code.

Can you get a debugging dump of the commands being sent each way?
(particularly if you see a bunch of APPEND commands before the reply
acknowledgement, as detailed at [1])

-tkc

[1]
http://www.ietf.org/rfc/rfc3502.txt on page 4

I took a strace of all network related sys calls from Python and Perl. It seems both don't do multi appends and waiting for the OK after appending one mail. But the Python version is getting a completely different response.

Python: "* 204 EXISTS\r\n* 1 RECENT\r\nNBGC3 OK [APPENDUID 1343283946 204] Append completed.\r\n"
Perl: "2 OK [APPENDUID 1343283946 255] Append completed.\r\n"

The Python lib seems to trigger a EXISTS and RECENT lookup on every append, which could be the performance difference. I couldn't figure out why, yet!

Python strace: http://aboutsimon.com/py.strace
Perl strace: http://aboutsimon.com/pl.strace

Strings are shortened to 40 chars.


Simon
-- 
http://mail.python.org/mailman/listinfo/python-list

Reply via email to