Bugs item #1389051, was opened at 2005-12-23 19:11
Message generated for change (Comment added) made by effbot
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1389051&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Fredrik Lundh (effbot)
Assigned to: Nobody/Anonymous (nobody)
Summary: imaplib causes excessive fragmentation for large documents
Initial Comment:
When fetching large documents via SSL, the imaplib
attempts to read it all in one chunk, but the SSL
socket layer only returns ~16k at a time.
The result is that Python will end up allocating, say,
a 15 megabyte block, shrink it to a few kilobytes,
occasionally allocate a medium-sized block (to hold
the list of chunks), and repeat this again and again
and again. Not all malloc implementations can reuse
the (15 megabytes minus a few kilobyte) block when
allocating the next 15 megabyte block. In a worst
case scenario, you'll need some 13 gigabytes of
virtual memory to read a 15 megabyte message...
A simple solution is to change
data = self.sslobj.read(size-read)
to
data = self.sslobj.read(min(size-read, 16384))
For more on this, see this thread:
http://groups.google.com/group/comp.lang.python/browse_
frm/thread/3737500bac287575/d715bf614a86e786
</F>
----------------------------------------------------------------------
>Comment By: Fredrik Lundh (effbot)
Date: 2005-12-25 11:57
Message:
Logged In: YES
user_id=38376
As noted in that thread, the same problem applies to non-
SSL accesses. The problematic line is:
data = self._sock.recv(recv_size)
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1389051&group_id=5470
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
