Bugs item #1092502, was opened at 2004-12-29 03:09
Message generated for change (Comment added) made by a_lauer
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1092502&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Platform-specific
Status: Open
Resolution: None
Priority: 5
Submitted By: bacchusrx (bacchusrx)
Assigned to: Nobody/Anonymous (nobody)
Summary: Memory leak in socket.py on Mac OS X 10.3
Initial Comment:
Some part of socket.py leaks memory on Mac OS X 10.3 (both with
the python 2.3 that ships with the OS and with python 2.4).
I encountered the problem in John Goerzen's offlineimap.
Transfers of messages over a certain size would cause the program
to bail with malloc errors, eg
*** malloc: vm_allocate(size=5459968) failed (error code=3)
*** malloc[13730]: error: Can't allocate region
Inspecting the process as it runs shows that python's total memory
size grows wildly during such transfers.
The bug manifests in _fileobject.read() in socket.py. You can
replicate the problem easily using the attached example with "nc -l
-p 9330 < /dev/zero" running on some some remote host.
The way _fileobject.read() is written, socket.recv is called with the
larger of the minimum rbuf size or whatever's left to be read.
Whatever is received is then appended to a buffer which is joined
and returned at the end of function.
It looks like each time through the loop, space for recv_size is
allocated but not freed, so if the loop runs for enough iterations,
python exhausts the memory available to it.
You can sidestep the condition if recv_size is small (like
_fileobject.default_bufsize small).
I can't replicate this problem with python 2.3 on FreeBSD 4.9 or
FreeBSD 5.2, nor on Mac OS X 10.3 if the logic from
_fileobject.read() is re-written in Perl (for example).
----------------------------------------------------------------------
Comment By: Andreas Lauer (a_lauer)
Date: 2005-11-10 08:42
Message:
Logged In: YES
user_id=1376343
The problem also occurs in rare cases under Windows XP with
Python 2.3.4. I Suspect the code line
recv_size = max(self._rbufsize, left)
in socket.py to be a part of the problem.
In the case that I investigated, this caused >600 allocations
of up to 5 MBytes (which came in 8 KB packets).
Sure, the memory allocator should be able to handle this in
_socket.recv (first it allocates the X MBytes buffer, which
is later
resized with _PyString_Resize), but it I think the correct
line in socket.py
is
recv_size = min(self._rbufsize, left).
At least, after this my problem was gone.
----------------------------------------------------------------------
Comment By: Bob Ippolito (etrepum)
Date: 2005-01-02 03:25
Message:
Logged In: YES
user_id=139309
that code paste is missing an "int i" at the beginning of main..
----------------------------------------------------------------------
Comment By: Bob Ippolito (etrepum)
Date: 2005-01-02 03:23
Message:
Logged In: YES
user_id=139309
#include <unistd.h>
#define NUM_ALLOCATIONS 100000
#define ALLOC_SIZE 10485760
#define ALLOC_RESIZE 1492
int main(int argc, char **argv) {
/* exiting will free all this leaked memory */
for (i = 0; i < NUM_ALLOCATIONS; i++) {
void *orig_ptr, *new_ptr;
size_t new_size, orig_size;
orig_ptr = malloc(ALLOC_SIZE);
orig_size = malloc_size(orig_ptr);
if (orig_ptr == NULL) {
printf("failure to malloc %d\n", i);
abort();
}
new_ptr = realloc(orig_ptr, ALLOC_RESIZE);
new_size = malloc_size(new_ptr);
printf("resized %d[%p] -> %d[%p]\n",
orig_size, orig_ptr, new_size, new_ptr);
if (new_ptr == NULL) {
printf("failure to realloc %d\n", i);
abort();
}
}
return 0;
}
----------------------------------------------------------------------
Comment By: Bob Ippolito (etrepum)
Date: 2005-01-02 03:22
Message:
Logged In: YES
user_id=139309
Ok. I've tracked it down. realloc(...) on Darwin doesn't actually resize
memory unless it *has* to. For shrinking an allocation, it does not have
to, therefore realloc(...) with a smaller size is a no-op.
It seems that this may be a misunderstanding by Python. The man page
for realloc(...) does not say that it will EVER free memory, EXCEPT in the
case where it has to allocate a larger region.
I'll attach an example that demonstrates this outside of Python.
----------------------------------------------------------------------
Comment By: bacchusrx (bacchusrx)
Date: 2005-01-02 00:01
Message:
Logged In: YES
user_id=646321
I've been able to replicate the problem reliably on both 10.3.5 and
10.3.7. I've attached two more examples to demonstrate:
Try this: Do, "dd if=/dev/zero of=./data bs=1024 count=10240" and save
server.pl wherever you put "data". Have three terminals open. In one,
run "perl server.pl -s0.25". In another, run "top -ovsize" and in the third
run "python example2.py".
After about 100 iterations, python's vsize is +1GB (just about the value
of cumulative_req in example2.py) and if left running will cause a
malloc error at around 360 iterations with a vsize over 3.6GB (again, just
about what cumulative_req reports). Mind you, we've only received
~512kbytes.
server.pl differs from the netcat method in that it (defaults) to sending
only 1492 bytes at a time (configurable with the -b switch) and sleeps for
however many seconds specified with the -s switch. This guarantees
enough iterations to raise the error each time around. When omittting
the -s switch to server.pl, I don't get the error, but throughput is good
enough that the loop in readFromSockUntil() only runs a few times.
----------------------------------------------------------------------
Comment By: Bob Ippolito (etrepum)
Date: 2005-01-01 07:27
Message:
Logged In: YES
user_id=139309
I just played with a bit more. If I catch the MemoryError and try again,
most of the time it will work (sometimes on the second try). These
malloc faults seem to be some kind of temporary condition.
----------------------------------------------------------------------
Comment By: Bob Ippolito (etrepum)
Date: 2005-01-01 07:18
Message:
Logged In: YES
user_id=139309
I can't reproduce this on either version of Python a 10.3.7 machine w/
1gb ram. Python's total memory usage seems stable to me even if the
read is in a while loop.
I can't see anything in sock_recv or _fileobject.read that will in any way
leak memory.
With a really large buffer size (always >17mb, but it does vary with each
run) it will get a memory error but the Python process doesn't grow
beyond 50mb at the samples I looked at. That's pretty much the amount
of RAM I'd expect it to use.
It is kind of surprising it doesn't want to allocate a buffer of that size,
because I have the RAM for it.. but I don't think this is a bug.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1092502&group_id=5470
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
