Bugs item #1092502, was opened at 2004-12-28 21:09 Message generated for change (Comment added) made by etrepum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1092502&group_id=5470
Category: Python Library Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: bacchusrx (bacchusrx) Assigned to: Nobody/Anonymous (nobody) Summary: Memory leak in socket.py on Mac OS X 10.3 Initial Comment: Some part of socket.py leaks memory on Mac OS X 10.3 (both with the python 2.3 that ships with the OS and with python 2.4). I encountered the problem in John Goerzen's offlineimap. Transfers of messages over a certain size would cause the program to bail with malloc errors, eg *** malloc: vm_allocate(size=5459968) failed (error code=3) *** malloc[13730]: error: Can't allocate region Inspecting the process as it runs shows that python's total memory size grows wildly during such transfers. The bug manifests in _fileobject.read() in socket.py. You can replicate the problem easily using the attached example with "nc -l -p 9330 < /dev/zero" running on some some remote host. The way _fileobject.read() is written, socket.recv is called with the larger of the minimum rbuf size or whatever's left to be read. Whatever is received is then appended to a buffer which is joined and returned at the end of function. It looks like each time through the loop, space for recv_size is allocated but not freed, so if the loop runs for enough iterations, python exhausts the memory available to it. You can sidestep the condition if recv_size is small (like _fileobject.default_bufsize small). I can't replicate this problem with python 2.3 on FreeBSD 4.9 or FreeBSD 5.2, nor on Mac OS X 10.3 if the logic from _fileobject.read() is re-written in Perl (for example). ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2005-01-01 21:22 Message: Logged In: YES user_id=139309 Ok. I've tracked it down. realloc(...) on Darwin doesn't actually resize memory unless it *has* to. For shrinking an allocation, it does not have to, therefore realloc(...) with a smaller size is a no-op. It seems that this may be a misunderstanding by Python. The man page for realloc(...) does not say that it will EVER free memory, EXCEPT in the case where it has to allocate a larger region. I'll attach an example that demonstrates this outside of Python. ---------------------------------------------------------------------- Comment By: bacchusrx (bacchusrx) Date: 2005-01-01 18:01 Message: Logged In: YES user_id=646321 I've been able to replicate the problem reliably on both 10.3.5 and 10.3.7. I've attached two more examples to demonstrate: Try this: Do, "dd if=/dev/zero of=./data bs=1024 count=10240" and save server.pl wherever you put "data". Have three terminals open. In one, run "perl server.pl -s0.25". In another, run "top -ovsize" and in the third run "python example2.py". After about 100 iterations, python's vsize is +1GB (just about the value of cumulative_req in example2.py) and if left running will cause a malloc error at around 360 iterations with a vsize over 3.6GB (again, just about what cumulative_req reports). Mind you, we've only received ~512kbytes. server.pl differs from the netcat method in that it (defaults) to sending only 1492 bytes at a time (configurable with the -b switch) and sleeps for however many seconds specified with the -s switch. This guarantees enough iterations to raise the error each time around. When omittting the -s switch to server.pl, I don't get the error, but throughput is good enough that the loop in readFromSockUntil() only runs a few times. ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2005-01-01 01:27 Message: Logged In: YES user_id=139309 I just played with a bit more. If I catch the MemoryError and try again, most of the time it will work (sometimes on the second try). These malloc faults seem to be some kind of temporary condition. ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2005-01-01 01:18 Message: Logged In: YES user_id=139309 I can't reproduce this on either version of Python a 10.3.7 machine w/ 1gb ram. Python's total memory usage seems stable to me even if the read is in a while loop. I can't see anything in sock_recv or _fileobject.read that will in any way leak memory. With a really large buffer size (always >17mb, but it does vary with each run) it will get a memory error but the Python process doesn't grow beyond 50mb at the samples I looked at. That's pretty much the amount of RAM I'd expect it to use. It is kind of surprising it doesn't want to allocate a buffer of that size, because I have the RAM for it.. but I don't think this is a bug. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1092502&group_id=5470 _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
