[ python-Bugs-1257255 ] tarfile local name is local, should be abspath
Bugs item #1257255, was opened at 2005-08-12 04:26 Message generated for change (Comment added) made by gustaebel You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1257255&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Martin Blais (blais) Assigned to: Nobody/Anonymous (nobody) Summary: tarfile local name is local, should be abspath Initial Comment: I ran into a bug using the tarfile module with compression from a directory that did not exist anymore, and this exhibits a bug in the tarfile module. I'm not completely sure how to fix it with certainty, so instead of filing a quick patch, I submit a bug so people familiar with the module can have a look first. The problem is that when you open a tarfile with gz or bz2 compression, the TarFile object is opened with a filename that is not absolute, just local (basename is called) but the actual file is using a file-like object. Now, when you add a new entry, there is a check in the TarFile.add method that checks if we're not adding the archive itself into the tarfile, and this method calls abspath. Calling abspath on the name that has been basename'd is incorrect and a bug. It happens not to fail nor cause any problems when called from an existing directory (the test returns false), but it does not do the job it is supposed to. When the CWD has been deleted, calling abspath fails with an OSError, which brings out the problem. Some code to reproduce the bug is provided in an attachment. -- Comment By: Lars Gustäbel (gustaebel) Date: 2005-08-15 13:39 Message: Logged In: YES user_id=642936 Could you please attach the test code you mentioned? -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1257255&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1257255 ] tarfile local name is local, should be abspath
Bugs item #1257255, was opened at 2005-08-12 02:26 Message generated for change (Comment added) made by blais You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1257255&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Martin Blais (blais) Assigned to: Nobody/Anonymous (nobody) Summary: tarfile local name is local, should be abspath Initial Comment: I ran into a bug using the tarfile module with compression from a directory that did not exist anymore, and this exhibits a bug in the tarfile module. I'm not completely sure how to fix it with certainty, so instead of filing a quick patch, I submit a bug so people familiar with the module can have a look first. The problem is that when you open a tarfile with gz or bz2 compression, the TarFile object is opened with a filename that is not absolute, just local (basename is called) but the actual file is using a file-like object. Now, when you add a new entry, there is a check in the TarFile.add method that checks if we're not adding the archive itself into the tarfile, and this method calls abspath. Calling abspath on the name that has been basename'd is incorrect and a bug. It happens not to fail nor cause any problems when called from an existing directory (the test returns false), but it does not do the job it is supposed to. When the CWD has been deleted, calling abspath fails with an OSError, which brings out the problem. Some code to reproduce the bug is provided in an attachment. -- >Comment By: Martin Blais (blais) Date: 2005-08-15 17:34 Message: Logged In: YES user_id=10996 Oops, that was silly. My apologies. Here goes the file... -- Comment By: Lars Gustäbel (gustaebel) Date: 2005-08-15 11:39 Message: Logged In: YES user_id=642936 Could you please attach the test code you mentioned? -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1257255&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1208304 ] urllib2's urlopen() method causes a memory leak
Bugs item #1208304, was opened at 2005-05-25 02:20
Message generated for change (Comment added) made by bwelling
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1208304&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Extension Modules
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Petr Toman (manekcz)
Assigned to: Nobody/Anonymous (nobody)
Summary: urllib2's urlopen() method causes a memory leak
Initial Comment:
It seems that the urlopen(url) methd of the urllib2 module
leaves some undestroyable objects in memory.
Please try the following code:
==
if __name__ == '__main__':
import urllib2
a = urllib2.urlopen('http://www.google.com')
del a # or a = None or del(a)
# check memory on memory leaks
import gc
gc.set_debug(gc.DEBUG_SAVEALL)
gc.collect()
for it in gc.garbage:
print it
==
In our code, we're using lots of urlopens in a loop and
the number of unreachable objects grows beyond all
limits :) We also tried a.close() but it didn't help.
You can also try the following:
==
def print_unreachable_len():
# check memory on memory leaks
import gc
gc.set_debug(gc.DEBUG_SAVEALL)
gc.collect()
unreachableL = []
for it in gc.garbage:
unreachableL.append(it)
return len(str(unreachableL))
if __name__ == '__main__':
print "at the beginning", print_unreachable_len()
import urllib2
print "after import of urllib2", print_unreachable_len()
a = urllib2.urlopen('http://www.google.com')
print 'after urllib2.urlopen', print_unreachable_len()
del a
print 'after del', print_unreachable_len()
==
We're using WindowsXP with latest patches, Python 2.4
(ActivePython 2.4 Build 243 (ActiveState Corp.) based on
Python 2.4 (#60, Nov 30 2004, 09:34:21) [MSC v.1310
32 bit (Intel)] on win32).
--
Comment By: Brian Wellington (bwelling)
Date: 2005-08-15 11:13
Message:
Logged In: YES
user_id=63197
The real problem we were seeing wasn't the memory leak, it
was a file descriptor leak. Leaking references within the
interpreter is bad, but the garbage collector will
eventually notice that the system is out of memory and clean
them. Leaking file descriptors is much worse, as gc won't
be triggered when the process has reached it's limit, and
the process will start failing with "Too many file descriptors".
To easily show this problem, run the following from an
interactive python interpreter:
import urllib2
f = urllib2.urlopen('http://www.google.com')
f.close()
and from another window, run "lsof -p ".
It should show a TCP socket in CLOSE_WAIT, which means the
file descriptor is still open. I'm seeing weirdness on
Fedora Core 4 today that I didn't see last week where after
a few seconds, the file descriptor is listed as "can't
identify protocol" instead of TCP, but that's not too
relevant, since it's still open.
Repeating the urllib2.urlopen()/close() pairs of statements
in the interpreter will cause more fds to be leaked, which
can also be seen by lsof.
--
Comment By: Sean Reifschneider (jafo)
Date: 2005-08-12 15:30
Message:
Logged In: YES
user_id=81797
I've just tried it again using the current CVS version as
well as the version installed with Fedora Core 4, and in
both cases I was able to run over 100,000 retrievals of
http://127.0.0.1/test.html and http://127.0.0.1/google.html.
test.html is just "it works" and google.html was generated
with "wget -O google.html http://google.com/";.
I was able to reproduce this before, but now am not. My
urllib2.py includes the r.recv=r.read line. I have upgraded
from FC3 to FC4, could this be something related to an OS or
library interaction? I was going to try to confirm the last
message, but now I can't reproduce the failure.
--
Comment By: Brian Wellington (bwelling)
Date: 2005-08-11 19:22
Message:
Logged In: YES
user_id=63197
We just ran into this same problem, and worked around it by
simply removing the 'r.recv = r.read' line in urllib2.py,
and creating a recv alias to the read function in
HTTPResponse ('recv = read' in the class).
Not sure if this is the best solution, but it seems to work.
--
Comment By: Sean Reifschneider (jafo)
Date: 2005-06-28 20:52
Message:
Logged In: YES
user_id=81797
I give up, this code is kind of a maze of twisty little
passages. I did try doing "a.fp.close()" and that didn't
seem to help at all. Couldn't really make any progress on
that though. I also tried doing a "if a.headers.fp:
a.headers.fp.close()", whic
[ python-Bugs-900092 ] hotshot.stats.load
Bugs item #900092, was opened at 2004-02-19 03:05
Message generated for change (Comment added) made by bwarsaw
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=900092&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 2.4
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: Simon Dahlbacka (sdahlbac)
Assigned to: Brett Cannon (bcannon)
Summary: hotshot.stats.load
Initial Comment:
trying to do a
hotshot.stats.load("myprofiling_file.prof")
fails with assertionerror
assert not self._stack
python 2.3.2 on WinXP
--
>Comment By: Barry A. Warsaw (bwarsaw)
Date: 2005-08-15 14:14
Message:
Logged In: YES
user_id=12800
Patches applied for Python 2.4.2 and 2.5a1
--
Comment By: Barry A. Warsaw (bwarsaw)
Date: 2005-07-08 11:02
Message:
Logged In: YES
user_id=12800
900092-patch-2.txt fixes the test suite for the extra return
event.
--
Comment By: Barry A. Warsaw (bwarsaw)
Date: 2005-07-07 19:26
Message:
Logged In: YES
user_id=12800
See 900092-patch.txt for a candidate patch against Python
2.4.1. Props to Justin Campbell for most of the heavily
lifting (but you can blame me for any problems ;).
This fix restore the tracing of a 'return' event for
exceptions that cause a function to exit.
--
Comment By: Greg Chapman (glchapman)
Date: 2004-11-08 18:54
Message:
Logged In: YES
user_id=86307
Well, the superficial fix doesn't work (sorry for posting
too soon). It turns out that PyTrace_EXCEPTION is sent for
any exception, not just one causing the function to exit.
So I guess the best fix may be to have hotshot always
install its profiler_callback to handle CALLS and RETURNS,
and then optionally install the tracer_callback to handle
only LINEs. Anyway, that works for the one test case I've
been using (a runcall of a function which simply does
"import pickle").
By the way, I'm testing with 2.4b1.
--
Comment By: Greg Chapman (glchapman)
Date: 2004-11-08 18:32
Message:
Logged In: YES
user_id=86307
I ran into this today, so I decided to look into it. It
looks to me like the problem only happens if you profile
with lineevents enabled. In that case, hotshot uses the
tracer_callback function (in _hotshot.c) to dispatch trace
events. This function explicitly ignores exception returns
(PyTrace_EXCEPTION), which can lead to an unbalanced stack
of calls/returns when the log is loaded (if an profiled
function exits with an exception).
It seems on the surface that tracer_callback ought to handle
exceptions the same way as normal returns. This would be
consistent with what happens when profiler_callback is used,
since PyEval_EvalFrame dispatches sends a Py_RETURN to
c_profilefunc when exiting because of an exception.
--
Comment By: Barry A. Warsaw (bwarsaw)
Date: 2004-09-27 10:41
Message:
Logged In: YES
user_id=12800
Could this be related to 1019882?
--
Comment By: Johannes Gijsbers (jlgijsbers)
Date: 2004-09-24 18:00
Message:
Logged In: YES
user_id=469548
Hmm, the file was too big, even though it was compressed.
I've uploaded it to
http://home.student.uva.nl/johannes.gijsbers/roundup.prof.bz2
now.
--
Comment By: Johannes Gijsbers (jlgijsbers)
Date: 2004-09-24 17:58
Message:
Logged In: YES
user_id=469548
While the original report isn't very useful, I've ran into
this problem multiple times as well. I can reproduce it
using the attached profile file (compressed because of the
large size) and the following console session:
Python 2.3.4 (#2, Jul 5 2004, 09:15:05)
[GCC 3.3.4 (Debian 1:3.3.4-2)] on linux2
Type "help", "copyright", "credits" or "license" for more
information.
>>> import hotshot.stats
>>> stats = hotshot.stats.load('roundup.prof')
Traceback (most recent call last):
File "", line 1, in ?
File "/usr/lib/python2.3/hotshot/stats.py", line 12, in load
return StatsLoader(filename).load()
File "/usr/lib/python2.3/hotshot/stats.py", line 51, in load
assert not self._stack
AssertionError
>>>
I'm not sure who's baby hotshot really is, so I'm leaving
this unassigned.
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=900092&group_id=5470
___
Python-bugs-li
[ python-Bugs-1260171 ] subprocess: more general (non-buffering) communication
Bugs item #1260171, was opened at 2005-08-15 14:15 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1260171&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Feature Request Status: Open Resolution: None Priority: 5 Submitted By: Ian Bicking (ianbicking) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess: more general (non-buffering) communication Initial Comment: Right now you can use subprocess.Popen.communicate() to make communication with subprocesses much easier and less likely to block than communicating directly with .stdin, .stdout, etc. However, that requires completely buffering the input and output. The functionality of communicate() (which is somewhat complex because of platform issues) could be made more general fairly easily. The current functionality of communicate could then be implemented in terms of that new method. I attached a function I'm using which does that for the posix systems (basically turning Popen's posix communicate into a function with some modifications). Replace "proc" with "self" (and give the function a better name) and you'd have a method. If patch 1175984 was accepted, then this wouldn't be that much of an issue: http://sourceforge.net/tracker/index.php?func=detail&aid=1175984&group_id=5470&atid=305470 -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1260171&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
