[ python-Bugs-1622896 ] Exception when compressing certain data with bz2
Bugs item #1622896, was opened at 2006-12-27 05:26
Message generated for change (Comment added) made by nnorwitz
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1622896&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
>Category: Extension Modules
Group: Python 2.5
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Alex Gontmakher (gsasha)
Assigned to: Nobody/Anonymous (nobody)
Summary: Exception when compressing certain data with bz2
Initial Comment:
Looks like an out-of-bounds array access... might be a security problem.
The attached file includes a script which, when executed, tries to pack the two
given directories with a bz2 compressor. On my machine (stock 32 bit Ubuntu
Edgy), the program fails with the following exception:
Traceback (most recent call last):
File "test.py", line 13, in ?
block = compressor.compress(open("compress-0067/"+file, "rb").read())
ValueError: the bz2 library has received wrong parameters
The problem occurs under either python2.4 or 2.5 (I don't have other versions
to test with).
Sorry, the file is large... I tried to reduce the example to smaller number of
files etc., but no such luck. The file is too large to be submitted here as an
attachment, so I have uploaded it to
http://www.cs.technion.ac.il/~gsasha/testcase.tar.bz2
--
>Comment By: Neal Norwitz (nnorwitz)
Date: 2006-12-28 00:27
Message:
Logged In: YES
user_id=33168
Originator: NO
I don't understand why you think this is an array out of bounds. I ran
your test case under valgrind and it reported no problems.
I can reproduce the problem. I have attached a patch that fixes the
problem for me. I am not certain it's the correct fix however. The unit
tests pass with this modification.
File Added: bz2.diff
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1622896&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1545837 ] array.array borks on deepcopy
Bugs item #1545837, was opened at 2006-08-24 11:49
Message generated for change (Comment added) made by twouters
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1545837&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.5
Status: Open
Resolution: Accepted
Priority: 9
Private: No
Submitted By: Václav Haisman (wilx)
>Assigned to: Thomas Wouters (twouters)
Summary: array.array borks on deepcopy
Initial Comment:
Hi,
I think there is a bug arraymodule.c this line:
{"__deepcopy__",(PyCFunction)array_copy,METH_NOARGS,
copy_doc},
should probably have METH_O instead of METH_NOARGS
there, since according to docs and the prototype of the
array_copy() function there is one parameter.
--
>Comment By: Thomas Wouters (twouters)
Date: 2006-12-28 10:11
Message:
Logged In: YES
user_id=34209
Originator: NO
The 2.5 branch at the time was still pre-2.5.0, and we didn't want to make
this change between release candidate and release. It's safe to be checked
into release25-maint now. I'll do it sometime tonight.
--
Comment By: Neal Norwitz (nnorwitz)
Date: 2006-12-28 08:59
Message:
Logged In: YES
user_id=33168
Originator: NO
Thomas, was there any reason this wasn't checked in to 2.5? I'm guessing
it was just forgotten. I don't see any mention in Misc/NEWS either.
--
Comment By: Thomas Wouters (twouters)
Date: 2006-08-29 09:35
Message:
Logged In: YES
user_id=34209
Not unless you want another release candidate. copy.deepcopy
has never worked on array instances, so it's not a
release-preventing bug (but each bugfix may *add* a
release-preventing bug by accident :)
--
Comment By: Raymond Hettinger (rhettinger)
Date: 2006-08-28 15:32
Message:
Logged In: YES
user_id=80475
Should this be fixed in the release candidate?
--
Comment By: Thomas Wouters (twouters)
Date: 2006-08-24 20:50
Message:
Logged In: YES
user_id=34209
Thanks! Fixed in the trunk (which is 2.6-to-be) revision
51565, and it will also be fixed for Python 2.4.4 and 2.5.1.
It's unfortunately just a bit too late for 2.5.0.
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1545837&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1622896 ] Exception when compressing certain data with bz2
Bugs item #1622896, was opened at 2006-12-27 15:26
Message generated for change (Comment added) made by gsasha
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1622896&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Extension Modules
Group: Python 2.5
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Alex Gontmakher (gsasha)
Assigned to: Nobody/Anonymous (nobody)
Summary: Exception when compressing certain data with bz2
Initial Comment:
Looks like an out-of-bounds array access... might be a security problem.
The attached file includes a script which, when executed, tries to pack the two
given directories with a bz2 compressor. On my machine (stock 32 bit Ubuntu
Edgy), the program fails with the following exception:
Traceback (most recent call last):
File "test.py", line 13, in ?
block = compressor.compress(open("compress-0067/"+file, "rb").read())
ValueError: the bz2 library has received wrong parameters
The problem occurs under either python2.4 or 2.5 (I don't have other versions
to test with).
Sorry, the file is large... I tried to reduce the example to smaller number of
files etc., but no such luck. The file is too large to be submitted here as an
attachment, so I have uploaded it to
http://www.cs.technion.ac.il/~gsasha/testcase.tar.bz2
--
>Comment By: Alex Gontmakher (gsasha)
Date: 2006-12-28 11:17
Message:
Logged In: YES
user_id=47707
Originator: YES
I saw that the effect occurs only when one compressor finishes and is
destroyed, and then the second one gives the "wrong parameters". Usually,
for such an effect to happen, the first compressor must have some permanent
effect on the system - and since it was destroyed before the second one was
created, it was only natural to assume that the reason was an out of bounds
access.
My approach is that such a bug should be suspected as an out-of-bounds
access until shown othervise, especially in a language like Python where it
is natural to assume that such problems shouldn't occur and that it is safe
to use security-wise. Of course I'll be happy to learn that it's just a
simple bug.
On a related note, will the fix be backported to all the relevant Python
versions? If you understand why the bug happens (I don't, sorry, have no
idea of internals of Python), you might be able to generate a compact test
case that would capture the erroneous behavior.
Best regards,
Alex
--
Comment By: Neal Norwitz (nnorwitz)
Date: 2006-12-28 10:27
Message:
Logged In: YES
user_id=33168
Originator: NO
I don't understand why you think this is an array out of bounds. I ran
your test case under valgrind and it reported no problems.
I can reproduce the problem. I have attached a patch that fixes the
problem for me. I am not certain it's the correct fix however. The unit
tests pass with this modification.
File Added: bz2.diff
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1622896&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1619659 ] htonl, ntohl don't handle negative longs
Bugs item #1619659, was opened at 2006-12-20 11:42 Message generated for change (Comment added) made by rhamphoryncus You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1619659&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Adam Olsen (rhamphoryncus) Assigned to: Nobody/Anonymous (nobody) Summary: htonl, ntohl don't handle negative longs Initial Comment: >>> htonl(-5) -67108865 >>> htonl(-5L) Traceback (most recent call last): File "", line 1, in ? OverflowError: can't convert negative value to unsigned long It works fine in 2.1 and 2.2, but fails in 2.3, 2.4, 2.5. htons, ntohs do not appear to have the bug, but I'm not 100% sure. -- >Comment By: Adam Olsen (rhamphoryncus) Date: 2006-12-28 14:37 Message: Logged In: YES user_id=12364 Originator: YES I forgot to mention it, but the only reason htonl should get passed a negative number is that it (and possibly struct?) produce a negative number. Changing them to always produce positive numbers may be an alternative solution. Or we may want to do both, always producing positive while also accepting negative. -- Comment By: Mark Roberts (mark-roberts) Date: 2006-12-26 02:24 Message: Logged In: YES user_id=1591633 Originator: NO >From man page for htonl and friends: #include uint32_t htonl(uint32_t hostlong); uint16_t htons(uint16_t hostshort); uint32_t ntohl(uint32_t netlong); uint16_t ntohs(uint16_t netshort); Python does call these underlying functions in Modules/socketmodule.c. The problem comes from that PyLong_AsUnsignedLong() called in socket_htonl() specifically checks to see that the value cannot be less than 0. The error checking was rather exquisite, I might add. - Mark -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1619659&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1623890 ] module docstring for subprocess is wrong
Bugs item #1623890, was opened at 2006-12-28 13:49 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1623890&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Neal Norwitz (nnorwitz) Assigned to: Nobody/Anonymous (nobody) Summary: module docstring for subprocess is wrong Initial Comment: The module docstring for subprocess is wrong. It says: communicate(input=None) Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate. The optional stdin argument should be a string to be sent to the child process, or None, if no data should be sent to the child. I'm not sure how to word the first stdin, but the second one should definitely be input, not stdin. Also need to verify the docstring for the communicate method. I'm guessing this affects Python 2.4 and later. Looking at 2.4.1? right now. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1623890&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1565525 ] gc allowing tracebacks to eat up memory
Bugs item #1565525, was opened at 2006-09-26 06:58 Message generated for change (Comment added) made by ghazel You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1565525&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Interpreter Core Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Greg Hazel (ghazel) Assigned to: Nobody/Anonymous (nobody) Summary: gc allowing tracebacks to eat up memory Initial Comment: Attached is a file which demonstrates an oddity about traceback objects and the gc. The observed behaviour is that when the tuple from sys.exc_info() is stored on an object which is inside the local scope, the object, thus exc_info tuple, are not collected even when both leave scope. If you run the test with "s.e = sys.exc_info()" commented out, the observed memory footprint of the process quickly approaches and sits at 5,677,056 bytes. Totally reasonable. If you uncomment that line, the memory footprint climbs to 283,316,224 bytes quite rapidly. That's a two order of magnitude difference! If you uncomment the "gc.collect()" line, the process still hits 148,910,080 bytes. This was observed in production code, where exc_info tuples are saved for re-raising later to get the stack- appending behaviour tracebacks and 'raise' perform. The example includes a large array to simulate application state. I assume this is bad behaviour occurring because the traceback object holds frames, and those frames hold a reference to the local objects, thus the exc_info tuple itself, thus causing a circular reference which involves the entire stack. Either the gc needs to be improved to prevent this from growing so wildly, or the traceback objects need to (optionally?) hold frames which do not have references or have weak references instead. -- >Comment By: Greg Hazel (ghazel) Date: 2006-12-29 01:29 Message: Logged In: YES user_id=731668 Originator: YES > If you are saying that it is unavoidable in your > application: I have difficulties believing this. For > example, you could do > > s.e = sys.exc_info()[:2] > > This would drop the traceback, and thus not create a cyclic > reference. Since, in the program you present, the traceback > is never used, this looks like a "legal" simplification (of > course, you don't use s.e at all in this program, so I can > only guess that you don't need the traceback in your real > application). "This was observed in production code, where exc_info tuples are saved for re-raising later to get the stack- appending behaviour tracebacks and 'raise' perform." I would like the traceback object so that I can re-raise the error. I can stringify it as tim_one suggests, but that can't be used with 'raise' and 'try','except' later. It is not important for my application to have all the references that the traceback object contains, which is what is causing the massive memory requirement. If I could replace the exc_info()[2] with a traceback look-alike that only held file, line, etc information for printing a standard traceback, that would solve this problem. -- Comment By: A.M. Kuchling (akuchling) Date: 2006-10-26 20:48 Message: Logged In: YES user_id=11375 A quick grep of the stdlib turns up various uses of sys.exc_info that do put it in a local variable., e.g. doctest._exception_traceback, unittest._exc_info_to_string, SimpleXMLRPCServer._marshaled_dispatch. Do these all need to be fixed? -- Comment By: Tim Peters (tim_one) Date: 2006-09-28 03:48 Message: Logged In: YES user_id=31435 [Martin] > tim_one: Why do you think your proposed modification of > introducing get_traceback would help? The frame foo still > refers to s (which is an O), and s.e will still refer > to the traceback that includes foo. Sorry about that! It was an illusion, of course. I wanted to suggest a quick fix, and "tested it" too hastily in a program that didn't actually bloat with /or/ without it. For the OP, I had need last year of capturing a traceback and (possibly) displaying it later in ZODB. It never would have occurred to me to try saving away exc_info(), though. Instead I used the `traceback` module to capture the traceback output (a string), which was (possibly) displayed later, with annotations, by a different thread. No cycles, no problems. BTW, I must repeat that there is no simple-minded way to 'repair' this. That isn't based on general principle, but on knowledge of how Python is implemented. -- Comment By: Martin v. Löwis (loewis) Date: 2006-09-28 03:03 Mess
[ python-Bugs-1623890 ] module docstring for subprocess is wrong
Bugs item #1623890, was opened at 2006-12-28 13:49 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1623890&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library >Group: Python 2.5 >Status: Closed >Resolution: Fixed Priority: 5 Private: No Submitted By: Neal Norwitz (nnorwitz) >Assigned to: Neal Norwitz (nnorwitz) Summary: module docstring for subprocess is wrong Initial Comment: The module docstring for subprocess is wrong. It says: communicate(input=None) Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate. The optional stdin argument should be a string to be sent to the child process, or None, if no data should be sent to the child. I'm not sure how to word the first stdin, but the second one should definitely be input, not stdin. Also need to verify the docstring for the communicate method. I'm guessing this affects Python 2.4 and later. Looking at 2.4.1? right now. -- >Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-28 19:02 Message: Logged In: YES user_id=33168 Originator: YES Committed revision 53187. (2.5) Committed revision 53188. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1623890&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-583975 ] gethostbyaddr lag
Bugs item #583975, was opened at 2002-07-19 11:41
Message generated for change (Comment added) made by nnorwitz
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=583975&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: None
>Status: Closed
>Resolution: Works For Me
Priority: 5
Private: No
Submitted By: Jason R. Mastaler (jasonrm)
Assigned to: Nobody/Anonymous (nobody)
Summary: gethostbyaddr lag
Initial Comment:
For more info, also see
http://mail.python.org/pipermail/python-list/2002-July/113706.html
Perl's gethostbyaddr doesn't seem to have this problem
as shown
below. Should I report this in the bug tracker?
$ time perl -MSocket -lwe 'print +(gethostbyaddr
inet_aton("datavortex.net"), AF_INET)[0]'
datavortex.net
real0m0.063s
user0m0.050s
sys 0m0.010s
$ time python2 -c 'from socket import * ; print
gethostbyaddr("datavortex.net")[0]'
datavortex.net
real0m20.176s
user0m0.070s
sys 0m0.020s
--
>Comment By: Neal Norwitz (nnorwitz)
Date: 2006-12-28 23:38
Message:
Logged In: YES
user_id=33168
Originator: NO
Not fully diagnosed, no. I suspect (based on reading the comments here)
that the other programs didn't use the threaded version where python did.
That's the only reasonable explanation I can come up with. I'm closing
this since we could never reproduce. I don't recall ever seeing another
similar bug report.
--
Comment By: A.M. Kuchling (akuchling)
Date: 2006-12-22 05:11
Message:
Logged In: YES
user_id=11375
Originator: NO
Was the cause of this bug ever diagnosed? Should this report remain open,
or be closed?
--
Comment By: Jason R. Mastaler (jasonrm)
Date: 2003-07-24 20:03
Message:
Logged In: YES
user_id=85984
This problem has cropped up again for another
TMDA user, so we still have this bug in Python.
In short, just as with the previous report, this
user sees a 20 second delay with
socket.gethostbyaddr() on a hostname which
all other tools (nslookup, host, mail progs, etc)
look up instantaneously.
In other words, Python is the only app on his
system with this problem. I had him try
Python 2.3c1 to no avail.
As with the previous user, he is running Linux
(Redhat):
Linux sparge 2.4.20-18.7 #1 Thu May 29 06:51:53 EDT 2003
i686 unknown
glibc-2.2.5-43
Any other information I should provide or any
diagnostics I can have the user run?
--
Comment By: Jason R. Mastaler (jasonrm)
Date: 2002-11-02 20:40
Message:
Logged In: YES
user_id=85984
The problem was under Python 2.2.
Though now the user reports that he can no longer
reproduce the problem, despite not having done any
Python upgrades or system configuration changes.
We might just have to chuck this one up to
FM (Funny Magic).
--
Comment By: Neal Norwitz (nnorwitz)
Date: 2002-11-02 13:14
Message:
Logged In: YES
user_id=33168
Jason, still with us? What version of python were you
having the problem with? 2.1.x? 2.2.x? Do you have the
problem with 2.2? Have you looked at patch #604210
(http://python.org/604210)? Can you test that?
--
Comment By: Jason R. Mastaler (jasonrm)
Date: 2002-09-06 17:20
Message:
Logged In: YES
user_id=85984
Still having the problem, but I'm unsure how to
proceed.
nslookup and host both return instantly when querying
datavortex.net.
Only Python seems to exhibit this problem, but it
still could be a system misconfiguration. This is the
only time I've ever seen/heard of this behavior.
--
Comment By: Neal Norwitz (nnorwitz)
Date: 2002-09-06 15:23
Message:
Logged In: YES
user_id=33168
Jason, are you still having this problem? Do you have
anything to report?
--
Comment By: Neal Norwitz (nnorwitz)
Date: 2002-07-22 15:37
Message:
Logged In: YES
user_id=33168
Looking at the output, the problem is definitely in
gethostbyaddr_r().
This is what python calls from Modules/socketmodule.c
socket_gethostbyaddr().
Notice: the first attempt to send to the first DNS server
"207.69.188.185" looks like it fails after 5 seconds. DNS
#2 is attempted "207.69.188.186", this send also takes 5
seconds, back to #1 ...
The poll is being done in gethostbyaddr_r() (ie, libc). If
you want to break in a debugger, gethost...() should be
around line 2216 in python 2.2. You can also put prints in.
As for why perl doesn't have the sam
