[issue9216] FIPS support for hashlib
Dave Malcolm added the comment: I've refreshed this patch against the latest version of the code in hg. In an attempt to make it easier to review, I've split it up into four (so far) thematic patches, which apply in sequence. -- Added file: http://bugs.python.org/file23172/0001-Rework-_hashlib-caching-moving-per-hash-cached-data-.patch ___ Python tracker <http://bugs.python.org/issue9216> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9216] FIPS support for hashlib
Changes by Dave Malcolm : Added file: http://bugs.python.org/file23173/0002-Add-error-handling-to-initialization-of-_hashlib.patch ___ Python tracker <http://bugs.python.org/issue9216> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9216] FIPS support for hashlib
Changes by Dave Malcolm : Added file: http://bugs.python.org/file23174/0003-Add-optional-usedforsecurity-argument-in-various-pla.patch ___ Python tracker <http://bugs.python.org/issue9216> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9216] FIPS support for hashlib
Changes by Dave Malcolm : Added file: http://bugs.python.org/file23175/0004-_hashlib-Add-selftest-for-FIPS-mode-and-usedforsecur.patch ___ Python tracker <http://bugs.python.org/issue9216> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9216] FIPS support for hashlib
Dave Malcolm added the comment: [and yes, I used git to generate the 4 patches; sorry ] -- ___ Python tracker <http://bugs.python.org/issue9216> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9216] FIPS support for hashlib
Dave Malcolm added the comment: The cumulative effect of the above patches (to _hashlib) are equivalent to what I've applied downstream to python 2 in RHEL 6.0 and Fedora 17 onwards, and python 3 in Fedora 17 onwards. In those environments I've additionally patched hashlib to only use _hashlib, rather than falling back on _md5 etc, since otherwise you get confusing error messages from hashlib.md5() when it defers to _md5 due to FIPS enforcement. In my downstream builds we can be sure of building against OpenSSL, but this other part of the patch seems less appropriate for upstream python, given that upstream python tries to be flexible in terms of its dependencies. Hope this makes sense. -- ___ Python tracker <http://bugs.python.org/issue9216> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12913] Add a debugging howto
Dave Malcolm added the comment: > >> - running Python in gdb > > This is somewhat orthogonal, but the devguide/gdb page doesn't say how to > > start running Python > > in gdb (it might be obvious to people used to use gdb, but it should still > > be mentioned). > If the devguide is updated, I will be content with just one line containing > one link in my howto. Currently we have this wiki page: > http://wiki.python.org/moin/DebuggingWithGdb I also remember a ubuntu wiki > page with more useful info but can’t find the bookmark. > > > FTR, I will use these resources: > https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/ > and http://www.jeetworks.org/node/99 There's also: http://docs.python.org/devguide/gdb.html -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue12913> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13390] Hunt memory allocations in addition to reference leaks
Changes by Dave Malcolm : -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue13390> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13534] test_cmath fails on ppc with glibc-2.14.90 due to optimized architecture-specific "hypot"
New submission from Dave Malcolm :
On ppc64, on this box, with glibc-2.14.90-19.ppc64, test_cmath fails with:
==
FAIL: test_specific_values (test.test_cmath.CMathTests)
--
Traceback (most recent call last):
File "/builddir/build/BUILD/Python-2.7.2/Lib/test/test_cmath.py", line 352,
in test_specific_values
msg=error_message)
File "/builddir/build/BUILD/Python-2.7.2/Lib/test/test_cmath.py", line 112,
in rAssertAlmostEqual
'{!r} and {!r} are not sufficiently close'.format(a, b))
AssertionError: acos0200: acos(complex(1.620686051868302e+308,
1.0308426226285283e+308))
Expected: complex(0.5665082609382622, -710.5420687424156)
Received: complex(0.5665082609382622, -inf)
Received value insufficiently close to expected value.
It turns out (after much debugging) that libm in this version of glibc has an
architecture-specific implementation of hypot(3) which returns inf in many of
the test cases in tests/cmath_testcases.txt, when the non-architecture-specific
implementation returns large finite values.
See downstream bug in Fedora's bug tracker:
https://bugzilla.redhat.com/show_bug.cgi?id=750811
for more details.
Seen with the Python 2.7.2 tarball, but the code doesn't appear to have changed
in hg in either the default or the 2.7 branches.
I will also file a bug in glibc's bug tracker about this.
--
components: Tests
messages: 148893
nosy: dmalcolm
priority: normal
severity: normal
status: open
title: test_cmath fails on ppc with glibc-2.14.90 due to optimized
architecture-specific "hypot"
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4
___
Python tracker
<http://bugs.python.org/issue13534>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13534] test_cmath fails on ppc with glibc-2.14.90 due to optimized architecture-specific "hypot"
Dave Malcolm added the comment: Reported in glibc's bug tracker as: http://sourceware.org/bugzilla/show_bug.cgi?id=13472 -- ___ Python tracker <http://bugs.python.org/issue13534> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13534] test_cmath fails on ppc with glibc-2.14.90 due to buggy architecture-specific "hypot"
Dave Malcolm added the comment: The glibc bug has been fixed in that project's git repo: http://repo.or.cz/w/glibc.git/commit/850fb039cec802072f70ed9763927881bbbf639c -- title: test_cmath fails on ppc with glibc-2.14.90 due to optimized architecture-specific "hypot" -> test_cmath fails on ppc with glibc-2.14.90 due to buggy architecture-specific "hypot" ___ Python tracker <http://bugs.python.org/issue13534> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13534] test_cmath fails on ppc with glibc-2.14.90 due to buggy architecture-specific "hypot"
Dave Malcolm added the comment: I agree. I filed this here in case anyone else ran into it, but given that this is really a glibc bug (now fixed), I'm closing this out as "won't fix". -- resolution: -> wont fix status: open -> closed ___ Python tracker <http://bugs.python.org/issue13534> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Changes by Dave Malcolm : -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue13703> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: "handle out of range in select()"
Changes by Dave Malcolm : -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue10527> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: "handle out of range in select()"
Dave Malcolm added the comment: The analogous code within Modules/selectmodule.c uses #ifdef HAVE_POLL to guard the poll-using code, to support non-Windows platforms that don't have "poll". Presumably a patch for this should do the same. -- ___ Python tracker <http://bugs.python.org/issue10527> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: "handle out of range in select()"
Dave Malcolm added the comment: Also, I see that Modules/selectmodule.c has some painful-looking workarounds involving "HAVE_BROKEN_POLL", which presumably would also be applicable here. -- ___ Python tracker <http://bugs.python.org/issue10527> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: "handle out of range in select()"
Dave Malcolm added the comment: [for reference: issue 11743 covered Antoine's rewrite of the connection class to be pure python, for 3.3 (re msg138310)] -- ___ Python tracker <http://bugs.python.org/issue10527> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12509] test_gdb fails on debug build when builddir != srcdir
New submission from Dave Malcolm : test_gdb.py fails when builddir != srcdir: the regex tries to match lines like this: #0 builtin_id (self=, v=()) at ../Python/bltinmodule.c:919 but isn't expecting the "../" before the "Python/bltinmodule.c" 2.7 uses a different regexp, and I don't think it's affected by an analogous problem. -- assignee: dmalcolm components: None files: fix-test_gdb-regexp.patch keywords: patch messages: 139960 nosy: dmalcolm, haypo priority: normal severity: normal stage: patch review status: open title: test_gdb fails on debug build when builddir != srcdir versions: Python 3.1, Python 3.2 Added file: http://bugs.python.org/file22600/fix-test_gdb-regexp.patch ___ Python tracker <http://bugs.python.org/issue12509> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11254] distutils doesn't byte-compile .py files to __pycache__ during installation
Dave Malcolm added the comment: FWIW, I've filed a bug about this issue for Fedora 15's python3 package here: https://bugzilla.redhat.com/show_bug.cgi?id=722578 Looks like this has led to an extra .pyc in the old location for every Python 3 .py file in Fedora 15's various python3-* rpm packages (as generated by distutils). rpm's post-processing has been generating .pyc files in the (correct) __pycache__ location (via rpmbuild's brp-python-bytecompile script, which uses "compileall"). For examples, see that Fedora bug report; so far it's affected every built rpm I've looked at. I believe the impact of the extra .pyc files is merely wasted disk space within our rpm-packaged Python 3.2 stack. I'm thinking of applying the patch to our downstream python3.src.rpm -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue11254> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12605] Enhancements to gdb 7 debugging hooks
New submission from Dave Malcolm : I'm attaching patches to handle some more "events" in the gdb7 debugging hooks for CPython (aka Tools/gdb/libpython.py). Currently, the hooks only care about C frames that are the bytecode interpreter (i.e. PyEval_EvalFrameEx) This patch changes this, dividing C frames into: - "python frames": - "bytecode frames" i.e. PyEval_EvalFrameEx - "other python frames": C frames that are of interest from a python perspective, but aren't bytecode - everything else so that the "py-bt", "py-up" and "py-down" commands will now work on the other kinds of "python" frames, in addition to the bytecode frames. Specifically, the following new kinds of C frame are displayed: - waiting on the GIL - garbage-collection - CFunctions (these weren't displayed before) This should assist when debugging multithreaded crashes, to more easily get a sense of what every thread is doing. Examples: Showing a garbage-collection: (gdb) py-bt Garbage-collecting File "", line 4, in foo File "", line 6, in bar File "", line 7, in (gdb) py-bt-full #1 Garbage-collecting #2 #5 Frame 0x720d6d48, for file , line 4, in foo () #8 Frame 0x720d6b70, for file , line 6, in bar () #11 Frame 0x77fb1d80, for file , line 7, in () Showing an invocation of "time.sleep()": (gdb) py-bt-full #0 #3 Frame 0x718aa060, for file , line 3, in foo () #6 Frame 0x720d6b70, for file , line 5, in bar () #9 Frame 0x77fb1d80, for file , line 6, in () Showing multiple threads, where all but one are waiting for the GIL: Thread 5 (Thread 0x7fffeb5fe700 (LWP 10716)): Traceback (most recent call first): Waiting for the GIL File "", line 10, in run File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 737, in _bootstrap_inner self.run() File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 710, in _bootstrap self._bootstrap_inner() Thread 4 (Thread 0x7fffebfff700 (LWP 10715)): Traceback (most recent call first): Waiting for the GIL File "", line 10, in run File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 737, in _bootstrap_inner self.run() File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 710, in _bootstrap self._bootstrap_inner() Thread 3 (Thread 0x70dea700 (LWP 10714)): Traceback (most recent call first): Waiting for the GIL File "", line 10, in run File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 737, in _bootstrap_inner self.run() File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 710, in _bootstrap self._bootstrap_inner() Thread 2 (Thread 0x717eb700 (LWP 10713)): Traceback (most recent call first): Waiting for the GIL File "", line 10, in run File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 737, in _bootstrap_inner self.run() File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 710, in _bootstrap self._bootstrap_inner() Thread 1 (Thread 0x77fdb700 (LWP 10709)): Traceback (most recent call first): File "", line 18, in -- assignee: dmalcolm components: Demos and Tools keywords: needs review messages: 140828 nosy: dmalcolm, haypo priority: normal severity: normal status: open title: Enhancements to gdb 7 debugging hooks versions: Python 2.7, Python 3.2, Python 3.3 ___ Python tracker <http://bugs.python.org/issue12605> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12605] Enhancements to gdb 7 debugging hooks
Changes by Dave Malcolm : -- keywords: +patch Added file: http://bugs.python.org/file22712/more-frames-in-gdb-hooks-py3k.patch ___ Python tracker <http://bugs.python.org/issue12605> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12605] Enhancements to gdb 7 debugging hooks
Dave Malcolm added the comment: (On 2.7, I needed import_site=True to get the new tests to work from a fresh build: "import time" wasn't being found otherwise) -- Added file: http://bugs.python.org/file22713/more-frames-in-gdb-hooks-2.7.patch ___ Python tracker <http://bugs.python.org/issue12605> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12326] Linux 3: code should avoid using sys.platform == 'linux2'
Dave Malcolm added the comment: Another datapoint: For Fedora 16, I haven't done any downstream patching (so far), because we hadn't run into any downstream problems. I did some digging into why we're _not_ experiencing issues. Currently for Fedora 16, we're shipping kernel-3.0 with python-2.7.2-4.fc16.x86_64 and python is reporting: $ python -c"import sys; print(sys.platform)" linux2 I investigated why we have this discrepancy: "uname" with the build environment for that RPM happens to be reporting a kernel-2*, whereas we're shipping a kernel-3*: http://koji.fedoraproject.org/koji/taskinfo?taskID=3187563 What's happening here is that although the chroot that the build was done in [1] has: kernel-3.0-0.rc6.git0.1.fc16.x86_64.rpm running "uname" in the chroot environment is reporting the kernel that's actually running, outside the chroot, which was: 2.6.32 and thus we have: checking MACHDEP... linux2 within the build log [2] So in this case, "sys.platform"'s final digit is reporting the major release of the kernel running outside the chroot-ed build environment (ironically bearing even less relationship to that of the currently-running kernel :( ) Hope this is helpful [1] http://koji.fedoraproject.org/koji/rpmlist?buildrootID=1096117%20&start=50&order=nvr&type=component [2] http://kojipkgs.fedoraproject.org/packages/python/2.7.2/4.fc16/data/logs/x86_64/build.log -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue12326> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12326] Linux 3: code should avoid using sys.platform == 'linux2'
Dave Malcolm added the comment: Note that PyPy is also affected by this issue; see https://bugs.pypy.org/issue832 For CPython, I'm of the opinion that: - the final digit of sys.platform as-is for "linux*" is effectively meaningless - that no code should be relying on the final digit of sys.platform (thankfully this is now recommended by: http://docs.python.org/library/sys.html#sys.platform ) - unfortunately there is code out there that checks against "linux2" and thus does rely on this value - patching CPython to force this to read "linux2" may be necessary for some downstream distributors of Python, to maximize compatibility with such broken code. For CPython, "sys.platform" currently reports on the difference between whether "uname" reported linux2 or linux3 at build time, which is currently meaningless (see msg142219 above about our chroot-ed build environment). For example, in RHEL we may at some future time upgrade our build farm to linux 3, but still provision our build trees within it for linux 2; this may mean that our build farm starts reporting "linux3" when rebuilding security updates for python 2.2, 2.3, 2.4 or 2.6, even when building against kernel-2.*'s user-space. If this happens, I'd be inclined to patch those builds of Python back to "linux2". It would be entirely meaningless and only damaging for one of our security updates of, say, Python 2.2 to shift sys.platform from "linux2" to "linux3" simply because of the kernel that was running in the build environment (as opposed to the kernel headers exposed to the compiler, and other such aspects of the kernel exposed in user-space). FWIW, my opinion is currently: - in 3.3, sys.platform on linux should be simply "linux" - for 2.7 and 3.2, sys.platform should be forced to "linux2" (and indeed, I anticipate doing this for earlier releases that I still maintain downstream) - existing documentation should say that on linux, sys.platform begins with "linux", and that programmers should avoid relying upon the suffix - I don't see the need for more adding access to more details of the build platform (and I can poke holes in any such plan, if anyone wants me to: what would it contain? what about the case where the user-space files e.g. headers aren't the same as the uname? etc) -- ___ Python tracker <http://bugs.python.org/issue12326> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12872] --with-tsc crashes on ppc64
New submission from Dave Malcolm :
On 64-bit PPC builds configured --with-tsc, Python segfaults within the first
function call in the bytecode interpreter.
Upon investigation this is due to this code in Python/ceval.c:
32 typedef unsigned long long uint64;
33
34 /* PowerPC support.
35 "__ppc__" appears to be the preprocessor definition to detect on OS
X, whereas
36 "__powerpc__" appears to be the correct one for Linux with GCC
37 */
38 #if defined(__ppc__) || defined (__powerpc__)
39
40 #define READ_TIMESTAMP(var) ppc_getcounter(&var)
41
42 static void
43 ppc_getcounter(uint64 *v)
44 {
45 register unsigned long tbu, tb, tbu2;
46
47loop:
48 asm volatile ("mftbu %0" : "=r" (tbu) );
49 asm volatile ("mftb %0" : "=r" (tb) );
50 asm volatile ("mftbu %0" : "=r" (tbu2));
51 if (__builtin_expect(tbu != tbu2, 0)) goto loop;
52
53 /* The slightly peculiar way of writing the next lines is
54 compiled better by GCC than any other way I tried. */
55 ((long*)(v))[0] = tbu;
56 /*((long*)(v))[1] = tb; */ /* < this is the bug */
57 }
58
59 #elif defined(__i386__)
(gdb) p sizeof(long)
$44 = 8
(gdb) p sizeof(uint64)
$45 = 8
Looks like lines 55 and 56 are erroneously assuming that a long is 4 bytes on
this arch: line 56 above is trashing the next value beyond the timer value,
which is on the machine's stack, corrupting local variables within
PyEval_EvalFrameEx.
The code has been this way since ppc_getcounter was added, in:
http://hg.python.org/cpython/rev/f455bbe7ea7e
I may have broken this in:
http://hg.python.org/cpython/rev/419ca089d365/
which was for:
http://bugs.python.org/issue10655
by (perhaps) generalizing support from ppc to (ppc and ppc64) (not sure about
this).
I'm attaching a patch which splits the ppc tsc support into 32-bit and 64-bit
parts, and which also removes an aliasing violation on 32-bit ppc.
(I'm tracking this downstream in Fedora's bug tracker as
https://bugzilla.redhat.com/show_bug.cgi?id=698726 )
--
components: Interpreter Core
files: python-2.7.2-tsc-on-ppc.patch
keywords: patch
messages: 143281
nosy: dmalcolm
priority: normal
severity: normal
stage: patch review
status: open
title: --with-tsc crashes on ppc64
type: crash
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4
Added file: http://bugs.python.org/file23079/python-2.7.2-tsc-on-ppc.patch
___
Python tracker
<http://bugs.python.org/issue12872>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9623] test_site.py has a couple of stray self.assertTrue calls that test for equality
New submission from Dave Malcolm : test_site.py has a couple of assertions of the form self.assertTrue(len(foo), some number) which appear to be incorrect, and should read: self.assertEqual(len(foo), some number) or assertEquals (that file uses both methods). r76047 fixed some of these, but a couple remain (introduced in r74526) in both 2.7 branch and py3k. Patch attached (for 2.7 branch) -- components: Tests keywords: easy, needs review, patch messages: 114069 nosy: dmalcolm priority: normal severity: normal stage: patch review status: open title: test_site.py has a couple of stray self.assertTrue calls that test for equality type: behavior versions: Python 2.7, Python 3.2, Python 3.3 ___ Python tracker <http://bugs.python.org/issue9623> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9623] test_site.py has a couple of stray self.assertTrue calls that test for equality
Dave Malcolm added the comment: Hopefully actually attaching the patch this time I've used assertEqual in the lines I've touched. I haven't touched the other lines, in order to isolate the semantic fix from the stylistic one. -- Added file: http://bugs.python.org/file18549/fix-stray-assertTrue-calls.patch ___ Python tracker <http://bugs.python.org/issue9623> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9628] runtests.sh -x doesn't work with more than two args (sed error)
New submission from Dave Malcolm : runtests.sh -x fails to work with more than two tests; for example, running: $ ./runtests.sh -x test_httplib test_http_cookies test_dl erroneously runs test_dl By default, "sed -e s" only substitutes the first match - the invocations within runtests.sh need to add the trailing "g" flag to substitute all matches. >From "info sed": The `s' command can be followed by zero or more of the following FLAGS: `g' Apply the replacement to _all_ matches to the REGEXP, not just the first. Am attaching a patch. (Seen with sed-4.2.1 on Fedora 13) -- components: Tests files: fix-sed-invocations-in-runtests.sh.patch keywords: easy, needs review, patch, patch messages: 114134 nosy: dmalcolm priority: normal severity: normal stage: patch review status: open title: runtests.sh -x doesn't work with more than two args (sed error) versions: Python 3.1, Python 3.2, Python 3.3 Added file: http://bugs.python.org/file18554/fix-sed-invocations-in-runtests.sh.patch ___ Python tracker <http://bugs.python.org/issue9628> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9628] runtests.sh -x doesn't work with more than two args (sed error)
Dave Malcolm added the comment: FWIW it works with two args, but adding the third fails; echoing PAT indicates the generated regex is missing the "|" separator: ^(test_httplib|test_http_cookies test_dl)$ -- ___ Python tracker <http://bugs.python.org/issue9628> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9629] SIZEOF_SOCKET_T used in longobject.h but undefined
Dave Malcolm added the comment: Looks like a dup of issue 4835 -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue9629> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9629] SIZEOF_SOCKET_T used in longobject.h but undefined
Changes by Dave Malcolm : -- resolution: -> duplicate status: open -> closed superseder: -> SIZEOF_SOCKET_T not defined ___ Python tracker <http://bugs.python.org/issue9629> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9631] Python 2.7 installation issue for Linux gcc-4.1.0-3 (Fedora Core 5?)
Dave Malcolm added the comment: (BTW, the versioning seems slightly misleading: "(Red Hat 4.1.0-3)" refers to the version of GCC, not of the operating system. You appear to be running gcc-4.1.0-3, which I believe was shipped in Fedora Core 5. Updating "title" metadata of the bug accordingly) -- nosy: +dmalcolm title: Python 2.7 installation issue for Linux Red Hat 4.1 -> Python 2.7 installation issue for Linux gcc-4.1.0-3 (Fedora Core 5?) ___ Python tracker <http://bugs.python.org/issue9631> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9635] RFE(patch): add Py_BREAKPOINT and sys.breakpoint hooks
New submission from Dave Malcolm :
It's sometimes useful to be able to programatically inject a breakpoint when
debugging CPython.
For example, sometimes you want a conditional breakpoint, but the logic
involved is too complex to be expressed in the debugger (e.g. runtime
complexity of evaluating the conditional in the debugger process, or deficiency
of the debugger itself).
I'm attaching a patch which:
- adds a Py_BREAKPOINT macro to pyport.h This is available as a quick and
dirty way of hardcoding a breakpoint in code (e.g. in extension modules); so
that when you need to you can put of these in (perhaps guarded by C-level
conditionals):
if (complex_conditional()) {
Py_BREAKPOINT();
}
- when Py_BREAKPOINT is defined, adds a sys.breakpoint() method. This means
that you can add C-level breakpoints to Python scripts, perhaps guarded by
python-level conditionals:
if foo and bar and not baz:
sys.breakpoint()
Naturally this is highly system-dependent. Only tested on Linux (Fedora 13
x86_64 with gcc-4.4.4). The Windows implementation was copied from
http://bugs.python.org/issue8863 but I don't have a Windows box to test it on.
I note that the GLib library within GNOME has a similar idea with a
G_BREAKPOINT macro, which has accumulated a collection of platform-specific
logic:
http://git.gnome.org/browse/glib/tree/glib/gbacktrace.h
(unfortunately that's LGPLv2+ licensed)
Doesn't yet have a unit test.
Note that when running on Linux when _not_ under a debugger, the default for
SIGTRAP is to get a coredump:
Trace/breakpoint trap (core dumped)
so people should be strongly discouraged from adding these calls to their code.
--
components: Interpreter Core, Library (Lib)
files: add-sys.breakpoint.patch
keywords: patch, patch
messages: 114301
nosy: dmalcolm, haypo
priority: normal
severity: normal
stage: patch review
status: open
title: RFE(patch): add Py_BREAKPOINT and sys.breakpoint hooks
type: feature request
versions: Python 3.2, Python 3.3
Added file: http://bugs.python.org/file18572/add-sys.breakpoint.patch
___
Python tracker
<http://bugs.python.org/issue9635>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9635] RFE(patch): add Py_BREAKPOINT and sys.breakpoint hooks
Dave Malcolm added the comment: > Note that when running on Linux when _not_ under a debugger, the > default for SIGTRAP is to get a coredump: > Trace/breakpoint trap (core dumped) > so people should be strongly discouraged from adding these calls to > their code. Looks like Windows' DebugBreak has similar behavior here; according to: http://msdn.microsoft.com/en-us/library/ms679297(VS.85).aspx "If the process is not being debugged, the function uses the search logic of a standard exception handler. In most cases, this causes the calling process to terminate because of an unhandled breakpoint exception." -- ___ Python tracker <http://bugs.python.org/issue9635> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9687] dbmmodule.c:dbm_contains fails on 64bit big-endian (test_dbm.py fails) when built against gdbm (int vs Py_ssize_t)
New submission from Dave Malcolm :
With a clean build of release27-maint (r84317), test_dbm.py fails on ppc64 with
this error:
File "test_dbm.py", line 24, in test_keys
self.assert_(k in self.d)
AssertionError
I'm building gainst gdbm-1.8.0 (specifically, on a prerelease of RHEL6, with
gdbm-devel-1.8.0-36.el6.ppc64)
All of the headers define datum as:
typedef struct {
char *dptr;
int dsize;
} datum;
Note the use of "int" for dsize.
This fragment of code in python's Modules/dbmmodule.c:dbm_contains:
if (PyString_AsStringAndSize(v, (char **)&key.dptr,
(Py_ssize_t *)&key.dsize)) {
return -1;
}
appears to assume that
sizeof(datum.dsize) == sizeof(Py_ssize_t)
which is not correct on these architectures:
(gdb) p sizeof(key.dsize)
$25 = 4
(gdb) p sizeof(Py_ssize_t)
$26 = 8
On ppc64, when PyString_AsStringAndSize writes the 0x1 value
for the ob_size of "a" to &key.dsize, I believe the 0x part is written
to &key.size, and the 0x0001 part is written to the 4 bytes following it,
due to the incorrect cast from (int*) to (Py_ssize_t*)
Thankfully
(gdb) p sizeof(key)
$28 = 16
so it writes this value to padding within the "datum key", rather than
corrupting the stack.
The dbm_fetch() invocation is thus passed a 0 dsize, and doesn't find the key,
hence the test fails.
The various other uses with that source file appear correct:
(i) there are various PyArg_Parse* calls using s#, with int, which is correct,
given the absence of the PY_SSIZE_T_CLEAN macro.
(ii) there are various calls of PyString_FromStringAndSize(, datum.dsize),
which I believe is correct: I believe the compiler will coerce this int to the
wider Py_ssize_t type.
I'm attaching a patch which (I hope) correctly coerces the size of the key from
Py_ssize_t to "int" within gdb_contains.
--
components: Extension Modules
files: fix-dbm_contains-on-64bit-bigendian.patch
keywords: patch, patch
messages: 114934
nosy: dmalcolm
priority: normal
severity: normal
stage: patch review
status: open
title: dbmmodule.c:dbm_contains fails on 64bit big-endian (test_dbm.py fails)
when built against gdbm (int vs Py_ssize_t)
type: behavior
versions: Python 2.7, Python 3.1, Python 3.2
Added file:
http://bugs.python.org/file18644/fix-dbm_contains-on-64bit-bigendian.patch
___
Python tracker
<http://bugs.python.org/issue9687>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9687] dbmmodule.c:dbm_contains fails on 64bit big-endian (test_dbm.py fails) when built against gdbm (int vs Py_ssize_t)
Dave Malcolm added the comment: Note to self: I'm tracking this one in RH's downstream tracker as: https://bugzilla.redhat.com/show_bug.cgi?id=626756 -- ___ Python tracker <http://bugs.python.org/issue9687> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9688] object.__basicsize__ is erroneously0
Changes by Dave Malcolm : -- nosy: dmalcolm priority: normal severity: normal status: open title: object.__basicsize__ is erroneously0 ___ Python tracker <http://bugs.python.org/issue9688> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9688] object.__basicsize__ is erroneously 0 on big-endian 64-bit machines (int vs Py_ssize_t)
New submission from Dave Malcolm :
On 64-bit bigendian machines (ppc64 and s390x), I'm seeing:
>>> print object.__basicsize__
0
(Discovered via a segfault in Jinja2 tries to use ctypes to manipulate
ob_refcnt of variables, and gets the wrong location, corrupting the objects
instead; see https://bugzilla.redhat.com/show_bug.cgi?id=627347 )
struct _typeobject declares tp_basicsize and tp_itemsize as Py_ssize_t:
{
...
Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */
...
}
but type_members defines them as T_INT:
{"__basicsize__", T_INT, offsetof(PyTypeObject,tp_basicsize),READONLY},
{"__itemsize__", T_INT, offsetof(PyTypeObject, tp_itemsize), READONLY},
Hence when accessing "object.__basicsize__", PyMember_GetOne reads it as a
T_INT, which gets it as 0 (incorrect). Accessing it as Py_ssize_t reads it as
16 (correct)
(gdb) p *(Py_ssize_t*)addr
$9 = 16
(gdb) p *(int*)addr
$10 = 0
I'm attaching a patch which changes them to use T_PYSSIZE_T and adds a selftest.
--
keywords: +patch
stage: -> patch review
title: object.__basicsize__ is erroneously0 -> object.__basicsize__ is
erroneously 0 on big-endian 64-bit machines (int vs Py_ssize_t)
type: -> behavior
versions: +Python 2.7
Added file: http://bugs.python.org/file18646/fix-typeobject-T_INT-usage.patch
___
Python tracker
<http://bugs.python.org/issue9688>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8998] add crypto routines to stdlib
Changes by Dave Malcolm : -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue8998> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9738] Document the encoding of functions bytes arguments of the C API
Dave Malcolm added the comment: > I think either of these is correct: > - a UTF-8-encoded string > - a string encoded in UTF-8 Possibly use the word "buffer" here, rather than "string", as "string" may suggest the "str" type. Or even: "NUL-terminated buffer of UTF-8-encoded bytes", or whatnot. (sorry for bikeshedding) -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue9738> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9188] test_gdb fails for UCS2 builds with UCS2 gdb
Dave Malcolm added the comment: Tested with UCS4 gdb: (gdb) python import sys; print hex(sys.maxunicode) 0x10 using latest py3k. Works with both UCS2 and UCS4 builds of py3k -- ___ Python tracker <http://bugs.python.org/issue9188> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9188] test_gdb fails for UCS2 builds with UCS2 gdb
Dave Malcolm added the comment: where "works" means: all tests in test_gdb.py were run, and passed. This was with a --with-pydebug-build in each case -- ___ Python tracker <http://bugs.python.org/issue9188> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9188] test_gdb fails for UCS2 builds with UCS2 gdb
Dave Malcolm added the comment: One minor quibble with the patch: In the line: while i < field_length: you're trusting that field_length has a sane value. If field_str points somewhere readable, and field_length is huge (e.g. 0xfff), then gdb could sit there for a while reading all that data until it either completes, or hits memory that it can't read (leading to a python exception within gdb). In other places in that code I used safe_range() to limit iterations to a safety threshold; I suggest the use of safety_limit(field_length) But this is a minor nit. I don't think this -- ___ Python tracker <http://bugs.python.org/issue9188> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9188] test_gdb fails for UCS2 builds with UCS2 gdb
Dave Malcolm added the comment: "I don't think this" was of course a browser misfire; I meant to type "I don't think this should block committing this fix" -- ___ Python tracker <http://bugs.python.org/issue9188> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9807] deriving configuration information for different builds with the same prefix
Dave Malcolm added the comment:
For reference, the patch that I'm currently applying to Fedora's build of
Python-3.2a1 can be seen at:
http://pkgs.fedoraproject.org/gitweb/?p=python3.git;a=blob_plain;f=python-3.2a1-debug-build.patch;hb=HEAD
It appears to be very similar to Matthias' patch (it was originally based on an
earlier version of Debian's python 2 patch, which I fixed up against Fedora's
python 2, changed some aspects I wasn't happy with, then ported to python 3.1,
then fixed up to 3.2a1 IIRC)
For further reference, Fedora's python3.spec has these comments that I wrote on
the patch:
# Patch to support building both optimized vs debug stacks DSO ABIs, sharing
# the same .py and .pyc files, using "_d.so" to signify a debug build of an
# extension module.
#
# Based on Debian's patch for the same,
#
http://patch-tracker.debian.org/patch/series/view/python2.6/2.6.5-2/debug-build.dpatch
#
# (which was itself based on the upstream Windows build), but with some
# changes:
#
# * Debian's patch to dynload_shlib.c looks for module_d.so, then module.so,
# but this can potentially find a module built against the wrong DSO ABI. We
# instead search for just module_d.so in a debug build
#
# * We remove this change from configure.in's build of the Makefile:
# SO=$DEBUG_EXT.so
# so that sysconfig.py:customize_compiler stays with shared_lib_extension='.so'
# on debug builds, so that UnixCCompiler.find_library_file can find system
# libraries (otherwise "make sharedlibs" fails to find system libraries,
# erroneously looking e.g. for "libffi_d.so" rather than "libffi.so")
#
# * We change Lib/distutils/command/build_ext.py:build_ext.get_ext_filename
# to add the _d there, when building an extension. This way, "make sharedlibs"
# can build ctypes, by finding the sysmtem libffi.so (rather than failing to
# find "libffi_d.so"), and builds the module as _ctypes_d.so
#
# * Similarly, update build_ext:get_libraries handling of Py_ENABLE_SHARED by
# appending "_d" to the python library's name for the debug configuration
#
# * We modify Modules/makesetup to add the "_d" to the generated Makefile
# rules for the various Modules/*.so targets
#
# This may introduce issues when building an extension that links directly
# against another extension (e.g. users of NumPy?), but seems more robust when
# searching for external libraries
#
# * We don't change Lib/distutils/command/build.py: build.build_purelib to
# embed plat_specifier, leaving it as is, as pure python builds should be
# unaffected by these differences (we'll be sharing the .py and .pyc files)
#
# * We introduce DEBUG_SUFFIX as well as DEBUG_EXT:
# - DEBUG_EXT is used by ELF files (names and SONAMEs); it will be "_d" for
# a debug build
# - DEBUG_SUFFIX is used by filesystem paths; it will be "-debug" for a
# debug build
#
# Both will be empty in an optimized build. "_d" contains characters that
# are valid ELF metadata, but this leads to various ugly filesystem paths (such
# as the include path), and DEBUG_SUFFIX allows these paths to have more natural
# names. Changing this requires changes elsewhere in the distutils code.
#
# * We add DEBUG_SUFFIX to PYTHON in the Makefile, so that the two
# configurations build parallel-installable binaries with different names
# ("python-debug" vs "python").
#
# * Similarly, we add DEBUG_SUFFIX within python-config and
# python$(VERSION)-config, so that the two configuration get different paths
# for these.
#
# * Patch runtests.sh to support supplying a value for PYTHON, so that we can
# run the tests against each of the builds
--
___
Python tracker
<http://bugs.python.org/issue9807>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9807] deriving configuration information for different builds with the same prefix
Dave Malcolm added the comment: In reply to Barry A. Warsaw: >@dmalcolm: I suspect you can reduce your diff for Python 3.2 now that PEP 3149 >has landed. Yeah, the patch I linked to is against the 3.2a1 tarball; I hoped to regenerate it for 3.2a2 but have been swamped this week :( -- ___ Python tracker <http://bugs.python.org/issue9807> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8998] add crypto routines to stdlib
Dave Malcolm added the comment: On Fri, 2010-09-17 at 23:11 +, Antoine Pitrou wrote: > Antoine Pitrou added the comment: > > > How about nss? As a bonus, this would also avoid making more work for > > Fedora (<http://fedoraproject.org/wiki/FedoraCryptoConsolidation>). > > Well, similar question: what will it bring and who will do the work? :) > (Fedora perhaps?) Possibly me - if you'll take my patches :) -- ___ Python tracker <http://bugs.python.org/issue8998> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8998] add crypto routines to stdlib
Dave Malcolm added the comment: I should note that I can't touch anything to do with Elliptic Curve crypto. I don't know if I can comment on the reasons for that. -- ___ Python tracker <http://bugs.python.org/issue8998> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8998] add crypto routines to stdlib
Dave Malcolm added the comment: FWIW, one of my RH colleagues (John Dennis) has written a set of Python bindings for NSS: http://fedoraproject.org/wiki/Features/PythonNSS (Though that seems to me to be a slightly different thing to a general-purpose crypto lib that happens to be written using NSS as an implementation detail) -- ___ Python tracker <http://bugs.python.org/issue8998> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6706] asyncore's accept() is broken
Dave Malcolm added the comment: giampaolo: did you ever rewrite the patch? For reference to other users: http://code.google.com/p/pyftpdlib/source/browse/trunk/pyftpdlib/ftpserver.py Note the complexity of the two handle_accept implementations in that file; both of them begin: try: sock, addr = self.accept() except TypeError: # sometimes accept() might return None (see issue 91) return except socket.error, err: # ECONNABORTED might be thrown on *BSD (see issue 105) if err[0] != errno.ECONNABORTED: logerror(traceback.format_exc()) return else: # sometimes addr == None instead of (ip, port) (see issue 104) if addr == None: return -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue6706> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9960] test_structmembers fails on s390x (bigendian 64-bit): int/Py_ssize_t issue
New submission from Dave Malcolm : test test_structmembers crashed -- : string too long Traceback (most recent call last): File "/builddir/build/BUILD/Python-2.7/Lib/test/regrtest.py", line 863, in runtest_inner the_package = __import__(abstest, globals(), locals(), []) File "/builddir/build/BUILD/Python-2.7/Lib/test/test_structmembers.py", line 12, in 9.9, 10.1010101010, "hi") ValueError: string too long _testcapimodule.c: test_structmembers_new's fmt has: "s#" and these args: &s, &string_len for grabbing this data: const char *s = NULL; Py_ssize_t string_len = 0; However, the module doesn't define PY_SSIZE_T_CLEAN, which leads to &string_len being treated as an (int*) rather than a (Py_ssize_t*) and thus written to with just the first 32-bits of the size, rather than the full size. The PyArgs_ call without PY_SSIZE_T_CLEAN writes the size of the string (2) through (int*)&string_len, writing 0x0002 to the first 4 bytes, setting the 64-bit "string_len" value to 0x0002 i.e. 2^34, wildly too large for the iirc 5 byte buffer. Confirmed in gdb: (gdb) p string_len $2 = 8589934592 (gdb) p /x string_len $3 = 0x2 Am attaching a patch (against 2.7 maintenance branch) which defines PY_SSIZE_T_CLEAN; doing so requires updating another int to be a Py_ssize_t in that module, within test_u_code. http://docs.python.org/c-api/arg.html lists "u# (Unicode) [Py_UNICODE *, int]" and has no reference to the effect of PY_SSIZE_T_CLEAN on the "u#" format specifier. My reading of Python/getargs.c is that this macro does affect "u#" in a manner analogous to "s#"; does this documentation need clarifying? -- components: Tests files: python-test_structmembers.patch keywords: easy, patch messages: 117448 nosy: dmalcolm priority: normal severity: normal stage: patch review status: open title: test_structmembers fails on s390x (bigendian 64-bit): int/Py_ssize_t issue type: behavior versions: Python 2.7 Added file: http://bugs.python.org/file19029/python-test_structmembers.patch ___ Python tracker <http://bugs.python.org/issue9960> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4111] Add Systemtap/DTrace probes
Dave Malcolm added the comment: Updated py3k version of patch. Changes: - renamed the probepoints: "function__entry" -> "frame__entry" "function__return" -> "frame__exit" as I believe this better describes what these do - added a test suite: Lib/test/test_systemtap.py - reformatted to use 4-space indentation - slightly reworked the configure.in code Some error-handling issue remain (see the FIXMEs in the code) Tested and working on Fedora 13 x86_64, using: systemtap-1.3-1.fc13.x86_64 kernel-2.6.33.3-84.fc13.x86_64 kernel-devel-2.6.33.3-84.fc13.x86_64 kernel-headers-2.6.33.3-84.fc13.x86_64 kernel-debuginfo-2.6.33.3-84.fc13.x86_64 kernel-debuginfo-common-x86_64-2.6.33.3-84.fc13.x86_64 -- Added file: http://bugs.python.org/file19037/py3k-add-systemtap-2010-09-27.patch ___ Python tracker <http://bugs.python.org/issue4111> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9675] segfault: PyDict_SetItem: Assertion `value' failed.
Changes by Dave Malcolm : -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue9675> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9518] PyModuleDef_HEAD_INIT does not explicitly initialize all fields of m_base
Dave Malcolm added the comment: Thanks. The code in question is a wrapper to a security-sensitive library (user-space SELinux code), hence the compilation warnings have been turned up as much as possible. The .c code in question is generated by SWIG, and that does indeed appear to be writing out full initializers for PyTypeObject instances (and the other associated structs). It appears to be just Python 3's PyModuleDef_HEAD_INIT macro that leaves fields uninitialized (hence this patch). The gory details of the SWIG-generated code can be seen at: http://userspace.selinuxproject.org/trac/browser/libselinux/src/selinuxswig_wrap.c (and the .i files in that directory) Although it's not on by default gcc will issue a "missing initializer" warning when fields aren't initialized when "-Wmissing-field-initializers" is enabled (in this case, due to the use of "-W"). This becomes an error with -Werror. Whether or not this is a useful warning isn't clear to me, but it seems to be reasonable to suppress the warning given that as-is, people who use gcc's "-W" catch-all will run into this on all Python 3 modules, and that the patch is trivial, and that this case gives no warnings when building such code against Python 2.* Hope this makes sense. -- ___ Python tracker <http://bugs.python.org/issue9518> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8863] Display Python backtrace on SIGSEGV, SIGFPE and fatal error
Dave Malcolm added the comment: It looks like this doesn't yet have any test cases. You probably should invoke a child python process that crashes and examine the output (perhaps running some/all of the examples in Lib/test/crashers ?); you may want to "steal" some of the wrapper code from Lib/test/test_gdb.py to do this. Test ideas: - generate a segfault, verify that the output is sane - generate a FPE (likewise) - perhaps run all of the crashers, and ensure that something sane happens (e.g. stack overflow when the limit is set to something that's beyond what the OS/CPU can cope with). Also, please test the interaction of this with the debugger (with gdb, at any rate): as I see it, this ought to gracefully get out of the way if you're running python under a debugger. See Lib/test/test_gdb.py for more examples of how to detect gdb, and invoke it in batch mode from a test suite. -- ___ Python tracker <http://bugs.python.org/issue8863> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8863] Display Python backtrace on SIGSEGV, SIGFPE and fatal error
Dave Malcolm added the comment: One other concern: many OSes (e.g. Linux distributions) implement some kind of system-wide crash-catching utility; for example in Fedora we have ABRT ( https://fedorahosted.org/abrt/wiki ). I'm not sure yet exactly how these are implemented, but we'd want to avoid breaking them: segfaults of a system-provided /usr/bin/python (or /usr/bin/python3 ) should continue to be detected by such tools. -- ___ Python tracker <http://bugs.python.org/issue8863> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4111] Add Systemtap/DTrace probes
Dave Malcolm added the comment: Updated version of patch attached (against py3k; r85426) I've changed the names back to "function__entry" and "function__return". -- ___ Python tracker <http://bugs.python.org/issue4111> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8863] Display Python backtrace on SIGSEGV, SIGFPE and fatal error
Dave Malcolm added the comment: > About SIGBUS: I don't know this signal. Is it used on Linux? If not, on > which OS is it used? Yes, IIRC typically you'll only see it on RISC CPUs that require instructions to be word-aligned; you typically see it if the program counter jumps to a broken address; the SIGBUS happens when the CPU tries to fetch the code. -- ___ Python tracker <http://bugs.python.org/issue8863> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7425] [PATCH] Improve the robustness of "pydoc -k" in the face of broken modules
Dave Malcolm added the comment: I notice this issue is in stage "unit test needed". It's not clear to me how to add a unit test for this: one idea I had is to create a broken module in some subdir somewhere, and invoke "pydoc -k" as a subprocess with PYTHONPATH containing the extra subdir (and assert no traceback seen in stderr of the child). Does that sound sane? -- ___ Python tracker <http://bugs.python.org/issue7425> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9807] deriving configuration information for different builds with the same prefix
Dave Malcolm added the comment: Summarizing IRC discussion: Tested on Fedora 13 x86_64 with: --enable-shared --with-wide-unicode and with confdir != srcdir with: ../configure --enable-shared --with-wide-unicode --with-pydebug Mostly working but, test_distutils fails: test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase) ... /usr/bin/ld: cannot find -lpython3.2 collect2: ld returned 1 exit status Each build makes a "libpython3.2.a" -- ___ Python tracker <http://bugs.python.org/issue9807> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10140] Tools/scripts/pathfix.py: add option to preserve timestamps
New submission from Dave Malcolm : I'm attempting to fix up the build in my Fedora/RHEL packages of Python to preserve timestamps of .py files as far as possible during the build, so that .pyc/.pyo files end up with predictable embedded mtime values and thus predictable hashes. Am attaching a patch to the py3k branch which adds a "-p" option to the Tools/script/pathfix.py script, which makes it preserve the mtime of the input files. -- components: Build files: pathfix-preserve-timestamps.patch keywords: needs review, patch messages: 119092 nosy: dmalcolm priority: normal severity: normal stage: patch review status: open title: Tools/scripts/pathfix.py: add option to preserve timestamps type: feature request versions: Python 3.2 Added file: http://bugs.python.org/file19276/pathfix-preserve-timestamps.patch ___ Python tracker <http://bugs.python.org/issue10140> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4111] Add Systemtap/DTrace probes
Dave Malcolm added the comment: Updated version of the patch; this ought to contain Include/pydtrace.d: $ diffstat /tmp/py3k-add-systemtap-2010-10-25.patch Include/pydtrace.d | 10 Lib/test/test_systemtap.py | 89 ++ Makefile.pre.in| 10 Python/ceval.c | 75 + configure | 576 +++-- configure.in | 34 ++ pyconfig.h.in |3 7 files changed, 522 insertions(+), 275 deletions(-) Patch contains FIXMEs (sorry), which clearly need addressing. As for the objectives, do folks here agree with the "Performance" notes in http://bugs.python.org/issue4111#msg100173 ? -- Added file: http://bugs.python.org/file19358/py3k-add-systemtap-2010-10-25.patch ___ Python tracker <http://bugs.python.org/issue4111> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4111] Add Systemtap/DTrace probes
Dave Malcolm added the comment: Updated patch, removing the FIXMEs, and slightly reworking the test code. I've wrapped the whole of get_frame_marker_info with a PyErr_Fetch/PyErr_Restore pair: the PyUnicode_AsUTF8String calls could fail with a MemoryError, and we don't want to confuse the regular exception handling within ceval. I'm not sure how to write a unit test to test for this: perhaps we could corrupt the __code__ instance so that co_filename is not a PyUnicodeObject, leading to a TypeError within the function, but that's a readonly attribute. Any ideas? I've also added a unit test for a non-ASCII script (Lib/test/systemtap_sample_☠.py), containing a non-ASCII identifier (文字化け). The non-ASCII script name (Lib/test/systemtap_sample_☠.py) may be controversial: do we have anything like that in the source tree yet? Is there any risk of messing up the build across platforms, or of impacting the Hg migration? Still to-do: - address the "Undefined symbol" issues seen by jcea. - documentation - doublecheck performance -- Added file: http://bugs.python.org/file19360/py3k-add-systemtap-2010-10-25-002.patch ___ Python tracker <http://bugs.python.org/issue4111> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4111] Add Systemtap/DTrace probes
Dave Malcolm added the comment: > It would be better to generate the sample dynamically, so that users > with an incompatible locale or filesystem aren't prevented from checking > out the source. Thanks: am attaching updated patch: I've removed Lib/test/systemtap_sample_☠.py, and now generate a similarly-named file during the test, using test.support.TEST_FN and unlink Still TODO: - address pitrou's concerns about PyUnicode_AsUTF8String from msg119580 - address the "Undefined symbol" issues seen by jcea (msg119563 onwards) - documentation - doublecheck performance - perhaps add a systemtap "tapset", and demo code using it (like I did in Fedora's python3 RPMs) - anything else I've missed :) -- Added file: http://bugs.python.org/file19363/py3k-add-systemtap-2010-10-25-003.patch ___ Python tracker <http://bugs.python.org/issue4111> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4111] Add Systemtap/DTrace probes
Dave Malcolm added the comment: I should note that I've only ever been testing this with SystemTap, on Linux. I don't have a box with DTrace, and I've never directly used it. I wouldn't today be able to diagnose a buildbot failure related to DTrace (I believe I would with systemtap, fwiw). Are there any DTrace experts around who would be willing to handle the DTrace side of this? If not, would it be reasonable to make this issue be only explicitly about "systemtap"? (e.g. perhaps change the "configure" argument accordingly?) Alternatively, given that this bug originally started as an RFE about DTrace, should we split out systemtap as a separate RFE? I'm sorry if I've "muddied the waters" by doing this. For example, the only test coverage I've written (test_systemtap.py) checks for the presence of a "stap" executable, and skips the tests if it's not found. I'm not familiar enough with DTrace to create the same for DTrace. FWIW (in response to IRC question): "thread_indent" is documented here: http://sourceware.org/systemtap/SystemTap_Beginners_Guide/systemtapscript-handler.html#thread_indent It looks like it may be systemtap-specific; however the only usage is within test_systemtap.py, guarded by the presence of a "stap" binary, skipping the tests if it is not found. -- ___ Python tracker <http://bugs.python.org/issue4111> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10195] Memory allocation fault-injection?
New submission from Dave Malcolm : We were chatting on #python-dev on possible ways of testing the correct handling of "MemoryError". Attached is one idea: adding a sys._inject_malloc_failure() hook, letting you inject a memory-allocation (or reallocation) failure some number of allocations in the future: >>> import sys [52733 refs] >>> 2 + 2 4 [52733 refs] >>> sys._inject_malloc_failure(50) [52733 refs] >>> 2 + 2 MemoryError [52747 refs] >>> 2 + 2 4 [52747 refs] I'm not sure how to make this useful; perhaps it could instead compare with the "serialno" in Objects/obmalloc.c so that you could set up a specific numbered allocation and have it fail (or perhaps a range of serial numbers, and expose "serialno" within the "sys" module?) Another idea might be to randomly have some proportion of allocations fail. Perhaps the test suite could have an option where it runs each set of tests in a separate subprocess, and sets a command-line switch to inject "random" malloc failures? (storing the seed, so that they're reproducible, on one machine at least). -- components: Tests files: py3k-inject-malloc-failure.txt messages: 119586 nosy: dmalcolm, pitrou priority: normal severity: normal status: open title: Memory allocation fault-injection? type: feature request versions: Python 3.2 Added file: http://bugs.python.org/file19365/py3k-inject-malloc-failure.txt ___ Python tracker <http://bugs.python.org/issue10195> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10195] Memory allocation fault-injection?
Dave Malcolm added the comment:
Attached is a new approach to doing this, based on "Out-Of-Memory Testing"
within http://sqlite.org/testing.html
This reads environment variables, and injects a fault at the given value of
"serialno", and (optionally) ongoing failures afterwards.
I used this to find a specific bug in Python/pythonrun.c (fix is the first hunk
of the patch): if moduleName is NULL, then Py_DECREF will read through NULL.
Potentially this gives the low-level machinery to support adding SQLite's
approach to be added to regrtest. Doing so would imply running each test many
tens of thousands of times, so perhaps we could run "-c pass" to establish at
what serialno the interpreter has fully started up, then use that as a starting
point when testing other scripts/modules.
I implemented a toy version of this as Lib/test/test_malloc_fault.py, which
sits in an infinite loop injecting individual allocation failures when starting
up sys.executable as a subprocess.
For low numbers, this throws up segfaults within _Py_ReadyTypes' call to
PyType_Ready(&PyType_Type), where PyExc_MemoryError is set but has not yet been
initialized (its ob_type is NULL):
/* this will probably fail since there's no memory and hee,
hee, we have to instantiate this class
*/
Running this interactively with a large value for PYTHONMALLOCINJECTFAULTSAT
leads to an interesting failure mode within PyRun_InteractiveLoopFlags(): every
call to PyRun_InteractiveOneFlags immediately returns (the
PyUnicode_FromString("stdin") in within PySys_GetObject() fails to allocate
memory); this leads to a tight loop sending the total refcount to stderr:
[52812 refs]
[52812 refs]
[52812 refs]
(etc)
--
Added file: http://bugs.python.org/file19372/py3k-inject-malloc-failure-002.txt
___
Python tracker
<http://bugs.python.org/issue10195>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9942] Allow memory sections to be OS MERGEABLE
Dave Malcolm added the comment: One possible use for this: mark the "str" buffers of PyUnicodeObject instances when demarshalling docstrings from disk; in theory these ought not to change, and can be quite large: the bulk of the memory overhead is stored in a separate allocation from the object, and thus isn't subjected to the ob_refcnt twiddling. No idea if it's worth it though; the syscall overhead might slow down module import; also, KSM works at the level of 4K pages, and it's not clear that the allocations would line up nicely with pages. FWIW, various related ideas here: http://dmalcolm.livejournal.com/4183.html Again, no idea if these are worthwhile, this was a brainstorm on my blog, and some of the ideas would involve major surgery to CPython to implement. -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue9942> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1346238] A constant folding optimization pass for the AST
Changes by Dave Malcolm : -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue1346238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9635] Add Py_BREAKPOINT and sys.breakpoint hooks
Dave Malcolm added the comment: Adding updated version of patch, which adds documentation to sys.rst and adds a unit test. I'm a little wary of this: it seems useful but also too much like a self-destruct button for my taste. -- Added file: http://bugs.python.org/file19457/py3k-add-breakpoint-2010-11-01-001.patch ___ Python tracker <http://bugs.python.org/issue9635> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9635] Add Py_BREAKPOINT and sys._breakpoint hooks
Dave Malcolm added the comment: I renamed it from sys.breakpoint to sys._breakpoint, since this is CPython-specific -- title: Add Py_BREAKPOINT and sys.breakpoint hooks -> Add Py_BREAKPOINT and sys._breakpoint hooks ___ Python tracker <http://bugs.python.org/issue9635> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10288] Remove
New submission from Dave Malcolm : Issue 5793 rationalized all usage of C "character" handling to use "Py_"-prefixed locale-unaware macros, at the "char" level. In particular, this comment was added in two places to Include/bytes_methods.h in r72044: http://svn.python.org/view/python/branches/py3k/Include/bytes_methods.h?view=diff&r1=72043&r2=72044 /* These are left in for backward compatibility and will be removed in 2.8/3.2 */ Given that 3.2 is coming soon, is it time to remove these? (also, the reference to "2.8" caught my eye) Attached is a patch to py3k which removes them, and fixes up various users that were still in the source tree. Am I right in thinking that the undef and redefinition of the various lower-case macros from was already intended to be removed? (given that this messes about with a standard C library) -- files: py3k-remove-old-char-compat-macros.patch keywords: patch messages: 120185 nosy: dmalcolm, eric.smith, mark.dickinson priority: normal severity: normal stage: patch review status: open title: Remove versions: Python 3.2, Python 3.3 Added file: http://bugs.python.org/file19459/py3k-remove-old-char-compat-macros.patch ___ Python tracker <http://bugs.python.org/issue10288> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10288] Remove deprecated C "character" handling macros ISUPPER() etc
Changes by Dave Malcolm : -- title: Remove -> Remove deprecated C "character" handling macros ISUPPER() etc ___ Python tracker <http://bugs.python.org/issue10288> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10288] Remove deprecated C "character" handling macros ISUPPER() etc
Dave Malcolm added the comment: Thanks for reviewing. If I'm reading things correctly, the ISUPPER et al macros were added in 2.6 and 3.0, and deprecated in 2.7 and 3.1. Tested with a full run of "-m test.regrtest -uall" here (x86_64 Fedora 13), with both 2-byte and 4-byte unicode; no failures: 342 tests OK. 7 tests skipped: test_gdb test_kqueue test_ossaudiodev test_startfile test_winreg test_winsound test_zipfile64 Those skips are all expected on linux2. I've fixed up the issue number in Misc/NEWS in my local tree. The patch as-is doesn't make sense for 3.1 or 2.7 (the macros should remain within the maintenance branches, in deprecated form) - do I need to explicitly mark this revision (e.g. using "svnmerge.py block"?) -- ___ Python tracker <http://bugs.python.org/issue10288> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10288] Remove deprecated C "character" handling macros ISUPPER() etc
Dave Malcolm added the comment: Committed to py3k in r86210 -- ___ Python tracker <http://bugs.python.org/issue10288> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1346238] A constant folding optimization pass for the AST
Dave Malcolm added the comment: FWIW, I'm working on fixing up the this patch to work against py3k; I'm assuming there's still interest in the AST visitor + specific optimization passes approach. -- ___ Python tracker <http://bugs.python.org/issue1346238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10331] test_gdb failure when warnings printed out
Dave Malcolm added the comment: Seems to relate to this gdb feature: http://sourceware.org/ml/gdb-patches/2005-05/msg00637.html Barry: is your ~/.gdbinit world writable? I can cook up a patch to ignore such warnings -- ___ Python tracker <http://bugs.python.org/issue10331> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10331] test_gdb failure when warnings printed out
Dave Malcolm added the comment: Alternatively, it looks like having it owned by another user would do this. For curiosity's sake, what's the output of: ls -al /home/barry/.gdbinit on that buildbot? Given that it's a security warning, should this simply be treated as misconfiguration, and be fixed on the buildbot? -- ___ Python tracker <http://bugs.python.org/issue10331> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1346238] A constant folding optimization pass for the AST
Dave Malcolm added the comment: I've been working on this against the py3k branch. I'm attaching what I've got so far. I foolishly didn't check the tlee-ast-optimize branch, instead using file6850 as a base. Rune Holm/titanstar, I assume you've signed a PSF contributor agreement? Changes since titanstar's original patch: - rework to apply against py3k - reformatted tabs to 4-space indentation; tried to reformat to PEP-7 as much as possible - added stmt types: With_kind, Nonlocal_kind - added expr types: IfExp_kind, Bytes_kind, SetComp_kind, DictComp_kind, Ellipsis_Kind - removed Print_kind, Exec_kind, Repr_kind - reworked Raise_kind - added "col_offset" and "arena" arguments; pass in the PyArena from the compiler as the context of the visitor - removal of all "free_expr" and "asdl_seq_free" calls on the assumption that PyArena now handles all of this (am I correct in thinking this?) - String -> Bytes in create_ast_from_constant_object - added test_optimize selftest suite, though this is based on bytecode disassembly, rather than direct inspection of the AST - I've fixed it up so it compiles and passes regrtest, but I suspect I've missed optimization possibilities I did a little performance testing using the py3k version of the benchmark suite; currently it's a slight regression for some tests, a slight improvement for others; nothing impressive yet. Thomas Lee's AST optimization branch (branched from r62457) has lots of interesting work: e.g. http://svn.python.org/view/python/branches/tlee-ast-optimize/Python/optimize.c?view=log This appears to not be quite the same starting point; he added a PyCF_NO_OPTIMIZE flag to Include/pythonrun.h (and other places), which seems like a good way to see the effect of the optimization pass. He also removed the peepholer; maybe it's worth doing that, but it seems worth at least keeping the test suite around to ensure a lack of regressions. I can look at cherrypicking Thomas' work/porting it to py3k. Re: "aiming high": I'd love to add new optimizations, but it's not clear to me what's permissable. In particular, is it permissable for an optimization pass to assume that there are no external modifications to the locals within a frame? It's possible to write code like this: frame = inspect.currentframe() inspect.getouterframes(frame)[-depth][0].f_locals[name] = value to manipulate locals; whether or not this actually affects running code in the current implementation of CPython seems hit-or-miss to me right now, I think depending on exactly when fastlocals get written back to the f_locals dictionary (I could have miswritten the precise code). By strategically manipulating locals in other frames, we can break pretty-much any typical compiler optimization: locals can appear or change from under us, or change attribute values, or gain side-effects to their __getattr__ (e.g. writing to disk). If it is permissable for an optimization pass to assume that there are no external modifications to the locals within a frame, then issue 4264 might be one to investigate: this is a patch on top of Tom Lee's work (to do local type-inference to replace list.append with LIST_APPEND). Ideas for other optimizations would be most welcome. -- Added file: http://bugs.python.org/file19513/py3k-ast-optimization-2010-11-05-001.patch ___ Python tracker <http://bugs.python.org/issue1346238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1346238] A constant folding optimization pass for the AST
Dave Malcolm added the comment: Another optimization idea: detect local dictionaries that are only ever used in non-mutating ways, and convert them to constants, rather than rebuilding the dict from scratch each time. See e.g. htmlparser.py:adjustSVGAttributes etc within the bm_html5lib benchmark (though this doesn't seem to be ported to py3k yet) -- ___ Python tracker <http://bugs.python.org/issue1346238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10331] test_gdb failure when warnings printed out
Dave Malcolm added the comment: Closing "won't fix", since this was misconfiguration; filtering out such warnings might mask a security issue. -- resolution: -> wont fix status: open -> closed ___ Python tracker <http://bugs.python.org/issue10331> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10391] obj2ast's error handling can lead to python crashing with a C-level assertion failure
New submission from Dave Malcolm : In various places within the generated Python/Python-ast.c, error handling generates a repr() and raises exceptions accordingly. Currently in py3k the generated code uses PyBytes_AS_STRING() on the repr. My understanding is that repr() should be a PyUnicodeObject, not a PyBytesObject. This seems to be unchanged from r63682, which was a mass-change of PyString to PyBytes from 2 years ago. This leads to a python crashing with an assertion failure: test_compile_ast (__main__.TestSpecifics) ... python: Python/Python-ast.c:5835: obj2ast_expr: Assertion `((PyObject*)(tmp))->ob_type))->tp_flags & ((1L<<27))) != 0)' failed. when invoking compile() on certain (malformed) trees of ast objects. -- components: Interpreter Core messages: 120967 nosy: dmalcolm priority: normal severity: normal status: open title: obj2ast's error handling can lead to python crashing with a C-level assertion failure type: crash versions: Python 3.1, Python 3.2, Python 3.3 ___ Python tracker <http://bugs.python.org/issue10391> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10391] obj2ast's error handling can lead to python crashing with a C-level assertion failure
Dave Malcolm added the comment: The attached patch: - extends the ast error-handling selftest with code that triggers this crash (on unpatched code) - fixes Parser/asdl_c.py to generate code using _PyUnicode_AS_STRING instead - contains the generated changes to Python/Python-ast.c FWIW, it's not clear to what extent _PyUnicode_AS_STRING is deprecated, Include/unicodeobject.h currently has: *** This API is for interpreter INTERNAL USE ONLY and will likely *** be removed or changed for Python 3.1. but I hope it won't be, it's far too useful. -- keywords: +patch stage: -> patch review Added file: http://bugs.python.org/file19572/py3k-issue-10391-fix-ast-error-handling.patch ___ Python tracker <http://bugs.python.org/issue10391> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10391] obj2ast's error handling can lead to python crashing with a C-level assertion failure
Changes by Dave Malcolm : -- nosy: +benjamin.peterson, brett.cannon, georg.brandl, ncoghlan ___ Python tracker <http://bugs.python.org/issue10391> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10399] AST Optimization: inlining of function calls
New submission from Dave Malcolm :
In msg#120541 of issue#1346238 Raymond suggested to "aim high", so here goes...
I'm opening this as a separate bug as it's a very different approach to the
patches in that bug; adding those on the nosy list for that bug. Sorry in
advance about the epic length of this comment.
I've been experimenting with AST optimizations, and have a (very) primitive
implementation of function-call inlining. As is, it's deeply flawed, but in a
spirit of "Release Early, Release Often" I though I'd post what I have so far,
and try to enumerate the rest of the work that would need doing to get this
into, say Python 3.3
The attached patch adds an AST optimization pass to Python/compiler.c. It does
this by adding an __optimizer__.py module: the compiler converts the AST to a
Python representation using ast2mod, and calls out to
__optimizer__.optimize_ast(). This can then (potentially) apply a series of
manipulations to the tree. The result is then converted back from python to
ast objects, and compilation proceeds as normal on the modified AST tree.
I initially was experimenting with a .c implementation, but moving to .py makes
things _much_ easier to develop and debug. In particular, I'm using graphviz's
"dot" tool to generate before/after visualizations of the AST.
The optimizer doesn't try to optimize itself (or anything that it imports), to
avoid infinite recursion when we have a cold .pyo cache.
Currently I'm doing the AST optimization before symbol table generation. This
means that the inlining is deeply flawed, as it has no knowledge of the scope
of names. A robust implementation would compare the scopes of the callsite and
that of the function body, and remap locals accordingly. The current
implementation renames all name references in the function body, which is
clearly wrong for e.g. references to globals. See below for notes on that.
Here's my test code::
def function_to_be_inlined(x, y, z):
return (2 * x * y) + z
print(function_to_be_inlined(3, 4, 5))
Here's what it compiles to after going through the inliner (clearly, line
numbering needs some work). Note the removal of the CALL_FUNCTION of our
target call site; the remaining CALL_FUNCTION is to "print":
2 0 LOAD_CONST 0 (", line 2>)
3 MAKE_FUNCTION0
6 STORE_NAME 0 (function_to_be_inlined)
4 9 LOAD_CONST 1 (3)
12 STORE_NAME 1 (__inline1f22840__x)
15 LOAD_CONST 2 (4)
18 STORE_NAME 2 (__inline1f22840__y)
21 LOAD_CONST 3 (5)
24 STORE_NAME 3 (__inline1f22840__z)
259 27 LOAD_CONST 4 (2)
30 LOAD_NAME1 (__inline1f22840__x)
33 BINARY_MULTIPLY
34 LOAD_NAME2 (__inline1f22840__y)
37 BINARY_MULTIPLY
38 LOAD_NAME3 (__inline1f22840__z)
41 BINARY_ADD
42 STORE_NAME 4 (__inline1f22840returnval__)
260 45 LOAD_NAME5 (print)
48 LOAD_NAME4 (__inline1f22840returnval__)
51 CALL_FUNCTION1
54 POP_TOP
55 LOAD_CONST 5 (None)
58 RETURN_VALUE
The idea is that a further optimization pass would go through and ellide the
unnecessary store-to-local/load-from-local instructions, followed by const
folding, getting this down to:
LOAD_CONST ()
MAKE_FUNCTION
STORE_NAME (function_to_be_inlined)
; inlined callsite:
LOAD_NAME (print)
LOAD_CONST (29)
CALL_FUNCTION 1
The biggest issue here is dealing with runtime differences between which
function was inlined, and which function is being called, either via
monkeypatching, or in method calls - we can inline intra-method calls within
one class, but we have to cope with overriding of those methods in subclasses.
Thinking aloud of a way of solving this (this isn't implemented yet):
add to AST:
Specialize(expr name,
stmt* specialized,
stmt* generalized)
where you have some interpretation of name (e.g. "self.do_something"), and
carry two different implementations.
so that e.g.
class Something:
def foo(self, x, y, z):
... # some code
self.bar(x, y, z)
... # more code
the:
Call(Attribute(Name(id='self'), id='bar'), args=[etc])
becomes
Specialize(name=Attribute(Name(id='self'), id='bar'),
specialized=[inlined implementation of Something.bar for
"self"],
[issue10399] AST Optimization: inlining of function calls
Changes by Dave Malcolm : Added file: http://bugs.python.org/file19584/before.png ___ Python tracker <http://bugs.python.org/issue10399> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10399] AST Optimization: inlining of function calls
Changes by Dave Malcolm : Added file: http://bugs.python.org/file19585/after.png ___ Python tracker <http://bugs.python.org/issue10399> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10924] Adding salt and Modular Crypt Format to crypt library.
Changes by Dave Malcolm : -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue10924> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11410] Use GCC visibility attrs in PyAPI_*
Changes by Dave Malcolm : -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue11410> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)
Changes by Dave Malcolm : -- nosy: +loewis ___ Python tracker <http://bugs.python.org/issue9263> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10288] Remove deprecated C "character" handling macros ISUPPER() etc
Changes by Dave Malcolm : -- resolution: -> accepted stage: patch review -> committed/rejected status: open -> closed versions: -Python 3.3 ___ Python tracker <http://bugs.python.org/issue10288> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9635] Add Py_BREAKPOINT and sys._breakpoint hooks
Changes by Dave Malcolm : -- nosy: +loewis ___ Python tracker <http://bugs.python.org/issue9635> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11393] Integrate faulthandler module into Python 3.3
Dave Malcolm added the comment: Various thoughts/nitpicking: - is it possible to indicate with a coding convention (e.g. comments) which parts of the code are intended to be called from a signal handler? It seems worth making this explicit. Or perhaps put it all in one file? - within tests.py, check_enabled and check_disabled seem to me to be misnamed; it's not at all clear what they do. I'd suggest renaming "get_output" to "run_code", perhaps (adding a docstring specifying the returned value) "check_enabled" seems to mean "assertCodeLeadsToOutput" or somesuch. Within backtrace.py: - do all platforms supported by Python have a concept of numeric filedescriptors? I was wondering if FILE* might be a better abstraction here (with flushing), then read http://bugs.python.org/issue8863#msg124385 which gives the reason: fprintf etc are not signal-safe - all of the calls to "write" ignore the return code, leading to warnings from GCC. I don't think there's any good way to handle errors from these calls, though. Might be nice to also have SIGABRT (as per a c-level assertion failure), exposed NB: on Fedora/RHEL we also have a whole-system crash detection system (called "abrt": https://fedorahosted.org/abrt/ ), and in theory, that means that for me, crash reports get run using the gdb pretty-print hooks. I'm wondering to what extent this would interract with whole-system crash-detection tools: would it e.g. mask a SIGSEGV, so that the crash is not seen by that system? -- ___ Python tracker <http://bugs.python.org/issue11393> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11706] Build from hg fails with ancient hg version (0.9.3)
New submission from Dave Malcolm : With hg/mercurial 0.9.3, the build of CPython fails with this error: gcc -pthread -c -fno-strict-aliasing -g -O0 -Wall -Wstrict-prototypes-I. -IInclude -I./Include-DPy_BUILD_CORE \ -DHGVERSION="\"`LC_ALL=C hg id -i .`\"" \ -DHGTAG="\"`LC_ALL=C hg id -t .`\"" \ -DHGBRANCH="\"`LC_ALL=C hg id -b .`\"" \ -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c hg identify: option -i not recognized hg identify: option -t not recognized hg identify: option -b not recognized ./Modules/getbuildinfo.c: In function ‘Py_GetBuildInfo’: ./Modules/getbuildinfo.c:37: error: missing terminating " character ./Modules/getbuildinfo.c:37: error: expected expression before ‘)’ token ./Modules/getbuildinfo.c:38: error: missing terminating " character ./Modules/getbuildinfo.c:38: error: missing terminating " character ./Modules/getbuildinfo.c:39: error: missing terminating " character ./Modules/getbuildinfo.c:39: error: missing terminating " character ./Modules/getbuildinfo.c:45: error: ‘buildinfo’ undeclared (first use in this function) ./Modules/getbuildinfo.c:45: error: (Each undeclared identifier is reported only once ./Modules/getbuildinfo.c:45: error: for each function it appears in.) ./Modules/getbuildinfo.c:49: warning: control reaches end of non-void function ./Modules/getbuildinfo.c: In function ‘_Py_hgversion’: ./Modules/getbuildinfo.c:54: error: missing terminating " character ./Modules/getbuildinfo.c:54: warning: ‘return’ with no value, in function returning non-void ./Modules/getbuildinfo.c: In function ‘_Py_hgidentifier’: ./Modules/getbuildinfo.c:61: error: missing terminating " character ./Modules/getbuildinfo.c:61: error: expected expression before ‘;’ token ./Modules/getbuildinfo.c:65: error: missing terminating " character ./Modules/getbuildinfo.c:65: error: expected expression before ‘;’ token make: *** [Modules/getbuildinfo.o] Error 1 What's happening is that "hg id" doesn't recognize the given options, and thus the various HG macros get set to insane multi-line strings containing usage information, but without quote characters, rather than to empty string literals [1]. This means that the post-processed code isn't valid C. The workaround presumably is to upgrade to a more recent version of mercurial. I'm reporting this here in the bug tracker in case anyone else runs into this problem. [1] Adding "-v" to the gcc invocation demonstrates that the values being passed are: -DHGVERSION="hg identify print information about the working copy Print a short summary of the current state of the repo. This summary identifies the repository state using one or two parent hash identifiers, followed by a "+" if there are uncommitted changes in the working directory, followed by a list of tags for this revision. aliases: id" -DHGTAG="hg identify print information about the working copy Print a short summary of the current state of the repo. This summary identifies the repository state using one or two parent hash identifiers, followed by a "+" if there are uncommitted changes in the working directory, followed by a list of tags for this revision. aliases: id" -DHGBRANCH="hg identify print information about the working copy Print a short summary of the current state of the repo. This summary identifies the repository state using one or two parent hash identifiers, followed by a "+" if there are uncommitted changes in the working directory, followed by a list of tags for this revision. aliases: id" -- components: Build messages: 132440 nosy: dmalcolm priority: low severity: normal status: open title: Build from hg fails with ancient hg version (0.9.3) type: compile error versions: Python 2.7, Python 3.2, Python 3.3 ___ Python tracker <http://bugs.python.org/issue11706> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11706] Build from hg fails in Modules/getbuildinfo.c when built using ancient hg version (0.9.3)
Dave Malcolm added the comment: Works OK with mercurial 1.3.1 (specifically, this is on a RHEL5 box, using mercurial-1.3.1-3.el5.i386.rpm from EPEL5) Closing this out, and changing title to help people searching for this. Not sure of the exact version of mercurial that's required. -- resolution: -> works for me stage: -> committed/rejected status: open -> closed title: Build from hg fails with ancient hg version (0.9.3) -> Build from hg fails in Modules/getbuildinfo.c when built using ancient hg version (0.9.3) ___ Python tracker <http://bugs.python.org/issue11706> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10517] test_concurrent_futures crashes with "--with-pydebug" on RHEL5 with "Fatal Python error: Invalid thread state for this thread"
Dave Malcolm added the comment: I tried again, and I'm still able to reproduce this bug on a RHEL5 box with cpython --with-pydebug as of a recent checkout (69030:00217100b9e7 as it happens): $ ./python -c "import multiprocessing.managers ; mpp = multiprocessing.Pool(4); sm = multiprocessing.managers.SyncManager(); sm.start()" Fatal Python error: Invalid thread state for this thread [66448 refs] -- ___ Python tracker <http://bugs.python.org/issue10517> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10632] multiprocessing generates a fatal error
Dave Malcolm added the comment: This is indeed looking like it's RHEL-specific. I was about to close it out as a duplicate of issue 10517, but I'm wondering why Brian chose to open it as a separate bug. -- ___ Python tracker <http://bugs.python.org/issue10632> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4111] Add Systemtap/DTrace probes
Dave Malcolm added the comment: jcea: I notice that on 2011-02-22 you made these changes: assignee: dmalcolm -> dino.viehland nosy: +dino.viehland versions: +Python 3.3 -Python 3.2 Did you mean to change the assignee, or was this an accident? -- ___ Python tracker <http://bugs.python.org/issue4111> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11830] "import decimal" fails in Turkish locale
New submission from Dave Malcolm : For Python 2 (here with 2.7.1): $ python -c 'import locale; locale.setlocale(locale.LC_ALL, "tr_TR.UTF-8"); import decimal' Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python2.7/decimal.py", line 3715, in val = globals()[globalname] KeyError: 'ROUND_CEiLiNG' The issue is that 'round_ceiling'.upper() is 'ROUND_CEiLiNG' in the Turkish locale, rather than 'ROUND_CEILING', as one might expect. A workaround for this may be to convert the str instances to unicode first, then call upper on them, then convert back to str. This would work since upper() for a unicode instance is locale-independent as per issue 1528802. (though there seems to have been some debate there). Patch attached, though it doesn't yet contain a test case. Only affects Python 2; with Python 3, the symbols are already stored as unicode internally. Reported downstream as: https://bugzilla.redhat.com/show_bug.cgi?id=694928 which has links to various other reports on this -- components: Library (Lib) files: decimal.py.patch keywords: patch messages: 133550 nosy: dmalcolm priority: normal severity: normal status: open title: "import decimal" fails in Turkish locale versions: Python 2.7 Added file: http://bugs.python.org/file21622/decimal.py.patch ___ Python tracker <http://bugs.python.org/issue11830> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11845] Refcounting error in compute_slice_indices in rangeobject.c
Changes by Dave Malcolm : -- nosy: +dmalcolm ___ Python tracker <http://bugs.python.org/issue11845> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
