[issue46147] Support AddressSanitizer in Windows build
New submission from anthony shaw : I'd like to compile my C-extensions with ASAN for testing in Windows, but they cannot be loaded as the host python.exe process needs to be compiled with ASAN. https://docs.microsoft.com/en-us/cpp/sanitizers/asan?view=msvc-170#ide-msbuild The EnableASAN flag would be a setting in vcxproj files within the PCBuild solution, and a flag in the build.bat to enable it. -- components: Build, Windows messages: 409006 nosy: anthonypjshaw, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: Support AddressSanitizer in Windows build type: enhancement ___ Python tracker <https://bugs.python.org/issue46147> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46147] Support AddressSanitizer in Windows build
anthony shaw added the comment: After some experimentation, this can be done if you enable ASAN in all projects except python3dll.vcxproj I think it would make sense (if this were supported) to have a flag in build.bat like there is for pgo `build.bat --asan` -- ___ Python tracker <https://bugs.python.org/issue46147> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46704] Parser API not checking for null-terminator
New submission from anthony shaw :
In tokenizer.c, the translate_newlines() function does a `strlen()` on the
input string, if the string is not null-terminated, e.g.
'\xbe' this leads to a heap-buffer-overflow. The overflow is not exploitable,
but if there are further changes to the parser, it might be worth using a
strlen() alternative, like strnlen().
static char *
translate_newlines(const char *s, int exec_input, struct tok_state *tok) {
int skip_next_lf = 0;
size_t needed_length = strlen(s) + 2, final_length;
This leads to a heap-buffer-overflow detected by ASAN in a simple reproducible
example, calling PyRun_StringFlags() from the LLVM fuzzer:
fuzz_target(47084,0x11356f600) malloc: nano zone abandoned due to inability to
preallocate reserved vm space.
Dictionary: 35 entries
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3034498392
INFO: Loaded 1 modules (43 inline 8-bit counters): 43 [0x10a2b71e8,
0x10a2b7213),
INFO: Loaded 1 PC tables (43 PCs): 43 [0x10a2b7218,0x10a2b74c8),
INFO:1 files found in ../Tests/fuzzing/corpus
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than
4096 bytes
=
==47084==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x60203131 at pc 0x00010bd1d555 bp 0x7ff7b5da0590 sp 0x7ff7b5d9fd50
READ of size 2 at 0x60203131 thread T0
#0 0x10bd1d554 in wrap_strlen+0x184
(libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x15554)
#1 0x10b12132b in translate_newlines+0x1b (Python:x86_64+0x5d32b)
#2 0x10b12071c in _PyParser_ASTFromString+0x1ac (Python:x86_64+0x5c71c)
#3 0x10b2f86de in PyRun_StringFlags+0x5e (Python:x86_64+0x2346de)
#4 0x10a25ec6b in CompileCode(char const*) fuzz_target.cpp:54
#5 0x10a25f247 in LLVMFuzzerTestOneInput fuzz_target.cpp:68
#6 0x10a27aff3 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*,
unsigned long) FuzzerLoop.cpp:611
#7 0x10a27c3c4 in
fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector >&) FuzzerLoop.cpp:804
#8 0x10a27c859 in fuzzer::Fuzzer::Loop(std::__1::vector >&) FuzzerLoop.cpp:857
#9 0x10a26aa5f in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char
const*, unsigned long)) FuzzerDriver.cpp:906
#10 0x10a298e42 in main FuzzerMain.cpp:20
#11 0x1134f44fd in start+0x1cd (dyld:x86_64+0x54fd)
0x60203131 is located 0 bytes to the right of 1-byte region
[0x60203130,0x60203131)
allocated by thread T0 here:
#0 0x10bd58a0d in wrap__Znam+0x7d
(libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x50a0d)
#1 0x10a27af02 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*,
unsigned long) FuzzerLoop.cpp:596
#2 0x10a27c3c4 in
fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector >&) FuzzerLoop.cpp:804
#3 0x10a27c859 in fuzzer::Fuzzer::Loop(std::__1::vector >&) FuzzerLoop.cpp:857
#4 0x10a26aa5f in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char
const*, unsigned long)) FuzzerDriver.cpp:906
#5 0x10a298e42 in main FuzzerMain.cpp:20
#6 0x1134f44fd in start+0x1cd (dyld:x86_64+0x54fd)
SUMMARY: AddressSanitizer: heap-buffer-overflow
(libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x15554) in wrap_strlen+0x184
Shadow bytes around the buggy address:
0x1c0405d0: fa fa 02 fa fa fa 02 fa fa fa 02 fa fa fa 02 fa
0x1c0405e0: fa fa 02 fa fa fa 02 fa fa fa 02 fa fa fa 02 fa
0x1c0405f0: fa fa 03 fa fa fa 03 fa fa fa 03 fa fa fa 03 fa
0x1c040600: fa fa 01 fa fa fa 01 fa fa fa 01 fa fa fa 01 fa
0x1c040610: fa fa 00 00 fa fa 00 fa fa fa 00 fa fa fa 00 00
=>0x1c040620: fa fa 00 fa fa fa[01]fa fa fa fd fa fa fa fd fd
0x1c040630: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 04 fa
0x1c040640: fa fa 00 00 fa fa 01 fa fa fa 01 fa fa fa 01 fa
0x1c040650: fa fa fd fa fa fa fd fa fa fa fd fd fa fa 01 fa
0x1c040660: fa fa 00 00 fa fa 01 fa fa fa fd fa fa fa fd fa
0x1c040670: fa fa 01 fa fa fa 06 fa fa fa 00 00 fa fa 06 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user:f7
Container overflow: fc
Array cookie:ac
Intra object redzone:bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone:cb
Shadow gap: cc
==47084==ABORTING
MS: 0 ; base unit:
artifact_prefix='./'; Test unit written to
./crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64:
zsh: abort ./fuzz_target -dict=../Tests/fuzzing/python.dict -o
[issue46707] Parser hanging on stacked { tokens
New submission from anthony shaw :
Providing an (invalid) input to the parser causes an exponentially-slow DoS to
the Python executable in 3.10.
e.g.
python3.10 -c "{:"
takes ~2 seconds
python3.10 -c ":"
takes ~22 seconds
Tested this all the way up to
d```{{{ef f():y
which took over an hour
--
components: Parser
keywords: 3.10regression
messages: 412972
nosy: anthonypjshaw, lys.nikolaou, pablogsal
priority: normal
severity: normal
status: open
title: Parser hanging on stacked { tokens
type: crash
___
Python tracker
<https://bugs.python.org/issue46707>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46707] Parser hanging on stacked { tokens
Change by anthony shaw : -- versions: +Python 3.10, Python 3.11 ___ Python tracker <https://bugs.python.org/issue46707> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43567] regen.vcxproj cannot regenerate some necessary files
anthony shaw added the comment: Guido, regen.vcxproj targets 142 as the SDK version, which is most likely a mistake. The other projects are part of the main PCBuild.sln solution, which has a variable for the base SDK version. If you need to change it quickly, you can either open it in VS and right click on the project and change the properties to an older SDK, or edit the vcxproj file itself. I can make this change in a patch, but because its a standalone project, it doesn't share the base SDK version with the others. I've reviewed Steve's patch and it would fix this because it changes regen into a build target instead of a build project, so it doesn't specify the SDK version at all. Steve's implementation is also much, much cleaner! -- nosy: +anthony shaw ___ Python tracker <https://bugs.python.org/issue43567> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44692] Const unfolding in parser with negative numbers doesn't match float/int behaviour
New submission from anthony shaw : Powers with negative bases do not observe the same rules that float_pow/long_pow do when it comes to returning negative results/ Example: > -2 ** 2 -4 >>> x = -2 >>> y = 2 >>> x ** y 4 >>> -2. ** 2. -4.0 >>> x = -2. >>> y = 2. >>> x ** y 4.0 Tested on 3.8, 3.9 and 3.10, they all have the same bug. -- components: Parser messages: 397922 nosy: anthonypjshaw, lys.nikolaou, pablogsal priority: normal severity: normal status: open title: Const unfolding in parser with negative numbers doesn't match float/int behaviour type: behavior versions: Python 3.10, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue44692> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44692] Const folding in parser with negative numbers doesn't match float/int behaviour
Change by anthony shaw : -- title: Const unfolding in parser with negative numbers doesn't match float/int behaviour -> Const folding in parser with negative numbers doesn't match float/int behaviour ___ Python tracker <https://bugs.python.org/issue44692> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44692] Const folding in parser with negative numbers doesn't match float/int behaviour
anthony shaw added the comment: Thanks! I didn't realise it applied the negative operator instead of loading the constant value. -- ___ Python tracker <https://bugs.python.org/issue44692> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39144] Align ctags and etags targets and include Python stdlib
New submission from anthony shaw : make tags will include - Modules/_ctypes/ where as make TAGS will not. Also, neither include the Python source files for the standard library, which both etags and ctags are capable of handling. PR to follow -- components: Build messages: 358917 nosy: anthony shaw priority: normal severity: normal status: open title: Align ctags and etags targets and include Python stdlib type: enhancement ___ Python tracker <https://bugs.python.org/issue39144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39144] Align ctags and etags targets and include Python stdlib
anthony shaw added the comment: Also, make tags will reset the "tags" file, whereas, make TAGS will not. If you ran "make tags" then "make TAGS" you will have a corrupt etags file -- ___ Python tracker <https://bugs.python.org/issue39144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39144] Align ctags and etags targets and include Python stdlib
Change by anthony shaw : -- keywords: +patch pull_requests: +17166 stage: -> patch review pull_request: https://github.com/python/cpython/pull/17721 ___ Python tracker <https://bugs.python.org/issue39144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39160] ./configure --help has inconsistencies in style
New submission from anthony shaw : I've noticed that ./configure --help is inconsistent. - The way default values are shared - The way enumerated - The verbs used (e.g. enable, set) - Some --with-xyx and some --with(out-xyz) - Some start with capitals, others don't Also, many of the flags could use additional explanation as to their purpose, or reference the rST file in the doc that explains what they do. PR to follow -- components: Build messages: 359014 nosy: anthony shaw priority: normal severity: normal status: open title: ./configure --help has inconsistencies in style ___ Python tracker <https://bugs.python.org/issue39160> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39160] ./configure --help has inconsistencies in style
anthony shaw added the comment:
Current
Optional Features:
--disable-option-checking ignore unrecognized --enable/--with options
--disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no)
--enable-FEATURE[=ARG] include FEATURE [ARG=yes]
--enable-universalsdk[=SDKDIR]
Build fat binary against Mac OS X SDK
--enable-framework[=INSTALLDIR]
Build (MacOSX|Darwin) framework
--enable-shared disable/enable building shared python library
--enable-profiling enable C-level code profiling
--enable-optimizations Enable expensive, stable optimizations (PGO, etc).
Disabled by default.
--enable-loadable-sqlite-extensions
support loadable extensions in _sqlite module
--enable-ipv6 Enable ipv6 (with ipv4) support
--disable-ipv6 Disable ipv6 support
--enable-big-digits[=BITS]
use big digits for Python longs [[BITS=30]]
Optional Packages:
--with-PACKAGE[=ARG]use PACKAGE [ARG=yes]
--without-PACKAGE do not use PACKAGE (same as --with-PACKAGE=no)
--with-universal-archs=ARCH
select architectures for universal build ("32-bit",
"64-bit", "3-way", "intel", "intel-32", "intel-64",
or "all")
--with-framework-name=FRAMEWORK
specify an alternate name of the framework built
with --enable-framework
--with-cxx-main=
compile main() and link python executable with C++
compiler
--with-suffix=.exe set executable suffix
--with-pydebug build with Py_DEBUG defined
--with-trace-refs enable tracing references for debugging purpose
--with-assertions build with C assertions enabled
--with-lto Enable Link Time Optimization in any build. Disabled
by default.
--with-hash-algorithm=[fnv|siphash24]
select hash algorithm
--with-address-sanitizer
enable AddressSanitizer (asan)
--with-memory-sanitizer enable MemorySanitizer (msan)
--with-undefined-behavior-sanitizer
enable UndefinedBehaviorSanitizer (ubsan)
--with-libs='lib1 ...' link against additional libs
--with-system-expat build pyexpat module using an installed expat
library
--with-system-ffi build _ctypes module using an installed ffi library
--with-system-libmpdec build _decimal module using an installed libmpdec
library
--with-tcltk-includes='-I...'
override search for Tcl and Tk include files
--with-tcltk-libs='-L...'
override search for Tcl and Tk libs
--with-dbmliborder=db1:db2:...
order to check db backends for dbm. Valid value is a
colon separated string with the backend names
`ndbm', `gdbm' and `bdb'.
--with(out)-doc-strings disable/enable documentation strings
--with(out)-pymallocdisable/enable specialized mallocs
--with(out)-c-locale-coercion
disable/enable C locale coercion to a UTF-8 based
locale
--with-valgrind Enable Valgrind support
--with(out)-dtrace disable/enable DTrace support
--with-libm=STRING math library
--with-libc=STRING C library
--with(out)-computed-gotos
Use computed gotos in evaluation loop (enabled by
default on supported compilers)
--with(out)-ensurepip=[=upgrade]
"install" or "upgrade" using bundled pip
--with-openssl=DIR root of the OpenSSL directory
--with-ssl-default-suites=[python|openssl|STRING]
Override default cipher suites string, python: use
Python's preferred selection (default), openssl:
leave OpenSSL's defaults untouched, STRING: use a
custom string, PROTOCOL_SSLv2 ignores the setting
--
___
Python tracker
<https://bugs.python.org/issue39160>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39160] ./configure --help has inconsistencies in style
anthony shaw added the comment:
Updated version based on principles:
- Should not specify --with(out) in help string
- Sentences should start with lowercase (as autoconf generates the builtin
string with lower case)
- --with-xyx=VALUE will be described as "overriding"
- --with-xyz will be described as "enabling" said feature
- Mac flags should have the same descriptions as what is in Mac/README.rst
- default value should be in brackets (if known)
- Enumerated options should be shown as =VAL1|VAL2 or [=VAL1|VAL2] if optional
Optional Features:
--disable-option-checking ignore unrecognized --enable/--with options
--disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no)
--enable-FEATURE[=ARG] include FEATURE [ARG=yes]
--enable-universalsdk[=SDKDIR]
create a universal binary build. SDKDIR specifies
which macOS SDK should be used to perform the build,
see Mac/README.rst. (default is no)
--enable-framework[=INSTALLDIR]
create a Python.framework rather than a traditional
Unix install. optional INSTALLDIR specifies the
installation path. see Mac/README.rst (default is
no)
--enable-shared enable building a shared Python library (default is
no)
--enable-profiling enable C-level code profiling with gprof (default is
no)
--enable-optimizations enable expensive, stable optimizations (PGO, etc.)
(default is no)
--enable-loadable-sqlite-extensions
support loadable extensions in _sqlite module, see
Doc/library/sqlite3.rst (default is no)
--enable-ipv6 enable ipv6 (with ipv4) support, see
Doc/library/socket.rst (default is yes if supported)
--enable-big-digits[=15|30]
use big digits (30 or 15 bits) for Python longs
(default is no)]
Optional Packages:
--with-PACKAGE[=ARG]use PACKAGE [ARG=yes]
--without-PACKAGE do not use PACKAGE (same as --with-PACKAGE=no)
--with-universal-archs=ARCH
specify the kind of universal binary that should be
created. this option is only valid when
--enable-universalsdk is set; options are:
("32-bit", "64-bit", "3-way", "intel", "intel-32",
"intel-64", or "all") see Mac/README.rst
--with-framework-name=FRAMEWORK
specify the name for the python framework on macOS
only valid when --enable-framework is set. see
Mac/README.rst (default is 'Python')
--with-cxx-main[=COMPILER]
compile main() and link Python executable with C++
compiler specified in COMPILER (default is $CXX)
--with-suffix=SUFFIXset executable suffix to SUFFIX (default is '.exe')
--with-pydebug build with Py_DEBUG defined (default is no)
--with-trace-refs enable tracing references for debugging purpose
(default is no)
--with-assertions build with C assertions enabled (default is no)
--with-lto enable Link-Time-Optimization in any build (default
is no)
--with-hash-algorithm=[fnv|siphash24]
select hash algorithm for use in Python/pyhash.c
(default is SipHash24)
--with-address-sanitizer
enable AddressSanitizer memory allocation
controller, 'asan' (default is no)
--with-memory-sanitizer enable MemorySanitizer memory allocation controller,
'msan' (default is no)
--with-undefined-behavior-sanitizer
enable UndefinedBehaviorSanitizer memory allocation
controller, 'ubsan' (default is no)
--with-libs='lib1 ...' link against additional libs (default is no)
--with-system-expat build pyexpat module using an installed system expat
library, see Doc/library/pyexpat.rst (default is no)
--with-system-ffi build _ctypes module using an installed ffi library,
see Doc/library/ctypes.rst (default is system
dependant)
--with-system-libmpdec build _decimal module using an installed libmpdec
library, see Doc/library/decimal.rst (default is no)
--with-tcltk-includes='-I...'
override search for Tcl and Tk include files
--with-tcltk-
[issue39160] ./configure --help has inconsistencies in style
Change by anthony shaw : -- keywords: +patch pull_requests: +17185 stage: -> patch review pull_request: https://github.com/python/cpython/pull/17747 ___ Python tracker <https://bugs.python.org/issue39160> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39160] ./configure --help has inconsistencies in style
Change by anthony shaw : -- nosy: +anthonypjshaw ___ Python tracker <https://bugs.python.org/issue39160> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39160] ./configure --help has inconsistencies in style
Change by anthony shaw : -- nosy: +twouters ___ Python tracker <https://bugs.python.org/issue39160> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39162] setup.py not picking up tkinter headers
Change by anthony shaw : -- nosy: +ned.deily ___ Python tracker <https://bugs.python.org/issue39162> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39162] setup.py not picking up tkinter headers
New submission from anthony shaw : ./configure && make -j4 is returning: Failed to build these modules: _tkinter I'm running macOS 10.15.2, with the SDK installed using `xcode-select --install` (no funny business) /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Tk.framework/ xcrun --show-sdk-path /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk If I debug setup.py, I can see it's eagerly returning the wrong path (/System) On a REPL, if I do: >>> import setup >>> setup.macosx_sdk_root() '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk' So the path is correct, but I can see in setup.py that detect_tkinter_darwin() will break inside the loop if /System/Library/Frameworks/Tk.framework/Versions/Current exists (which it does), but it doesn't contain the headers, it contains 3 directories: Resources Tk _CodeSignature I think detect_tkinter_darwin should be updated so that framework_dirs scans macosx_sdk_root first, but I don't know what other scenarios this might break. I'd be happy to submit a patch for my scenario but it looks like this whole function needs tests so each platform change and scenario as they come up. There's loads of other issues on BPO related. -- components: Build messages: 359026 nosy: anthonypjshaw priority: normal severity: normal status: open title: setup.py not picking up tkinter headers type: compile error ___ Python tracker <https://bugs.python.org/issue39162> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39162] setup.py not picking up tkinter headers
anthony shaw added the comment: I also can't see how this condition would ever match, since macosx_sdk_root() is not in the list being iterated upon, https://github.com/python/cpython/blob/master/setup.py#L1774 -- ___ Python tracker <https://bugs.python.org/issue39162> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39162] setup.py not picking up tkinter headers
Change by anthony shaw : -- keywords: +patch pull_requests: +17189 stage: -> patch review pull_request: https://github.com/python/cpython/pull/17753 ___ Python tracker <https://bugs.python.org/issue39162> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39162] setup.py not picking up tkinter headers
anthony shaw added the comment: This comes down to a logic bug in detect_tkinter_darwin(), shown in https://github.com/python/cpython/pull/17753/files#r361956102 -- ___ Python tracker <https://bugs.python.org/issue39162> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39185] Add quiet and detailed verbosity levels to build.bat
New submission from anthony shaw : The build.bat script (windows build) has a flag for verbose, which sets the msbuild verbosity level to normal. The default level is minimal. The quiet and detailed levels would also be useful for development. -- components: Windows messages: 359178 nosy: anthonypjshaw, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: Add quiet and detailed verbosity levels to build.bat type: enhancement ___ Python tracker <https://bugs.python.org/issue39185> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39185] Add quiet and detailed verbosity levels to build.bat
Change by anthony shaw : -- keywords: +patch pull_requests: +17224 stage: -> patch review pull_request: https://github.com/python/cpython/pull/17791 ___ Python tracker <https://bugs.python.org/issue39185> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39186] Windows installer instructions refer to mercurial
New submission from anthony shaw : Very minor, but the instructions in Tools/msi/readme.txt tell the user to ensure hg.exe is in PATH, but the scripts use Git. -- components: Windows messages: 359179 nosy: anthonypjshaw, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: Windows installer instructions refer to mercurial ___ Python tracker <https://bugs.python.org/issue39186> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39186] Windows installer instructions refer to mercurial
Change by anthony shaw : -- keywords: +patch pull_requests: +17225 stage: -> patch review pull_request: https://github.com/python/cpython/pull/17792 ___ Python tracker <https://bugs.python.org/issue39186> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34946] inspect.getcallargs is marked as deprecated in documentation, but doesn't issue a DeprecationWarning when used
anthony shaw added the comment: Resolved by another bpo -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue34946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39498] Signpost security considerations in library
New submission from anthony shaw : Within the documentation, there are some really important security considerations for standard library modules. e.g. subprocess, ssl, pickle, xml. There is currently no "index" of these, so you have to go hunting for them. They're easter eggs within the docs. There isn't a unique admonition type either, so you have to search across many criteria. In particular for security researchers, it would be useful to consolidate and signpost these security best-practices in one index. PR to follow, -- assignee: docs@python components: Documentation messages: 361009 nosy: anthonypjshaw, docs@python priority: normal severity: normal status: open title: Signpost security considerations in library type: enhancement ___ Python tracker <https://bugs.python.org/issue39498> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39498] Signpost security considerations in library
Change by anthony shaw : -- keywords: +patch pull_requests: +17647 stage: -> patch review pull_request: https://github.com/python/cpython/pull/18272 ___ Python tracker <https://bugs.python.org/issue39498> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40432] Pegen regenerate project for Windows not working
New submission from anthony shaw : The additional tasks in the MSBuild project for pegen regeneration are not functional: - Setting PYTHONPATH= inline cannot be done in Windows using that method - The task does not inherit environment variables that way - The path to the peg_generator module is in Unix path format PR to follow -- assignee: anthonypjshaw components: Build messages: 367613 nosy: anthonypjshaw, pablogsal priority: normal severity: normal status: open title: Pegen regenerate project for Windows not working versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue40432> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40432] Pegen regenerate project for Windows not working
Change by anthony shaw : -- keywords: +patch pull_requests: +19108 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19785 ___ Python tracker <https://bugs.python.org/issue40432> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40490] peg_generator module has unused imports
New submission from anthony shaw : the tools/peg_generator package contains a number of unused imports -- components: Interpreter Core messages: 368005 nosy: anthonypjshaw, pablogsal priority: normal severity: normal stage: needs patch status: open title: peg_generator module has unused imports type: enhancement versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue40490> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40490] peg_generator module has unused imports
Change by anthony shaw : -- keywords: +patch pull_requests: +19202 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/19891 ___ Python tracker <https://bugs.python.org/issue40490> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34956] _tkinter built on macOS 10.14 does not link to Tcl and Tk in /Library/Frameworks
Change by anthony shaw : -- nosy: +anthonypjshaw nosy_count: 4.0 -> 5.0 pull_requests: +19480 pull_request: https://github.com/python/cpython/pull/17753 ___ Python tracker <https://bugs.python.org/issue34956> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42521] Debug (-d) mode not working in 3.9
New submission from anthony shaw : I noticed since the new parser implementation, the debug mode in the tokeniser is no longer working. This is the case for 3.9.0 and also 3.9.1rc1. Running python3.9 with a simple application like this: # Demo application def my_function(): proceed Does not output anything > python3.9 -d ~/PycharmProjects/cpython-book-samples/13/test_tokens.py > Produces no output Whereas python3.10 (the latest alpha) outputs the expected results -- messages: 382236 nosy: anthony shaw, lys.nikolaou, pablogsal priority: normal severity: normal status: open title: Debug (-d) mode not working in 3.9 versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue42521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42521] Debug (-d) mode not working in 3.9
anthony shaw added the comment: thanks, it would be great if this were documented in the --help section like the other options which require a debug build (-X showrefcount) -- ___ Python tracker <https://bugs.python.org/issue42521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43292] xml.ElementTree iterparse filehandle left open
anthony shaw added the comment: which version of Python were you doing this on? The function in question is https://github.com/python/cpython/blob/master/Lib/xml/etree/ElementTree.py#L1233-L1278 -- nosy: +anthonypjshaw ___ Python tracker <https://bugs.python.org/issue43292> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43292] xml.ElementTree iterparse filehandle left open
anthony shaw added the comment:
Example script attached works perfectly on macOS
> python3.9 test.py
[('default',
'http://schemas.openxmlformats.org/officeDocument/2006/extended-properties'),
('vt', 'http://schemas.openxmlformats.org/officeDocument/2006/docPropsVTypes')]
[('default',
'http://schemas.openxmlformats.org/officeDocument/2006/extended-properties'),
('vt', 'http://schemas.openxmlformats.org/officeDocument/2006/docPropsVTypes')]
--
nosy: +anthonypjshaw
___
Python tracker
<https://bugs.python.org/issue43292>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41888] Duplicate Opcode value in master and 3.9rc2
New submission from anthony shaw : I noticed that HAVE_ARGUMENT and STORE_NAME have the same value (90) in Include/opcode.h in both the source code and in the 2bd31b5f revision (3.9.0rc2). This must be creating some weird bugs. https://github.com/python/cpython/blob/582aaf19e8b94a70c1f96792197770d604ba0fdf/Include/opcode.h#L70-L71 -- components: Interpreter Core messages: 377688 nosy: anthonypjshaw priority: normal severity: normal status: open title: Duplicate Opcode value in master and 3.9rc2 type: compile error versions: Python 3.10, Python 3.9 ___ Python tracker <https://bugs.python.org/issue41888> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41888] Duplicate Opcode value in master and 3.9rc2
anthony shaw added the comment: No, this is intentional. sorry. closing https://github.com/python/cpython/blob/34cd3e9f6a87f9c50edac893b0d5ae46c4e48ee3/Doc/library/dis.rst#L1194 -- ___ Python tracker <https://bugs.python.org/issue41888> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41888] Duplicate Opcode value in master and 3.9rc2
anthony shaw added the comment: Or is this some sort of obscure month python reference!? -- ___ Python tracker <https://bugs.python.org/issue41888> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41888] Duplicate Opcode value in master and 3.9rc2
Change by anthony shaw : -- stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue41888> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35488] pathlib Path.match does not behave as described
New submission from anthony shaw :
The documentation for pathlib PurePath.match(needle) says it accepts
"glob-style pattern.".
For an absolute path with a recursive pattern (**) it doesn't match correct for
more than 1 directory level.
All of the assertions in the attached file should pass.
The issue I've seen is on the attached file. I'm using Python 3.7.1 and have
also tested this against Python 3.6.6 with the pathlib module on PyPi.
Absolute path glob'ing with a recursive pattern works as expected:
entries = pathlib.Path('/var').glob('/var/**/*.log')
Once this issue is confirmed, I would be happy to test & contribute a fix
--
components: Library (Lib)
files: test_pathlib.py
messages: 331782
nosy: anthony shaw
priority: normal
severity: normal
status: open
title: pathlib Path.match does not behave as described
versions: Python 3.7
Added file: https://bugs.python.org/file47996/test_pathlib.py
___
Python tracker
<https://bugs.python.org/issue35488>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35488] pathlib Path.match does not behave as described
Change by anthony shaw : -- keywords: +patch pull_requests: +10410 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35488> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35488] pathlib Path.match does not behave as described
anthony shaw added the comment: Raised a PR for the test. Will look into doc PR -- ___ Python tracker <https://bugs.python.org/issue35488> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35488] pathlib Path.match does not behave as described
anthony shaw added the comment: Yes, this is similar to https://bugs.python.org/issue29249 In that issue, it suggests this feature is not supported, but that is neither documented, nor tested. -- nosy: -serhiy.storchaka ___ Python tracker <https://bugs.python.org/issue35488> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35668] low test coverage for idlelib
New submission from anthony shaw : idlelib is one of the lesser-tested libraries in cpython: https://codecov.io/gh/python/cpython/tree/master/Lib/idlelib Raising this issue and also volunteering to extend the test module to get coverage across major behaviours and functions that are missing tests. -- assignee: terry.reedy components: IDLE, Library (Lib) messages: 333077 nosy: anthony shaw, terry.reedy priority: normal severity: normal status: open title: low test coverage for idlelib type: enhancement ___ Python tracker <https://bugs.python.org/issue35668> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35668] Improve test coverage for idlelib
anthony shaw added the comment: thanks terry, Some great pointers there, I'll review the exiting work and the README doc. With regards to my experience, I have quite extensive experience with python testing. Most of which would be open-source on my Github profile https://github.com/tonybaloney Some of the larger Python projects I've contributed test suites to would be SaltStack, Apache Libcloud and StackStorm. I've contributed to the tox project and pytest. I also write tutorials on Python testing (mostly for beginners) like this one https://realpython.com/python-testing/ I admit I don't have much experience with tkinter. It does sound like a challenge, but definitely one that I'm willing to research and approach responsibly. -- ___ Python tracker <https://bugs.python.org/issue35668> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35668] Improve test coverage for idlelib
anthony shaw added the comment: I was looking at the debugger.py module as being a good place the start, writing test cases for the Idb, Debugger, StackViewer and NamespaceViewer by patching out the dependant components (bdb, Idb, etc. I might start there, raise a PR against it and do a module at a time, then work my way up to some of the trickier, tkinter-based modules. -- ___ Python tracker <https://bugs.python.org/issue35668> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35668] Improve test coverage for idlelib
anthony shaw added the comment: thanks Cheryl, here's my branch https://github.com/tonybaloney/cpython/tree/idlelib_tests I've already seen some code which is 17 years old! -- ___ Python tracker <https://bugs.python.org/issue35668> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35668] Improve test coverage for idlelib
Change by anthony shaw : -- keywords: +patch pull_requests: +10913 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35668> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35668] Improve test coverage for idlelib
Change by anthony shaw : -- keywords: +patch, patch pull_requests: +10913, 10914 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35668> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35668] Improve test coverage for idlelib
Change by anthony shaw : -- keywords: +patch, patch, patch, patch pull_requests: +10913, 10914, 10915, 10916 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35668> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35668] Improve test coverage for idlelib
Change by anthony shaw : -- keywords: +patch, patch, patch pull_requests: +10913, 10914, 10916 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35668> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35690] IDLE: Fix and test debugger.
Change by anthony shaw : -- keywords: +patch pull_requests: +10971 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issue35690> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35668] Improve test coverage for idlelib
Change by anthony shaw : -- pull_requests: +10972 ___ Python tracker <https://bugs.python.org/issue35668> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35668] Improve test coverage for idlelib
Change by anthony shaw : -- pull_requests: +10972, 10973 ___ Python tracker <https://bugs.python.org/issue35668> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30029] Compiler "'await' outside function" error message is unreachable
New submission from anthony shaw: This is related to issue26188, Using await in a simple statement (outside of an async def method) raises SyntaxError with the unhelpful message "invalid syntax". It seems obvious once you've read PEP492 in detail, but I think that as more and more developers use async/await this will stump lots of people. I've been trying to pick apart where this constraint is raised to see whether I can help with a PR, I've been through Grammar, Parser and then the AST and Compiler. Looking at https://github.com/python/cpython/blob/master/Python/compile.c#L4307-L4319 I can see there are helpful error messages, but during the tokenizer phase it checks that the syntax cannot be used unless you're within an async def method https://github.com/python/cpython/blob/6f0eb93183519024cb360162bdd81b9faec97ba6/Parser/tokenizer.c#L1574-L1583 I can't reproduce this in a REPL for 3.7.0a0, it never gets any further than the grammar or tokenizer phase. -- messages: 291398 nosy: anthonypjshaw priority: normal severity: normal status: open title: Compiler "'await' outside function" error message is unreachable versions: Python 3.7 ___ Python tracker <http://bugs.python.org/issue30029> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30029] Compiler "'await' outside function" error message is unreachable
anthony shaw added the comment: yey! I figured it out!! -- ___ Python tracker <http://bugs.python.org/issue30029> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30029] Compiler "'await' outside function" error message is unreachable
Changes by anthony shaw : -- pull_requests: +1215 ___ Python tracker <http://bugs.python.org/issue30029> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30029] Compiler "'await' outside function" error message is unreachable
Changes by anthony shaw : -- stage: -> resolved status: open -> closed ___ Python tracker <http://bugs.python.org/issue30029> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36386] segfault on PyUnicode_DecodeFSDefaultAndSize for uninitialized Py
New submission from anthony shaw : If for whatever reason, Py_Initialize() has not been run or failed to run, any call to Py_CompileStringFlags will call PyUnicode_DecodeFSDefault and the reference to interp will be NULL. There is currently no null reference check in PyUnicode_DecodeFSDefaultAndSize which causes a segfault. https://github.com/python/cpython/blob/master/Objects/unicodeobject.c#L3736-L3737 is the offending line. It might be better to catch the null pointer and raise an unrecoverable error there? Error: signal 11: 0 ceval-prof 0x0001066310f3 handler + 35 1 libsystem_platform.dylib0x7fff6adddb3d _sigtramp + 29 2 ??? 0x 0x0 + 0 3 ceval-prof 0x000106734536 PyUnicode_DecodeFSDefault + 38 4 ceval-prof 0x000106879514 Py_CompileStringExFlags + 36 5 ceval-prof 0x000106631280 main + 320 6 libdyld.dylib 0x7fff6abf2ed9 start + 1 -- components: Interpreter Core messages: 338521 nosy: anthony shaw priority: normal severity: normal status: open title: segfault on PyUnicode_DecodeFSDefaultAndSize for uninitialized Py versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue36386> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36386] segfault on PyUnicode_DecodeFSDefaultAndSize for uninitialized Py
anthony shaw added the comment: This applies to PyUnicode_EncodeFSDefault as well, it has the same issue -- ___ Python tracker <https://bugs.python.org/issue36386> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36386] segfault on PyUnicode_DecodeFSDefaultAndSize for uninitialized Py
anthony shaw added the comment: I'm expecting a "this is not a bug, why would the interpreter not be initialized", but it would be nice to get a friendly error message since this is a public API. IF so, am also happy to submit a PR with a fix -- ___ Python tracker <https://bugs.python.org/issue36386> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36386] segfault on PyUnicode_DecodeFSDefaultAndSize for uninitialized Py
anthony shaw added the comment: This is because PyUnicode_DecodeFSDefaultAndSize calls _PyInterpreterState_GET_UNSAFE(), which already documents the potential NULL return value. /* Get the current interpreter state. The macro is unsafe: it does not check for error and it can return NULL. The caller must hold the GIL. See also _PyInterpreterState_Get() and _PyGILState_GetInterpreterStateUnsafe(). */ #define _PyInterpreterState_GET_UNSAFE() (_PyThreadState_GET()->interp) > Python has a *very large* C API. It doesn't seem worth it to me to modify > every single Python function to detect when the API is misused. Understood, happy for this to be closed. Aware that I was misusing the API :-) -- ___ Python tracker <https://bugs.python.org/issue36386> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36420] f_trace_opcodes setting and accessing opcodes
New submission from anthony shaw : The f_trace_opcodes flag for sys.settrace in 3.7 are proving tricky. I must be missing something but it's not clear how it helps in tracing the opcode about to be executed because it runs before opcode and oparg variables are set by NEXTOPARG(), so the only way to establish the opcode is to look at the frame code and work out the next instruction in the stack. The documentation references dis, but if you call that for a traceback or using the frame code, you only have the last instruction, not the next one? def trace(frame, event, args): frame.f_trace_opcodes = True if event == 'opcode': disassemble(frame.f_code, frame.f_lasti) return frame It looks like the emitting of the opcode event needs to come after NEXTOPARG(), but that means if the tracing function were to add any instructions to the stack, that would no longer work. Alternatively, the opcode could be calculated and added as an argument. -- components: Interpreter Core messages: 338772 nosy: anthony shaw, ncoghlan priority: normal severity: normal status: open title: f_trace_opcodes setting and accessing opcodes versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue36420> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36420] f_trace_opcodes setting and accessing opcodes
Change by anthony shaw : -- resolution: -> not a bug stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue36420> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36420] f_trace_opcodes setting and accessing opcodes
anthony shaw added the comment:
Took a while, but I worked out a solution:
import sys
import dis
import traceback
import io
def t(frame, event, args):
frame.f_trace_opcodes=True
stack = traceback.extract_stack(frame)
pad = " "*len(stack) + "|"
if event == 'opcode':
with io.StringIO() as out:
dis.disco(frame.f_code, frame.f_lasti, file=out)
lines = out.getvalue().split('\n')
[print(f"{pad}{l}") for l in lines]
elif event == 'call':
print(f"{pad}Calling {frame.f_code}")
elif event == 'return':
print(f"{pad}Returning {args}")
elif event == 'line':
print(f"{pad}Changing line to {frame.f_lineno}")
else:
print(f"{pad}{frame} ({event} - {args})")
print(f"{pad}--")
return t
sys.settrace(t)
eval('"-".join([letter for letter in "hello"])')
--
___
Python tracker
<https://bugs.python.org/issue36420>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36500] Add "regen-*" equivalent projects for Windows builds
New submission from anthony shaw : Now that pgen is written in Python, it'd be useful for Windows users to be able to rebuild grammar and tokens into the parser table. The current hook (make regen-grammar) is built into the Makefile. Add support for VS2017+ vcxproj files to call the script directly -- components: Build messages: 339309 nosy: anthony shaw priority: normal severity: normal status: open title: Add "regen-*" equivalent projects for Windows builds versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue36500> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36500] Add "regen-*" equivalent projects for Windows builds
Change by anthony shaw : -- keywords: +patch pull_requests: +12583 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36500> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36500] Add "regen-*" equivalent projects for Windows builds
Change by anthony shaw : -- nosy: +brett.cannon, steve.dower ___ Python tracker <https://bugs.python.org/issue36500> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36500] Add "regen-*" equivalent projects for Windows builds
anthony shaw added the comment: Project now also does: regen-symbol (regenerate Lib/symbol.py) and regen-keyword (regenerate Lib/keyword.py) -- ___ Python tracker <https://bugs.python.org/issue36500> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36500] Add "regen-*" equivalent projects for Windows builds
Change by anthony shaw : -- nosy: +pablogsal ___ Python tracker <https://bugs.python.org/issue36500> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36551] Optimize list comprehensions with preallocate size and protect against overflow
New submission from anthony shaw : List comprehensions currently create a series of opcodes inside a code object, the first of which is BUILD_LIST with an oparg of 0, effectively creating a zero-length list with a preallocated size of 0. If you're doing a simple list comprehension on an iterator, e.g. def foo(): a = iterable return [x for x in a] Disassembly of at 0x109db2c40, file "", line 3>: 3 0 BUILD_LIST 0 2 LOAD_FAST0 (.0) >>4 FOR_ITER 8 (to 14) 6 STORE_FAST 1 (x) 8 LOAD_FAST1 (x) 10 LIST_APPEND 2 12 JUMP_ABSOLUTE4 >> 14 RETURN_VALUE The list comprehension will do a list_resize on the 4, 8, 16, 25, 35, 46, 58, 72, 88th iterations, etc. This PR preallocates the list created in a list comprehension to the length of the iterator using PyObject_LengthHint(). It uses a new BUILD_LIST_PREALLOC opcode which builds a list with the allocated size of PyObject_LengthHint(co_varnames[oparg]). [x for x in iterable] compiles to: Disassembly of at 0x109db2c40, file "", line 3>: 3 0 BUILD_LIST_PREALLOC 0 2 LOAD_FAST0 (.0) >>4 FOR_ITER 8 (to 14) 6 STORE_FAST 1 (x) 8 LOAD_FAST1 (x) 10 LIST_APPEND 2 12 JUMP_ABSOLUTE4 >> 14 RETURN_VALUE If the comprehension has ifs, then it will use the existing BUILD_LIST opcode Testing using a range length of 1 ./python.exe -m timeit "x=list(range(1)); [y for y in x]" Gives 392us on the current 3.8 branch and 372us with this change (about 8-10% faster) the longer the iterable, the bigger the impact. This change also catches the issue that a very large iterator, like a range object : [a for a in range(2**256)] Would cause the 3.8< interpreter to consume all memory and crash because there is no check against PY_SSIZE_MAX currently. With this change (assuming there is no if inside the comprehension) is now caught and thrown as an OverflowError: >>> [a for a in range(2**256)] Traceback (most recent call last): File "", line 1, in File "", line 1, in OverflowError: Python int too large to convert to C ssize_t -- components: Interpreter Core messages: 339586 nosy: anthony shaw, ncoghlan priority: normal severity: normal status: open title: Optimize list comprehensions with preallocate size and protect against overflow versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue36551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36551] Optimize list comprehensions with preallocate size and protect against overflow
Change by anthony shaw : -- keywords: +patch pull_requests: +12644 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36552] Replace OverflowError with ValueError when calculating length of range objects > PY_SIZE_MAX
New submission from anthony shaw : When calculating length of range() objects that have an r->length > PY_SIZE_MAX, the underlying PyLong_AsSsize_t() function will raise an OverflowError: >>> a = list(range(2**256)) Traceback (most recent call last): File "", line 1, in OverflowError: Python int too large to convert to C ssize_t >>> a = range(2**256) >>> len(a) Traceback (most recent call last): File "", line 1, in OverflowError: Python int too large to convert to C ssize_t This is expected behaviour, but to the average user, who won't know what ssize_t is, or what this has to do with Python int, the user message is confusing and OverflowError is the symptom but not the cause. The cause is that the length sent to range was in a value too large to calculate. This patch changes OverflowError to ValueError to hint to the user that the value sent to the range object constructor is too large. >>> a = list(range(2**256)) Traceback (most recent call last): File "", line 1, in ValueError: Range object too large to calculate length (Overflow Error) >>> a = range(2**256) >>> len(a) Traceback (most recent call last): File "", line 1, in ValueError: Range object too large to calculate length (Overflow Error) -- components: Library (Lib) messages: 339589 nosy: anthony shaw priority: normal severity: normal status: open title: Replace OverflowError with ValueError when calculating length of range objects > PY_SIZE_MAX versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue36552> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36552] Replace OverflowError with ValueError when calculating length of range objects > PY_SIZE_MAX
Change by anthony shaw : -- keywords: +patch pull_requests: +12646 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36552> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36551] Optimize list comprehensions with preallocate size and protect against overflow
Change by anthony shaw : -- type: -> enhancement ___ Python tracker <https://bugs.python.org/issue36551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36551] Optimize list comprehensions with preallocate size and protect against overflow
anthony shaw added the comment: The opcode would not solely apply to this specific use case. I could seek another way of implementing the same behaviour without an additional opcode? -- ___ Python tracker <https://bugs.python.org/issue36551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35488] pathlib Path.match does not behave as described
Change by anthony shaw : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue35488> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36551] Optimize list comprehensions with preallocate size and protect against overflow
anthony shaw added the comment:
> How about converting `[x for x in it]` to `[*it]` in AST?
I should have been more explicit, this patch improves the performance of all
list comprehensions that don’t have an if clause.
Not just
[x for x in y]
but:
d = {} # some sort of dictionary
[f”{k} — {v}” for k, v in d.items()]
a = iterable
[val**2 for val in a]
Would all use BUILD_LIST_PREALLOC and use a LengthHint.
I can do another speed test for those other scenarios.
Most of the stdlib packages have these sorts of list comps, including those in
the default site.py.
--
nosy: -pablogsal
___
Python tracker
<https://bugs.python.org/issue36551>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36551] Optimize list comprehensions with preallocate size and protect against overflow
anthony shaw added the comment: > This might cause a MemoryError when the __length_hint__ of the source returns > a too large value, even when the actual size of the comprehension is smaller, > e.g.: The current implementation of list comprehensions raise neither a memoryerror or overflow error. They will consume all available memory and crash the interpreter. This patch raises an OverflowError before execution instead of just looping until memory heap exhaustion -- ___ Python tracker <https://bugs.python.org/issue36551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36551] Optimize list comprehensions with preallocate size and protect against overflow
anthony shaw added the comment: > This patch makes it slow for small iterators That is a one-off cost for the __length_hint__ of the range object specifically. Objects with a known length (lists, sets, tuples) would not have that overhead. I can run a more useful set of benchmarks against this. So the +0.6us would be the same for ranges 8-16. Then less for 16-25, then again for 25-35 as the removal of the reallocation process has a more significant factor for larger ranges. -- nosy: -pablogsal ___ Python tracker <https://bugs.python.org/issue36551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36551] Optimize list comprehensions with preallocate size and protect against overflow
anthony shaw added the comment: > In such case, current behavior works. And your patch will raise > OverflowError. Try [x for x in range(2**1000)] in a REPL. It doesn’t raise anything, it tries to create a list that will eventually exceed PY_SIZE_MAX, but it only crashes once it reaches that iteration. This raises an OverflowError instead, the same way: len(range(2**1000)) raises an OverflowError -- ___ Python tracker <https://bugs.python.org/issue36551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36551] Optimize list comprehensions with preallocate size and protect against overflow
anthony shaw added the comment: > If your patch uses __length_hint__, it is bug. iterator will return 2**1000 for __length_hint__, but produce no item on iteration. It raises an OverflowError because of the goto https://github.com/python/cpython/pull/12718/files#diff-7f17c8d8448b7b6f90549035d2147a9fR2493 this could just as easily set size to 0. I put `goto error` given the opportunity to handle an expected fault gracefully. -- ___ Python tracker <https://bugs.python.org/issue36551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36551] Optimize list comprehensions with preallocate size and protect against overflow
anthony shaw added the comment: > If your patch uses __length_hint__, it is bug. I’m not sure I understand this comment, PEP424 says “This is useful for presizing containers when building from an iterable.“ This patch uses __length_hint__ to presize the list container for a list comprehension. -- ___ Python tracker <https://bugs.python.org/issue36551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36551] Optimize list comprehensions with preallocate size and protect against overflow
anthony shaw added the comment: Have just optimized some of the code and pushed another change as 69dce1c552. ran both master and 69dce1c552 using pyperformance with PGO: ➜ ~ python3.8 -m perf compare_to master.json 69dce1c552.json --table +-+-+-+ | Benchmark | master | 69dce1c552 | +=+=+=+ | 2to3| 432 ms | 426 ms: 1.02x faster (-2%) | +-+-+-+ | chaos | 157 ms | 155 ms: 1.01x faster (-1%) | +-+-+-+ | crypto_pyaes| 154 ms | 153 ms: 1.00x faster (-0%) | +-+-+-+ | dulwich_log | 123 ms | 124 ms: 1.00x slower (+0%) | +-+-+-+ | fannkuch| 603 ms | 600 ms: 1.01x faster (-1%) | +-+-+-+ | float | 153 ms | 154 ms: 1.01x slower (+1%) | +-+-+-+ | go | 323 ms | 326 ms: 1.01x slower (+1%) | +-+-+-+ | hexiom | 13.6 ms | 13.5 ms: 1.01x faster (-1%) | +-+-+-+ | json_dumps | 18.1 ms | 17.9 ms: 1.01x faster (-1%) | +-+-+-+ | logging_format | 13.2 us | 13.8 us: 1.05x slower (+5%) | +-+-+-+ | logging_silent | 266 ns | 280 ns: 1.05x slower (+5%) | +-+-+-+ | logging_simple | 12.4 us | 13.1 us: 1.06x slower (+6%) | +-+-+-+ | meteor_contest | 145 ms | 132 ms: 1.10x faster (-9%) | +-+-+-+ | nbody | 179 ms | 172 ms: 1.04x faster (-4%) | +-+-+-+ | nqueens | 138 ms | 134 ms: 1.03x faster (-3%) | +-+-+-+ | pathlib | 56.4 ms | 55.6 ms: 1.01x faster (-1%) | +-+-+-+ | pickle | 15.0 us | 15.4 us: 1.03x slower (+3%) | +-+-+-+ | pickle_pure_python | 620 us | 617 us: 1.01x faster (-1%) | +-+-+-+ | raytrace| 696 ms | 691 ms: 1.01x faster (-1%) | +-+-+-+ | regex_compile | 242 ms | 243 ms: 1.00x slower (+0%) | +-+-+-+ | scimark_monte_carlo | 140 ms | 143 ms: 1.02x slower (+2%) | +-+-+-+ | scimark_sparse_mat_mult | 5.90 ms | 5.94 ms: 1.01x slower (+1%) | +-+-+-+ | spectral_norm | 194 ms | 196 ms: 1.01x slower (+1%) | +-+-+-+ | sympy_str | 246 ms | 245 ms: 1.00x faster (-0%) | +-+-+-+ | telco | 8.42 ms | 8.31 ms: 1.01x faster (-1%) | +-+-+-+ | unpack_sequence | 59.2 ns | 59.7 ns: 1.01x slower (+1%) | +-+-+-+ | unpickle| 21.2 us | 21.4 us: 1.01x slower (+1%) | +-+-+-+ | unpickle_list | 5.73 us | 5.81 us: 1.01x slower (+1%) | +-+-+-+ | unpickle_pure_python| 471 us | 467 us: 1.01x faster (-1%) | +-+-+-+ | xml_etree_iterparse | 142 ms | 143 ms: 1.01x slower (+1%) | +-+-+-+ | xml_etree_generate | 139 ms | 137 ms: 1.02x faster (-2%) | +-+-+-+ | xml_etree_process | 109 ms | 108 ms: 1.01x faster (-1%) | +-+-+-+ Not significant (21): deltablue; django_template; html5lib; json_loads; mako; pickle_dict; pickle_list; pidigits; python_startup; python_startup_no_site; regex_dna; regex_effbot; regex_v8
[issue36551] Optimize list comprehensions with preallocate size and protect against overflow
anthony shaw added the comment: >I am -1 for this optimization because it affects only one particular case >(neither other kinds of comprehensions, nor generator expressions, nor list >comprehensions with conditions) and even in this case it is small. It is possible to add a lot of other optimizations for other cases which will sped up them to 50% or 100%, but we do not do this, because every such optimization has a cost. It increases the amount of code which should be maintained and covered by tests, it adds small overhead in common cases to speed up an uncommon case, and increasing the code base can negatively affect surrounding code (just because the CPU cache and registers are used inappropriate and the compiler optimizes less important paths). Understood, I had hoped this change would have a broader impact. The additional opcode is not ideal either. > In addition, while this change speed up list comprehensions for long list, it > slows down them for short lists. Short lists are more common. I've been profiling this today, basically, this implementation receives the `list_iter`, `range_iter`, etc. There is no standard object model for an iterator's length, _PyObject_HasLen would return false because it neither implements tp_as_sequence nor, tp_as_mapping (rightly so). What this has uncovered (so hopefully there's some value from this whole experience!) is that __length_hint__ for iterators is _really_ inefficient. Take a list_iterator for example: PyObject_LengthHint will call, _PyObject_HasLen, which returns false, which then goes to call _PyObject_LookupSpecial, then _PyObject_CallNoArg, which calls listiter_len, which calls PyList_GET_SIZE which returns a Py_ssize_t, which is then converted to a PyLong via PyLong_FromSsize_t, which is then returned back to PyObject_LengthHint, which then PyLong_AsSsize_t is run to convert the PyLong back into a Py_ssize_t The Py_ssize_t is then finally returned to the caller! My conclusion was that the list comprehension should be initialized to the length of the target, before GET_ITER is run. This would remove the overhead for range objects, because you could simply call _PyObject_HasLen, which would return true for dict, list, tuple and set, but false for range objects (which is what you want). The issue is that GET_ITER is called outside the code object for the comprehension, so you'd have to pass an additional argument to the comprehension generator. This is way outside of my expertise, but the only way I can see to find a sizeable benefit, with minimal code and no edge cases which are slower. Thanks for your time -- ___ Python tracker <https://bugs.python.org/issue36551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36782] Add tests for the datetime C API
anthony shaw added the comment: Reserving this issue (in mentored sprint) -- nosy: +anthony shaw ___ Python tracker <https://bugs.python.org/issue36782> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36782] Add tests for the datetime C API
Change by anthony shaw : -- nosy: +edison.abahurire ___ Python tracker <https://bugs.python.org/issue36782> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35534] SIGSEGV in stackdepth_walk
anthony shaw added the comment: stackdepth_walk no longer exists in Python 3, this is a 2.7 specific issue. opcode_stack_effect in compile.c will have the expected stack effect. If there is a mismatch between the Opcodes compiled with the distribution and the ones in the compile.c opcode_stack_effect, it can cause a crash. Ensure that `make regen-all` is run before compiling with a clean build to avoid this. -- nosy: +anthony shaw versions: -Python 3.6 ___ Python tracker <https://bugs.python.org/issue35534> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34214] Pylint recusion stack overflow abort
anthony shaw added the comment: I agree with Jerome, that handling recursion errors is a part of a 3rd party package and not a bug with CPython. -- nosy: +anthony shaw ___ Python tracker <https://bugs.python.org/issue34214> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31372] Add SSLSocket.get_verify_result()
anthony shaw added the comment: Please review the PR again, some of the code no longer applies as it targets master and the PR is quite old. -- nosy: +anthony shaw ___ Python tracker <https://bugs.python.org/issue31372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32430] Simplify Modules/Setup{,.dist,.local}
anthony shaw added the comment: PR GH-5062 is still open in GitHub, is this PR still required? -- nosy: +anthony shaw ___ Python tracker <https://bugs.python.org/issue32430> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31841] Several methods of collections.UserString do not return instances of UserString or its subclasses
anthony shaw added the comment: This issue has been open for some time and the PR references an upstream branch that no longer exists. Can this request be closed, or has the conversation yet to be resolved? -- nosy: +anthony shaw ___ Python tracker <https://bugs.python.org/issue31841> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15987] Provide a way to compare AST nodes for equality recursively
anthony shaw added the comment: This discussion is inconclusive and targets an old version of CPython, can this issue be closed? -- nosy: +anthony shaw ___ Python tracker <https://bugs.python.org/issue15987> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15987] Provide a way to compare AST nodes for equality recursively
anthony shaw added the comment: Closing issue, PR branch has since been removed and targets Python 3.4 -- nosy: +anthonypjshaw resolution: -> out of date stage: test needed -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue15987> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24263] unittest cannot load module whose name starts with Unicode
anthony shaw added the comment: The original PR refers to a branch that no longer exists, but the behaviour documented still applies to master. There were some changes to the test loader, but none that fixed this issue. -- nosy: +anthonypjshaw ___ Python tracker <https://bugs.python.org/issue24263> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34820] binascii.c:1578:1: error: the control flow of function ‘binascii_crc32’ does not match its profile data (counter ‘arcs’)
anthony shaw added the comment: There is not enough information to triage this issue and since the last request there has been no reply. Closing issue. -- nosy: +anthonypjshaw stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue34820> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
