[issue43498] "dictionary changed size during iteration" error in _ExecutorManagerThread
Thomas Petazzoni added the comment: I can confirm we are seeing the same issue when building Python 3.9 in the context of Buildroot. See http://autobuild.buildroot.net/results/ae6/ae6c4ab292589a4e4442dfb0a1286349a9bf4d29/build-end.log for an example build result. This happens since we have added 48-cores (96 threads) build machines to our build farm, which dramatically increased the build parallelism. -- nosy: +thomas-petazzoni ___ Python tracker <https://bugs.python.org/issue43498> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43498] "dictionary changed size during iteration" error in _ExecutorManagerThread
Thomas Petazzoni added the comment: For the record: we're seeing this issue ~50 times a day on our build infrastructure. -- ___ Python tracker <https://bugs.python.org/issue43498> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Thomas Petazzoni added the comment: No, because it's not simply about the size of the installed Python standard library: it's also about the number of dependencies to build before being able to build Python. For example, a normal Python installation requires OpenSSL, libncurses, and lots of other things. On many embedded systems, those are not needed. -- ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Thomas Petazzoni added the comment: But this expectation is not true: if dependencies are not available, Python silently disables the build of certain modules. So this story of making the standard library always has all the modules does not really stand. -- ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Changes by Thomas Petazzoni : -- type: -> enhancement ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
New submission from Thomas Petazzoni: In the context of space-constrained embedded Linux systems, installing the entire set of Python modules and extensions is not necessarily desirable. For example, all the test modules, as well as certain extensions requiring third-parties libraries are often unnecessary, and uselessly consume precious storage on the embedded Linux system. While we could certainly remove these undesired modules and extensions manually, it is much more convenient to have configuration options to selectively enable and disable them. Another very strong benefit of having configuration options is that we can actually *disable* the build of these unneeded modules and extensions, therefore saving a lot of build time, which is very nice when you're repeatedly cross-compiling an entire embedded Linux system. The proposed set of patches add several --enable-/--disable- options to enable/disable certain Python modules and extensions. These patches have been part of Buildroot, an embedded Linux build system (used for example by Google, and many embedded processor vendors, as well as a huge number of embedded system makers) for a while, and are useful to all our users using Python on their embedded Linux systems. Instead of carrying them around, we would like to have them merged in upstream Python. Of course, we are definitely open to discussion on the approach to take to implement this configurability, and I'm ready to rework the patches according to the comments received here. Thanks! -- components: Build messages: 207802 nosy: thomas-petazzoni priority: normal severity: normal status: open title: Provide configure options to enable/disable Python modules and extensions versions: Python 3.5 ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Changes by Thomas Petazzoni : -- keywords: +patch Added file: http://bugs.python.org/file33380/0001-Add-infrastructure-to-disable-the-build-of-certain-e.patch ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Changes by Thomas Petazzoni : Added file: http://bugs.python.org/file33382/0003-Add-an-option-to-disable-pydoc.patch ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Changes by Thomas Petazzoni : Added file: http://bugs.python.org/file33383/0004-Add-an-option-to-disable-lib2to3.patch ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Changes by Thomas Petazzoni : Added file: http://bugs.python.org/file33381/0002-Add-an-option-to-disable-installation-of-test-module.patch ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Changes by Thomas Petazzoni : Added file: http://bugs.python.org/file33384/0005-Add-option-to-disable-the-sqlite3-module.patch ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Changes by Thomas Petazzoni : Added file: http://bugs.python.org/file33385/0006-Add-an-option-to-disable-the-tk-module.patch ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Changes by Thomas Petazzoni : Added file: http://bugs.python.org/file33387/0008-Add-an-option-to-disable-expat.patch ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Changes by Thomas Petazzoni : Added file: http://bugs.python.org/file33386/0007-Add-an-option-to-disable-the-curses-module.patch ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Changes by Thomas Petazzoni : Added file: http://bugs.python.org/file33389/0010-Add-an-option-to-disable-NIS.patch ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Changes by Thomas Petazzoni : Added file: http://bugs.python.org/file33391/0012-Add-an-option-to-disable-IDLE.patch ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Changes by Thomas Petazzoni : Added file: http://bugs.python.org/file33388/0009-Add-an-option-to-disable-CJK-codecs.patch ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Changes by Thomas Petazzoni : Added file: http://bugs.python.org/file33390/0011-Add-an-option-to-disable-unicodedata.patch ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20211] setup.py: do not add invalid header locations
New submission from Thomas Petazzoni: In the cross-compilation case, setup.py incorrectly adds /usr/include to self.compiler.include_dirs, and results in the following invalid compilation line: /home/thomas/projets/buildroot/output/host/usr/bin/arm-none-linux-gnueabi-gcc -fPIC -Wno-unused-result -Werror=declaration-after-statement -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -pipe -Os -I./Include -I/usr/include -I. -IInclude -I/home/thomas/projets/buildroot/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/include -I/home/thomas/projets/buildroot/output/build/python3-3.4.0b1/Include -I/home/thomas/projets/buildroot/output/build/python3-3.4.0b1 -c /home/thomas/projets/buildroot/output/build/python3-3.4.0b1/Modules/_struct.c -o build/temp.linux-arm-3.4/home/thomas/projets/buildroot/output/build/python3-3.4.0b1/Modules/_struct.o cc1: warning: include location "/usr/include" is unsafe for cross-compilation [-Wpoison-system-directories] The -I/usr/include is wrong when cross compiling, so we disable adding INCLUDEDIR and LIBDIR from the host when cross compiling. -- components: Cross-Build files: 0001-setup.py-do-not-add-invalid-header-locations.patch keywords: patch messages: 207805 nosy: thomas-petazzoni priority: normal severity: normal status: open title: setup.py: do not add invalid header locations type: compile error versions: Python 3.4 Added file: http://bugs.python.org/file33393/0001-setup.py-do-not-add-invalid-header-locations.patch ___ Python tracker <http://bugs.python.org/issue20211> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20212] distutils: fix build_ext check to find whether we're building Python or not
New submission from Thomas Petazzoni: The build_ext logic uses sys.executable.startswith(os.path.join(sys.exec_prefix, "bin")) to determine whether we're building a third-party Python extension, or a built-in Python extension. However, this check is wrong in cross-compilation mode, because the host Python interpreter might very well be installed in its prefix, when it is used to cross-compile the target modules and extensions. The current check would mis-detect this as we're building third-party Python modules, while we are in fact building the internal Python modules of the target Python. Therefore, use the existing sysconfig.python_build variable, which provides the information of whether we're building Python itself or not in a correct way. -- components: Cross-Build files: 0002-distutils-fix-build_ext-check-to-find-whether-we-re-.patch keywords: patch messages: 207806 nosy: thomas-petazzoni priority: normal severity: normal status: open title: distutils: fix build_ext check to find whether we're building Python or not type: compile error versions: Python 3.4 Added file: http://bugs.python.org/file33394/0002-distutils-fix-build_ext-check-to-find-whether-we-re-.patch ___ Python tracker <http://bugs.python.org/issue20212> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20213] Change the install location of _sysconfigdata.py
New submission from Thomas Petazzoni: The _sysconfigdata.py module contains definitions that are needed when building Python modules. In cross-compilation mode, when building Python extensions for the target, we need to use the _sysconfigdata.py of the target Python while executing the host Python. However until now, the _sysconfigdata.py module was installed in build/lib.- directory, together with a number of architecture-specific shared objects, which cannot be used with the host Python. To solve this problem, this patch moves _sysconfigdata.py to a separate location, build/sysconfigdata.-/, and only this directory gets added to the PYTHONPATH of the host Python interpreter when building Python modules for the target. -- components: Cross-Build files: 0003-Change-the-install-location-of-_sysconfigdata.py.patch keywords: patch messages: 207807 nosy: thomas-petazzoni priority: normal severity: normal status: open title: Change the install location of _sysconfigdata.py type: compile error versions: Python 3.4 Added file: http://bugs.python.org/file33395/0003-Change-the-install-location-of-_sysconfigdata.py.patch ___ Python tracker <http://bugs.python.org/issue20213> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20213] Change the install location of _sysconfigdata.py
Thomas Petazzoni added the comment: The patch is not working as it should be, so I withdraw it for the moment. -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue20213> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Thomas Petazzoni added the comment: Hello, On Sat, 04 Jun 2016 23:33:00 +, Matthias Klose wrote: > fyi, this came up again at the 2016 Language Summit, the request here > was to provide some minimal packages which are just enough to run a > cloud image (the cloud-init package requires a somehow minimal > python). Thanks for the feedback! Definitely glad to hear that some other folks are interested in reducing the footprint of the default Python installation. Since embedded people have been ignored, hopefully the cloud people will have more luck in this endeavor :) Thanks! Thomas -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com -- ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()
Thomas Petazzoni added the comment: The original problem is that Python wants to generate random numbers at *startup*. Are those random numbers really used for crypto-related activities? I doubt it. So isn't the proper solution to have two functions, one delivering random numbers that are usable for crypto-related activities, and which would potentially block, and a second one that delivers random numbers that are not appropriate for crypto stuff. This second function can be used at Python startup to replace what is done currently. It is most likely perfectly fine if Python blocks when explicitly asked to generate cryptographically secure random numbers. But not when simply starting the interpreter. -- ___ Python tracker <http://bugs.python.org/issue26839> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26852] add the '--enable-legacy-pyc-files' option to configure
Thomas Petazzoni added the comment: I can also say that in the Buildroot project, we have patches to get rid of the PEP3147 stuff. Indeed, the PEP3147 forces one to have both the .py *and* the .pyc file for a given module. If you have only the .pyc file, Python does not recognize it and it cannot be imported. By removing the PEP3147 functionality, we are able to keep only the .pyc files on the target, therefore dividing roughly by two the size of the Python installation. -- ___ Python tracker <http://bugs.python.org/issue26852> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26852] add the '--enable-legacy-pyc-files' option to configure
Thomas Petazzoni added the comment: See https://git.buildroot.org/buildroot/tree/package/python3/0016-Add-importlib-fix-for-PEP-3147-issue.patch -- ___ Python tracker <http://bugs.python.org/issue26852> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27640] add the '--disable-test-suite' option to configure
Thomas Petazzoni added the comment: We have a similar patch in Buildroot (see https://git.buildroot.org/buildroot/tree/package/python3/0017-Add-an-option-to-disable-installation-of-test-module.patch) so we would be very happy to see this patch merged. Note that we also have many more patches to disable various parts of the Python standard installation (https://git.buildroot.org/buildroot/tree/package/python3/). So either we have one option for each feature (like we have implemented), or a more general config option --disable-feature=test,this,that. See also bug http://bugs.python.org/issue20210 which is related, and was also proposing a patch to disable the test suite. -- ___ Python tracker <http://bugs.python.org/issue27640> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26852] add the '--enable-legacy-pyc-files' option to configure
Thomas Petazzoni added the comment: No, if you remove PEP3147 (like the Buildroot patch does), and install only the .pyc files, you have a smaller Python installation, and nothing gets "re-generated", since what you already have are the .pyc files. The .py files are not even installed on the target system. -- ___ Python tracker <http://bugs.python.org/issue26852> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23767] Library and include paths not added when cross compiling on localized sytem
Thomas Petazzoni added the comment: I confirm seeing the same issue. This patch fixes it. -- nosy: +thomas-petazzoni ___ Python tracker <http://bugs.python.org/issue23767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25420] "import random" blocks on entropy collection on Linux with low entropy
Thomas Petazzoni added the comment:
I can confirm that I'm affected by the same issue. Booting a simple Linux
system on a Qemu ARM platform, the python startup hangs during 25 seconds due
to the call to getrandom(). I am not doing anything with Python, just starting
the Python interpreter:
# strace -t -o strace.log python
random: nonblocking pool is initialized
Python 3.5.0 (default, Dec 23 2015, 15:11:18)
[GCC 5.1.1 20150608] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
# grep -A 2 getrandom strace.log
14:43:50
getrandom("\245\362a=\305\32Z\263\364\352j\223\0017\302q\361M\336+\2722>[", 24,
0) = 24
14:44:35 ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0
14:44:35 mmap2(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x76baf000
As you can see, 25 seconds blocked due to the getrandom() system call. Makes
the Python interpreter not really usable anymore. I would understand if Python
would do when I need to generate cryptographically secure random numbers. But
at this point, I am just starting the interpreter, nothing else.
This is a regression from Python 3.4.3.
--
nosy: +thomas-petazzoni
___
Python tracker
<http://bugs.python.org/issue25420>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25420] "import random" blocks on entropy collection on Linux with low entropy
Thomas Petazzoni added the comment: Obviously I did my math wrong: it waits 45 seconds in getrandom(), not 25 seconds. See my strace log. -- ___ Python tracker <http://bugs.python.org/issue25420> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20210] Provide configure options to enable/disable Python modules and extensions
Thomas Petazzoni added the comment: @Mark I would be happy to, but if you refer to the previous discussion about this bug report, the feedback was quite negative. And since I'm not really willing to do some clean up to finally get the patches rejected, I'd like to at least have 1/ an agreement on the principle itself, and 2/ some guidance as to what needs to be changed in the currently proposed patches to get them accepted. -- ___ Python tracker <http://bugs.python.org/issue20210> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
