Re: [Python-Dev] Location of CI for Windows Embeddable Distro
I've successfully automated the build of the embedded zip on Windows for 3.6 on my internal CI using tools/msi script. Moving onto linux, can you please summarize the primary challenges/differences for creating the embeddable zip for linux? Gerald R. Wiltse [email protected] On Wed, May 1, 2019 at 2:47 PM Gerald Wiltse wrote: > Yes, I'll help with this, and I try to get signed up with an account for > the issue tracker ASAP. > > I've just started trying to build create the embeddable package for > windows locally (all from source). Once I have that automated, I'll take a > naive stab at the Linux layout. I'm sure i'll soon hit the additional > complications you're talking about and come back to you with questions. > > Gerald R. Wiltse > [email protected] > > > > On Wed, May 1, 2019 at 1:57 PM Steve Dower wrote: > >> On 01May2019 1305, Gerald Wiltse wrote: >> > That does help, thanks so much for the fast response! I now see how >> > it's executed on Azure pipelines also, so that should be enough for me >> > to reverse engineer the rest. >> > >> > Followup questions if you have time: >> > 1. It seems that directory doesn't exist in the 3.6 branches/tags, so >> it >> > must have been done a different way back then. Do you think it's >> > back-portable to 3.6? >> >> Yep, it should work with 3.6. There were scripts under Tools/msi back >> then that did the same thing, but with the Store app I rewrote it to be >> more flexible. >> >> > 2. One of our objectives is to effectively create a nearly identical >> > embeddable package for linux. Do you think the layout code is generic >> > enough to be modified to create a linux equivalent? >> >> Probably not, it's under the "PC" directory because it is entirely >> CPython specific. >> >> I'd love to be able to have a similar embeddable distro for Linux (and >> macOS), and I'll be investing some of my time in working towards this, >> but it's far more complicated than simply laying out the right files. If >> you're interested in helping with this, then we're in the right place to >> discuss it, but we'll likely have to move to the issue tracker at some >> point once we figure out who else wants/needs to be involved. >> >> > Obviously, there are many alternative methods and documents for >> creating >> > "embeddable" pythons on linux, but the windows package has all the >> > characteristics we want, and of course we'd like our Windows and Linux >> > process build and packaging processes to be as similar as possible, so >> > it seems attractive to try to port this layout script to linux. >> >> "make install" already does the same thing, but the install location >> matters at build time, so it's a little more complex to make it work as >> an embedded runtime. I'm sure there are people who know how to make it >> work though. >> >> Cheers, >> Steve >> > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files
On 2019-05-01, 06:46 GMT, Serhiy Storchaka wrote: > These lines were added for purpose. They are needed for > running tests in separate file as a script. > > $ ./python Lib/unittest/test/testmock/testcallable.py -v > test_attributes (__main__.TestCallable) ... ok Isn't the standard way how to run one module just? $ ./python -mtest -v testmock.testcallable Best, Matěj -- https://matej.ceplovi.cz/blog/, Jabber: [email protected] GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 As long as we are thinking of natural values we must say that the sun looks down on nothing half so good as a household laughing together over a meal, or two friends talking over a pint of beer, or a man alone reading a book that interests him; and that all economies, politics, laws, armies, and institutions, save insofar as they prolong and multiply such scenes, are a mere ploughing the sand and sowing the ocean, a meaningless vanity and vexation of the spirit. Collective activities are, of course, necessary, but this is the end to which they are necessary. -- C.S. Lewis, “Membership” in “The Weight of Glory” ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Location of CI for Windows Embeddable Distro
With minimal patching of tools/msi/make_zip.py, I was able to create an equivalent zip package on Linux. It doesn't work out of the box of course. I had to do two commands My biggest concern is that even if the layout is "correct" on linux, the python runtime might not actually support being run like this on Linux. Perhaps the code for using python36.zip as a landmark and source of modules is only implemented on Windows. Do you know if that is the case? tmp$ export LD_LIBRARY_PATH=$(pwd) tmp$ export PYTHONHOME=$(pwd) tmp$ export PYTHONPATH=$(pwd) tmp$ ./python Fatal Python error: Py_Initialize: Unable to get the locale encoding ModuleNotFoundError: No module named 'encodings' Current thread 0x7fdde9eb1680 (most recent call first): Aborted (core dumped) Here's the full directory structure, error, and patch I created. Any additional feedback would be great. https://gist.github.com/solvingj/87305e0f70e14d738d6989ba2f41352f Gerald R. Wiltse [email protected] On Thu, May 2, 2019 at 11:24 AM Gerald Wiltse wrote: > I've successfully automated the build of the embedded zip on Windows for > 3.6 on my internal CI using tools/msi script. > > Moving onto linux, can you please summarize the primary > challenges/differences for creating the embeddable zip for linux? > > Gerald R. Wiltse > [email protected] > > > > On Wed, May 1, 2019 at 2:47 PM Gerald Wiltse > wrote: > >> Yes, I'll help with this, and I try to get signed up with an account for >> the issue tracker ASAP. >> >> I've just started trying to build create the embeddable package for >> windows locally (all from source). Once I have that automated, I'll take a >> naive stab at the Linux layout. I'm sure i'll soon hit the additional >> complications you're talking about and come back to you with questions. >> >> Gerald R. Wiltse >> [email protected] >> >> >> >> On Wed, May 1, 2019 at 1:57 PM Steve Dower >> wrote: >> >>> On 01May2019 1305, Gerald Wiltse wrote: >>> > That does help, thanks so much for the fast response! I now see how >>> > it's executed on Azure pipelines also, so that should be enough for me >>> > to reverse engineer the rest. >>> > >>> > Followup questions if you have time: >>> > 1. It seems that directory doesn't exist in the 3.6 branches/tags, so >>> it >>> > must have been done a different way back then. Do you think it's >>> > back-portable to 3.6? >>> >>> Yep, it should work with 3.6. There were scripts under Tools/msi back >>> then that did the same thing, but with the Store app I rewrote it to be >>> more flexible. >>> >>> > 2. One of our objectives is to effectively create a nearly identical >>> > embeddable package for linux. Do you think the layout code is generic >>> > enough to be modified to create a linux equivalent? >>> >>> Probably not, it's under the "PC" directory because it is entirely >>> CPython specific. >>> >>> I'd love to be able to have a similar embeddable distro for Linux (and >>> macOS), and I'll be investing some of my time in working towards this, >>> but it's far more complicated than simply laying out the right files. If >>> you're interested in helping with this, then we're in the right place to >>> discuss it, but we'll likely have to move to the issue tracker at some >>> point once we figure out who else wants/needs to be involved. >>> >>> > Obviously, there are many alternative methods and documents for >>> creating >>> > "embeddable" pythons on linux, but the windows package has all the >>> > characteristics we want, and of course we'd like our Windows and Linux >>> > process build and packaging processes to be as similar as possible, so >>> > it seems attractive to try to port this layout script to linux. >>> >>> "make install" already does the same thing, but the install location >>> matters at build time, so it's a little more complex to make it work as >>> an embedded runtime. I'm sure there are people who know how to make it >>> work though. >>> >>> Cheers, >>> Steve >>> >> ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version
Hi,
Thanks to Steve Dower's feedback, I enhanced and completed my PEP 587.
Main changes:
* It is now possible to read the configuration and then modify the
read configuration. For example, new directories can be added to
PyConfig.module_search_paths (see the example below and the example in
the PEP)
* PyConfig is now "dynamic" by default: strings are duplicated and
PyConfig_Clear() must be called to release memory
* PyConfig now only uses wchar_t* for strings (unicode): char* (bytes)
is no longer used. I had to hack CPython internals for that :-)
* I added a "_config_version" private field to PyPreConfig and
PyConfig to prepare the backward compatibility for future changes.
* I removed the Open Question section: all known issues have been fixed.
During the Language Summit, Brett Cannon said that Steve Dower
declined the offer to be the BDFL-delegate for this PEP. Thomas
Wouters proposed himself to be the new BDFL-delegate.
Example to read the configuration, append a directory to sys.path
(module_search_paths) and then initialize Python with this
configuration:
void init_python(void)
{
PyInitError err;
PyConfig config = PyConfig_INIT;
err = PyConfig_Read(&config);
if (_Py_INIT_FAILED(err)) {
goto fail;
}
err = PyWideStringList_Append(&config.module_search_paths,
L"/path/to/more/modules");
if (_Py_INIT_FAILED(err)) {
goto fail;
}
err = Py_InitializeFromConfig(&config);
if (_Py_INIT_FAILED(err)) {
goto fail;
}
PyConfig_Clear(&config);
return;
fail:
PyConfig_Clear(&config);
Py_ExitInitError(err);
}
The HTML version will be online shortly:
https://www.python.org/dev/peps/pep-0587/
Full text below.
Victor
PEP: 587
Title: Python Initialization Configuration
Author: Nick Coghlan , Victor Stinner
Discussions-To: [email protected]
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Mar-2019
Python-Version: 3.8
Abstract
Add a new C API to configure the Python Initialization providing finer
control on the whole configuration and better error reporting.
Rationale
=
Python is highly configurable but its configuration evolved organically:
configuration parameters is scattered all around the code using
different ways to set them (mostly global configuration variables and
functions). A straightforward and reliable way to configure Python is
needed. Some configuration parameters are not accessible from the C API,
or not easily.
The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as
input whereas the Python filesystem encoding is set during the
initialization.
This PEP is a partial implementation of PEP 432 which is the overall
design. New fields can be added later to ``PyConfig`` structure to
finish the implementation of the PEP 432 (add a new partial
initialization which allows to configure Python using Python objects to
finish the full initialization).
Python Initialization C API
===
This PEP proposes to add the following new structures, functions and
macros.
New structures (4):
* ``PyConfig``
* ``PyInitError``
* ``PyPreConfig``
* ``PyWideStringList``
New functions (16):
* ``Py_PreInitialize(config)``
* ``Py_PreInitializeFromArgs(config, argc, argv)``
* ``Py_PreInitializeFromWideArgs(config, argc, argv)``
* ``PyWideStringList_Append(list, item)``
* ``PyConfig_DecodeLocale(config_str, str)``
* ``PyConfig_SetString(config_str, str)``
* ``PyConfig_Read(config)``
* ``PyConfig_SetArgv(config, argc, argv)``
* ``PyConfig_SetWideArgv(config, argc, argv)``
* ``PyConfig_Clear(config)``
* ``Py_InitializeFromConfig(config)``
* ``Py_InitializeFromArgs(config, argc, argv)``
* ``Py_InitializeFromWideArgs(config, argc, argv)``
* ``Py_UnixMain(argc, argv)``
* ``Py_ExitInitError(err)``
* ``Py_RunMain()``
New macros (9):
* ``PyPreConfig_INIT``
* ``PyConfig_INIT``
* ``Py_INIT_OK()``
* ``Py_INIT_ERR(MSG)``
* ``Py_INIT_NO_MEMORY()``
* ``Py_INIT_EXIT(EXITCODE)``
* ``Py_INIT_IS_ERROR(err)``
* ``Py_INIT_IS_EXIT(err)``
* ``Py_INIT_FAILED(err)``
This PEP also adds ``_PyRuntimeState.preconfig`` (``PyPreConfig`` type)
and ``PyInterpreterState.config`` (``PyConfig`` type) fields to these
internal structures. ``PyInterpreterState.config`` becomes the new
reference configuration, replacing global configuration variables and
other private variables.
PyWideStringList
``PyWideStringList`` is a list of ``wchar_t*`` strings.
Example to initialize a string from C static array::
static wchar_t* argv[2] = {
L"-c",
L"pass",
};
PyWideStringList config_argv = PyWideStringList_INIT;
config_argv.length = Py_ARRAY_LENGTH(argv);
config_argv.items = argv;
``PyWideStringList`` structure fields:
* ``length`` (``Py_ssize_t``)
* ``items`` (``wchar_t**``)
Methods:
* ``PyInitError PyWideStringLi
Re: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version
On Thursday, May 02, 2019 Victor Stinner wrote: > According to this > * ``run_command`` (``wchar_t*``): ``-c COMMAND`` argument > * ``run_filename`` (``wchar_t*``): ``python3 SCRIPT`` argument > * ``run_module`` (``wchar_t*``): ``python3 -m MODULE`` argument this > ``-c COMMAND````run_module = COMMAND`` should read "run_command = COMMAND". Typo, not? ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Location of CI for Windows Embeddable Distro
Just an update based on testing. If I zip the modules up into python36.zip (whether in .py or .pyc form) the linux binary does not find them. If I copy them into the embedded root, alongside the python executable, it finds them as expected. This just makes it seem more like an issue where Linux binary does not respect .zip file as a source of modules. Gerald R. Wiltse [email protected] On Thu, May 2, 2019 at 2:56 PM Gerald Wiltse wrote: > With minimal patching of tools/msi/make_zip.py, I was able to create an > equivalent zip package on Linux. It doesn't work out of the box of > course. I had to do two commands > > My biggest concern is that even if the layout is "correct" on linux, the > python runtime might not actually support being run like this on Linux. > Perhaps the code for using python36.zip as a landmark and source of modules > is only implemented on Windows. Do you know if that is the case? > > tmp$ export LD_LIBRARY_PATH=$(pwd) > tmp$ export PYTHONHOME=$(pwd) > tmp$ export PYTHONPATH=$(pwd) > tmp$ ./python > Fatal Python error: Py_Initialize: Unable to get the locale encoding > ModuleNotFoundError: No module named 'encodings' > > Current thread 0x7fdde9eb1680 (most recent call first): > Aborted (core dumped) > > Here's the full directory structure, error, and patch I created. Any > additional feedback would be great. > > https://gist.github.com/solvingj/87305e0f70e14d738d6989ba2f41352f > > Gerald R. Wiltse > [email protected] > > > > On Thu, May 2, 2019 at 11:24 AM Gerald Wiltse > wrote: > >> I've successfully automated the build of the embedded zip on Windows for >> 3.6 on my internal CI using tools/msi script. >> >> Moving onto linux, can you please summarize the primary >> challenges/differences for creating the embeddable zip for linux? >> >> Gerald R. Wiltse >> [email protected] >> >> >> >> On Wed, May 1, 2019 at 2:47 PM Gerald Wiltse >> wrote: >> >>> Yes, I'll help with this, and I try to get signed up with an account for >>> the issue tracker ASAP. >>> >>> I've just started trying to build create the embeddable package for >>> windows locally (all from source). Once I have that automated, I'll take a >>> naive stab at the Linux layout. I'm sure i'll soon hit the additional >>> complications you're talking about and come back to you with questions. >>> >>> Gerald R. Wiltse >>> [email protected] >>> >>> >>> >>> On Wed, May 1, 2019 at 1:57 PM Steve Dower >>> wrote: >>> On 01May2019 1305, Gerald Wiltse wrote: > That does help, thanks so much for the fast response! I now see how > it's executed on Azure pipelines also, so that should be enough for me > to reverse engineer the rest. > > Followup questions if you have time: > 1. It seems that directory doesn't exist in the 3.6 branches/tags, so it > must have been done a different way back then. Do you think it's > back-portable to 3.6? Yep, it should work with 3.6. There were scripts under Tools/msi back then that did the same thing, but with the Store app I rewrote it to be more flexible. > 2. One of our objectives is to effectively create a nearly identical > embeddable package for linux. Do you think the layout code is generic > enough to be modified to create a linux equivalent? Probably not, it's under the "PC" directory because it is entirely CPython specific. I'd love to be able to have a similar embeddable distro for Linux (and macOS), and I'll be investing some of my time in working towards this, but it's far more complicated than simply laying out the right files. If you're interested in helping with this, then we're in the right place to discuss it, but we'll likely have to move to the issue tracker at some point once we figure out who else wants/needs to be involved. > Obviously, there are many alternative methods and documents for creating > "embeddable" pythons on linux, but the windows package has all the > characteristics we want, and of course we'd like our Windows and Linux > process build and packaging processes to be as similar as possible, so > it seems attractive to try to port this layout script to linux. "make install" already does the same thing, but the install location matters at build time, so it's a little more complex to make it work as an embedded runtime. I'm sure there are people who know how to make it work though. Cheers, Steve >>> ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version
Le jeu. 2 mai 2019 à 16:20, Edwin Zimmerman a écrit : > > ``-c COMMAND````run_module = COMMAND`` > should read "run_command = COMMAND". Typo, not? Oops, you're right: it's a typo. Now fixed: ``-c COMMAND````run_command = COMMAND`` Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
