Re: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
On 1 August 2018 at 08:14, Victor Stinner wrote: > Hi, > > I finished my work on the _PyCoreConfig structure: it's a C structure > in Include/pystate.h which has many fields used to configure Python > initialization. In Python 3.6 and older, these parameters were scatted > around the code, and it was hard to get an exhaustive list of it. > > This work is linked to the Nick Coghlan's PEP 432 "Restructuring the > CPython startup sequence": > https://www.python.org/dev/peps/pep-0432/ > > Right now, the new API is still private. Nick Coghlan splitted the > initialization in two parts: "core" and "main". I'm not sure that this > split is needed. It is, because one of the aims is to make it clear when frozen bytecode (ala importlib) and other builtin modules can start to be used as part of the configuration process. That point is when the core configuration completes. By contrast, main interpreter configuration is only complete when you have full filesystem access as well (including external imports). That separation also means that an embedding application can choose *not* to proceed to the second step, and thus limit imports to the modules built in to the application. > We should see what to do, but it would be nice to > make the _PyCoreConfig API public! IMHO it's way better than the old > way to configuration Python initialization. Step 1 will be to bring PEP 432 up to date with what actually happened - we learned a lot from your and Eric's efforts in actually implementing the draft design as a private API, but the current PEP still reflects the original design concept. Thanks for all your work on this! It's exciting to see it finally coming to fruition :) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
On 2 August 2018 at 19:49, Victor Stinner wrote: > About that, I'm working on a subproject: abandon Py_xxx global > configuration variables to replace them with accessing > interpreter->core_config->xxx. I'm not sure yet if it's a good idea or > not, but it would allow to have two interpreters to have their own > different configuration. Imagine two interpreters with different > sys.path running in isolated mode. Or maybe an interpreter without > importlib? > > One of the issue is that we have now two copies of the same option. > For example, Py_BytesWarningFlag and > interpreter->core_config->bytes_warning. That's why I would like to > switch to core_config. One of the challenges we have around those is the backwards compatibility implications for embedding applications, so I suspect the earliest we'll be able to make that change is in the release after the new initialisation API becomes public. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
On 3 August 2018 at 01:59, Victor Stinner wrote: > 2018-08-02 1:18 GMT+02:00 Eric Snow : >> The "core" config is basically the config for the runtime. In fact, >> PEP 432 renamed "core" to "runtime". Please keep the firm distinction >> between the runtime and the (main) interpreter. > > There is already something called _PyRuntime but it's shared between > all interpreters. > > _PyCoreConfig is already *per* interpreter. > > Would you mind to elaborate what you mean by the "main interpreter"? I > don't see anything obvious in the current code about what is a "main > interpreter". Technically, I don't see anything like that. > > I'm still not convinced that we need _PyMainInterpreterConfig: > _PyCoreConfig contains the same information. Is it really worth it to > duplicate all _PyCoreConfig (more than 36 fields) in > _PyMainInterpreterConfig? _PyMainInterpreterConfig adds a third copy > of many paramters: another opportunity to introduce an inconsistency. > Right now, an interpreter contains both: core and main > configurations... The issue is massive scope creep in the "core config": currently you need to fully specify everything in order to even use the builtin data structs. That's not the design goal of PEP 432: the idea is to have an absolutely bare minimum set of settings that gives a working C API, but won't let you access the host operating system. I wasn't reigning you in on it because there were real problems to be solved in getting the multi-stage start up to work at all given the current code structure. Now that it's working though, we should be looking to move settings back out of coreconfig, and reducing the amount of startup work that needs to be done using raw C code that can't rely on the CPython C API. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
On 4 August 2018 at 08:37, Victor Stinner wrote: > It seems like the PEP 432 proposes an API designed from scratch as the > target API. I started from the 28 years old C code and I tried to > cleanup the code. Our method is different, so it's not surprising that > the result is different :-) No, you didn't start from the 28 year old C code - you started from the private initial implementation of PEP 432, as per https://www.python.org/dev/peps/pep-0432/#implementation-strategy (and PEP 432 in turn was born from the Python 3.3 changes needed to integrate importlib properly into the __main__ initialisation process) Eric and I merged that [1], because it become apparent while I was working on the core settings management framework that actually migrating individual settings needed to happen in-tree, as the alternative of continuing to maintain it out of tree would at best result in an enormous unreviewable patch, and more likely never result in a patch at all (as the churn rate on CPython was high enough to cause regular conflicts even with the framework branch, let alone once we started migrating individual settings). The current private API doesn't meet some of the original design goals of the PEP, but the time to reconcile that (and figure out whether it's the API design or the implementation that should change) is while we're updating and reviewing the PEP against the current implementation - while everything remained private, it didn't make sense to throw any additional roadblocks in the way of the excellent work you were doing. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Error message for wrong number of arguments
On 4 August 2018 at 05:34, Jeroen Demeyer wrote: > On 2018-07-30 17:28, Nick Coghlan wrote: >> >> I would, and I think it would make sense for the PEP to cite improving >> consistency (and reducing code duplication?) in that regard as an >> advantage of the PEP. > > > I'm not sure to which PEP you are referring (PEP 580 or a new PEP?). After > thinking a bit about the issue of error messages, I realized that PEP 580 > would make this easier to fix (to be clear: there are ways to fix it without > PEP 580, I'm just saying that PEP 580 makes it easier). There are two > related reasons for this: > > * The existing code which calls the actual underlying C function doesn't > have access to the Python-level function object. So it can't know whether > it's a function (where self doesn't count) or a method (where self counts). > > * Armin Rigo suggested to use a new flag to indicate this difference: that > would certainly work for Argument Clinic (just have Argument Clinic add that > flag). For methods defined without Argument Clinic, we cannot require such a > new flag though. We could still add the flag at runtime, but it's far from > clear if we can freely change the flags inside a PyMethodDef at runtime (at > least, no existing code that I know does that). > > PEP 580 solves the first issue by having the function object available and > it solves the second issue by not relying on PyMethodDef at all for calling > functions/methods. The second issue especially can be generalized as: PEP > 580 makes the implementation of functions/methods much less rigid, making it > easier to change the implementation. > > So maybe this can be seen as yet another advantage of PEP 580. Yes, this is the kind of framing of the issue that I had in mind :) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
On 31 July 2018 at 17:45, Antoine Pitrou wrote: > On Tue, 31 Jul 2018 09:27:03 +0200 > Jeroen Demeyer wrote: >> On 2018-07-31 08:58, Antoine Pitrou wrote: >> > I think Stefan is right that we >> > should push people towards Cython and alternatives, rather than direct >> > use of the C API (which people often fail to use correctly, in my >> > experience). >> >> I know this probably isn't the correct place to bring it up, but I'm >> sure that CPython itself could benefit from using Cython. For example, >> most of the C extensions in Modules/ could be written in Cython. > > We don't depend on any third-party Python modules. Adding a Cython > dependency for CPython development would be a tough sell. > > Also, a C extension can be built-in (linked statically into the > interpreter), which I think would be hard to do with Cython. It'd be *really* nice to at least be able to write some of the C API tests directly in Cython rather than having to fiddle about with splitting the test between the regrtest parts that actually define the test case and the extension module parts that expose the interfaces that we want to test. So just as we have a split between the core interpreter components needed to freeze importlib and the full Python interpreter, it could be very interesting to have a split between the builtin and extension modules that are required to bootstrap Cython, and those that can instead *rely* on Cython as part of their build process. While actually doing that would likely mean introducing a venv dependency into the later stages of the build process, we already have such a dependency for the docs build, and that seems to work OK. Cheers, Nick. P.S. Note that even though static linking Cython generated modules with CPython doesn't work right just now, experiments like https://mdqinc.com/blog/2011/08/statically-linking-python-with-cython-generated-modules-and-packages/ suggest that it shouldn't take too many adjustments to the build process to get it to work by default. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use of Cython
Hi Nick, Le 04/08/2018 à 15:13, Nick Coghlan a écrit : > > It'd be *really* nice to at least be able to write some of the C API > tests directly in Cython rather than having to fiddle about with > splitting the test between the regrtest parts that actually define the > test case and the extension module parts that expose the interfaces > that we want to test. Actually, I think testing the C API is precisely the kind of area where you don't want to involve a third-party, especially not a moving target (Cython is actively maintained and generated code will vary after each new Cython release). Besides, Cython itself calls the C API, which means you might end up involuntarily testing the C API against itself. If anything, testing the C API using ctypes or cffi would probably be more reasonable... assuming we get ctypes / cffi to compile everywhere, which currently isn't the case. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use of Cython
Antoine Pitrou schrieb am 04.08.2018 um 15:57: > Le 04/08/2018 à 15:13, Nick Coghlan a écrit : >> >> It'd be *really* nice to at least be able to write some of the C API >> tests directly in Cython rather than having to fiddle about with >> splitting the test between the regrtest parts that actually define the >> test case and the extension module parts that expose the interfaces >> that we want to test. > > Actually, I think testing the C API is precisely the kind of area where > you don't want to involve a third-party, especially not a moving target > (Cython is actively maintained and generated code will vary after each > new Cython release). Besides, Cython itself calls the C API, which > means you might end up involuntarily testing the C API against itself. > > If anything, testing the C API using ctypes or cffi would probably be > more reasonable... assuming we get ctypes / cffi to compile everywhere, > which currently isn't the case. I agree that you would rather not want to let Cython (or another tool) generate the specific code that tests a specific C-API call, but you could still use Cython to get around writing the setup, validation and unittest boilerplate code in C. Basically, a test could then look something like this (probably works, although I didn't test it): from cpython.object cimport PyObject from cpython.list cimport PyList_Append def test_PyList_Append_on_empty_list(): # setup code l = [] assert len(l) == 0 value = "abc" pyobj_value = value refcount_before = pyobj_value.ob_refcnt # conservative test call, translates to the expected C code, # although with exception propagation if it returns -1: errcode = PyList_Append(l, value) # validation assert errcode == 0 assert len(l) == 1 assert l[0] is value assert pyobj_value.ob_refcnt == refcount_before + 1 If you don't want the exception handling, you can define your own declaration of PyList_Append() that does not have it. But personally, I'd rather use try-except in my test code than manually taking care of cleaning up (unexpected) exceptions. What we do in Cython, BTW, is write doctests in compiled ".pyx" files. That allows us to execute certain parts of a test in Python (the doctest code) and other parts in Cython (the compiled functions/classes that have the doctests), and thus to do a direct comparison between Python and Cython. An example that you could find in a test ".pyx" file: def test_times2(x): """ doctest that gets executed by Python: >>> test_times2(3) == 3 * 2 True """ # Cython compiled code in a compiled function that gets tested: return x * 2 Given that CPython's current "_testcapimodule.c" is only a good 5000 lines long (divide that by the number of public C-API functions and macros!), I'm sure the above could help in improving the unit test coverage of the C-API quite quickly. Stefan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use of Cython
On 5 August 2018 at 00:46, Stefan Behnel wrote: > Antoine Pitrou schrieb am 04.08.2018 um 15:57: >> Actually, I think testing the C API is precisely the kind of area where >> you don't want to involve a third-party, especially not a moving target >> (Cython is actively maintained and generated code will vary after each >> new Cython release). Besides, Cython itself calls the C API, which >> means you might end up involuntarily testing the C API against itself. >> >> If anything, testing the C API using ctypes or cffi would probably be >> more reasonable... assuming we get ctypes / cffi to compile everywhere, >> which currently isn't the case. > > I agree that you would rather not want to let Cython (or another tool) > generate the specific code that tests a specific C-API call, but you could > still use Cython to get around writing the setup, validation and unittest > boilerplate code in C. Basically, a test could then look something like > this (probably works, although I didn't test it): > > from cpython.object cimport PyObject > from cpython.list cimport PyList_Append > > def test_PyList_Append_on_empty_list(): > # setup code > l = [] > assert len(l) == 0 > value = "abc" > pyobj_value = value > refcount_before = pyobj_value.ob_refcnt > > # conservative test call, translates to the expected C code, > # although with exception propagation if it returns -1: > errcode = PyList_Append(l, value) > > # validation > assert errcode == 0 > assert len(l) == 1 > assert l[0] is value > assert pyobj_value.ob_refcnt == refcount_before + 1 > > > If you don't want the exception handling, you can define your own > declaration of PyList_Append() that does not have it. But personally, I'd > rather use try-except in my test code than manually taking care of cleaning > up (unexpected) exceptions. Exactly, that's the kind of thing I had in mind. At the moment, writing a new dedicated C API test requires designing 4 things: 1. The test case itself (what action to take, which assertions to make about it) 2. The C code to make the API call you want to test 3. The Python->C interface for the test case from 1 to pass test values in to the code from 2 4. The C->Python interface to get state of interest from 2 back to the test case from 1 If we were able to use Cython to handle 3 & 4 rather than having to hand craft it for every test, then I believe it would significantly lower the barrier to testing the C API directly rather than only testing it indirectly through the CPython implementation. Having such a test suite available would then hopefully make it easier for other implementations to provide robust emulations of the public C API. ctypes & cffi likely wouldn't help as much in the case, since they don't eliminate the need to come up with custom code for parts 3 & 4, they just let you write that logic in Python rather than C. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
