[Python-Dev] PyPy on PySide6 is there: PyPy with a Gui
Hi Guido et. al., since May 2021 I have been working at running PyPy on PySide6, which was a difficult undertaking, since PyPy internally is quite a bit different. I declared the project to be ready-to-use when the Mandelbrot example of the PySide examples (examples/corelib/threads/mandelbrot.py) is working. This was finally solved this week on 2022-02-01, so we have the first advanced Gui working with PyPy and with the amazing result of speed: PyPy 3.8 works 10 times faster than the identical code on Python 3.10 and 5.5 times slower than the same example in C++ Qt. I think to send an official announce when this is available on pip. This effort marks the completion of my PyPy support, which began in 2003 and ended involuntarily in 2006 due to a stroke. All the best -- Chris -- Christian Tismer-Sperling:^) [email protected] Software Consulting : http://www.stackless.com/ Strandstraße 37 : https://github.com/PySide 24217 Schönberg : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/32XOK67BGMEX2UIYVVXMHOUME56O3GJ7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python
On 02. 02. 22 11:50, Stefan Behnel wrote: Petr Viktorin schrieb am 02.02.22 um 10:22: Moving off the internal (unstable) API would be great, but I don't think Cython needs to move all the way to the limited API. There are three "levels" in the C API: - limited API, with long-term ABI compatibility guarantees That's what "-DCYTHON_LIMITED_API -DPy_LIMITED_API=..." is supposed to do, which currently fails for much if not most code. - "normal" public API, covered by the backwards compatibility policy (users need to recompile for every minor release, and watch for deprecation warnings) That's probably close to what "-DCYTHON_LIMITED_API" does by itself as it stands. I can see that being a nice feature that just deserves a more suitable name. (The name was chosen because it was meant to also internally define "Py_LIMITED_API" at some point. Not sure if it will ever do that.) - internal API (underscore-prefixed names, `internal` headers, things documented as private) AFAIK, only the last one is causing trouble here. Yeah, and that's the current default mode on CPython. Beware that there are no guarantees on this API. It can change *at any time*. Technically, it can even change in point releases (only ABI must not change). We probably won't change it in a point release, especially not in a way that would break Cython, but it would still be great to move to the public API instead. Maybe we should advertise the two modes more. And make sure that both work. There are certainly issues with the current state of the "limited API" implementation, but that just needs work and testing. I wonder if it can it be renamed? "Limited API" has a specific meaning since PEP 384, and using it for the public API is adding to the general confusion in this area :( ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/CNIPFAWQY5KGV662SAY3LFMXEARYXVT2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
> On 2 Feb 2022, at 23:41, Eric Snow wrote: > > I […] > > Cons: > > * a little less convenient: adding a global string requires modifying > a separate file from the one where you actually want to use the string > * strings can get "orphaned" (I'm planning on checking in CI) > * some strings may never get used for any given ./python invocation > (not that big a difference though) The first two cons can probably be fixed by adding some indirection, with some markers at the place of use and a script that uses those to generate the C definitions. Although my gut feeling is that adding a the CI check you mention is good enough and adding the tooling for generating code isn’t worth the additional complexity. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/ ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/6EJSL7LBAZM4HL5THZDZGTYFS5HRAIPY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
On Wed, Feb 2, 2022 at 11:49 PM Eric Snow wrote: > FTR, here is the (private/internal) C-API affected by getting rid of > _Py_IDENTIFIER(): > > * 21 C-API functions with `_Py_Identifer` parameters - would be dropped >+ _PyUnicode_FromId() >+ _PyUnicode_EqualToASCIIId() >+ _PyObject_CallMethodId() >+ _PyObject_CallMethodId_SizeT() >+ _PyObject_CallMethodIdObjArgs() >+ _PyObject_VectorcallMethodId() >+ _PyObject_CallMethodIdNoArgs() >+ _PyObject_CallMethodIdOneArg() >+ _PyEval_GetBuiltinId() >+ _PyDict_GetItemId() >+ _PyDict_SetItemId() >+ _PyDict_DelItemId() >+ _PyDict_ContainsId() >+ _PyImport_GetModuleId() >+ _PyType_LookupId() >+ _PyObject_LookupSpecial() >+ _PyObject_GetAttrId() >+ _PyObject_SetAttrId() >+ _PyObject_LookupAttrId() >+ _PySys_GetObjectId() >+ _PySys_SetObjectId() I am well aware that we don't provide any backward compatibility warranty on the private and internal C API. *But*. People love to use they anyway. In the top 5000 PyPI projects, I found 11 projects using them: * Cython-0.29.26 (and so indirect most projects using Cython) * datatable-1.0.0 * frozendict-2.2.0 * multidict-6.0.2 * mypy-0.931 * pickle5-0.0.12 * pysqlite3-0.4.6 * ruamel.ordereddict-0.4.15 * scandir-1.10.0 * typed_ast-1.5.2 * zodbpickle-2.2.0 They use the these 17 functions: * _PyDict_ContainsId() * _PyDict_DelItemId() * _PyDict_GetItemId() * _PyDict_GetItemIdWithError() * _PyDict_SetItemId() * _PyEval_GetBuiltinId() * _PyObject_CallMethodId() * _PyObject_CallMethodIdNoArgs() * _PyObject_CallMethodIdObjArgs() * _PyObject_CallMethodIdOneArg() * _PyObject_GetAttrId() * _PyObject_LookupAttrId() * _PyObject_LookupSpecial() * _PyObject_SetAttrId() * _PySys_GetObjectId() * _PyUnicode_EqualToASCIIId() * _PyUnicode_FromId() If the _Py_IDENTIFIER() API is removed, it would be *nice* to provide a migrate path (tool?) to help these projects moving away the _Py_IDENTIFIER() API. Or at least do the work to update these 11 projects. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/7RMLIJHUWVBZFV747TFEHOE6LNBVQSMM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
On Wed, Feb 2, 2022 at 11:43 PM Eric Snow wrote: > My plan is to replace our use of _Py_IDENTIFIER() with statically > initialized string objects (as fields under _PyRuntimeState). That > involves the following: > > * add a PyUnicodeObject field (not a pointer) to _PyRuntimeState for > each string that currently uses _Py_IDENTIFIER() (or > _Py_static_string()) In bpo-39465, I made the _PyUnicode_FromId() compatible with running sub-interpreters in parallel (one GIL per interpreter). A "static" PyUnicodeObject would have to share the reference count between sub-interpreters, whereas Py_INCREF/Py_DECREF are not thread-safe: there is lock to prevent data races. Is there a way to push the "immortal objects" strategy discussed in bpo-40255? The deepfreeze already pushed some functions related to that, like _PyObject_IMMORTAL_INIT() in the internal C API. Moreover... deepfreeze already produces "immortal" PyUnicodeObject strings using the "ob_refcnt = 9" hack. IMO we should decide on a strategy. Either we move towards immortal objects (modify Py_INCREF/Py_DECREF to not modify the ref count if an object is immortal), or we make sure that no Python is shared between two Python interpreters. > I'd also like to actually get rid of _Py_IDENTIFIER(), along with > other related API including ~14 (private) C-API functions. Dropping > all that helps reduce maintenance costs. Is it required by your work on static strings, or is it more about removing the API which would no longer be consumed by Python itself? If it's not required, would it make sense to follow the PEP 387 deprecation (mark functions as deprecated, document the deprecation, and wait 2 releases to remove it)? Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/FFQ6N7K2BHG37JVFOACIEZLTKH43NV3L/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
By the way, Argument Clinic now produces way faster calling
conventions than hand-written METH_VARARGS with PyArg_ParseTuple(). It
would be make this tool available to 3rd party projects.
Either extract it and put it on PyPI, but it means that Python will
need to Python and pip install something to build itself... not good?
Or we can add it to stdlib. IMO we need to write more tests and more
documentation. Right now, test_clinic is "light". Another issue is
that it requires on the currently privte _PyArg_Parser structure. By
the way, this structure is causing crashes if sub-interpreters are run
in parallel (one GIL per interpreter) because of the
_PyArg_Parser.kwtuple tuple of keyword parameter names (tuple of str).
If Eric's tool becomes successful inside CPython, we may consider to
make it available to 3rd party projects as well. Such tool and
Argument Clinic and not so different than Cython which generates C
code to ease the development of C extensions modules.
Victor
On Thu, Feb 3, 2022 at 7:50 AM Inada Naoki wrote:
>
> +1 for overall.
>
> On Thu, Feb 3, 2022 at 7:45 AM Eric Snow wrote:
> >
> >
> > I'd also like to actually get rid of _Py_IDENTIFIER(), along with
> > other related API including ~14 (private) C-API functions. Dropping
> > all that helps reduce maintenance costs. However, at least one PyPI
> > project (blender) is using _Py_IDENTIFIER(). So, before we could get
> > rid of it, we'd first have to deal with that project (and any others).
> >
>
> It would be nice to provide something similar to _PY_IDENTIFIER, but
> designed (and documented) for 3rd party modules like this.
>
> ```
> typedef struct {
> Py_IDENTIFIER(foo);
> ...
> } Modstate;
> ...
> // in some func
> Modstate *state = (Modstate*)PyModule_GetState(module);
> PyObject_GetAttr(o, PY_IDENTIFIER_STR(state->foo));
> ...
> // m_free()
> static void mod_free(PyObject *module) {
> Modstate *state = (Modstate*)PyModule_GetState(module);
> Py_IDENTIFIER_CLEAR(state->foo);
> }
> ```
>
>
> --
> Inada Naoki
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/ZZ5QOZDOAO734SDRJGMXW6AJGAVEPUHE/
> Code of Conduct: http://python.org/psf/codeofconduct/
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/QNP7RDER74XU3DNHJP5EAZY6G6N4VYE7/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone
Hi Guido, My "north star", as you say, is the HPy "design" (not the actual HPy API). I would like to convert PyObject* to opaque handles: dereferencing a PyObject* pointer would simply fail with a compiler error. I'm working bottom-to-top: prepare PyObject and PyVarObject to become opaque, *and* top-to-bottom: prepare subclasses (structures "inheriting" from PyObject and PyVarObject) to become opaque like PyFrameObject. IMO if PyObject* becomes a handle, the migration to the HPy API should be much easier. So far, a large part of the work has been identifying which APIs are preventing to move the current C API to opaque handles. The second part has been fixing these APIs one by one, starting with changes which don't break the C API. In the last 5 years, I fixed many issues of the C API. Very few projects were impacted since I avoided incompatible changes on purpose. Now I reached the end of my (current) queue with the most controversial changes: incompatible C API changes. I wrote PEP 670 and 674 to share my overall rationale and communicate on the reasons why I consider that these changes will make our life easier next years. I'm well aware that in the short term, the benefits are very limited. But we cannot jump immediately to opaque PyObject* handles. This work must be done incrementally. On Thu, Feb 3, 2022 at 1:40 AM Guido van Rossum wrote: >> 1) First, my main worry is that we put a high pressure on maintainers >> of most important Python dependencies before the next of a new Python >> version, because we want them to handle the flow of incompatible C API >> changes before the final Python 3.x versions is released, to get them >> available when Python 3.x final is released. > > Hm, maybe we should reduce the flow. And e.g. reject PEP 674... We need to find a the right limit when introducing C API to not break too many C extensions "per Python release". Yes, PEP 674 is an example of incompatible C API change, but Python 3.11 already got many others which don't are unrelated to that PEP. For example, the optimization work done by your Microsoft team broke a bunch of projects using PyThreadState and PyFrameObject APIs. See the related What's New in Python 3.11 entries: https://docs.python.org/dev/whatsnew/3.11.html#id2 (I didn't keep track of which projects are affected by these changes.) I would like to ensure that it remains possible to optimize Python, but for that, we need to reduce the friction related to C API changes. What's the best approach for that remains an open question. I'm proposing some solutions, we are discussing advantages and drawbacks ;-) >> It annoys core developers who cannot change things in Python without >> getting an increasing number of complains about a large number of >> broken packages, sometimes with a request to revert. > > You are mostly talking about yourself here, right? Since the revert requests > were mostly aimed at you. :-) Latest requests for revert are not about C API changes that I made. Cython and the C API exception change: https://mail.python.org/archives/list/[email protected]/thread/RS2C53LDZPXHRR2VCY2G2YSPDVA4LNQU/ There are other requests for revert in Python 3.11 related to Python changes, not to the C API. So far, I only had to revert 2 changes about my C API work: * PyType_HasFeature(): the change caused a performance regression on macOS, sadly Python cannot be built with LTO. With LTO (all platforms but macOS), my change doesn't affect performances. * Py_TYPE() / Py_SIZE() change: I reverted my change to have more time to prepare affected projects. Two years later, I consider that this preparation work is now done (all affected projects are ready) and so I submitted the PEP 674. Affected projects (merged and pending fixes): https://www.python.org/dev/peps/pep-0674/#backwards-compatibility >> Moreover, it became common to ask multiple >> changes and multiple releases before a Python final release, since >> more incompatible changes are introduced in Python (before the beta1). > > Sorry, your grammar confuses me. Who is asking whom to do what here? Cython is the best example. During the development cycle of a new Python version, my Fedora team adapts Cython to the next Python and then request a release to get an official release supporting an alpha Python version. A release is not strictly required by us (we can apply downsteam patches), but it's more convenient for us and for people who cannot use our work-in-progress "COPR" (repository of RPM packages specific to Fedora). When we ask a project (using Cython) to merge our pull request, maintainers want to test it on the current Python alpha version, and it's not convenient when Cython is unusable. Between Python 3.x alpha1 and Python 3.x final, there might be multiple Cython releases to handle each time a bunch of C API incompatible changes. On Python and Cython sides, so far, there was no coordination to group incompatible changes, they land between alpha1 and beta1 in a
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
Victor Stinner wrote: > On Wed, Feb 2, 2022 at 11:49 PM Eric Snow [email protected] wrote: > In the top 5000 PyPI projects, I found 11 projects using them: > * Cython-0.29.26 (and so indirect most projects using Cython) I believe Cython is (for once) a false alarm here. I don't think it uses any of those functions. It has a comment that contains "_PyObject_LookupAttrId" - https://github.com/cython/cython/blob/8d2df028bf9536942b60670bf0aa80d6acc7464a/Cython/Utility/ObjectHandling.c#L1061 - This is a bit of code that we'd adapted from CPython and the comment just explains what the original line is. It's possible I've missed something of course, in which case let me know. David ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/5WICJJ5GYJE54JKPG56UY33YHHFQROHX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone
On Thu, Feb 3, 2022 at 9:27 AM Victor Stinner wrote: > Hi Guido, > [SNIP] > > On Thu, Feb 3, 2022 at 1:40 AM Guido van Rossum wrote: > > [SNIP] > > > > Maybe we need to help there. For example IIRC conda-forge will build > conda packages -- maybe we should offer a service like that for wheels? > > Tooling to automate the creation of wheel binary packages targeting > the stable ABI would help. C extensions likely need changing a few > function calls which are not supported by the limited C API. Someone > has to do this work. > The idea of having a service to build wheels for folks is an old one that I think a ton of people would benefit from. Currently, people typically get pointed to https://pypi.org/project/cibuildwheel/ as the per-project solution. But designing a safe way to build wheels from any sdist on PyPI, keeping such a service up, having the free processing to do it, etc. is unfortunately a big enough project that no one has stepped forward to try and tackle it. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/JYSDC7EVKPAITBEE4JJR6WNIMNBQYY44/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
On Thu, Feb 3, 2022 at 6:37 AM Victor Stinner wrote:
> By the way, Argument Clinic now produces way faster calling
> conventions than hand-written METH_VARARGS with PyArg_ParseTuple(). It
> would be make this tool available to 3rd party projects.
>
> Either extract it and put it on PyPI, but it means that Python will
> need to Python and pip install something to build itself... not good?
>
Since we check in the generated source you only need Python installed while
developing CPython, not to build it. And since that's already a requirement
to build the docs, freeze code, etc. I don't think that's an unreasonable
requirement.
We also don't have to break Argument Clinic out of our repo just to make it
available externally. There's nothing saying we can't have a subdirectory
of code that we directly rely on and publish separately for our development
needs. Could even add blurb into the repo that way if we wanted to.
-Brett
>
> Or we can add it to stdlib. IMO we need to write more tests and more
> documentation. Right now, test_clinic is "light". Another issue is
> that it requires on the currently privte _PyArg_Parser structure. By
> the way, this structure is causing crashes if sub-interpreters are run
> in parallel (one GIL per interpreter) because of the
> _PyArg_Parser.kwtuple tuple of keyword parameter names (tuple of str).
>
> If Eric's tool becomes successful inside CPython, we may consider to
> make it available to 3rd party projects as well. Such tool and
> Argument Clinic and not so different than Cython which generates C
> code to ease the development of C extensions modules.
>
> Victor
>
> On Thu, Feb 3, 2022 at 7:50 AM Inada Naoki wrote:
> >
> > +1 for overall.
> >
> > On Thu, Feb 3, 2022 at 7:45 AM Eric Snow
> wrote:
> > >
> > >
> > > I'd also like to actually get rid of _Py_IDENTIFIER(), along with
> > > other related API including ~14 (private) C-API functions. Dropping
> > > all that helps reduce maintenance costs. However, at least one PyPI
> > > project (blender) is using _Py_IDENTIFIER(). So, before we could get
> > > rid of it, we'd first have to deal with that project (and any others).
> > >
> >
> > It would be nice to provide something similar to _PY_IDENTIFIER, but
> > designed (and documented) for 3rd party modules like this.
> >
> > ```
> > typedef struct {
> > Py_IDENTIFIER(foo);
> > ...
> > } Modstate;
> > ...
> > // in some func
> > Modstate *state = (Modstate*)PyModule_GetState(module);
> > PyObject_GetAttr(o, PY_IDENTIFIER_STR(state->foo));
> > ...
> > // m_free()
> > static void mod_free(PyObject *module) {
> > Modstate *state = (Modstate*)PyModule_GetState(module);
> > Py_IDENTIFIER_CLEAR(state->foo);
> > }
> > ```
> >
> >
> > --
> > Inada Naoki
> > ___
> > Python-Dev mailing list -- [email protected]
> > To unsubscribe send an email to [email protected]
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/[email protected]/message/ZZ5QOZDOAO734SDRJGMXW6AJGAVEPUHE/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/QNP7RDER74XU3DNHJP5EAZY6G6N4VYE7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/NUPJY6G3OYEMYEAUA7BJX665A6PHDHW6/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
On Thu, Feb 3, 2022 at 6:31 AM Victor Stinner wrote: > On Wed, Feb 2, 2022 at 11:43 PM Eric Snow > wrote: > > My plan is to replace our use of _Py_IDENTIFIER() with statically > > initialized string objects (as fields under _PyRuntimeState). That > > involves the following: > > > > * add a PyUnicodeObject field (not a pointer) to _PyRuntimeState for > > each string that currently uses _Py_IDENTIFIER() (or > > _Py_static_string()) > > In bpo-39465, I made the _PyUnicode_FromId() compatible with running > sub-interpreters in parallel (one GIL per interpreter). > > A "static" PyUnicodeObject would have to share the reference count > between sub-interpreters, whereas Py_INCREF/Py_DECREF are not > thread-safe: there is lock to prevent data races. > > Is there a way to push the "immortal objects" strategy discussed in > bpo-40255? The deepfreeze already pushed some functions related to > that, like _PyObject_IMMORTAL_INIT() in the internal C API. > Moreover... deepfreeze already produces "immortal" PyUnicodeObject > strings using the "ob_refcnt = 9" hack. > > IMO we should decide on a strategy. Either we move towards immortal > objects (modify Py_INCREF/Py_DECREF to not modify the ref count if an > object is immortal), or we make sure that no Python is shared between > two Python interpreters. > Either we come to an agreement as a group (for against the concept and then how to do it), or if there's a stalemate it goes to the SC (with potentially a PEP to properly explain the nuances of the proposed solution). -Brett > > > > I'd also like to actually get rid of _Py_IDENTIFIER(), along with > > other related API including ~14 (private) C-API functions. Dropping > > all that helps reduce maintenance costs. > > Is it required by your work on static strings, or is it more about > removing the API which would no longer be consumed by Python itself? > > If it's not required, would it make sense to follow the PEP 387 > deprecation (mark functions as deprecated, document the deprecation, > and wait 2 releases to remove it)? > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my death. > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/FFQ6N7K2BHG37JVFOACIEZLTKH43NV3L/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/VFI6XGSCY2PVGHEHHDJY5QOT7MKM4Q5P/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
On Wed, Feb 2, 2022 at 2:48 PM Eric Snow wrote: > I'm planning on moving us to a simpler, more efficient alternative to > _Py_IDENTIFIER(), but want to see if there are any objections first > before moving ahead. Also see https://bugs.python.org/issue46541. > > _Py_IDENTIFIER() was added in 2011 to replace several internal string > object caches and to support cleaning up the cached objects during > finalization. A number of "private" functions (each with a > _Py_Identifier param) were added at that time, mostly corresponding to > existing functions that take PyObject* or char*. Note that at present > there are several hundred uses of _Py_IDENTIFIER(), including a number > of duplicates. > > My plan is to replace our use of _Py_IDENTIFIER() with statically > initialized string objects (as fields under _PyRuntimeState). That > involves the following: > > * add a PyUnicodeObject field (not a pointer) to _PyRuntimeState for > each string that currently uses _Py_IDENTIFIER() (or > _Py_static_string()) > * statically initialize each object as part of the initializer for > _PyRuntimeState > * add a macro to look up a given global string > * update each location that currently uses _Py_IDENTIFIER() to use the > new macro instead > > Pros: > > * reduces indirection (and extra calls) for C-API functions that need > the strings (making the code a little easier to understand and > speeding it up) > * the objects are referenced from a fixed address in the static data > section instead of the heap (speeding things up and allowing the C > compiler to optimize better) > * there is no lazy allocation (or lookup, etc.) so there are fewer > possible failures when the objects get used (thus less error return > checking) > * saves memory (at little, at least) > * if needed, the approach for per-interpreter is simpler > * helps us get rid of several hundred static variables throughout the code > base > * allows us to get rid of _Py_IDENTIFIER() and a bunch of related > C-API functions > * "deep frozen" modules can use the global strings > * commonly-used strings could be pre-allocated by adding > _PyRuntimeState fields for them > > Cons: > > * a little less convenient: adding a global string requires modifying > a separate file from the one where you actually want to use the string > * strings can get "orphaned" (I'm planning on checking in CI) > * some strings may never get used for any given ./python invocation > (not that big a difference though) > > I have a PR up (https://github.com/python/cpython/pull/30928) that > adds the global strings and replaces use of _Py_IDENTIFIER() in our > code base, except for in non-builtin stdlib extension modules. (Those > will be handled separately if we proceed.) The PR also adds a CI > check for "orphaned" strings. It leaves _Py_IDENTIFIER() for now, but > disallows any Py_BUILD_CORE code from using it. > > With that change I'm seeing a 1% improvement in performance (see > https://github.com/faster-cpython/ideas/issues/230). > > I'd also like to actually get rid of _Py_IDENTIFIER(), along with > other related API including ~14 (private) C-API functions. Dropping > all that helps reduce maintenance costs. However, at least one PyPI > project (blender) is using _Py_IDENTIFIER(). So, before we could get > rid of it, we'd first have to deal with that project (and any others). > datapoint: an internal code search turns up blender, multidict, and typed_ast as open source users of _Py_IDENTIFIER . Easy to clean up as PRs. There are a couple of internal uses as well, all of which are similarly easy to address and are only in code that is expected to need API cleanup tweaks between CPython versions. Overall I think addressing the broader strategy question among the performance focused folks is worthwhile though. -gps > > To sum up, I wanted to see if there are any objections before I start > merging anything. Thanks! > > -eric > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/DNMZAMB4M6RVR76RDZMUK2WRLI6KAAYS/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/BCQJ6AZMPTI2DGFQPC27RUIFJQGDIOQD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
Oh right, Cython seems to be a false positive.
A code search found 3 references to __Pyx_PyObject_LookupSpecial():
PYPI-2022-01-26-TOP-5000/Cython-0.29.26.tar.gz:
Cython-0.29.26/Cython/Compiler/ExprNodes.py: lookup_func_name =
'__Pyx_PyObject_LookupSpecial'
PYPI-2022-01-26-TOP-5000/Cython-0.29.26.tar.gz:
Cython-0.29.26/Cython/Compiler/Nodes.py: code.putln("%s =
__Pyx_PyObject_LookupSpecial(%s, %s); %s" % (
PYPI-2022-01-26-TOP-5000/Cython-0.29.26.tar.gz:
Cython-0.29.26/Cython/Utility/ObjectHandling.c: static CYTHON_INLINE
PyObject* __Pyx_PyObject_LookupSpecial(PyObject* obj, PyObject*
attr_name) {
Oh, that's not "_PyObject_LookupSpecial()", it doesn't use the
_Py_Identifier type:
static CYTHON_INLINE PyObject*
__Pyx_PyObject_LookupSpecial(PyObject* obj, PyObject* attr_name)
{ ... }
Victor
On Thu, Feb 3, 2022 at 7:27 PM wrote:
>
> Victor Stinner wrote:
> > On Wed, Feb 2, 2022 at 11:49 PM Eric Snow [email protected] wrote:
>
> > In the top 5000 PyPI projects, I found 11 projects using them:
> > * Cython-0.29.26 (and so indirect most projects using Cython)
>
> I believe Cython is (for once) a false alarm here. I don't think it uses any
> of those functions.
>
> It has a comment that contains "_PyObject_LookupAttrId" -
> https://github.com/cython/cython/blob/8d2df028bf9536942b60670bf0aa80d6acc7464a/Cython/Utility/ObjectHandling.c#L1061
> - This is a bit of code that we'd adapted from CPython and the comment just
> explains what the original line is.
>
> It's possible I've missed something of course, in which case let me know.
>
> David
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/5WICJJ5GYJE54JKPG56UY33YHHFQROHX/
> Code of Conduct: http://python.org/psf/codeofconduct/
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/DXXEYI7YPZCQO6VAPLVNVVAPH4LZ4KVU/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
On Wed, Feb 2, 2022 at 11:50 PM Inada Naoki wrote: > It would be nice to provide something similar to _PY_IDENTIFIER, but > designed (and documented) for 3rd party modules like this. I suppose I'd like to know what the value of _Py_IDENTIFIER() is for 3rd party modules. They can already use PyUnicode_InternFromString() to get a "global" object and then store it in their module state. I would not expect _Py_IDENTIFIER() to provide much of an advantage over that. Perhaps I'm missing something? If there is a real benefit then we should definitely figure out a good public API for it (if the current private one isn't sufficient). I won't be authoring that PEP though. :) -eric ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/AVJKKWITJPHUQTE2IXDYBCTQTKVPZPD7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
On Thu, Feb 3, 2022 at 6:46 AM Ronald Oussoren wrote: > Although my gut feeling is that adding a the CI check you mention is good > enough and adding the tooling for generating code isn’t worth the additional > complexity. Yeah, I came to the same conclusion. :) -eric ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/FAMDFSTN5KR2Z7LOVTK5GGF6YKR6G65Z/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
On Thu, Feb 3, 2022 at 7:17 AM Victor Stinner wrote: > In the top 5000 PyPI projects, I found 11 projects using them: > [snip[ > They use the these 17 functions: Thanks! That is super helpful. > If the _Py_IDENTIFIER() API is removed, it would be *nice* to provide > a migrate path (tool?) to help these projects moving away the > _Py_IDENTIFIER() API. Or at least do the work to update these 11 > projects. If something like _Py_IDENTIFIER() provides genuine value then we should consider a proper public API. Otherwise I agree that we should work with those projects to stop using it. I guess either way they should stop using the "private" API. :) -eric ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/IGXAXUSDBKYOOVVFSAUYLE5R5TXVZT4A/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone
On Wed, Feb 2, 2022 at 4:46 PM Guido van Rossum wrote: A few notes in this: > Maybe we need to help there. For example IIRC conda-forge will build conda > packages -- maybe we should offer a service like that for wheels? > Yes, conda-forge used a complex CI system to build binaries conda packages for a variety of Python versions. And it does have some support for development versions. ONce a "feedstock" is developed, it's remarkably painless to get all the binaries up and available. I imagine someone could borrow a bunch of that code to make a system to build wheels. In fact, ther is the MAcPYthon org: https://github.com/MacPython Which began as a place to share building scripts for Mac binaries, but has expanded to build wheels for multiple platforms for the scipy stack. I don't know how it works these days -- I am no longer involved since I discovered conda, but they seem to have some nice stuff there -- perhaps it could be leveraged for more projects. However: one of the challenges for building C extensions is that they often depend on external C libs -- and that is exactly the problem that conda was built to address. So in a sense, a conda-forge-like auto-build system is inherently easier for conda packages than binary wheels. Which doesn't mean it couldn't be done -- just that the challenge of third party libs would need to be addressed. In any case, someone would have to do the work, as usual. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/JKD3W3CA2OYWTQJ75X5FXN44OSR7PSGK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
On Thu, Feb 3, 2022 at 2:50 PM Eric Snow
wrote:
> On Wed, Feb 2, 2022 at 11:50 PM Inada Naoki
> wrote:
> > It would be nice to provide something similar to _PY_IDENTIFIER, but
> > designed (and documented) for 3rd party modules like this.
>
> I suppose I'd like to know what the value of _Py_IDENTIFIER() is for
> 3rd party modules. They can already use PyUnicode_InternFromString()
> to get a "global" object and then store it in their module state. I
> would not expect _Py_IDENTIFIER() to provide much of an advantage over
> that. Perhaps I'm missing something?
>
> If there is a real benefit then we should definitely figure out a good
> public API for it (if the current private one isn't sufficient). I
> won't be authoring that PEP though. :)
>
Why not read through some of that code and see what they are doing with it?
I imagine one advantage is that _Py_IDENTIFIER() can be used entirely local
to a function. E.g. (from _operator.c):
_Py_IDENTIFIER(partial);
functools = PyImport_ImportModule("functools");
if (!functools)
return NULL;
partial = _PyObject_GetAttrId(functools, &PyId_partial);
That's convenient since it means they don't have to pass module state
around.
--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
On Thu, Feb 3, 2022 at 7:26 AM Victor Stinner wrote: > In bpo-39465, I made the _PyUnicode_FromId() compatible with running > sub-interpreters in parallel (one GIL per interpreter). > > A "static" PyUnicodeObject would have to share the reference count > between sub-interpreters, whereas Py_INCREF/Py_DECREF are not > thread-safe: there is lock to prevent data races. Yeah, if we end up not being able to safely share the global string objects between interpreters then we would move them under PyInterpreterState. Currently I'm putting them under _PyRuntimeState. Doing that might reduce the performance benefits a little, since Py_GET_GLOBAL_STRING() would have to look up the interpreter to use (or we'd have to pass it in). That doesn't seem like much of a penalty though and doesn't impact the other benefits of the change. > Is there a way to push the "immortal objects" strategy discussed in > bpo-40255? I'm planning on circling back to that next week. > The deepfreeze already pushed some functions related to > that, like _PyObject_IMMORTAL_INIT() in the internal C API. > Moreover... deepfreeze already produces "immortal" PyUnicodeObject > strings using the "ob_refcnt = 9" hack. Note we only set the value really high as a safety precaution since these objects are all statically allocated. Eddie Elizondo's proposal involves a number of other key points, including keeping the refcount from changing. > IMO we should decide on a strategy. Either we move towards immortal > objects (modify Py_INCREF/Py_DECREF to not modify the ref count if an > object is immortal), or we make sure that no Python is shared between > two Python interpreters. +1 The catch is that things get messier when we make some objects per-interpreter while others stay runtime-global. I'm going to write a bit more about this next week, but the best strategy will probably be to first consolidate all the global objects under _PyRuntimeState first and then move them to PyInterpreterState all at once when we can do it safely. > > I'd also like to actually get rid of _Py_IDENTIFIER(), along with > > other related API including ~14 (private) C-API functions. Dropping > > all that helps reduce maintenance costs. > > Is it required by your work on static strings, or is it more about > removing the API which would no longer be consumed by Python itself? It is definitely not required for that. Rather, we won't need it any more so we should benefit from getting rid of it. The only blocker is that some 3rd party modules are using it. > If it's not required, would it make sense to follow the PEP 387 > deprecation (mark functions as deprecated, document the deprecation, > and wait 2 releases to remove it)? If you think it's worth it. It's a private API. I'd rather work to get 3rd party modules off it and then move on sooner. -eric ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/WR33NVCSIHOMN5X7YGCL2DHNCBQGKWAU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Moving away from _Py_IDENTIFIER().
On Thu, Feb 3, 2022 at 4:01 PM Guido van Rossum wrote:
> Why not read through some of that code and see what they are doing with it?
Yep, I'm planning on it.
> I imagine one advantage is that _Py_IDENTIFIER() can be used entirely local
> to a function.
Yeah, they'd have to put something like this in their module init:
state->partial_str = PyUnicode_InternFromString("partial");
if (state->partial_str == NULL) {
return NULL;
}
> E.g. (from _operator.c):
>
> _Py_IDENTIFIER(partial);
> functools = PyImport_ImportModule("functools");
> if (!functools)
> return NULL;
> partial = _PyObject_GetAttrId(functools, &PyId_partial);
>
> That's convenient since it means they don't have to pass module state around.
I might call that cheating. :) For an extension module this means
they are storing a little bit of their state in the
runtime/interpreter state instead of in their module state. Is there
precedent for that with any of our other API?
Regardless, the status quo certainly is simpler (if they aren't
already using module state in the function). Without _Py_IDENTIFER()
it would look like:
functools = PyImport_ImportModule("functools");
if (!functools)
return NULL;
my_struct *state = (my_struct*)PyModule_GetState(module);
if (state == NULL) {
Py_DECREF(functools);
return NULL;
}
partial = PyObject_GetAttr(functools, state->partial_str);
If they are already using the module state in their function then the
code would be simpler:
functools = PyImport_ImportModule("functools");
if (!functools)
return NULL;
partial = PyObject_GetAttr(functools, state->partial_str);
-eric
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/4625QOLXLZAAU2XNXEQM5W2JWX3FH4VM/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] [RELEASE] Python 3.11.0a5 is available
We needed to tame some angry buildbots, but after a small fight, we won with just some scratches! Here you have a shiny new alpha release: Python 3.11.0a5. https://www.python.org/downloads/release/python-3110a5/ **This is an early developer preview of Python 3.11** # Major new features of the 3.11 series, compared to 3.10 Python 3.11 is still in development. This release, 3.11.0a5 is the fifth of seven planned alpha releases. Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process. During the alpha phase, features may be added up until the start of the beta phase (2022-05-06) and, if necessary, may be modified or deleted up until the release candidate phase (2022-08-01). Please keep in mind that this is a preview release and its use is **not** recommended for production environments. Many new features for Python 3.11 are still being planned and written. Among the new major new features and changes so far: * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include Fine-Grained Error Locations in Tracebacks * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups and except* * [PEP 673](https://www.python.org/dev/peps/pep-0673/) -- Self Type * [PEP 646](https://www.python.org/dev/peps/pep-0646/)-- Variadic Generics * The [Faster Cpython Project](https://github.com/faster-cpython) is already yielding some exciting results: this version of CPython 3.11 is ~12% faster on the geometric mean of the [PyPerformance benchmarks]( speed.python.org), compared to 3.10.0. * Hey, **fellow core developer,** if a feature you find important is missing from this list, let me know. The next pre-release of Python 3.11 will be 3.11.0a6, currently scheduled for Monday, 2022-02-28. # More resources * [Online Documentation](https://docs.python.org/3.11/) * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule * Report bugs at [https://bugs.python.org](https://bugs.python.org). * [Help fund Python and its community](/psf/donations/). # And now for something completely different In physics, the Poynting vector (Umov-Poynting vector) represents the directional energy flux (the energy transfer per unit area per unit time) or power flow of an electromagnetic field. It is named after its discoverer John Henry Poynting who first derived it in 1884. Oliver Heaviside also discovered it independently in the more general form that recognises the freedom of adding the curl of an arbitrary vector field to the definition. The Poynting vector is used throughout electromagnetics in conjunction with Poynting’s theorem, the continuity equation expressing conservation of electromagnetic energy, to calculate the power flow in electromagnetic fields. # We hope you enjoy those new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Your friendly release team, Pablo Galindo @pablogsal Ned Deily @nad Steve Dower @steve.dower ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/S4UDXUMVUHMEE36TBFAJPNTJF4TEUYPX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone
On 2/3/2022 12:15 PM, Victor Stinner wrote: I'm working bottom-to-top: prepare PyObject and PyVarObject to become opaque, *and* top-to-bottom: prepare subclasses (structures "inheriting" from PyObject and PyVarObject) to become opaque like PyFrameObject. IMO if PyObject* becomes a handle, the migration to the HPy API should be much easier. It seems to me that moving PyObject* to be a handle leaves you in a place very similar to HPy. So why not just focus on making HPy suitable for developing C extensions, leave the existing C API alone, and eventually abandon the existing C API? Eric ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/BWCNBP26BZU2SYLCBVCXXVBMYUTSHE27/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PyPy on PySide6 is there: PyPy with a Gui
This is very cool Chris -- thanks! One question: and with the amazing result of speed: > > PyPy 3.8 works > 10 times faster than the identical code on Python 3.10 > and > 5.5 times slower than the same example in C++ Qt. > Is this primarily Python-QT interaction? or computing the mandelbrot set, for which I would expect to see performance numbers like that. Anyway, really cool in any case -- a major step for PyPy. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/YQGKNFGJKPMHDJ6LEDHXMLZRNNBA4356/ Code of Conduct: http://python.org/psf/codeofconduct/
