[Python-Dev] Re: name for new Enum decorator
On Thu, May 27, 2021 at 8:30 PM Ethan Furman wrote: > But what if we have something like: > > class Color(Flag): > RED = 1# 0001 > BLUE = 4 # 0100 > WHITE = 7 # 0111 > > As you see, WHITE is an "alias" for a value that does not exist in the > Flag (0010, or 2). That seems like it's probably an error. Are there use cases where you want to support gaps in the bits? If not, your decorator could accept which flags should be spanned. That seems useful, too. It would be a strictly stronger check. That would change the kind of name you want, though. Otherwise, some things that occur to me: factorable, factorizable, factorized, decomposable. The thinking here is that each named member should be capable of being decomposed / factored into individual values that themselves have names. --Chris > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/S6232IQIUAZKRXNPDPCS755LCVAFBE37/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] New pythoncapi_compat project adding Python 3.10 support to your C extensions without losing Python 2.7-3.9 support
Hi, What do you think of promoting the pythoncapi_compat project that I'm introducing below in the "C API: Porting to Python 3.10" section of What's New In Python 3.10? Should this project be moved under the GitHub psf organization to have a more "future proof" URL? I would like to promote this project to prepare C extensions maintainers for the following incompatible C API change (Py_TYPE) that I would like to push into Python 3.11: https://github.com/python/cpython/pull/26493 (Py_REFCNT was already converted to a static inline function in Python 3.10.) I already made this Py_TYPE change in Python 3.10, but I had to revert it since it broke too many projects. Since last year, I upgraded most of these broken projects, I created the pythoncapi_compat project, and I succeeded to use upgrade_pythoncapi.py script and copy the pythoncapi_compat.h header file in multiple C extensions. C extensions written with Cython are not affected. I already fixed Cython last year to emit code compatible with my incoming incompatible change. If it's not done yet, you only have to regenerate the C files using a recent Cython version. -- I wrote a new script which adds Python 3.10 support to your C extensions without losing Python 2.7 support: https://github.com/pythoncapi/pythoncapi_compat To add Python 3.10 support to your C extension, go to its source directory and run: /path/to/upgrade_pythoncapi.py . It upgrades all C files (.c) in the current directory and subdirectories. For example, it replaces "op->ob_type" with "Py_TYPE(op)". It creates an ".old" copy of patched files. Use the -o option to select operations: * -o Py_TYPE: only replace "obj->ob_type" with "Py_TYPE(obj)". * -o all,-PyMem_MALLOC: run all operations, but don't replace PyMem_MALLOC(...) with PyMem_Malloc(...). -- The upgrade_pythoncapi.py script relies on the pythoncapi_compat.h header file that I wrote to provide recent Python 3.9-3.11 C functions on old Python versions. Examples: Py_NewRef() and PyThreadState_GetFrame(). Functions are implemented as simple static inline functions to avoid requiring to link your extension to a dynamic library. You can already use the new Py_NewRef() and Py_IsNone() Python 3.10 functions in your projects without losing support for Python 2.7-3.9! -- The script also replaces "frame->f_back" with "_PyFrame_GetBackBorrow(frame)". The _PyFrame_GetBackBorrow() function doesn't exist in the Python C API, it's only provided by pythoncapi_compat.h to ease the migration of C extensions. I advise you to replace _PyFrame_GetBackBorrow() (borrowed reference) with PyFrame_GetBack() (strong reference). -- This project is related to my PEP 620 "Hide implementation details from the C API" which tries to make the C API more abstract to later allow to implement new optimization in CPython and to make other Python implementations like PyPy faster when running C extensions. Article on the creation of the pythoncapi project: https://vstinner.github.io/pythoncapi_compat.html The main drawback of this project is that it uses regular expressions to parse C code. Such "parser" can miss C code which has to be patched manually. In my experience, additional manual changes are really rare and take less than 1 minute on a very large C extension like numpy. -- This project only targets extension modules written in C by using directly the "Python.h" API. I advise you to use Cython or HPy to no longer be bothered with incompatible C API changes at every Python release ;-) * https://cython.org/ * https://hpy.readthedocs.io/ I hope that my script will facilitate migration of C extensions to HPy. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/KHDZGCNOYEDUTSPAATUDP55ZSSQM5RRC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: New pythoncapi_compat project adding Python 3.10 support to your C extensions without losing Python 2.7-3.9 support
On Thu, Jun 3, 2021 at 2:30 AM Joannah Nanjekye
wrote:
> Is HPy ready yet given IIRC, there is no even first release yet? I stand to
> be corrected.
> I reckon it will still go through a period of incompatible changes for some
> time/months too.
HPy is a great project, but even if 90% of top 4000 PyPI extensions
are converted to it, CPython will likely still want to support the
long tail of old C extensions written directly with the Python C API
("Python.h"). My PEP 620 addresses the specific case of CPython which
wants to maximize its backward compatibility. It's a trade-off between
optimizations/major code rewrite and backward compatibility. Sadly,
you have to break eggs to make an omelet (french expression) :-)
I suggest to use Cython or HPy to write *new* C extensions, since they
don't depend (directly) on the C API. At a new Python version, just
update Cython/HPy, and you're good. I hope that tomorrow, Cython/HPy
will even support the limited C API and the stable ABI to not even
have to rebuild wheel packages! Cython already has an (experimental?)
option to restrict emitted C code to the limited C API.
--
I would say that HPy is not complete nor mature yet. IMO it's already
worth it to start playing with it, try to convert your small C
extensions, and send back your early feedback to HPy developers!
HPy links:
* Website: https://hpyproject.org/
* Blog: https://hpyproject.org/blog/
* Documentation: https://docs.hpyproject.org/
* GitHub: https://github.com/hpyproject/hpy/
Status:
* May 2021 (blog): https://hpyproject.org/blog/posts/2021/05/may-status-update/
* April 2021 (doc):
https://docs.hpyproject.org/en/latest/overview.html#current-status-and-roadmap
* March 2021 (blog): https://hpyproject.org/blog/posts/2021/03/hello-hpy/
There is a 0.0.1 release on GitHub (Git tag created last January):
https://github.com/hpyproject/hpy/releases/tag/0.0.1
On PyPI, the 0.0.1 tarball doesn't contain any source: it was only
created to reserve the name on PyPI.
Victor
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/DM3YRZVFYEB672ZINR7HDXCGH3RLVEQ6/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: New pythoncapi_compat project adding Python 3.10 support to your C extensions without losing Python 2.7-3.9 support
I like the idea but I don't understand the full impact yet. That's my job to assess. For my research, I was also trying to have a similar compatibility hack for the changes I have had to do to allow GC integration. > This project only targets extension modules written in C by using > directly the "Python.h" API. I advise you to use Cython or HPy to no > longer be bothered with incompatible C API changes at every Python > release ;-) Is HPy ready yet given IIRC, there is no even first release yet? I stand to be corrected. I reckon it will still go through a period of incompatible changes for some time/months too. On Wed, Jun 2, 2021 at 8:47 PM Victor Stinner wrote: > Hi, > > What do you think of promoting the pythoncapi_compat project that I'm > introducing below in the "C API: Porting to Python 3.10" section of > What's New In Python 3.10? > > Should this project be moved under the GitHub psf organization to have > a more "future proof" URL? > > I would like to promote this project to prepare C extensions > maintainers for the following incompatible C API change (Py_TYPE) that > I would like to push into Python 3.11: > https://github.com/python/cpython/pull/26493 > (Py_REFCNT was already converted to a static inline function in Python > 3.10.) > > I already made this Py_TYPE change in Python 3.10, but I had to revert > it since it broke too many projects. Since last year, I upgraded most > of these broken projects, I created the pythoncapi_compat project, and > I succeeded to use upgrade_pythoncapi.py script and copy the > pythoncapi_compat.h header file in multiple C extensions. > > C extensions written with Cython are not affected. I already fixed > Cython last year to emit code compatible with my incoming incompatible > change. If it's not done yet, you only have to regenerate the C files > using a recent Cython version. > > -- > > I wrote a new script which adds Python 3.10 support to your C > extensions without losing Python 2.7 support: > https://github.com/pythoncapi/pythoncapi_compat > > To add Python 3.10 support to your C extension, go to its source > directory and run: > > /path/to/upgrade_pythoncapi.py . > > It upgrades all C files (.c) in the current directory and > subdirectories. For example, it replaces "op->ob_type" with > "Py_TYPE(op)". It creates an ".old" copy of patched files. > > Use the -o option to select operations: > > * -o Py_TYPE: only replace "obj->ob_type" with "Py_TYPE(obj)". > * -o all,-PyMem_MALLOC: run all operations, but don't replace > PyMem_MALLOC(...) with PyMem_Malloc(...). > > -- > > The upgrade_pythoncapi.py script relies on the pythoncapi_compat.h > header file that I wrote to provide recent Python 3.9-3.11 C functions > on old Python versions. Examples: Py_NewRef() and > PyThreadState_GetFrame(). Functions are implemented as simple static > inline functions to avoid requiring to link your extension to a > dynamic library. > > You can already use the new Py_NewRef() and Py_IsNone() Python 3.10 > functions in your projects without losing support for Python 2.7-3.9! > > -- > > The script also replaces "frame->f_back" with > "_PyFrame_GetBackBorrow(frame)". > > The _PyFrame_GetBackBorrow() function doesn't exist in the Python C > API, it's only provided by pythoncapi_compat.h to ease the migration > of C extensions. I advise you to replace _PyFrame_GetBackBorrow() > (borrowed reference) with PyFrame_GetBack() (strong reference). > > -- > > This project is related to my PEP 620 "Hide implementation details > from the C API" which tries to make the C API more abstract to later > allow to implement new optimization in CPython and to make other > Python implementations like PyPy faster when running C extensions. > > Article on the creation of the pythoncapi project: > https://vstinner.github.io/pythoncapi_compat.html > > The main drawback of this project is that it uses regular expressions > to parse C code. Such "parser" can miss C code which has to be patched > manually. In my experience, additional manual changes are really rare > and take less than 1 minute on a very large C extension like numpy. > > -- > > This project only targets extension modules written in C by using > directly the "Python.h" API. I advise you to use Cython or HPy to no > longer be bothered with incompatible C API changes at every Python > release ;-) > > * https://cython.org/ > * https://hpy.readthedocs.io/ > > I hope that my script will facilitate migration of C extensions to HPy. > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my death. > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/KHDZGCNOYEDUTSPAATUDP55ZSSQM5RRC/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Best, J
[Python-Dev] Re: name for new Enum decorator
On 5/27/21 8:24 PM, Ethan Furman wrote: > So, like the enum.unique decorator that can be used when duplicate names should be an error, > I'm adding a new decorator to verify that a Flag has no missing aliased values that can be > used when the programmer thinks it's appropriate... but I have no idea what to call it. > > Any nominations? Thank you everyone for your ideas! Instead of adding another single-purpose decorator, I'm toying with the idea of adding a general purpose decorator that accepts instructions. Something along the lines of: class EnumChecks(StrEnum): """ various conditions to check an enumeration for """ UNIQUE = "one name per value" CONTINUOUS = "no skipped values" DECOMPOSABLE_ALIASES = "flag aliases must have all bits named" def verify(enumeration, *checks): if UNIQUE in checks: # ensure no duplicates if CONTINUOUS in checks: # ensure no missing values if DECOMPOSABLE_ALIASES in checks: # ensure all multi-flag aliases are composed of named flags Thoughts? -- ~Ethan~ ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/DVFOYTEPFNUDXWIYS763XGTCWJ6GFL5U/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: name for new Enum decorator
On 6/2/2021 7:59 PM, Ethan Furman wrote: On 5/27/21 8:24 PM, Ethan Furman wrote: > So, like the enum.unique decorator that can be used when duplicate names should be an error, > I'm adding a new decorator to verify that a Flag has no missing aliased values that can be > used when the programmer thinks it's appropriate... but I have no idea what to call it. > > Any nominations? Thank you everyone for your ideas! Instead of adding another single-purpose decorator, I'm toying with the idea of adding a general purpose decorator that accepts instructions. Something along the lines of: class EnumChecks(StrEnum): """ various conditions to check an enumeration for """ UNIQUE = "one name per value" CONTINUOUS = "no skipped values" DECOMPOSABLE_ALIASES = "flag aliases must have all bits named" def verify(enumeration, *checks): if UNIQUE in checks: # ensure no duplicates if CONTINUOUS in checks: # ensure no missing values if DECOMPOSABLE_ALIASES in checks: # ensure all multi-flag aliases are composed of named flags Thoughts? Seems more forward-looking and extensible rather than a proliferation of decorators. I like that. And the EnumChecks provides a way to make a tradeoff between short names (although DECOMPOSABLE_ALIASES isn't particularly short, it could be made shorter given the explanations) and the explanations of them in the text value. However, for DECOMPOSABLE_ALIASES, not only is the name not short, the explanation isn't particularly clear either, and it doesn't sound like it would properly explain the case where you want to have a mask that is larger than the number of currently used individual "flags" (which maybe shouldn't be called "bits", but "flags"). That explanation confuses the terminology between "masks" calling them "flags" and "flags" calling them bits. But maybe my understanding of what you originally meant the term "flag" to mean in the Enum context is not clear... in any case, a terminology should be used consistently, and the terminology should differentiate between at least 3 cases: 1. single bit items (flag?) 2. multi-bit values that are not a superset of a group of related single bit items (group?) e.g. PURPLE = RED | BLUE (not including GREEN) 3. multi-bit values that are intended as a mask, to include all related single bit items as well as possibly reserving space for future, yet-undefined, related single bit items. (mask?) and maybe also 4. multi-bit fields containing a number, rather than a set of individual bits (field?) ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/VPAMZXZFKJ5SVGUWSIV6DB5EBHXXLMCV/ Code of Conduct: http://python.org/psf/codeofconduct/
