[Python-Dev] Re: [python-committers] [RELEASE] Python 3.11.0a3 is available

[Victor Stinner]

Yeah, another Python 3.11 alpha release!

The asyncore, asynchat and smtpd modules are back into Python 3.11!
They were removed but the removal has been reverted to respect the PEP
387 process: have two Python versions (3.10 and 3.11) emitting a
DeprecationWarning.

In general, I strongly advice you running your project and its test
suite with "python3 -W default" or PYTHONWARNINGS=default environment
variable to see DeprecationWarning and PendingDeprecationWarning, or
even "python3 -W error" or PYTHONWARNINGS=error to treat warnings as
errors. You can also try the development mode which shows these
warnings, and more, use "python3 -X dev" or PYTHONDEVMODE=1:

   https://docs.python.org/dev/library/devmode.html

Victor


On Thu, Dec 9, 2021 at 12:52 AM Pablo Galindo Salgado
 wrote:
>
> You can tell that we are slowly getting closer to the first beta as the 
> number of release blockers that we need to fix on every release starts to 
> increase  But we did it! Thanks to Steve Dower, Ned Deily, Christian Heimes, 
> Łukasz Langa and Mark Shannon that helped get things ready for this release :)
>
> Go get the new version here:
>
> https://www.python.org/downloads/release/python-3110a3/
>
> **This is an early developer preview of Python 3.11**
>
> # Major new features of the 3.11 series, compared to 3.10
>
> Python 3.11 is still in development.  This release, 3.11.0a3 is the third of 
> seven planned alpha releases.
>
> Alpha releases are intended to make it easier to test the current state of 
> new features and bug fixes and to test the release process.
>
> During the alpha phase, features may be added up until the start of the beta 
> phase (2022-05-06) and, if necessary, may be modified or deleted up until the 
> release candidate phase (2022-08-01).  Please keep in mind that this is a 
> preview release and its use is **not** recommended for production 
> environments.
>
> Many new features for Python 3.11 are still being planned and written.  Among 
> the new major new features and changes so far:
>
> * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include 
> Fine-Grained Error Locations in Tracebacks
> * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups 
> and except*
> * The [Faster Cpython Project](https://github.com/faster-cpython) is already 
> yielding some exciting results: this version of CPython 3.11 is ~12% faster 
> on the geometric mean of the [PyPerformance benchmarks](speed.python.org), 
> compared to 3.10.0.
>  * Hey, **fellow core developer,** if a feature you find important is missing 
> from this list, let me know.
>
> The next pre-release of Python 3.11 will be 3.11.0a4, currently scheduled for 
> Monday, 2022-01-03.
>
> # More resources
>
> * [Online Documentation](https://docs.python.org/3.11/)
> * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule
> * Report bugs at [https://bugs.python.org](https://bugs.python.org).
> * [Help fund Python and its community](/psf/donations/).
>
> # And now for something completely different
>
> Rayleigh scattering, named after the nineteenth-century British physicist 
> Lord Rayleigh is the predominantly elastic scattering of light or other 
> electromagnetic radiation by particles much smaller than the wavelength of 
> the radiation. For light frequencies well below the resonance frequency of 
> the scattering particle, the amount of scattering is inversely proportional 
> to the fourth power of the wavelength. Rayleigh scattering results from the 
> electric polarizability of the particles. The oscillating electric field of a 
> light wave acts on the charges within a particle, causing them to move at the 
> same frequency. The particle, therefore, becomes a small radiating dipole 
> whose radiation we see as scattered light. The particles may be individual 
> atoms or molecules; it can occur when light travels through transparent 
> solids and liquids but is most prominently seen in gases.
>
> The strong wavelength dependence of the scattering means that shorter (blue) 
> wavelengths are scattered more strongly than longer (red) wavelengths. This 
> results in the indirect blue light coming from all regions of the sky.
>
> # We hope you enjoy those new releases!
>
> Thanks to all of the many volunteers who help make Python Development and 
> these releases possible! Please consider supporting our efforts by 
> volunteering yourself or through organization contributions to the Python 
> Software Foundation.
>
> Your friendly release team,
> Pablo Galindo @pablogsal
> Ned Deily @nad
> Steve Dower @steve.dower
> ___
> python-committers mailing list -- python-committ...@python.org
> To unsubscribe send an email to python-committers-le...@python.org
> https://mail.python.org/mailman3/lists/python-committers.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-committ...@python.org/message/UJULODWK44IYR7HIITSBUQ75YC33FJUC/
> 

[Python-Dev] Re: Clarification regarding Stable ABI and _Py_*

[Petr Viktorin]

On 08. 12. 21 18:06, Eric Snow wrote:

On Wed, Dec 8, 2021 at 2:23 AM Petr Viktorin  wrote:

That really depends on what function we'd want to remove. There are
usually alternatives to deleting things, but the options depend on the
function. If we run out of other options we can make the function always
fail or make it leak memory.
And the regular backwards compatibility policy gives us 2 years to
figure something out :)


What about the various symbols listed in Misc/stable_abi.txt that were
accidentally added to the limited API?  Can we move toward dropping
them from the stable ABI?

Most notably, there are quite a few functions listed there that are in
the stable ABI but no longer in the limited API.  This implies that
either they were already deprecated in the limited API (and removed)
or they were just removed.  At least in some cases they were moved to
header files in Include/cpython or Include/internal.  So I would not
expect extensions to be using them.  This subset of those symbols
seems entirely appropriate to remove from the stable ABI.  Is that
okay?  Do we even need to bother deprecating them?  What about just
the "private" ones?

For example, I went to change/remove _PyThreadState_Init() (internal
API declared in Include/internal/pycore_pystate.h) and found that it
is in the stable ABI but not the limited API.  It's highly unlikely
anyone is using it and plan on double-checking.  As far as I can tell,
the function was accidentally exposed in the limited API and stable
ABI and later removed from the limited API.


It's possible to remove them just like _PyObject_GC_Malloc was removed, 
but check that it was unusable (e.g. not called from public macros) in 
all versions of Python from 3.2 up to now.
Could you check if this PR makes things clear? 
https://github.com/python/devguide/pull/778


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/T3M5AE6DFB2C6JAMCAMAIC2XQDNOKND5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [RELEASE] Python 3.11.0a3 is available

[Mark Shannon]

On 08/12/2021 11:51 pm, Pablo Galindo Salgado wrote:


* The [Faster Cpython Project](https://github.com/faster-cpython 
) is already yielding some exciting results: this 
version of CPython 3.11 is ~12% faster on the geometric mean of the [PyPerformance 
benchmarks](speed.python.org ), compared to 3.10.0.


Actually, it is quite a lot better than that at 19% on the standard benchmark 
suite :)

https://gist.github.com/markshannon/0ddfb0b705d23b863477d7f7f9f00ef1


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BDJFYAC5IL77MBYQMH56D7ST67GDBMN2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [RELEASE] Python 3.11.0a3 is available

[Pablo Galindo Salgado]

Thanks, Mark for the catch! I have updated it in the release page and all
announcements where I can edit the text :)

On Thu, 9 Dec 2021 at 12:34, Mark Shannon  wrote:

>
>
> On 08/12/2021 11:51 pm, Pablo Galindo Salgado wrote:
>
> > * The [Faster Cpython Project](https://github.com/faster-cpython <
> https://github.com/faster-cpython>) is already yielding some exciting
> results: this version of CPython 3.11 is ~12% faster on the geometric mean
> of the [PyPerformance benchmarks](speed.python.org <
> http://speed.python.org>), compared to 3.10.0.
>
> Actually, it is quite a lot better than that at 19% on the standard
> benchmark suite :)
>
> https://gist.github.com/markshannon/0ddfb0b705d23b863477d7f7f9f00ef1
>
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/BDJFYAC5IL77MBYQMH56D7ST67GDBMN2/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZHXCZTGL3RQN5TMALN6V6OCQEHZR32HZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Clarification regarding Stable ABI and _Py_*

[Eric Snow]

On Thu, Dec 9, 2021 at 1:56 AM Petr Viktorin  wrote:
> It's possible to remove them just like _PyObject_GC_Malloc was removed,
> but check that it was unusable (e.g. not called from public macros) in
> all versions of Python from 3.2 up to now.

That's what I expected.  Thanks.

> Could you check if this PR makes things clear?
> https://github.com/python/devguide/pull/778

Yeah, that text is super helpful.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TB7JEBQXUJJKK4SZVLCMUNOTRTD5KQ5C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Should isinstance call __getattribute__?

[Steven D'Aprano]

I'm looking for some guidance on a bug report involving isinstance and 
__getattribute__` please.

The issue is that if your class overloads `__getattribute__`, calling 
isinstance on an instance will call the overloaded `__getattribute__` 
method when looking up `__class__`, potentially causing isinstance to 
fail, or return the wrong result.

See b.p.o. #32683 

https://bugs.python.org/issue32683

I see no reason why this isn't working as designed, __getattribute__ is 
intended to overload attribute access, and that could include the 
`__class__` attribute. Am I wrong?

It has been suggested that isinstance should call `object.__getattribute__` 
and bypass the class' overloaded method, but I expect that would 
probably break objects which intentionally lie about their class. 
(Mocks? Stubs? Proxies?)



Thanks,

Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W55NIIXBI2QNZUMTGTZNTXZ7TZSWUQ7F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should isinstance call __getattribute__?

[Ronald Oussoren via Python-Dev]

> On 9 Dec 2021, at 16:41, Steven D'Aprano  wrote:
> 
> I'm looking for some guidance on a bug report involving isinstance and 
> __getattribute__` please.
> 
> The issue is that if your class overloads `__getattribute__`, calling 
> isinstance on an instance will call the overloaded `__getattribute__` 
> method when looking up `__class__`, potentially causing isinstance to 
> fail, or return the wrong result.
> 
> See b.p.o. #32683 
> 
> https://bugs.python.org/issue32683
> 
> I see no reason why this isn't working as designed, __getattribute__ is 
> intended to overload attribute access, and that could include the 
> `__class__` attribute. Am I wrong?
> 
> It has been suggested that isinstance should call `object.__getattribute__` 
> and bypass the class' overloaded method, but I expect that would 
> probably break objects which intentionally lie about their class. 
> (Mocks? Stubs? Proxies?)

https://mail.python.org/pipermail/python-dev/2015-October/141953.html 
 is an 
old thread about the difference between type(x)/Py_TYPE(x) and x.__class__ that 
contains some insight about this.

Proxy types are one use case, although with some sharp edges.

Ronald

—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UTFAVD5CW6DWESWXEBIIF2RHSKO6FVCC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Explicit markers for special C-API situations (re: Clarification regarding Stable ABI and _Py_*)

[Eric Snow]

(replying to 
https://mail.python.org/archives/list/python-dev@python.org/message/OJ65FPCJ2NVUFNZDXVNK5DU3R3JGLL3J/)

On Wed, Dec 8, 2021 at 10:06 AM Eric Snow  wrote:
> What about the various symbols listed in Misc/stable_abi.txt that were
> accidentally added to the limited API?  Can we move toward dropping
> them from the stable ABI?

tl;dr We should consider making classifications related to the stable
ABI harder to miss.



Knowing what is in the limited API is fairly straightforward. [1]
However, it's clear that identifying what is part of the stable ABI,
and why, is not so easy.  Currently, we must rely on
Misc/stable_abi.txt [2] (and the associated
Tools/scripts/stable_abi.py).  Documentation (C-API docs, PEPs,
devguide) help too.

Yet, there's a concrete disconnect here: the header files are by
definition the authoritative single-source-of-truth for the C-API and
it's too easy to forget about supplemental info in another file or
document.  This out-of-sight-out-of-mind situation is part of how we
accidentally added things to the limited API for a while. [3]

The stable ABI isn't the only area where we must identify different
subsets of the C-API.  However, in those other cases we use different
structural/naming conventions to explicitly group things.  Most
importantly, each of those conventions makes the grouping unavoidable
when reading the code. [4]  For example:

* closely related declarations go in the same header file (and then
also exposed via Include/Python.h)
* prefixes (e.g. Py_, PyDict_) provides similar grouping
* an additional underscore prefix identifies "private" C-API
* symbols are explicitly identified as part of the C-API via macros
(PyAPI_FUNC, PyAPI_DATA) [5]
* relatively recently, different directories correspond to different
API layers (Include, Include/cpython, Include/internal) [3]



Could we take a similar explicit, coupled-to-the-code approach to
identify when the different stable ABI situations apply?  Here's the
specific approach I had in mind, with macros similar to PyAPI_FUNC:

* PyAPI_ABI_FUNC - in stable ABI when it wouldn't be normally (e.g.
underscore prefix, in Include/internal)
* PyAPI_ABI_INDIRECT - exposed in stable ABI due to a macro
* PyAPI_ABI_ONLY - it only exists for ABI compatibility and isn't
actually used any more
* PyAPI_ABI_ACCIDENTAL - unintentionally added to limited API,
probably not used there

(...or perhaps use a PyABI_ prefix, though that's a bit easy to miss
when reading.)

As a reader I would find markers like this helpful in recognizing
those special situations, as well as the constraints those situations
impose on modification.  At the least such macros would indicate
something different is going on, and the macro name would be something
I could look up if I needed more info.  I expect others reading the
code would get comparable value.  I also expect tools like
Tools/scripts/stable_abi.py would benefit.

-eric


[1] in Include/*.h and not #ifndef Py_LIMITED_API (sadly also making
it easy to accidentally add things to the limited API, see [3])
[2] Before that you had to  rely on comments or external documents or,
in the worst case, work it out through careful study of the code,
commit history, and mailing list archives.
[3] The addition of Include/cpython and Include/internal helped us
stop accidentally adding to the limited API.
[4] It also makes the groupings deterministically discoverable by tools.
[5] explicit use of "extern" indicates a different intent
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7BSVTXDYCEOURQTLDRUXPXNPRYMM3I4G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Explicit markers for special C-API situations (re: Clarification regarding Stable ABI and _Py_*)

[Petr Viktorin]

I'll not get back to CPython until Tuesday, but I'll add a quick note
for now. It's a bit blunt for lack of time; please don't be offended.

If the code is the authoritative source of truth, we need a proper
parser to extract the information. But we can't really use an existing
parser (e.g. we need to navigate various #ifdef combinations), and
writing a correct (=tested) custom C parser is pretty expensive. C
declarations being "deterministically discoverable by tools" is a
myth.
I know you wrote a parser (kudos!), but unfortunately I don't trust it
enough to let it define the API. Bugs in the parser could result in
the API definition silently changing.

That's why the info is in a separate version-controlled file, which
must be explicitly modified. That file is the source of truth (or at
least intent).
There are also checks to ensure the code matches the manifest, so if
you break things the CI should let you know.
See the rationale in PEP 652:
https://www.python.org/dev/peps/pep-0652/#rationale

As for the types you mentioned:
* PyAPI_ABI_INDIRECT, PyAPI_ABI_ONLY - these should get a comment. I
don't think adding machine-readable metadata (and tooling for it)
would be worth it, but I won't block it.
* PyAPI_ABI_ACCIDENTAL - could be deprecated in the Limited API, and
later removed from it, becoming "PyAPI_ABI_ONLY".

On Thu, Dec 9, 2021 at 6:41 PM Eric Snow  wrote:
>
> (replying to 
> https://mail.python.org/archives/list/python-dev@python.org/message/OJ65FPCJ2NVUFNZDXVNK5DU3R3JGLL3J/)
>
> On Wed, Dec 8, 2021 at 10:06 AM Eric Snow  wrote:
> > What about the various symbols listed in Misc/stable_abi.txt that were
> > accidentally added to the limited API?  Can we move toward dropping
> > them from the stable ABI?
>
> tl;dr We should consider making classifications related to the stable
> ABI harder to miss.
>
> 
>
> Knowing what is in the limited API is fairly straightforward. [1]
> However, it's clear that identifying what is part of the stable ABI,
> and why, is not so easy.  Currently, we must rely on
> Misc/stable_abi.txt [2] (and the associated
> Tools/scripts/stable_abi.py).  Documentation (C-API docs, PEPs,
> devguide) help too.
>
> Yet, there's a concrete disconnect here: the header files are by
> definition the authoritative single-source-of-truth for the C-API and
> it's too easy to forget about supplemental info in another file or
> document.  This out-of-sight-out-of-mind situation is part of how we
> accidentally added things to the limited API for a while. [3]
>
> The stable ABI isn't the only area where we must identify different
> subsets of the C-API.  However, in those other cases we use different
> structural/naming conventions to explicitly group things.  Most
> importantly, each of those conventions makes the grouping unavoidable
> when reading the code. [4]  For example:
>
> * closely related declarations go in the same header file (and then
> also exposed via Include/Python.h)
> * prefixes (e.g. Py_, PyDict_) provides similar grouping
> * an additional underscore prefix identifies "private" C-API
> * symbols are explicitly identified as part of the C-API via macros
> (PyAPI_FUNC, PyAPI_DATA) [5]
> * relatively recently, different directories correspond to different
> API layers (Include, Include/cpython, Include/internal) [3]
>
> 
>
> Could we take a similar explicit, coupled-to-the-code approach to
> identify when the different stable ABI situations apply?  Here's the
> specific approach I had in mind, with macros similar to PyAPI_FUNC:
>
> * PyAPI_ABI_FUNC - in stable ABI when it wouldn't be normally (e.g.
> underscore prefix, in Include/internal)
> * PyAPI_ABI_INDIRECT - exposed in stable ABI due to a macro
> * PyAPI_ABI_ONLY - it only exists for ABI compatibility and isn't
> actually used any more
> * PyAPI_ABI_ACCIDENTAL - unintentionally added to limited API,
> probably not used there
>
> (...or perhaps use a PyABI_ prefix, though that's a bit easy to miss
> when reading.)
>
> As a reader I would find markers like this helpful in recognizing
> those special situations, as well as the constraints those situations
> impose on modification.  At the least such macros would indicate
> something different is going on, and the macro name would be something
> I could look up if I needed more info.  I expect others reading the
> code would get comparable value.  I also expect tools like
> Tools/scripts/stable_abi.py would benefit.
>
> -eric
>
>
> [1] in Include/*.h and not #ifndef Py_LIMITED_API (sadly also making
> it easy to accidentally add things to the limited API, see [3])
> [2] Before that you had to  rely on comments or external documents or,
> in the worst case, work it out through careful study of the code,
> commit history, and mailing list archives.
> [3] The addition of Include/cpython and Include/internal helped us
> stop accidentally adding to the limited API.
> [4] It also makes the groupings deterministically discoverable by tools.
> [5] explicit use of "extern" indicat

[Python-Dev] Re: Explicit markers for special C-API situations (re: Clarification regarding Stable ABI and _Py_*)

[Guido van Rossum]

Maybe we could start by having the tool regenerate the file and verifying
that it produces the same results? Then in the future we keep the file in
the repo so changes to it can be tracked separately, but we run the tool as
part of CI to make sure that its output still matches. This is what we do
for other generated files like opcode.h, parser.c and so on.

On Thu, Dec 9, 2021 at 10:31 AM Petr Viktorin  wrote:

> I'll not get back to CPython until Tuesday, but I'll add a quick note
> for now. It's a bit blunt for lack of time; please don't be offended.
>
> If the code is the authoritative source of truth, we need a proper
> parser to extract the information. But we can't really use an existing
> parser (e.g. we need to navigate various #ifdef combinations), and
> writing a correct (=tested) custom C parser is pretty expensive. C
> declarations being "deterministically discoverable by tools" is a
> myth.
> I know you wrote a parser (kudos!), but unfortunately I don't trust it
> enough to let it define the API. Bugs in the parser could result in
> the API definition silently changing.
>
> That's why the info is in a separate version-controlled file, which
> must be explicitly modified. That file is the source of truth (or at
> least intent).
> There are also checks to ensure the code matches the manifest, so if
> you break things the CI should let you know.
> See the rationale in PEP 652:
> https://www.python.org/dev/peps/pep-0652/#rationale
>
> As for the types you mentioned:
> * PyAPI_ABI_INDIRECT, PyAPI_ABI_ONLY - these should get a comment. I
> don't think adding machine-readable metadata (and tooling for it)
> would be worth it, but I won't block it.
> * PyAPI_ABI_ACCIDENTAL - could be deprecated in the Limited API, and
> later removed from it, becoming "PyAPI_ABI_ONLY".
>
> On Thu, Dec 9, 2021 at 6:41 PM Eric Snow 
> wrote:
> >
> > (replying to
> https://mail.python.org/archives/list/python-dev@python.org/message/OJ65FPCJ2NVUFNZDXVNK5DU3R3JGLL3J/
> )
> >
> > On Wed, Dec 8, 2021 at 10:06 AM Eric Snow 
> wrote:
> > > What about the various symbols listed in Misc/stable_abi.txt that were
> > > accidentally added to the limited API?  Can we move toward dropping
> > > them from the stable ABI?
> >
> > tl;dr We should consider making classifications related to the stable
> > ABI harder to miss.
> >
> > 
> >
> > Knowing what is in the limited API is fairly straightforward. [1]
> > However, it's clear that identifying what is part of the stable ABI,
> > and why, is not so easy.  Currently, we must rely on
> > Misc/stable_abi.txt [2] (and the associated
> > Tools/scripts/stable_abi.py).  Documentation (C-API docs, PEPs,
> > devguide) help too.
> >
> > Yet, there's a concrete disconnect here: the header files are by
> > definition the authoritative single-source-of-truth for the C-API and
> > it's too easy to forget about supplemental info in another file or
> > document.  This out-of-sight-out-of-mind situation is part of how we
> > accidentally added things to the limited API for a while. [3]
> >
> > The stable ABI isn't the only area where we must identify different
> > subsets of the C-API.  However, in those other cases we use different
> > structural/naming conventions to explicitly group things.  Most
> > importantly, each of those conventions makes the grouping unavoidable
> > when reading the code. [4]  For example:
> >
> > * closely related declarations go in the same header file (and then
> > also exposed via Include/Python.h)
> > * prefixes (e.g. Py_, PyDict_) provides similar grouping
> > * an additional underscore prefix identifies "private" C-API
> > * symbols are explicitly identified as part of the C-API via macros
> > (PyAPI_FUNC, PyAPI_DATA) [5]
> > * relatively recently, different directories correspond to different
> > API layers (Include, Include/cpython, Include/internal) [3]
> >
> > 
> >
> > Could we take a similar explicit, coupled-to-the-code approach to
> > identify when the different stable ABI situations apply?  Here's the
> > specific approach I had in mind, with macros similar to PyAPI_FUNC:
> >
> > * PyAPI_ABI_FUNC - in stable ABI when it wouldn't be normally (e.g.
> > underscore prefix, in Include/internal)
> > * PyAPI_ABI_INDIRECT - exposed in stable ABI due to a macro
> > * PyAPI_ABI_ONLY - it only exists for ABI compatibility and isn't
> > actually used any more
> > * PyAPI_ABI_ACCIDENTAL - unintentionally added to limited API,
> > probably not used there
> >
> > (...or perhaps use a PyABI_ prefix, though that's a bit easy to miss
> > when reading.)
> >
> > As a reader I would find markers like this helpful in recognizing
> > those special situations, as well as the constraints those situations
> > impose on modification.  At the least such macros would indicate
> > something different is going on, and the macro name would be something
> > I could look up if I needed more info.  I expect others reading the
> > code would get comparable value.  I also expect tools like
> > Tool

[Python-Dev] Re: Explicit markers for special C-API situations (re: Clarification regarding Stable ABI and _Py_*)

[Christian Heimes]

On 09/12/2021 19.26, Petr Viktorin wrote:

I'll not get back to CPython until Tuesday, but I'll add a quick note
for now. It's a bit blunt for lack of time; please don't be offended.

If the code is the authoritative source of truth, we need a proper
parser to extract the information. But we can't really use an existing
parser (e.g. we need to navigate various #ifdef combinations), and
writing a correct (=tested) custom C parser is pretty expensive. C
declarations being "deterministically discoverable by tools" is a
myth.
I know you wrote a parser (kudos!), but unfortunately I don't trust it
enough to let it define the API. Bugs in the parser could result in
the API definition silently changing.


There are other options than writing a new parser. GCC and Clang are 
flexible. For example GCC can be extended with plugins and custom 
attributes. We could extend the header files with custom attributes and 
then use a plugin to create an ABI file from the attributes.


I created a quick n' hack 
https://github.com/python/cpython/compare/main...tiran:gcc-pythonapi-plugin?expand=1 
as proof of concept.


The plugin takes

   PyAPI_ABI_FUNC(PyObject *) PyLong_FromLong(long);

and dumps the declaration as:

  extern struct PyObject * PyLong_FromLong (long int); "abi_func"

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N5ZTLGPLWHVI5UECVGVND43MBI4FAQN7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Explicit markers for special C-API situations (re: Clarification regarding Stable ABI and _Py_*)

[Jim J. Jewett]

Christian Heimes wrote:
> On 09/12/2021 19.26, Petr Viktorin wrote:

> > If the code is the authoritative source of truth, we need a proper
> > parser to extract the information.  ... unfortunately I don't trust it
> > enough to let it define the API. Bugs in the parser could result in
> > the API definition silently changing.

> There are other options than writing a new parser. GCC and Clang are 
> flexible. For example GCC can be extended with plugins and custom 
> attributes.

But they have the same problem ... it can be difficult to know if there is a 
subtle bug in someone's understanding of how the plugin interacts with, for 
example, nested ifndef.

The failure mode for an explicitly manually maintained text file is that 
something doesn't get added when it should, and the more conservative API 
consumers wait an extra release before using it.

-jJ



 We could extend the header files with custom attributes and 
> then use a plugin to create an ABI file from the attributes.
> I created a quick n' hack 
> https://github.com/python/cpython/compare/main...tiran:gcc-pythonapi-plugin?...
>  
> as proof of concept.
> The plugin takes
> PyAPI_ABI_FUNC(PyObject *) PyLong_FromLong(long);
> and dumps the declaration as:
> extern struct PyObject * PyLong_FromLong (long int); "abi_func"
> Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PKQFEIK75EWVTNMLB5CGBYLQANZG6QJH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Explicit markers for special C-API situations (re: Clarification regarding Stable ABI and _Py_*)

[Eric Snow]

On Thu, Dec 9, 2021, 11:26 Petr Viktorin  wrote:

> I'll not get back to CPython until Tuesday, but I'll add a quick note
> for now. It's a bit blunt for lack of time; please don't be offended.
>

Not at all. :)  The tooling is a secondary concern to my point.  Mostly, I
wish the declarations in the header files had the extra classifications,
rather than having to remember to refer to a separate text file.

>
-eric

>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OV7BOBOINCBWLZS3DZRWWJGY3BE4IOZB/
Code of Conduct: http://python.org/psf/codeofconduct/