[Python-Dev] Re: Speeding up CPython

[Thomas Wouters]

(For the record, I’m not replying as a PSF Director in this; I haven’t
discussed this with the rest of the Board yet. This just comes from the
Steering Council.)

The Steering Council discussed this proposal in our weekly meeting, last
week. It's a complicated subject with a lot of different facets to
consider. First of all, though, we want to thank you, Mark, for bringing
this to the table. The Steering Council and the PSF have been looking for
these kinds of proposals for spending money on CPython development. We need
ideas like this to have something to spend money on that we might collect
(e.g. via the new GitHub sponsors page), and also to have a good story to
potential (corporate) sponsors.

That said, we do have a number of things to consider here.

For background, funding comes in a variety of flavours. Most donations to
the PSF are general fund donations; the foundation is free to use it for
whatever purpose it deems necessary (within its non-profit mission). The
PSF Board and staff decide where this money has the biggest impact, as
there are a lot of things the PSF could spend it on.

Funds can also be earmarked for a specific purpose. Donations to PyPI (
donate.pypi.org) work this way, for example. The donations go to the PSF,
but are set aside specifically for PyPI expenses and development. Fiscal
sponsorship (https://www.python.org/psf/fiscal-sponsorees/) is similar, but
even more firmly restricted (and the fiscal sponsorees, not the PSF,
decides what to spend the money on).

A third way of handling funding is more targeted donations: sponsors donate
for a specific program. For example, GitHub donated money specifically for
the PSF to hire a project manager to handle the migration from
bugs.python.org to GitHub Issues. Ezio Melotti was contracted by the PSF
for this job, not GitHub, even though the funds are entirely donated by
GitHub. Similar to such targeted donations are grant requests, like the
several grants PyPI received and the CZI grant request for CPython that was
recently rejected (https://github.com/python/steering-council/issues/26).
The mechanics are a little different, but the end result is the same: the
PSF receives funds to achieve very specific goals.

Regarding donations to CPython development (as earmarked donations, or from
the PSF's general fund), the SC drew up a plan for investment that is
centered around maintenance: reducing the maintenance burden, easing the
load on volunteers where desired, working through our bug and PR backlog.
(The COVID-19 impact on PyCon and PSF funds put a damper on our plans, but
we used much of the original plan for the CZI grant request, for example.
Since that, too, fell through, we're hoping to collect funds for a reduced
version of the plan through the PSF, which is looking to add it as a
separate track in the sponsorship program.) Speeding up pure-Python
programs is not something we consider a priority at this point, at least
not until we can address the larger maintenance issues.

And it may not be immediately obvious from Mark's plans, but as far as we
can tell, the proposal is for speeding up pure-Python code. It will do
little for code that is hampered, speed-wise, by CPython's object model, or
threading model, or the C API. We have no idea how much this will actually
matter to users. Making pure-Python code execution faster is always
welcome, but it depends on the price. It may not be a good place to spend
$500k or more, and it may even not be considered worth the implementation
complexity.

Thinking specifically of corporate sponsorship, it's very much the question
if pure-Python code speedup is something companies would be willing to
invest serious funds in. Google's Unladen Swallow was such an investment,
and though it did deliver speedups (which were included in Python 2.7) and
even though Google has a lot of Python code, there was not enough interest
to keep it going. This may be different now, but finding out what
"customers" (in the broadest sense) actually want is an important first
step in asking for funding for a project like this. It's the kind of thing
normally done by a product manager, at least in the corporate world, and we
need that same effort and care put into it.

If we can potentially find the funds for this project, via the PSF's
general fund, earmarked funds or a direct corporate sponsor, we also have
to consider what we are actually delivering. Which performance metrics are
we improving? How are we measuring them, what benchmarks? What if the
sponsor has their own benchmarks they want to use? What about effects on
other performance metrics, ones the project isn't seeking to improve, are
they allowed to worsen? To what extent? How will that be measured? How will
we measure progress as the project continues? What milestones will we set?
What happens when there's disagreement about the result between the sponsor
and the people doing the work? What if the Steering Council or the core
developers -- as a body -- dec

[Python-Dev] Re: Speeding up CPython

[Antoine Pitrou]

Hello,

On Wed, 4 Nov 2020 13:27:50 +0100
Thomas Wouters  wrote:
> 
> And it may not be immediately obvious from Mark's plans, but as far as we
> can tell, the proposal is for speeding up pure-Python code. It will do
> little for code that is hampered, speed-wise, by CPython's object model, or
> threading model, or the C API. We have no idea how much this will actually
> matter to users. Making pure-Python code execution faster is always
> welcome, but it depends on the price. It may not be a good place to spend
> $500k or more, and it may even not be considered worth the implementation
> complexity.

FWIW, I think it would definitely be worth it.  Performance will be a
*major* hurdle for Python in the years to come (the other hurdle being
ease of deployment).

> Thinking specifically of corporate sponsorship, it's very much the question
> if pure-Python code speedup is something companies would be willing to
> invest serious funds in.

I would suggest for example talking to Quansight, Numfocus, the NVidia
Rapids team, and/or coiled.io .  There are areas of scientific computing
where better pure Python performance would help (one potential area is
the Dask scheduler, another is the Numba JIT compiler).

Another prominent area is server-side Web development, but I have noone
to suggest there :-)

Best regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RCB5JE22QQRNYU76IDAKYCF3VYWHUDFZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Paul Moore]

On Wed, 4 Nov 2020 at 13:14, Antoine Pitrou  wrote:
>
> On Wed, 4 Nov 2020 13:27:50 +0100
> Thomas Wouters  wrote:
> >
> > And it may not be immediately obvious from Mark's plans, but as far as we
> > can tell, the proposal is for speeding up pure-Python code. It will do
> > little for code that is hampered, speed-wise, by CPython's object model, or
> > threading model, or the C API. We have no idea how much this will actually
> > matter to users. Making pure-Python code execution faster is always
> > welcome, but it depends on the price. It may not be a good place to spend
> > $500k or more, and it may even not be considered worth the implementation
> > complexity.
>
> FWIW, I think it would definitely be worth it.  Performance will be a
> *major* hurdle for Python in the years to come (the other hurdle being
> ease of deployment).

I agree on both of these points, and I would love to see funding be
available for both of these items.

But having said that, I agree with the SC's position here. Getting
funding is only one part of the problem, project management and
co-ordination is absolutely necessary (we're talking about a $2M
project!) and would be a significant overhead. Even if the cost of
such resource could come from the funding, there's still a significant
cashflow problem with committing that resource prior to getting
funding, as well as a risk that the funding doesn't materialise and
the investment is lost.

I hope that we can find some way to realise the benefits Mark has
identified, but I can see why the SC has to prioritise the way they
have.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4NY6QKSD7375B24EM3MBI4HDHDGQRIB7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [RELEASE] Python 3.10.0a2 available for testing

[Victor Stinner]

Hi,

Currently, building a wheel packaging on Python 3.10 fails with:

AssertionError: would build wheel with unsupported tag ('cp310',
'cp310', 'linux_x86_64')

This bug is discussed in many places:

* PEP 641 -- Using an underscore in the version portion of Python 3.10
compatibility tags
  https://www.python.org/dev/peps/pep-0641/
* PEP 641 discussion:
  
https://discuss.python.org/t/pep-641-using-an-underscore-in-the-version-portion-of-python-3-10-compatibility-tags/5513
* CPython PR 20333: "bpo-40747: Make py_version_nodot 3_10 not 310"
  https://github.com/python/cpython/pull/20333
* wheel: "Fails to build wheel for Python 3.10"
  https://github.com/pypa/wheel/issues/354
* python3-setuptools: "python-setuptools fails to build with Python
3.10: AssertionError: would build wheel with unsupported tag ('cp310',
'cp310', 'linux_x86_64')"
https://bugzilla.redhat.com/show_bug.cgi?id=1891840

Victor

Le mar. 3 nov. 2020 à 18:45, Pablo Galindo Salgado
 a écrit :
>
> The engines of the secret release manager machine have finished producing a 
> new pre-release. Go get it here:
>
> https://www.python.org/downloads/release/python-3100a2/
>
> Major new features of the 3.10 series, compared to 3.9
>
> Python 3.10 is still in development. This releasee, 3.10.0a2 is the second of 
> six planned alpha releases. Alpha releases are intended to make it easier to 
> test the current state of new features and bug fixes and to test the release 
> process. During the alpha phase, features may be added up until the start of 
> the beta phase (2021-05-03) and, if necessary, may be modified or deleted up 
> until the release candidate phase (2021-10-04). Please keep in mind that this 
> is a preview release and its use is not recommended for production 
> environments.
>
> Many new features for Python 3.10 are still being planned and written. Among 
> the new major new features and changes so far:
>
> PEP 623 -- Remove wstr from Unicode
> PEP 604 -- Allow writing union types as X | Y
> PEP 612 -- Parameter Specification Variables
> PEP 626 -- Precise line numbers for debugging and other tools.
> (Hey, fellow core developer, if a feature you find important is missing from 
> this list, let Pablo know.)
> The next pre-release of Python 3.10 will be 3.10.0a3, currently scheduled for 
> 2020-12-07.
>
> And now for something completely different
>
> The cardinality (the number of elements) of infinite sets can be one of the 
> most surprising results of set theory. For example, there are the same amount 
> of even natural numbers than natural numbers (which can be even or odd). 
> There is also the same amount of rational numbers than natural numbers. But 
> on the other hand, there are more real numbers between 0 and 1 than natural 
> numbers! All these sets have infinite cardinality but turn out that some of 
> these infinities are bigger than others. These infinite cardinalities 
> normally are represented using aleph numbers. Infinite sets are strange 
> beasts indeed.
>
> Regards from cold London,
> Pablo Galindo Salgado
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/EYPALIF2V25SI4LBH5THTY2IM4VWDGXJ/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VLLHMEGA2TAPJBPZXYZPKL6I2FDRIEAZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Mark Shannon]

Hi Thomas,

I have to assume that this isn't a rejection of my proposal, since I 
haven't actually made a proposal to the SC yet :)


Thanks for the feedback though, it's very valuable to know the SC's 
thinking on this matter.


I have a few comments inline below.


On 04/11/2020 12:27 pm, Thomas Wouters wrote:


(For the record, I’m not replying as a PSF Director in this; I haven’t 
discussed this with the rest of the Board yet. This just comes from the 
Steering Council.)



The Steering Council discussed this proposal in our weekly meeting, last 
week. It's a complicated subject with a lot of different facets to 
consider. First of all, though, we want to thank you, Mark, for bringing 
this to the table. The Steering Council and the PSF have been looking 
for these kinds of proposals for spending money on CPython development. 
We need ideas like this to have something to spend money on that we 
might collect (e.g. via the new GitHub sponsors page), and also to have 
a good story to potential (corporate) sponsors.



That said, we do have a number of things to consider here.


For background, funding comes in a variety of flavours. Most donations 
to the PSF are general fund donations; the foundation is free to use it 
for whatever purpose it deems necessary (within its non-profit mission). 
The PSF Board and staff decide where this money has the biggest impact, 
as there are a lotof things the PSF could spend it on.



Funds can also be earmarked for a specific purpose. Donations to PyPI 
(donate.pypi.org ) work this way, for example. 
The donations go to the PSF, but are set aside specifically for PyPI 
expenses and development. Fiscal sponsorship 
(https://www.python.org/psf/fiscal-sponsorees/) is similar, but even 
more firmly restricted (and the fiscal sponsorees, not the PSF, decides +++

what to spend the money on).


A third way of handling funding is more targeted donations: sponsors 
donate for a specific program. For example, GitHub donated money 
specifically for the PSF to hire a project manager to handle the 
migration from bugs.python.org  to GitHub 
Issues. Ezio Melotti was contracted by the PSF for this job, not GitHub, 
even though the funds are entirely donated by GitHub. Similar to such 
targeted donations are grant requests, like the several grants PyPI 
received and the CZI grant request for CPython that was recently 
rejected (https://github.com/python/steering-council/issues/26). The 
mechanics are a little different, but the end result is the same: the 
PSF receives funds to achieve very specific goals.


I really don't want to take money away from the PSF. Ideally I would 
like the PSF to have more money.





Regarding donations to CPython development (as earmarked donations, or 
from the PSF's general fund), the SC drew up a plan for investment that 
is centered around maintenance: reducing the maintenance burden, easing 
the load on volunteers where desired, working through our bug and PR 
backlog. (The COVID-19 impact on PyCon and PSF funds put a damper on our 
plans, but we used much of the original plan for the CZI grant request, 
for example. Since that, too, fell through, we're hoping to collect 
funds for a reduced version of the plan through the PSF, which is 
looking to add it as a separate track in the sponsorship program.) 
Speeding up pure-Python programs is not something we consider a priority 
at this point, at least not until we can address the larger maintenance 
issues.


I too think we should improve the maintenance story.
But maintenance doesn't get anyone excited. Performance does.
By allocating part of the budget to maintenance we get performance *and* 
a better maintenance story. That's my goal anyway.


I think it is a lot easier to say to corporations, give us X dollars to 
speed up Python and you save Y dollars, than give us X dollars to 
improve maintenance with no quantifiable benefit to them.





And it may not be immediately obvious from Mark's plans, but as far as 
we can tell, the proposal is for speeding up pure-Python code. It will 
do little for code that is hampered, speed-wise, by CPython's object 
model, or threading model, or the C API. We have no idea how much this 
will actually matter to users. Making pure-Python code execution faster 
is always welcome, but it depends on the price. It may not be a good 
place to spend $500k or more, and it may even not be considered worth 
the implementation complexity.


I'll elaborate:

1. There will be a large total diff, but not that large an increase in 
code size; less than 1% of the current size of the C code base.


There would be an increase in the conceptual complexity of the interpreter,
but I'm hoping to largely offset that with better code organization.

It is perfectly possible to *improve* code quality,
if not necessarily size, while increasing performance.
Simpler code is often faster and better algorithms do not make worse code.

2. Th

[Python-Dev] Re: PEP 642: Constraint Pattern Syntax for Structural Pattern Matching

[Nick Coghlan]

On Tue., 3 Nov. 2020, 8:07 am Ethan Furman,  wrote:

> On 11/2/20 1:52 PM, Glenn Linderman wrote:
> > On 11/2/2020 1:42 PM, Guido van Rossum wrote:
> >> But we feel that `case x, x` can easily be misunderstood as "a tuple of
> two equal values"
> >
> > So what _is_ the syntax for "a tuple of two equal values" ?
> >
> > case x, ?x:  # comes to mind (not that it is in the PEP :))
>
> Using a guard statement:
>
>  case x, y if x == y
>

This example made me realise that I need to add test cases for "case x,
==x:" and "case x, is x:" to PEP 642's reference implementation (and text
to the PEP pointing out that explicit constraints can help address the
pattern back-reference problem).

Cheers,
Nick.


>
>
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PPBTGG54SUQBWKJR22VOVL7NQ7L2WBWY/
Code of Conduct: http://python.org/psf/codeofconduct/