[Python-Dev] My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

2020-06-09 Thread Petr Viktorin 

On 2020-06-05 16:32, Mark Shannon wrote:

Hi,

There have been a lot of changes both to the C API and to internal 
implementations to allow multiple interpreters in a single O/S process.


These changes cause backwards compatibility changes, have a negative 
performance impact, and cause a lot of churn.


While I'm in favour of PEP 554, or some similar model for parallelism in 
Python, I am opposed to the changes we are currently making to support it.



What are sub-interpreters?
--

A sub-interpreter is a logically independent Python process which 
supports inter-interpreter communication built on shared memory and 
channels. Passing of Python objects is supported, but only by copying, 
not by reference. Data can be shared via buffers.


Here's my biased take on the subject:

Interpreters are contexts in which Python runs. They contain 
configuration (e.g. the import path) and runtime state (e.g. the set of 
imported modules). An interpreter is created at Python startup 
(Py_InitializeEx), and you can create/destroy additional ones with 
Py_NewInterpreter/Py_EndInterpreter.

This is long-standing API that is used, most notably by mod_wsgi.

Many extension modules and some stdlib modules don't play well with the 
existence of multiple interpreters in a process, mainly because they use 
process-global state (C static variables) rather than some more granular 
scope.
This tends to result in nasty bugs (C-level crashes) when multiple 
interpreters are started in parallel (Py_NewInterpreter) or in sequence 
(several Py_InitializeEx/Py_FinalizeEx cycles). The bugs are similar in 
both cases.


Whether Python interpreters run sequentially or in parallel, having them 
work will enable a use case I would like to see: allowing me to call 
Python code from wherever I want, without thinking about global state. 
Think calling Python from an utility library that doesn't care about the 
rest of the application it's used in. I personally call this "the Lua 
use case", because light-weight, worry-free embedding is an area where 
Python loses to Lua. (And JS as well—that's a relatively recent 
development, but much more worrying.)


The part I have been involved in is moving away from process-global 
state. Process-global state can be made to work, but it is much safer to 
always default to module-local state (roughly what Python-language's 
`global` means), and treat process-global state as exceptions one has to 
think through. The API introduced in PEPs 384, 489, 573 (and future 
planned ones) aims to make module-local state possible to use, then 
later easy to use, and the natural default.


Relatively recently, there is an effort to expose interpreter creation & 
finalization from Python code, and also to allow communication between 
them (starting with something rudimentary, sharing buffers). There is 
also a push to explore making the GIL per-interpreter, which ties in to 
moving away from process-global state. Both are interesting ideas, but 
(like banishing global state) not the whole motivation for 
changes/additions. It's probably possible to do similar things with 
threads or subprocesses, sure, but if these efforts went away, the other 
issues would remain.


I am not too fond of the term "sub-interpreters", because it implies 
some kind of hierarchy. Of course, if interpreter creation is exposed to 
Python, you need some kind of "parent" to start the "child" and get its 
result when done. Also, due to some practical issues you might (sadly, 
currently) need some notion of "the main interpreter". But ideally, we 
can make interpreters entirely independent to allow the "Lua use case".
In the end-game of these efforts, I see Py_NewInterpreter transparently 
calling Py_InitializeEx if global state isn't set up yet, and similarly, 
Py_EndInterpreter turning the lights off if it's the last one out.

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/NLITVUIZQSUJ2F6XDTPMD7IP7FGTMNBA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

2020-06-09 Thread Guido van Rossum 
Petr, thanks for clearly stating your interests and goals for
subinterpreters. This lays to rest some of my own fears. I am still
skeptical that (even after the GIL is separated) they will enable
multi-core in ways that multiple processes couldn't handle just as well or
better, but your clear statement that *embedding* is the more important use
case helps me feel supportive of the concept.

On Tue, Jun 9, 2020 at 6:26 AM Petr Viktorin  wrote:

> On 2020-06-05 16:32, Mark Shannon wrote:
> > Hi,
> >
> > There have been a lot of changes both to the C API and to internal
> > implementations to allow multiple interpreters in a single O/S process.
> >
> > These changes cause backwards compatibility changes, have a negative
> > performance impact, and cause a lot of churn.
> >
> > While I'm in favour of PEP 554, or some similar model for parallelism in
> > Python, I am opposed to the changes we are currently making to support
> it.
> >
> >
> > What are sub-interpreters?
> > --
> >
> > A sub-interpreter is a logically independent Python process which
> > supports inter-interpreter communication built on shared memory and
> > channels. Passing of Python objects is supported, but only by copying,
> > not by reference. Data can be shared via buffers.
>
> Here's my biased take on the subject:
>
> Interpreters are contexts in which Python runs. They contain
> configuration (e.g. the import path) and runtime state (e.g. the set of
> imported modules). An interpreter is created at Python startup
> (Py_InitializeEx), and you can create/destroy additional ones with
> Py_NewInterpreter/Py_EndInterpreter.
> This is long-standing API that is used, most notably by mod_wsgi.
>
> Many extension modules and some stdlib modules don't play well with the
> existence of multiple interpreters in a process, mainly because they use
> process-global state (C static variables) rather than some more granular
> scope.
> This tends to result in nasty bugs (C-level crashes) when multiple
> interpreters are started in parallel (Py_NewInterpreter) or in sequence
> (several Py_InitializeEx/Py_FinalizeEx cycles). The bugs are similar in
> both cases.
>
> Whether Python interpreters run sequentially or in parallel, having them
> work will enable a use case I would like to see: allowing me to call
> Python code from wherever I want, without thinking about global state.
> Think calling Python from an utility library that doesn't care about the
> rest of the application it's used in. I personally call this "the Lua
> use case", because light-weight, worry-free embedding is an area where
> Python loses to Lua. (And JS as well—that's a relatively recent
> development, but much more worrying.)
>
> The part I have been involved in is moving away from process-global
> state. Process-global state can be made to work, but it is much safer to
> always default to module-local state (roughly what Python-language's
> `global` means), and treat process-global state as exceptions one has to
> think through. The API introduced in PEPs 384, 489, 573 (and future
> planned ones) aims to make module-local state possible to use, then
> later easy to use, and the natural default.
>
> Relatively recently, there is an effort to expose interpreter creation &
> finalization from Python code, and also to allow communication between
> them (starting with something rudimentary, sharing buffers). There is
> also a push to explore making the GIL per-interpreter, which ties in to
> moving away from process-global state. Both are interesting ideas, but
> (like banishing global state) not the whole motivation for
> changes/additions. It's probably possible to do similar things with
> threads or subprocesses, sure, but if these efforts went away, the other
> issues would remain.
>
> I am not too fond of the term "sub-interpreters", because it implies
> some kind of hierarchy. Of course, if interpreter creation is exposed to
> Python, you need some kind of "parent" to start the "child" and get its
> result when done. Also, due to some practical issues you might (sadly,
> currently) need some notion of "the main interpreter". But ideally, we
> can make interpreters entirely independent to allow the "Lua use case".
> In the end-game of these efforts, I see Py_NewInterpreter transparently
> calling Py_InitializeEx if global state isn't set up yet, and similarly,
> Py_EndInterpreter turning the lights off if it's the last one out.
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/NLITVUIZQSUJ2F6XDTPMD7IP7FGTMNBA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

[Python-Dev] [RELEASE] Python 3.9.0b3 is now available for testing

2020-06-09 Thread Łukasz Langa 
On behalf of the entire Python development community, and the currently serving 
Python release team in particular, I’m pleased to announce the release of 
Python 3.9.0b3. Get it here:

https://www.python.org/downloads/release/python-390b3/ 

Wait, Beta 3? What happened to Beta 2?

Beta 2? Speak of him no more. We disappeared him. He was a bad release. Truly 
awful. I get shivers just thinking about it. Never mention that name again in 
this house.

I mean, long story short, in Beta 2 you couldn’t do 
urllib.request.urlopen("https://www.python.org";).read() because it wouldn’t 
find root certificates due to a bug . Since 
this was a problem only apparent on an installed Python, it wasn’t identified 
by unit tests and was only found by Ned while he was testing his Mac installer. 
By the time we learned of the severity of the bug I already tagged and 
published the release on python.org . That’s why we 
couldn’t just re-do the release under the same version.

Sorry for the trouble. We’re tweaking our release process to catch this problem 
sooner in future releases. Now, back to regular programming…

This is a beta preview of Python 3.9

Python 3.9 is still in development. This release, 3.9.0b3, is the third of five 
planned beta release previews.

Beta release previews are intended to give the wider community the opportunity 
to test new features and bug fixes and to prepare their projects to support the 
new feature release.

Call to action

We strongly encourage maintainers of third-party Python projects to test with 
3.9 during the beta phase and report issues found to the Python bug tracker 
 as soon as possible. While the release is planned to 
be feature complete entering the beta phase, it is possible that features may 
be modified or, in rare cases, deleted up until the start of the release 
candidate phase (2020-08-10). Our goal is have no ABI changes after beta 5 and 
as few code changes as possible after 3.9.0rc1, the first release candidate. To 
achieve that, it will be extremely important to get as much exposure for 3.9 as 
possible during the beta phase.

Please keep in mind that this is a preview release and its use is not 
recommended for production environments.

Major new features of the 3.9 series, compared to 3.8

Some of the new major new features and changes in Python 3.9 are:

PEP 584 , Union Operators in dict

PEP 585 , Type Hinting Generics In 
Standard Collections

PEP 593 , Flexible function and 
variable annotations

PEP 602 , Python adopts a stable 
annual release cadence

PEP 616 , String methods to remove 
prefixes and suffixes

PEP 617 , New PEG parser for CPython

BPO 38379 , garbage collection does not 
block on resurrected objects;

BPO 38692 , os.pidfd_open added that allows 
process management without races and signals;

BPO 39926 , Unicode support updated to 
version 13.0.0;

BPO 1635741 , when Python is initialized 
multiple times in the same process, it does not leak memory anymore;

A number of Python builtins (range, tuple, set, frozenset, list, dict) are now 
sped up using PEP 590  vectorcall;

A number of Python modules (_abc, audioop, _bz2, _codecs, _contextvars, _crypt, 
_functools, _json, _locale, operator, resource, time, _weakref) now use 
multiphase initialization as defined by PEP 489 
;

A number of standard library modules (audioop, ast, grp, _hashlib, pwd, 
_posixsubprocess, random, select, struct, termios, zlib) are now using the 
stable ABI defined by PEP 384 .

(Hey, fellow core developer, if a feature you find important is missing from 
this list, let Łukasz know .)

The next pre-release, the fourth beta release of Python 3.9, will be 3.9.0b4. 
It is currently scheduled for 2020-06-29.

More resources

Online Documentation 
PEP 596 , 3.9 Release Schedule
Report bugs at https://bugs.python.org .
Help fund Python and its community .
Your friendly release team,
Ned Deily @nad 
Steve Dower @steve.dower 
Łukasz Langa @ambv ___
Python-Dev mailing list -- [email protected]
To unsu

[Python-Dev] Re: [python-committers] [RELEASE] Python 3.9.0b3 is now available for testing

2020-06-09 Thread Victor Stinner 
Le mar. 9 juin 2020 à 23:59, Łukasz Langa  a écrit :
> I mean, long story short, in Beta 2 you couldn’t do 
> urllib.request.urlopen("https://www.python.org";).read() because it wouldn’t 
> find root certificates due to a bug. Since this was a problem only apparent 
> on an installed Python, it wasn’t identified by unit tests and was only found 
> by Ned while he was testing his Mac installer.

https://bugs.python.org/issue40924

If certifi is critical for Python releases, would it be possible to
put a check in a CI somewhere to ensure that it always works?

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/VCYJO2EKB4QBLK2QONRA67KDVXQS5WYN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

2020-06-09 Thread Inada Naoki 
On Tue, Jun 9, 2020 at 10:28 PM Petr Viktorin  wrote:
>
> Relatively recently, there is an effort to expose interpreter creation &
> finalization from Python code, and also to allow communication between
> them (starting with something rudimentary, sharing buffers). There is
> also a push to explore making the GIL per-interpreter, which ties in to
> moving away from process-global state. Both are interesting ideas, but
> (like banishing global state) not the whole motivation for
> changes/additions.
>

Some changes for per interpreter GIL doesn't help sub interpreters so much.
For example, isolating memory allocator including free list and
constants between
sub interpreter makes sub interpreter fatter.
I assume Mark is talking about such changes.

Now Victor proposing move dict free list per interpreter state and the code
looks good to me.  This is a change for per interpreter GIL, but not
for sub interpreters.
https://github.com/python/cpython/pull/20645

Should we commit this change to the master branch?
Or should we create another branch for such changes?

Regards,
-- 
Inada Naoki  
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/L7JRFJLDLO6E4SDXYKDPTEIEDZK2PNR4/
Code of Conduct: http://python.org/psf/codeofconduct/