[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)
On 2020-06-10 04:43, Inada Naoki wrote: On Tue, Jun 9, 2020 at 10:28 PM Petr Viktorin wrote: Relatively recently, there is an effort to expose interpreter creation & finalization from Python code, and also to allow communication between them (starting with something rudimentary, sharing buffers). There is also a push to explore making the GIL per-interpreter, which ties in to moving away from process-global state. Both are interesting ideas, but (like banishing global state) not the whole motivation for changes/additions. Some changes for per interpreter GIL doesn't help sub interpreters so much. For example, isolating memory allocator including free list and constants between sub interpreter makes sub interpreter fatter. I assume Mark is talking about such changes. Now Victor proposing move dict free list per interpreter state and the code looks good to me. This is a change for per interpreter GIL, but not for sub interpreters. https://github.com/python/cpython/pull/20645 Should we commit this change to the master branch? Or should we create another branch for such changes? I think that most of all, the changes aimed at breaking up the GIL need a PEP, so that everyone knows what the changes are actually about -- and especially so that everyone knows the changes are happening. Note that neither PEP 554 (which itself isn't accepted yet) nor PEP 573 is related to breaking up the GIL. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/I4DURF74SJZ3PEILBWDVR2XHOZQKRZRH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)
Hi, I agree that embedding Python is an important use case and that we should try to leak less memory and better isolate multiple interpreters for this use case. There are multiple projects to enhance code to make it work better with multiple interpreters: * convert C extension modules to multiphase initialization (PEP 489) * move C extension module global variables (static ...) into a module state * convert static types to heap types * make free lists per interpreter * etc. From what I saw, the first side effect is that "suddenly", tests using subinterpreters start to report new reference leaks. Examples of issues and fixes: * https://github.com/python/cpython/commit/18a90248fdd92b27098cc4db773686a2d10a4d24: reference leak in the init function of the select module * https://github.com/python/cpython/commit/310e2d25170a88ef03f6fd31efcc899fe062da2c: reference cycles with encodings and _testcapi misuses PyModule_AddObject() * https://bugs.python.org/issue40050: _weakref and importlib * etc. In fact, none of these bugs is not new. I checked for a few: bugs were always there. It's just that previously, nobody paid attention to these leaks. Fixing subinterpreters helps to leak less memory even for the single interpreter (embed Python) use case. The problem is that Python never tried to clear everything at exit. One way to see the issue is the number of references at exit using a debug build, on the up-to-date master branch: $ ./python -X showrefcount -c pass [18645 refs, 6141 blocks] Python leaks 18,645 references at exit. Some of the work that I listed is tracked by https://bugs.python.org/issue1635741 which was created in 2007: "Py_Finalize() doesn't clear all Python objects at exit". Another way to see the issue is: $ PYTHONMALLOC=malloc valgrind ./python -c pass (...) ==169747== LEAK SUMMARY: ==169747==definitely lost: 48 bytes in 2 blocks ==169747==indirectly lost: 136 bytes in 6 blocks ==169747== possibly lost: 700,552 bytes in 5,677 blocks ==169747==still reachable: 5,450 bytes in 48 blocks ==169747== suppressed: 0 bytes in 0 blocks Python leaks around 700 KB at exit. Even if you ignore the "run multiple interpreters in parallel" and PEP 554 use cases, enhancing code to better work with subinterpreters also makes Python a better library to embed in applications and so is useful. Victor Le mer. 10 juin 2020 à 04:46, Inada Naoki a écrit : > > On Tue, Jun 9, 2020 at 10:28 PM Petr Viktorin wrote: > > > > Relatively recently, there is an effort to expose interpreter creation & > > finalization from Python code, and also to allow communication between > > them (starting with something rudimentary, sharing buffers). There is > > also a push to explore making the GIL per-interpreter, which ties in to > > moving away from process-global state. Both are interesting ideas, but > > (like banishing global state) not the whole motivation for > > changes/additions. > > > > Some changes for per interpreter GIL doesn't help sub interpreters so much. > For example, isolating memory allocator including free list and > constants between > sub interpreter makes sub interpreter fatter. > I assume Mark is talking about such changes. > > Now Victor proposing move dict free list per interpreter state and the code > looks good to me. This is a change for per interpreter GIL, but not > for sub interpreters. > https://github.com/python/cpython/pull/20645 > > Should we commit this change to the master branch? > Or should we create another branch for such changes? > > Regards, > -- > Inada Naoki > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/L7JRFJLDLO6E4SDXYKDPTEIEDZK2PNR4/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/INREEYUJLL47R4ZGJ2GGJDZSPX2ORMA6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)
Hi Petr, On 09/06/2020 2:24 pm, Petr Viktorin wrote: On 2020-06-05 16:32, Mark Shannon wrote: Hi, There have been a lot of changes both to the C API and to internal implementations to allow multiple interpreters in a single O/S process. These changes cause backwards compatibility changes, have a negative performance impact, and cause a lot of churn. While I'm in favour of PEP 554, or some similar model for parallelism in Python, I am opposed to the changes we are currently making to support it. What are sub-interpreters? -- A sub-interpreter is a logically independent Python process which supports inter-interpreter communication built on shared memory and channels. Passing of Python objects is supported, but only by copying, not by reference. Data can be shared via buffers. Here's my biased take on the subject: Interpreters are contexts in which Python runs. They contain configuration (e.g. the import path) and runtime state (e.g. the set of imported modules). An interpreter is created at Python startup (Py_InitializeEx), and you can create/destroy additional ones with Py_NewInterpreter/Py_EndInterpreter. This is long-standing API that is used, most notably by mod_wsgi. Many extension modules and some stdlib modules don't play well with the existence of multiple interpreters in a process, mainly because they use process-global state (C static variables) rather than some more granular scope. This tends to result in nasty bugs (C-level crashes) when multiple interpreters are started in parallel (Py_NewInterpreter) or in sequence (several Py_InitializeEx/Py_FinalizeEx cycles). The bugs are similar in both cases. Whether Python interpreters run sequentially or in parallel, having them work will enable a use case I would like to see: allowing me to call Python code from wherever I want, without thinking about global state. Think calling Python from an utility library that doesn't care about the rest of the application it's used in. I personally call this "the Lua use case", because light-weight, worry-free embedding is an area where Python loses to Lua. (And JS as well—that's a relatively recent development, but much more worrying.) This seems like a worthwhile goal. However I don't see why this requires having multiple Python interpreters in a single O/S process. The part I have been involved in is moving away from process-global state. Process-global state can be made to work, but it is much safer to always default to module-local state (roughly what Python-language's `global` means), and treat process-global state as exceptions one has to think through. The API introduced in PEPs 384, 489, 573 (and future planned ones) aims to make module-local state possible to use, then later easy to use, and the natural default. I don't agree. Process level state is *much* safer than module-local state. Suppose two interpreters, have both imported the same module. By using O/S processes to keep the interpreters separate, the hardware prevents the two copies of the module from interfering with each other. By sharing an address space the separation is maintained by trust and hoping that third party modules don't have too many bugs. I don't see how you can claim the later case if safer. Relatively recently, there is an effort to expose interpreter creation & finalization from Python code, and also to allow communication between them (starting with something rudimentary, sharing buffers). There is also a push to explore making the GIL per-interpreter, which ties in to moving away from process-global state. Both are interesting ideas, but (like banishing global state) not the whole motivation for changes/additions. It's probably possible to do similar things with threads or subprocesses, sure, but if these efforts went away, the other issues would remain. What other issues? Please be specific. I am not too fond of the term "sub-interpreters", because it implies some kind of hierarchy. Of course, if interpreter creation is exposed to Python, you need some kind of "parent" to start the "child" and get its result when done. Also, due to some practical issues you might (sadly, currently) need some notion of "the main interpreter". But ideally, we can make interpreters entirely independent to allow the "Lua use case". In the end-game of these efforts, I see Py_NewInterpreter transparently calling Py_InitializeEx if global state isn't set up yet, and similarly, Py_EndInterpreter turning the lights off if it's the last one out. I'll drop the "sub" from now on :) If each interpreter runs in its own process, then initializing an interpreter and initializing the "global" state are the same thing and wouldn't need a separate step. Cheers, Mark. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org
[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)
On Wed, Jun 10, 2020 at 5:37 AM Mark Shannon wrote: > By sharing an address space the separation is maintained by trust and hoping that third party modules don't have too many bugs. By definition, the use of any third-party module (or even the standard library itself) is by trust and the hope that they don't have too many bugs. Sure, this creates a potential new class of bugs, for those who use it, while also offering the chance to find and fix old bugs like Victor found. Mostly, though, it exposes lots of bad practices that people could mostly get away with as long as the assumption was that everything would always be single-threaded, single-process, and the entire software industry is moving away from those assumptions, so it's only logical that Python takes advantage of that shift instead of becoming another legacy language. In the meantime, modules can explicitly label themselves as single-interpreter only, requiring multiprocessing instead of threading or embedding to work correctly. Modules were more than happy to label themselves as 2.x only for a decade -Em ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/JKELVIZHMZRS4VPBNJFWSP5POACQ4ODU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)
On 6/10/2020 8:33 AM, Mark Shannon wrote: Hi Petr, On 09/06/2020 2:24 pm, Petr Viktorin wrote: On 2020-06-05 16:32, Mark Shannon wrote: Whether Python interpreters run sequentially or in parallel, having them work will enable a use case I would like to see: allowing me to call Python code from wherever I want, without thinking about global state. Think calling Python from an utility library that doesn't care about the rest of the application it's used in. I personally call this "the Lua use case", because light-weight, worry-free embedding is an area where Python loses to Lua. (And JS as well—that's a relatively recent development, but much more worrying.) This seems like a worthwhile goal. However I don't see why this requires having multiple Python interpreters in a single O/S process. I assume it would be so that my code could link with library A, which embeds Python, and library B, which also embeds Python. A and B have no knowledge of each other. The part I have been involved in is moving away from process-global state. Process-global state can be made to work, but it is much safer to always default to module-local state (roughly what Python-language's `global` means), and treat process-global state as exceptions one has to think through. The API introduced in PEPs 384, 489, 573 (and future planned ones) aims to make module-local state possible to use, then later easy to use, and the natural default. I don't agree. Process level state is *much* safer than module-local state. Suppose two interpreters, have both imported the same module. By using O/S processes to keep the interpreters separate, the hardware prevents the two copies of the module from interfering with each other. By sharing an address space the separation is maintained by trust and hoping that third party modules don't have too many bugs. I don't see how you can claim the later case if safer. I've always assumed that per-module state meant per-module, per-interpreter. Maybe I've misunderstood, in which case I agree with Mark. If per-module state isn't isolated per interpreter, that sort of kills the multiple interpreter model, in my mind. Eric ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/WOOPV4QKFFZSZ5QVXMC3JBEHPTVULH4S/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)
> On 10 Jun 2020, at 14:33, Mark Shannon wrote: > > Hi Petr, > > On 09/06/2020 2:24 pm, Petr Viktorin wrote: >> On 2020-06-05 16:32, Mark Shannon wrote: >>> Hi, >>> >>> There have been a lot of changes both to the C API and to internal >>> implementations to allow multiple interpreters in a single O/S process. >>> >>> These changes cause backwards compatibility changes, have a negative >>> performance impact, and cause a lot of churn. >>> >>> While I'm in favour of PEP 554, or some similar model for parallelism in >>> Python, I am opposed to the changes we are currently making to support it. >>> >>> >>> What are sub-interpreters? >>> -- >>> >>> A sub-interpreter is a logically independent Python process which supports >>> inter-interpreter communication built on shared memory and channels. >>> Passing of Python objects is supported, but only by copying, not by >>> reference. Data can be shared via buffers. >> Here's my biased take on the subject: >> Interpreters are contexts in which Python runs. They contain configuration >> (e.g. the import path) and runtime state (e.g. the set of imported modules). >> An interpreter is created at Python startup (Py_InitializeEx), and you can >> create/destroy additional ones with Py_NewInterpreter/Py_EndInterpreter. >> This is long-standing API that is used, most notably by mod_wsgi. >> Many extension modules and some stdlib modules don't play well with the >> existence of multiple interpreters in a process, mainly because they use >> process-global state (C static variables) rather than some more granular >> scope. >> This tends to result in nasty bugs (C-level crashes) when multiple >> interpreters are started in parallel (Py_NewInterpreter) or in sequence >> (several Py_InitializeEx/Py_FinalizeEx cycles). The bugs are similar in both >> cases. >> Whether Python interpreters run sequentially or in parallel, having them >> work will enable a use case I would like to see: allowing me to call Python >> code from wherever I want, without thinking about global state. Think >> calling Python from an utility library that doesn't care about the rest of >> the application it's used in. I personally call this "the Lua use case", >> because light-weight, worry-free embedding is an area where Python loses to >> Lua. (And JS as well—that's a relatively recent development, but much more >> worrying.) > > This seems like a worthwhile goal. However I don't see why this requires > having multiple Python interpreters in a single O/S process. The mod_wsgi use case seems to require this (he writes without having looked at its source code). I have another possible use case: Independent plugins written in Python in native applications written in other languages. That doesn’t mean that is worthwhile to complicate the CPython code base for these. I have no opinion on that, both because I haven’t been active for a while and because I haven’t looked at the impact the current work has had. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/GLKVB4JNZCZCXHNCF4F6VUBF7V6NKN5F/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)
Hi, as an user, the "lua use case" is right what I need at work. I realize that for python this is a niche case, and most users don't need any of this, but I hope it will useful to understand why having multiple independent interpreters in a single process can be an essential feature. The company I work for develop and sells a big C++ financial system with python embedded, providing critical flexibility to our customers. Python is used as a scripting language, with most cases having C++ calling a python script itself calling other C++ functions. Most of the times those scripts are in workloads I/O bound or where the time spent in python is negligible. But some workloads are really cpu bound and those tend to become GIL-bound, even with massive use of C++ helpers; some to the point that GIL-contention makes up over 80% of running time, instead of 1-5%. And every time our customers upgrade their server, they buy machines with more cores and the contention problem worsens. Obviously, our use case calls for per-thread separate interpreters: server processes run continuously and already consume gigabytes of RAM, so startup time or increased memory consumption are not issues. Shared state also is not needed, actually we try to avoid it as much as possible. In the end, removing process-global state is extremely interesting for us. Thank you, Riccardo ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/D6OSXZ7B6IAFPZN5VMZ6AEIWRPVO55I4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)
Eric V. Smith wrote: > On 6/10/2020 8:33 AM, Mark Shannon wrote: > > Hi Petr, > > On 09/06/2020 2:24 pm, Petr Viktorin wrote: > > On 2020-06-05 16:32, Mark Shannon wrote: > > Whether Python interpreters run sequentially or in parallel, having > > them work will enable a use case I would like to see: allowing me to > > call Python code from wherever I want, without thinking about global > > state. Think calling Python from an utility library that doesn't care > > about the rest of the application it's used in. I personally call > > this "the Lua use case", because light-weight, worry-free embedding > > is an area where Python loses to Lua. (And JS as well—that's a > > relatively recent development, but much more worrying.) > > This seems like a worthwhile goal. However I don't see why this > > requires having multiple Python interpreters in a single O/S process. > > I assume it would be so that my code could link with library A, which > embeds Python, and library B, which also embeds Python. A and B have no > knowledge of each other. > > The part I > > have been involved in is moving away from process-global > > state. Process-global state can be made to work, but it is much safer > > to always default to module-local state (roughly what > > Python-language's global means), and treat process-global state as > > exceptions one has to think through. The API introduced in PEPs 384, > > 489, 573 (and future planned ones) aims to make module-local state > > possible to use, then later easy to use, and the natural default. > > I don't agree. Process level state is much safer than module-local > > state. > > Suppose two interpreters, have both imported the same module. > > By using O/S processes to keep the interpreters separate, the hardware > > prevents the two copies of the module from interfering with each other. > > By sharing an address space the separation is maintained by trust and > > hoping that third party modules don't have too many bugs. > > I don't see how you can claim the later case if safer. > > I've always assumed that per-module state meant per-module, > per-interpreter. It _can_, but it isn't guaranteed because we are talking about C here and people do "interesting" things when they are handed that much flexibility. 😉 Plus a bunch of work has been done in the last few years to make per-interpreter state for modules be supported. -Brett > Maybe I've misunderstood, in which case I agree with > Mark. If per-module state isn't isolated per interpreter, that sort of > kills the multiple interpreter model, in my mind. > Eric ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/WEVPITVEFX72ONIEOJ5VQZSGOCPSRLSQ/ Code of Conduct: http://python.org/psf/codeofconduct/
