[Python-Dev] Re: subinterpreters and their possible impact on large extension projects
On 17. 12. 21 4:02, Jim J. Jewett wrote: Petr Viktorin wrote: In Python 3.11, Python still implements around 100 types as "static types" which are not compatible with subinterpreters, ... seems like changing it may break the C API *and* the stable ABI If sub-interpreters each need their own copy of even immutable built-in types, then what advantage do they have over separate processes? They need copies of all *Python* objects. A non-Python library may allow several Python wrappers/proxies for a single internal object, effectively sharing that object between subinterpreters. (Which is a problem for removing the GIL -- currently all operations done by such wrappers are protected by the GIL.) OK, so what is the advantage of having multiple interpreters? The only advantage I can see is that if you're embedding what are essentially several distinct python processes, you can still keep them all inside the single process used by the embedding program. But seems pretty far along the "they're already compiling anyhow; so the ABI isn't crucial" path. You should be able to use Python as an implementation detail of a library.For example, an application should be able to use several such libraries, without their Python runtimes influencing each other. See PEP 630 for some more details: https://www.python.org/dev/peps/pep-0630/#motivation ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/CQ36ECT4PFXQMPDIIDHCG2YFYFCAXDPZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Static types and subinterpreters running in parallel
On 16. 12. 21 20:24, Eric Snow wrote: On Thu, Dec 16, 2021 at 10:54 AM Guido van Rossum wrote: Eric has been looking into this. It's probably the only solution if we can't get immutable objects. Yep. I've investigated the following approach (for the objects exposed in the public and limited C-API): * add a pointer field to PyInterpreterState (or a sub-struct) for each of the objects * for the main interpreter, set those pointers to the existing statically declared objects * for subinterpreters make a copy (memcpy()?) and fix it up * add a lookup API and encourage extensions to use it * for 3.11+ change the symbols to macros: + in the internal C-API (Py_BUILD_CORE), the macro would resolve to the corresponding PyInterpreterState field + in the public C-API (and limited API extensions built with 3.11+), the macro would resolve to a call to a (non-inline) lookup function + for limited API extensions built against earlier Python versions we'd still export the existing symbols * limited API extensions built against pre-3.11 Python would only be allowed to run in the main interpreter on 3.11+ + they probably weren't built with subinterpreters in mind anyway There are still a number of details to sort out, but nothing that seems like a huge obstacle. Here are the ones that come to mind, along with other details, caveats, and open questions: * the static types exposed in the C-API are PyObject values rather than pointers + I solved this by dereferencing the result of the lookup function (Guido's idea), e.g. #define PyTuple_Type (*(_Py_GetObject_Tuple())) * there is definitely a penalty to using a per-interpreter lookup function + this would only apply to extension modules since internally we would access the PyInterpreterState fields directly + this is mostly a potential problem only when the object is directly referenced frequently (e.g. a tight loop), + the impact would probably center on use of the high-frequency singletons (None, True, False) and possibly with Py*_CheckExact() calls + would it be enough of a problem to be worth mitigating? how would we do so? * static types in extensions can't have tp_base set to a builtin type (since the macro won't resolve) + extensions that support subinterpreters (i.e. PEP 489) won't be using static types (a weak assumption) + extensions that do not support subinterpreters and still have static types would probably break + how to fix that? * limited API extensions built against 3.11+ but running under older Python versions would break? + how to fix that? With Py_LIMITED_API lower than 3.11, the old API/ABI should continue working. But with none of the new features. Or I guess we could start requiring users to compile stable-ABI extensions with the lowest Python version they support (which is currently just best practice). That would be a major change, though -- and a harsh reminder that the API is "limited", not "stable" like the ABI. All in all, immortals do seem like much better solution, if they can be made to work. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/ZDDAUMLNLWNTLGRQX3DH3MVAZY35EBVF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2021-12-10 - 2021-12-17) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open7180 (-15) closed 50665 (+93) total 57845 (+78) Open issues with patches: 2857 Issues opened (56) == #20741: Documentation archives should be available also in tar.xz form https://bugs.python.org/issue20741 reopened by iritkatriel #23522: Misleading note in Statistics module documentation https://bugs.python.org/issue23522 reopened by gvanrossum #29221: ABC Recursion Error on isinstance() with less than recursion l https://bugs.python.org/issue29221 reopened by serhiy.storchaka #43749: venv module does not copy the correct python exe https://bugs.python.org/issue43749 reopened by eryksun #44413: OverflowError: mktime argument out of range after 2019 https://bugs.python.org/issue44413 reopened by andrei.avk #46044: Update distutils documentation to say PyPI only accepts tar.g https://bugs.python.org/issue46044 opened by mbussonn #46045: NetBSD: do not use POSIX semaphores https://bugs.python.org/issue46045 opened by wiz #46050: [pathlib] Option so that OSError does not block glob in pathli https://bugs.python.org/issue46050 opened by matt32106 #46051: Make @atexit.register work for functions with arguments https://bugs.python.org/issue46051 opened by quapka #46052: IDLE: make Ctrl,Alt + IME non-ascii letter work on Windows https://bugs.python.org/issue46052 opened by anton.bryl #46053: NetBSD: ossaudio support incomplete https://bugs.python.org/issue46053 opened by wiz #46054: Incorrect error when parsing non-utf8 files https://bugs.python.org/issue46054 opened by pablogsal #46055: Speed up binary shifting operators https://bugs.python.org/issue46055 opened by xuxinhang #46061: Journal execution gives fatal error in Python 3.10.1 https://bugs.python.org/issue46061 opened by eaqrzn #46064: Permalinks to underscored documentation entries don't work. https://bugs.python.org/issue46064 opened by Fabian Dill #46065: re.findall takes forever and never ends https://bugs.python.org/issue46065 opened by ramzitra #46066: TypedDict alternative definition syntax with keyword args is c https://bugs.python.org/issue46066 opened by 97littleleaf11 #46067: SSLContext.set_npn_protocols broken in Python 3.10, tries to c https://bugs.python.org/issue46067 opened by diabonas #46068: Change use of warnings.warn to logging.warning in a few places https://bugs.python.org/issue46068 opened by andrei.avk #46070: _PyImport_FixupExtensionObject() regression causing a crash in https://bugs.python.org/issue46070 opened by graysky #46071: Graphlib documentation https://bugs.python.org/issue46071 opened by dam1784 #46072: Unify handling of stats in the CPython VM https://bugs.python.org/issue46072 opened by Mark.Shannon #46073: ast.unparse produces: 'FunctionDef' object has no attribute 'l https://bugs.python.org/issue46073 opened by TheRobotCarlson #46075: CookieJar.extract_cookies doesn't process cookies form local d https://bugs.python.org/issue46075 opened by keddad #46076: Document using __slots__ to provide per-attribute docstrings https://bugs.python.org/issue46076 opened by AlexWaygood #46077: Include sha256 hashes of release downloads in announcement com https://bugs.python.org/issue46077 opened by gregory.p.smith #46079: [doc] Broken URL in "Brief Tour of the Standard Library" https://bugs.python.org/issue46079 opened by vivekvashist #46080: argparse.BooleanOptionalAction with default=argparse.SUPPRESS https://bugs.python.org/issue46080 opened by felixfontein #46083: PyUnicode_FSConverter() has confusing reference semantics https://bugs.python.org/issue46083 opened by twouters #46084: Python 3.9.6 scan_dir returns filenotfound on long paths, but https://bugs.python.org/issue46084 opened by jschwar313 #46085: OrderedDict iterator allocates di_result unnecessarily https://bugs.python.org/issue46085 opened by Kevin Shweh #46086: Add ratio_min() function to the difflib library https://bugs.python.org/issue46086 opened by gibu #46088: Build hangs under Visual Studio in deepfreeze stage https://bugs.python.org/issue46088 opened by gvanrossum #46089: Problems with AF_PACKET sockets https://bugs.python.org/issue46089 opened by gallard #46090: C extensions can't swap out live frames anymore https://bugs.python.org/issue46090 opened by brandtbucher #46091: IndendationError from multi-line indented statements https://bugs.python.org/issue46091 opened by ucodery #46092: Fix/update missing parameters in function signatures for Built https://bugs.python.org/issue46092 opened by vivekvashist #46094: Missing unit test on unittest.TestResult to check for required https://bugs.python.org/issue46094 opened by DreamSh0t #46095: Warning about iterate/modify has unwarranted detail https://bugs.python.org/issue46095 open
[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL
fwiw we added immortal objects to Pyston and haven't run into any issues with it. The goal is a bit different: to eliminate common refcount operations for performance, which we can do a bit more of because we have a jit. And we don't mind if unaware code ends up changing the refcounts of immortal objects since it's no worse for us than before. So anyway maybe it's not super comparable for the issues discussed here, but at least we haven't run into any issues of extension modules being confused by very large reference counts. The one issue we do run into is that quite a few projects will test in debug mode that their c extension doesn't leak reference counts, and that no longer works for us because we don't update Py_RefTotal for immortal objects. kmod On Wed, Dec 15, 2021 at 2:02 PM Eric Snow wrote: > On Tue, Dec 14, 2021 at 11:19 AM Eric Snow > wrote: > > There is one solution that would help both of the above in a nice way: > > "immortal" objects. > > FYI, here are some observations that came up during some discussions > with the "faster-cpython" team today: > > * immortal objects should probably only be immutable ones (other than > ob_refcnt, of course) > * GC concerns are less of an issue if a really high ref count (bit) is > used to identify immortal objects > * ob_refcnt is part of the public API (sadly), so using it to mark > immortal objects may be sensitive to interference > * ob_refcnt is part of the stable ABI (even more sadly), affecting any > solution using ref counts > * using the ref count isn't the only viable approach; another would be > checking the pointer itself >+ put the object in a specific section of static data and compare > the pointer against the bounds >+ this avoids loading the actual object data if it is immortal >+ for objects that are mostly treated as markers (e.g. None), this > could have a meaningful impact >+ not compatible with dynamically allocated objects > > -eric > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/LVLFPOIOXM34NQ2G73BAXIRS4TIN74JV/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/RX7JKJTATP4IUIWB5SPUIARSYM2C56EQ/ Code of Conduct: http://python.org/psf/codeofconduct/
