[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2020-04-10 - 2020-04-17) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open7443 (+10) closed 44618 (+55) total 52061 (+65) Open issues with patches: 2932 Issues opened (40) == #25680: Selector.select() hangs when there is nothing to select https://bugs.python.org/issue25680 reopened by gvanrossum #40247: Logged out of user when running Tkinter https://bugs.python.org/issue40247 opened by Jordan #40249: __import__ doesn't honour globals https://bugs.python.org/issue40249 opened by stefan #40250: unable to comment out r'\u' string with triple quote marks https://bugs.python.org/issue40250 opened by Elmar Bucher #40251: selectors.KqueueSelector hangs on EOF, unlike other selectors https://bugs.python.org/issue40251 opened by russelldavis #40252: selectors.KqueueSelector should not be the default selector https://bugs.python.org/issue40252 opened by russelldavis #40253: Fix .py(w) file association with Pyhon 3 Windows installer https://bugs.python.org/issue40253 opened by virtualnobi #40254: pyspecific directives are not translatable https://bugs.python.org/issue40254 opened by cocoatomo #40255: Fixing Copy on Writes from reference counting https://bugs.python.org/issue40255 opened by eelizondo #40256: Python 3.8 Not Launching on Bootcamp Windows 10. https://bugs.python.org/issue40256 opened by YusufM #40257: Improve the use of __doc__ in pydoc https://bugs.python.org/issue40257 opened by serhiy.storchaka #40259: re.Scanner groups https://bugs.python.org/issue40259 opened by dchron #40260: modulefinder traceback regression starting on Windows https://bugs.python.org/issue40260 opened by barry-scott #40261: Build of Python where make is called from subprocess, within a https://bugs.python.org/issue40261 opened by rdbisme #40262: SSL recv_into requires the object to implement __len__ unlike https://bugs.python.org/issue40262 opened by tzickel #40263: ValueError exception on _winapi.WaitForMultipleObjects https://bugs.python.org/issue40263 opened by Ray Donnelly #40269: Inconsistent complex behavior with (-1j) https://bugs.python.org/issue40269 opened by rushilu #40272: ModuleNotFoundEror thrown by system python while accessing it https://bugs.python.org/issue40272 opened by gavin #40273: mappingproxy isn't reversible https://bugs.python.org/issue40273 opened by jack1142 #40275: test.support has way too many imports https://bugs.python.org/issue40275 opened by vstinner #40278: pathlib Path.replace raises OSError when target exists https://bugs.python.org/issue40278 opened by selik #40279: Documentation example of module init function lacks error hand https://bugs.python.org/issue40279 opened by scoder #40280: Consider supporting emscripten/webassembly as a build target https://bugs.python.org/issue40280 opened by Simon Biggs #40283: Documentation of turtle.circle() https://bugs.python.org/issue40283 opened by guchao #40284: Add mapping methods to types.SimpleNamespace https://bugs.python.org/issue40284 opened by rhettinger #40288: atexit module should not be loaded more than once per interpre https://bugs.python.org/issue40288 opened by vstinner #40289: "highlighting" how to get Python's Scripts directory in the do https://bugs.python.org/issue40289 opened by thautwarm #40291: socket library support for CAN_J1939 https://bugs.python.org/issue40291 opened by karlding #40293: Tag libffi build and sources in cpython-source-deps for 3.9.0b https://bugs.python.org/issue40293 opened by indygreg #40296: help(list[int]) fails https://bugs.python.org/issue40296 opened by serhiy.storchaka #40297: test_socket.CANTest is broken at HEAD on master https://bugs.python.org/issue40297 opened by karlding #40300: logging.Formatter crashes when default_msec_format is None. https://bugs.python.org/issue40300 opened by felixxm #40301: zipfile module: new feature (two lines of code), useful for te https://bugs.python.org/issue40301 opened by massimosala #40304: Classes created using type() don't need to explicitly inherit https://bugs.python.org/issue40304 opened by boris #40305: Fix server_close() method for ThreadingHTTPServer and TCPServe https://bugs.python.org/issue40305 opened by ROUX antoine2 #40306: Enhancement request for SSLContext - flag to handle trailing d https://bugs.python.org/issue40306 opened by bli2020 #40307: multiprocessing.BaseManager does not retain Client type. https://bugs.python.org/issue40307 opened by amol #40308: Intermittent failure of test_os.TestScandir.test_attributes on https://bugs.python.org/issue40308 opened by Dennis Sweeney #40309: âunmatched parenâ for space before parenthesis in Py_Build https://bugs.python.org/issue40309 opened by Wrzlprmft #40311: docs.python.org banner - release blocker for 2.7.18 https://bugs.python.org/issue4031
[Python-Dev] PEP 554 for 3.9 or 3.10?
Hi all, It was nice to speak with many of you at the online language summit this week. It was the next best thing to seeing you all in person! :) With the 3.9 feature freeze coming up I'm considering options for PEP 554. I'm hopeful to have a pronouncement from Antoine in the near future. If that comes in time for 3.9, we will have the implementation ready to go. It is already nearly complete and the remaining work should be done in the next week or so. I'm looking for feedback on a nagging worry I have with merging PEP 554 into 3.9 (if it is accepted) without having the per-interpreter GIL. See below. I'll update the PEP later with relevant info from this discussion. wishing-you-were-all-here-ly yours, -eric + (++ Dilemma Many folks have conflated PEP 554 with having a per-interpreter GIL. In fact, I was careful to avoid any mention of parallelism or the GIL in the PEP. Nonetheless some are expecting that when PEP 554 lands we will reach multi-core nirvana. While PEP 554 might be accepted and the implementation ready in time for 3.9, the separate effort toward a per-interpreter GIL is unlikely to be sufficiently done in time. That will likely happen in the next couple months (for 3.10). So...would it be sufficiently problematic for users if we land PEP 554 in 3.9 without per-interpreter GIL? Options Here are the options as I see them (if the PEP is accepted in time for 3.9): 1. merge PEP 554 into 3.9 even if per-interpreter GIL doesn't get into 3.9 (they get parallelism for free in 3.10) 2. like 1, but mark the module as provisional until per-interpreter GIL lands 3. do not merge PEP 554 until per-interpreter GIL is merged 4. like 3, but publish a 3.9-only module to PyPI in the meantime Context Like I said, PEP 554 purposefully does not talk about parallelism or the GIL. Instead it focuses strictly on the following (with an emphasis on minimalism): * exposing the existing C-API * supporting a new concurrency model (CSP) Regarding that first one, the value is in exposing the existing functionality to a broader group of people. keep in mind that subinterpreters currently have various other limitations (aside from sharing the GIL): * nothing about them has been optimized (e.g. interpreter startup, data sharing, data passing) * extension modules with process-global state may break * some bugs still remain in the corners Currently, there is a chicken-and-egg problem. It is unlikely that there will be sufficient motivation to address those limitations until the community starts really using subinterpreters. Hence PEP 554. My Position At this point I think we should go with option #1 (or possibly #2). IMHO, we would be fine merging PEP 554 without per-interpreter GIL. Here's why: * the two objectives of the PEP have value for the community (as explained in the PEP) * the sooner we can get the functionality of subinterpreters out to the broader world, the better * I don't think the lack of parallelism in 3.9 will trip anyone up My only concern is that folks try them out in 3.9, get frustrated by the limitations, and then there is a mental barrier to trying them again in the future when the limitations have been reduced or eliminated. However, I don't think that will be a problem for people that would benefit from serious use of subinterpreters. Furthermore, if we're clear that "for now" subinterpreters still share the GIL in 3.9 then I'm not so worried about folks getting confused. In fact, as long as we're clear about all the current limitations then there isn't much downside to option #1. Marking the module as "provisional" may help communicate that there are limitations that impact use. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/3HVRFWHDMWPNR367GXBILZ4JJAUQ2STZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
Eric Snow wrote: > Hi all, > It was nice to speak with many of you at the online language summit > this week. It was the next best thing to seeing you all in person! :) > With the 3.9 feature freeze coming up I'm considering options for PEP > > I'm hopeful to have a pronouncement from Antoine in the near > future. If that comes in time for 3.9, we will have the > implementation ready to go. It is already nearly complete and the > remaining work should be done in the next week or so. > > I'm looking for feedback on a nagging worry I have with merging PEP > 554 into 3.9 (if it is accepted) without having the per-interpreter > GIL. See below. I'll update the PEP later with relevant info from > this discussion. > wishing-you-were-all-here-ly yours, > -eric > + (++ > Dilemma > Many folks have conflated PEP 554 with having a per-interpreter GIL. > In fact, I was careful to avoid any mention of parallelism or the GIL > in the PEP. Nonetheless some are expecting that when PEP 554 lands we > will reach multi-core nirvana. > While PEP 554 might be accepted and the implementation ready in time > for 3.9, the separate effort toward a per-interpreter GIL is unlikely > to be sufficiently done in time. That will likely happen in the next > couple months (for 3.10). > So...would it be sufficiently problematic for users if we land PEP 554 > in 3.9 without per-interpreter GIL? > Options > Here are the options as I see them (if the PEP is accepted in time for 3.9): > > 1. merge PEP 554 into 3.9 even if per-interpreter GIL doesn't get into 3.9 > (they get parallelism for free in 3.10) > 2. like 1, but mark the module as provisional until per-interpreter GIL lands > 3. do not merge PEP 554 until per-interpreter GIL is merged > 4. like 3, but publish a 3.9-only module to PyPI in the meantime My vote is for #2. No need to rush it and it gives you feedback on the API from the public before it gets locked in for backwards-compatibility. And you do know you could have done this as a poll on Discourse, right? 😉 -Brett > > Context > Like I said, PEP 554 purposefully does not talk about parallelism or > the GIL. Instead it focuses strictly on the following (with an > emphasis on minimalism): > > exposing the existing C-API > supporting a new concurrency model (CSP) > > Regarding that first one, the value is in exposing the existing > functionality to a broader group of people. keep in mind that > subinterpreters currently have various other limitations (aside from > sharing the GIL): > > nothing about them has been optimized (e.g. interpreter startup, > data sharing, data passing) > extension modules with process-global state may break > some bugs still remain in the corners > > Currently, there is a chicken-and-egg problem. It is unlikely that > there will be sufficient motivation to address those limitations until > the community starts really using subinterpreters. Hence PEP 554. > My Position > At this point I think we should go with option #1 (or possibly #2). > IMHO, we would be fine merging PEP 554 without per-interpreter GIL. > Here's why: > > the two objectives of the PEP have value for the community (as > explained in the PEP) > the sooner we can get the functionality of subinterpreters out to > the broader world, the better > I don't think the lack of parallelism in 3.9 will trip anyone up > > My only concern is that folks try them out in 3.9, get frustrated by > the limitations, and then there is a mental barrier to trying them > again in the future when the limitations have been reduced or > eliminated. However, I don't think that will be a problem for people > that would benefit from serious use of subinterpreters. > Furthermore, if we're clear that "for now" subinterpreters still share > the GIL in 3.9 then I'm not so worried about folks getting confused. > In fact, as long as we're clear about all the current limitations then > there isn't much downside to option #1. > Marking the module as "provisional" may help communicate that there > are limitations that impact use. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/WCMLX56RRZYGTWQFIL2D2D3FHEI2CZI2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
Hi Eric, Le ven. 17 avr. 2020 à 20:56, Eric Snow a écrit : > With the 3.9 feature freeze coming up I'm considering options for PEP > 554. I'm hopeful to have a pronouncement from Antoine in the near > future. If that comes in time for 3.9, we will have the > implementation ready to go. It is already nearly complete and the > remaining work should be done in the next week or so. I have concerns about shared data. The current implementation of CPython and subinterpreters still shares singletons. Moreover, I understand that the PEP proposes to first directly share PyObject between subinterpreters, like bytes objects. Isolating subinterpreters is a big task and so all issues cannot be solved at once. So I am not with making some compromises for a first milestone. The problem is to define which semantics can be exposed in the public API for the first milestone. -- Sharing directly singletons like None can become a performance kill once subinterpreters will run in parallel: https://bugs.python.org/issue39511 Mark Shannon summarized: "Having two CPUs write to the same cache line is a well known performance problem. There's nothing special about CPython here. The proper name for it seems to be "cache line ping-pong", but a search for "false sharing" might be more informative." The problem is that PyObject.ob_refcnt should be protected by atomic operations or locks if these objects are shared directly. I proposed to have singletons per interpreter, some people would prefer immortal singletons. -- For bytes or buffer objects, I understand that you propose to share the exact same PyObject objects between interpreters, at least in the first implementation. It may be better to have one proxy object in each interpreter which would control which interpreters can read and which interpreters can write into the object. It's a similar but more complex issue than singletons. -- What is the plan for Python 3.9? What semantics would be exposed in Python 3.9? Should we wait until these questions are answered before making the API public? Victor > I'm looking for feedback on a nagging worry I have with merging PEP > 554 into 3.9 (if it is accepted) without having the per-interpreter > GIL. See below. I'll update the PEP later with relevant info from > this discussion. > > wishing-you-were-all-here-ly yours, > > -eric > > + (++ > > Dilemma > > > Many folks have conflated PEP 554 with having a per-interpreter GIL. > In fact, I was careful to avoid any mention of parallelism or the GIL > in the PEP. Nonetheless some are expecting that when PEP 554 lands we > will reach multi-core nirvana. > > While PEP 554 might be accepted and the implementation ready in time > for 3.9, the separate effort toward a per-interpreter GIL is unlikely > to be sufficiently done in time. That will likely happen in the next > couple months (for 3.10). > > So...would it be sufficiently problematic for users if we land PEP 554 > in 3.9 without per-interpreter GIL? > > Options > > > Here are the options as I see them (if the PEP is accepted in time for 3.9): > > 1. merge PEP 554 into 3.9 even if per-interpreter GIL doesn't get into > 3.9 (they get parallelism for free in 3.10) > 2. like 1, but mark the module as provisional until per-interpreter GIL lands > 3. do not merge PEP 554 until per-interpreter GIL is merged > 4. like 3, but publish a 3.9-only module to PyPI in the meantime > > Context > > > Like I said, PEP 554 purposefully does not talk about parallelism or > the GIL. Instead it focuses strictly on the following (with an > emphasis on minimalism): > > * exposing the existing C-API > * supporting a new concurrency model (CSP) > > Regarding that first one, the value is in exposing the existing > functionality to a broader group of people. keep in mind that > subinterpreters currently have various other limitations (aside from > sharing the GIL): > > * nothing about them has been optimized (e.g. interpreter startup, > data sharing, data passing) > * extension modules with process-global state may break > * some bugs still remain in the corners > > Currently, there is a chicken-and-egg problem. It is unlikely that > there will be sufficient motivation to address those limitations until > the community starts really using subinterpreters. Hence PEP 554. > > My Position > > > At this point I think we should go with option #1 (or possibly #2). > IMHO, we would be fine merging PEP 554 without per-interpreter GIL. > Here's why: > > * the two objectives of the PEP have value for the community (as > explained in the PEP) > * the sooner we can get the functionality of subinterpreters out to > the broader world, the better > * I don't think the lack of parallelism in 3.9 will trip anyone up > > My only concern is that folks try them out in 3.9, get frustrated by > the limitations, and then there is a mental barrier to trying
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, Apr 17, 2020 at 1:38 PM Brett Cannon wrote: > Eric Snow wrote: > > 1. merge PEP 554 into 3.9 even if per-interpreter GIL doesn't get into 3.9 > > (they get parallelism for free in 3.10) > > 2. like 1, but mark the module as provisional until per-interpreter GIL > > lands > > 3. do not merge PEP 554 until per-interpreter GIL is merged > > 4. like 3, but publish a 3.9-only module to PyPI in the meantime > > My vote is for #2. No need to rush it and it gives you feedback on the API > from the public before it gets locked in for backwards-compatibility. Yeah, I'd be fine with that. Actually, making the module provisional is already part of the PEP. :) > And you do know you could have done this as a poll on Discourse, right? Yeah, I thought of that after the fact. Then again, discourse isn't the tool I already had open. :) -eric ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/CDVYBRIZ77BJN7WGGSLWULZ22SLB2TB5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, Apr 17, 2020 at 1:39 PM Victor Stinner wrote: > I have concerns about shared data. > > The current implementation of CPython and subinterpreters still shares > singletons. Moreover, I understand that the PEP proposes to first > directly share PyObject between subinterpreters, like bytes objects. Hmm, the PEP explicitly says: Note that objects are not shared between interpreters since they are tied to the interpreter in which they were created. Instead, the objects' data is passed between interpreters. See the Shared data section for more details about sharing between interpreters. > Isolating subinterpreters is a big task and so all issues cannot be > solved at once. So I am not with making some compromises for a first > milestone. +1 To me, one of the most important things is to get the functionality out there as soon as possible, even if it is limited (but not useless). > The problem is to define which semantics can be exposed in the public > API for the first milestone. > > [snip] > > What is the plan for Python 3.9? What semantics would be exposed in Python > 3.9? > > Should we wait until these questions are answered before making the API > public? The PEP proposes minimal functionality. We can relax that in future versions of Python on a case-by-case basis. If you are specifically referring to object sharing, the PEP already says there is none. -eric ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/QVCKZVUECJQ3DCXO2GQ7HZA4HNAWB734/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
Le ven. 17 avr. 2020 à 20:56, Eric Snow a écrit : > keep in mind that > subinterpreters currently have various other limitations (aside from > sharing the GIL): > > * nothing about them has been optimized (e.g. interpreter startup, > data sharing, data passing) > * extension modules with process-global state may break > * some bugs still remain in the corners Honestly, I don't think that we are far from being able to move the GIL lock per interpreter. But this change alone is useless, since we still have too many C extensions which are not ready for that: * C extension modules should be converted to multiphase initialization (PEP 489) * Statically allocated types should be converted to heap allocated types (PyType_FromSpec) * A module state should be added to a bunch of C extension modules * Many C extension modules don't implement the GC protocol, since it wasn't really needed previously: this should now be done (implement traverse, clear, free) * These two changes require PEP 573 to be implemented to get access the module state in functions where it wasn't possible previously The good news is that there is an on-going effort to complete this task: https://bugs.python.org/issue1635741 More and more C extension modukes are converted. The bad news is that better isolating subinterpreters "suddently" triggers old issues. See for example https://bugs.python.org/issue40217 "The garbage collector doesn't take in account that objects of heap allocated types hold a strong reference to their type" which is a very complex issue. I also fixed many reference cycles and other bugs, discovered after some specific C extensions were modified. There are still tons of "static" variables all around the code and still no technical solution to have "per interpreter" variable. We need something like Thread-Locale State, but for interpreters in the C API. PyInterpreter.dict is not convenient in C. Well, I'm not writing that for Eric who is already well aware of these issues, but for others ;-) > At this point I think we should go with option #1 (or possibly #2). > IMHO, we would be fine merging PEP 554 without per-interpreter GIL. I would prefer PEP 554 to be restricted to the communication between interpreters. Isolating subinterpreters should be out of the scope of this PEP, since it's a different problem and we cannot solve all problems in a single PEP. See what happened with PEP 432 which tried to solve multiple related problems: PEP 587 fixed the preinitialization and the configuration, but not the "path configuration" issue. Excluding the "path configuration" from the PEP 432 in the PEP 587 made possible to get a PEP "done". Not sure if a PEP is needed to isolate subinterpreters. There are already many issues and https://github.com/ericsnowcurrently/multi-core-python/ to track the work. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/U65UDU5V64EPYHPBLQKOEU3A23ZXIBY7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, Apr 17, 2020 at 1:39 PM Victor Stinner wrote: > Sharing directly singletons like None can become a performance kill > once subinterpreters will run in parallel: > https://bugs.python.org/issue39511 > > Mark Shannon summarized: "Having two CPUs write to the same cache line > is a well known performance problem. There's nothing special about > CPython here. The proper name for it seems to be "cache line > ping-pong", but a search for "false sharing" might be more > informative." > > The problem is that PyObject.ob_refcnt should be protected by atomic > operations or locks if these objects are shared directly. I agree. Sharing objects between subinterpreters is a recipe for pain. That's why the PEP says we will not. :) > I proposed to have singletons per interpreter, some people would > prefer immortal singletons. Yeah, we definitely have to sort that out before we make the GIL per-interpreter. > For bytes or buffer objects, I understand that you propose to share > the exact same PyObject objects between interpreters, at least in the > first implementation. > > It may be better to have one proxy object in each interpreter which > would control which interpreters can read and which interpreters can > write into the object. It's a similar but more complex issue than > singletons. It isn't the same Python object. It is the buffer that gets shared (for objects that support the buffer protocol). However, I suppose that it will have a similar problem to what you described above, since it is the same (read-only) memory in both interpreters. I'm not too worried about that at the moment though, particularly if we make the module provisional. -eric ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/FT6UOCKNOHAZVNNU6V5X6TYWCXFPWMWM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, Apr 17, 2020 at 1:54 PM Victor Stinner wrote: > Honestly, I don't think that we are far from being able to move the > GIL lock per interpreter. But this change alone is useless, since we > still have too many C extensions which are not ready for that: Agreed. This is a big part of why I don't think we can make the GIL per-interpreter for 3.9. However, there are some things we could do to mitigate the problem if we want to aim for 3.9: * only allow the main interpreter to import extension modules that do not implement PEP 489 support + in subinterpreters it would be an ImportError * allow subinterpreters to be created either isolated (not sharing the GIL, etc.) or not isolated (sharing the GIL, etc.) + the default is "not isolated" (but only while the "interpreters" module is provisional) and users must opt in to isolated mode + this would allow extension authors to more easily check their modules for subinterpreter support, without causing excessive bug reports from users trying out subinterpreters I also am not convinced that in the next month we can work out all the possible corner cases problems that I expect will pop up once we have a per-interpreter GIL. Then again, arguably that could be addressed after the feature freeze if we think those possible issues could be sorted out in a month or two. :) > I would prefer PEP 554 to be restricted to the communication between > interpreters. > > Isolating subinterpreters should be out of the scope of this PEP, > since it's a different problem and we cannot solve all problems in a > single PEP. Isolating interpreters isn't part of PEP 554, other than that objects are not shared between interpreters. > Not sure if a PEP is needed to isolate subinterpreters. There are > already many issues and > https://github.com/ericsnowcurrently/multi-core-python/ to track the > work. Agreed. I don't know that anyone had or has any intention of adding a PEP related to isolation (i.e. moving the GIL to PyInterpreterState). -eric ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/QBY4JIXMYNWJOY3POH6YV35FWI2H6Z7I/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, Apr 17, 2020 at 11:50 AM Eric Snow wrote: > Dilemma > > > Many folks have conflated PEP 554 with having a per-interpreter GIL. > In fact, I was careful to avoid any mention of parallelism or the GIL > in the PEP. Nonetheless some are expecting that when PEP 554 lands we > will reach multi-core nirvana. > > While PEP 554 might be accepted and the implementation ready in time > for 3.9, the separate effort toward a per-interpreter GIL is unlikely > to be sufficiently done in time. That will likely happen in the next > couple months (for 3.10). > > So...would it be sufficiently problematic for users if we land PEP 554 > in 3.9 without per-interpreter GIL? > > Options > > > Here are the options as I see them (if the PEP is accepted in time for 3.9): > > 1. merge PEP 554 into 3.9 even if per-interpreter GIL doesn't get into > 3.9 (they get parallelism for free in 3.10) > 2. like 1, but mark the module as provisional until per-interpreter GIL lands > 3. do not merge PEP 554 until per-interpreter GIL is merged > 4. like 3, but publish a 3.9-only module to PyPI in the meantime I think some perspective might be useful here :-). The last time we merged a new concurrency model in the stdlib, it was asyncio. In that case, the process went something like: - We started with two extremely mature libraries (Twisted + Tornado) with long histories of real-world use - The asyncio designers (esp. Guido) did a very extensive analysis of these libraries' design choices, spoke to the maintainers about what they'd learned from hard experience, etc. - Asyncio was initially shipped outside the stdlib to allow for testing and experimentation, and at this stage it was used to build non-trivial projects (e.g. the aiohttp project's first commits use tulip, not asyncio) - When it was eventually added to the stdlib, it was still marked provisional for multiple python releases, and underwent substantial and disruptive changes during this time - Even today, the limitations imposed by the stdlib release cycle still add substantial difficulty to maintaining asyncio OTOH, AFAICT the new concurrency model in PEP 554 has never actually been used, and it isn't even clear whether it's useful at all. Designing useful concurrency models is *stupidly* hard. And on top of that, it requires major reworks of the interpreter internals + disrupts the existing C extension module ecosystem -- which is very different from asyncio, where folks who didn't use it could just ignore it. So to me, it's kind of shocking that you'd even bring up the possibility of merging PEP 554 as-is, without even a provisional marker. And if it's possible for it to live on PyPI, then why would we even consider putting it into the stdlib? Personally, I'm still leaning towards thinking that the whole subinterpreter project is fundamentally flawed, and that on net we'd be better off removing support for them entirely. But that's a more complex and nuanced question that I'm not 100% certain of, while the idea of merging it for 3.9 seems like a glaringly obvious bad idea. I know you want folks to consider PEP 554 on its own merits, ignoring the GIL-splitting work, but let's be realistic: purely as a concurrency framework, there's at least a dozen more mature/featureful/compelling options in the stdlib and on PyPI, and as an isolation mechanism, subinterpreters have been around for >20 years and in that time they've found 3 users and no previous champions. Obviously the GIL stuff is the only reason PEP 554 might be worth accepting. Or if PEP 554 is really a good idea on its own merits, purely as a new concurrency API, then why not build that concurrency API on top of multiprocessing and put it on PyPI and let real users try it out? One more thought. Quoting from Poul Henning-Kemp's famous email at bikeshed.org: > Parkinson shows how you can go in to the board of directors and get approval for building a multi-million or even billion dollar atomic power plant, but if you want to build a bike shed you will be tangled up in endless discussions. > > Parkinson explains that this is because an atomic plant is so vast, so expensive and so complicated that people cannot grasp it, and rather than try, they fall back on the assumption that somebody else checked all the details before it got this far. Richard P. Feynmann gives a couple of interesting, and very much to the point, examples relating to Los Alamos in his books. > > A bike shed on the other hand. Anyone can build one of those over a weekend, and still have time to watch the game on TV. So no matter how well prepared, no matter how reasonable you are with your proposal, somebody will seize the chance to show that he is doing his job, that he is paying attention, that he is *here*. Normally, when people reference this story they focus on the bikeshed, hence the term "bikeshedding". But for PEP 554, you're building a nuclear power plant :-). The whole conglomeration of a new concurrency
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, Apr 17, 2020 at 2:59 PM Nathaniel Smith wrote: > I think some perspective might be useful here :-). > > The last time we merged a new concurrency model in the stdlib, it was asyncio. > > [snip] > > OTOH, AFAICT the new concurrency model in PEP 554 has never actually > been used, and it isn't even clear whether it's useful at all. Perhaps I didn't word things quite right. PEP 554 doesn't provide a new concurrency model so much as it provides functionality that could probably be used as the foundation for one. Ultimately the module proposed in the PEP does the following: * exposes the existing subinterpreters functionality almost as-is * provides a minimal way to pass information between subinterpreters (which you don't need in C but do in Python code) * adds a few minor conveniences like propagating exceptions and making it easier to share buffers safely So the comparison with asyncio isn't all that fair. > Designing useful concurrency models is *stupidly* hard. And on top of > that, it requires major reworks of the interpreter internals + Nearly all of the "rework" is worth doing for other good reasons. Furthermore, I'd call at most a few of the reworks "major". > disrupts the existing C extension module ecosystem -- which is very > different from asyncio, where folks who didn't use it could just > ignore it. The concern is users opening issues saying "your extension won't work in subinterpreters", right? (You brought up the possible burden on extension authors in discussions several years ago, for which I'm still grateful.) How is that different from any other feature, new or not? "Your library doesn't provide an awaitable API." "Your library doesn't support pickling." "Your library doesn't implement the buffer protocol." PEP 554 doesn't introduce some new kind of impact. So (unless I've misunderstood), by your reasoning we wouldn't add any new features for which library authors might have to change something. Are you suggesting that the burden from PEP 554 will be larger than saying "then don't try to use our extension in subinterpreters"? Are you concerned about users reporting bugs that surface when an incompatible extension is used in a subinterpreter? That shouldn't be a problem if we raise ImportError if an extension that does not support PEP 489 is imported in a subinterpreter. FWIW, the impact to extension authors is the one thing about which I still have any meaningful uncertainty and worry. Various people have explained to me how it won't be a big problem, but I'm still nervous about it. I just don't think my worry is large than the actual risk (and possible cost). > So to me, it's kind of shocking that you'd even bring up the > possibility of merging PEP 554 as-is, without even a provisional > marker. Unsurprisingly it isn't shocking to me. :) From my point of view it seems okay. However, I'll be the first to recognize how hard it can be to see things from a different perspective. Hence I started this thread. :) > And if it's possible for it to live on PyPI, then why would we > even consider putting it into the stdlib? Given that we're exposing functionality of the CPython runtime I don't see the point in keeping this out of the stdlib. Furthermore, there are use cases to explore for subinterpreters in our test suite that we can address only if the "interpreters" module is part of the CPython repo. So why keep it hidden away and then publish the exact same thing on PyPI? > Personally, I'm still > leaning towards thinking that the whole subinterpreter project is > fundamentally flawed, and that on net we'd be better off removing > support for them entirely. By which I imagine you mean drop the subinterpreters API and not actually get rid of all the architecture related to PyInterpreterState (which is valuable from a code health perspective). > But that's a more complex and nuanced > question that I'm not 100% certain of, while the idea of merging it > for 3.9 seems like a glaringly obvious bad idea. Yeah, I remember your position from previous conversations (and still appreciate your feedback). :) > I know you want folks to consider PEP 554 on its own merits, ignoring > the GIL-splitting work, but let's be realistic: purely as a > concurrency framework, there's at least a dozen more > mature/featureful/compelling options in the stdlib and on PyPI, and as > an isolation mechanism, subinterpreters have been around for >20 years > and in that time they've found 3 users and no previous champions. > Obviously the GIL stuff is the only reason PEP 554 might be worth > accepting. Saying it's "obviously" the "only" reason is a bit much. :) PEP 554 exposes existing functionality that hasn't been all that popular (until recently for some reason ) mostly because it is old, was never publicized (until recently), and involved using the C-API. As soon as folks learn about it they want it, for various reasons including (relative) isolation and reduced resource
