Re: [Python-Dev] SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register
I feel like you are actively undermining attempts to prevent exploitation of known vulnerabilities because the software in question is currently too slow. For a 4-8% performance penalty, we could just add the CFLAGS to the build now and not worry about it. I give up. On Friday, September 21, 2018, Franklin? Lee wrote: > On Thu, Sep 20, 2018 at 2:10 PM Wes Turner wrote: > > > > On Thursday, September 20, 2018, Stefan Ring > wrote: > >> > >> On Tue, Sep 18, 2018 at 8:38 AM INADA Naoki > wrote: > >> > >> > I think this topic should split to two topics: (1) Guard Python > >> > process from Spectre/Meltdown > >> > attack from other process, (2) Prohibit Python code attack other > >> > processes by using > >> > Spectre/Meltdown. > >> > >> (3) Guard Python from performance degradation by overly aggressive > >> Spectre "mitigation". > > > > > > > Spectre has the potential of having a greater impact on cloud > providers than Meltdown. Whereas Meltdown allows unauthorized applications > to read from privileged memory to obtain sensitive data from processes > running on the same cloud server, Spectre can allow malicious programs to > induce a hypervisor to transmit the data to a guest system running on top > of it. > > - Private SSL certs > > - Cached keys and passwords in non-zeroed RAM > > - [...] > > > > https://en.wikipedia.org/wiki/Spectre_(security_vulnerability) > > It's true that the attacks should worry cloud providers. Doesn't that > mean that companies like Amazon, Microsoft (Steve), and Docker should > have done analyses on CPython's vulnerability to these exploits? > Has/should/can anyone officially representing Python contact the > companies and ask them? > > When I followed your quote to find the context, I found it uses, as > its source, a Forbes article. The source cited by THAT article is > Daniel Gruss, who was one of the researchers. Should someone from the > PSF contact the researchers? Steve says he spoke to some of them to > judge whether the proposed compiler flags would help, and decided > against it. > > Absent of expert input, here's my non-expert take: That quote requires > an OS-level fix. A Python program without the proper permissions can't > do such things unless there is a vulnerability with the OS, and it is > extremely unlikely for anyone to update Python for Spectre but not > update the OS (and they'd be screwed in any case). And even if there > is a vulnerability in the OS, maybe the way to exploit it is by using > arbitrary Python execution (which you need before you can TRY to use > Spectre) on this Python interpreter. You can then write a new binary > file and run THAT, and it will be fast enough. That's not something > you can fix about CPython. > > Also, (again with my understanding) the problem of Spectre and > Meltdown are that you can escape sandboxes and the like, such as the > user/kernel divide, or a software sandbox like that provided by a > JavaScript VM. For CPython to be "vulnerable" to these attacks, it > needs to have some kind of sandbox or protection to break out of. > Instead, we sometimes have sandboxes AROUND CPython (like Jupyter) or > WITHIN CPython. I don't see how it makes sense to talk about a sandbox > escape FOR CPython (yet). > > Your original post linked to a discussion about Linux using those > build flags. Linux is a kernel, and has such protections that can be > bypassed, so it has something to worry about. Malicious code can be > native code, which (to my understanding) will be fast enough to > exploit the cache miss time. Here's Google's article about the > retpoline and why it helps: > https://support.google.com/faqs/answer/7625886 > > As of yet, you have quoted passages that have little relevance to > interpreter devs, especially non-JIT interpreters, and you have linked > to entire articles for non-experts with little relevance to > interpreter devs. This doesn't show that you have any better of an > understanding than I have, which is less than the understanding that > some of the Python devs have, and much less than what Steve has. In > short, it LOOKS like you don't know what you're talking about. If you > have a different and deeper understanding of the problem, then you > need to show it, and say why there is a problem for CPython > specifically. Or find someone who can do that for you. > > > Here's one: > > https://github.com/Eugnis/spectre-attack/blob/master/Source.c > > > > Is this too slow in CPython with: > > - Coroutines (asyncio (tulip)) > > - PyPy JIT * > > - Numba JIT * > > - C Extensions * > > - Cython * > > > > * Not anyone here's problem. > > C extensions are obviously fast enough. I think most of the other > starred examples are fast enough, but it's probably more subtle than I > think and requires further analysis by their devs. I also think > there's something important I'm still missing about what's required > and what it can do. > > I don't see what coroutines have to do with it. Coroutines are still > Python cod
Re: [Python-Dev] SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register
There is much reason to believe it won't help, and you have given little reason to believe that it will help, or that you even properly understand how the flag mitigates Spectre in programs where it does help. I am not undermining your attempts for the sake of performance. I am trying to help you understand why no one is taking you seriously, and suggesting what you can do so that they will. One way is to explain why an unsandboxed interpreter needs to protect against a sandbox escape. On Sep 21, 2018 3:12 AM, "Wes Turner" wrote: I feel like you are actively undermining attempts to prevent exploitation of known vulnerabilities because the software in question is currently too slow. For a 4-8% performance penalty, we could just add the CFLAGS to the build now and not worry about it. I give up. On Friday, September 21, 2018, Franklin? Lee wrote: > On Thu, Sep 20, 2018 at 2:10 PM Wes Turner wrote: > > > > On Thursday, September 20, 2018, Stefan Ring > wrote: > >> > >> On Tue, Sep 18, 2018 at 8:38 AM INADA Naoki > wrote: > >> > >> > I think this topic should split to two topics: (1) Guard Python > >> > process from Spectre/Meltdown > >> > attack from other process, (2) Prohibit Python code attack other > >> > processes by using > >> > Spectre/Meltdown. > >> > >> (3) Guard Python from performance degradation by overly aggressive > >> Spectre "mitigation". > > > > > > > Spectre has the potential of having a greater impact on cloud > providers than Meltdown. Whereas Meltdown allows unauthorized applications > to read from privileged memory to obtain sensitive data from processes > running on the same cloud server, Spectre can allow malicious programs to > induce a hypervisor to transmit the data to a guest system running on top > of it. > > - Private SSL certs > > - Cached keys and passwords in non-zeroed RAM > > - [...] > > > > https://en.wikipedia.org/wiki/Spectre_(security_vulnerability) > > It's true that the attacks should worry cloud providers. Doesn't that > mean that companies like Amazon, Microsoft (Steve), and Docker should > have done analyses on CPython's vulnerability to these exploits? > Has/should/can anyone officially representing Python contact the > companies and ask them? > > When I followed your quote to find the context, I found it uses, as > its source, a Forbes article. The source cited by THAT article is > Daniel Gruss, who was one of the researchers. Should someone from the > PSF contact the researchers? Steve says he spoke to some of them to > judge whether the proposed compiler flags would help, and decided > against it. > > Absent of expert input, here's my non-expert take: That quote requires > an OS-level fix. A Python program without the proper permissions can't > do such things unless there is a vulnerability with the OS, and it is > extremely unlikely for anyone to update Python for Spectre but not > update the OS (and they'd be screwed in any case). And even if there > is a vulnerability in the OS, maybe the way to exploit it is by using > arbitrary Python execution (which you need before you can TRY to use > Spectre) on this Python interpreter. You can then write a new binary > file and run THAT, and it will be fast enough. That's not something > you can fix about CPython. > > Also, (again with my understanding) the problem of Spectre and > Meltdown are that you can escape sandboxes and the like, such as the > user/kernel divide, or a software sandbox like that provided by a > JavaScript VM. For CPython to be "vulnerable" to these attacks, it > needs to have some kind of sandbox or protection to break out of. > Instead, we sometimes have sandboxes AROUND CPython (like Jupyter) or > WITHIN CPython. I don't see how it makes sense to talk about a sandbox > escape FOR CPython (yet). > > Your original post linked to a discussion about Linux using those > build flags. Linux is a kernel, and has such protections that can be > bypassed, so it has something to worry about. Malicious code can be > native code, which (to my understanding) will be fast enough to > exploit the cache miss time. Here's Google's article about the > retpoline and why it helps: > https://support.google.com/faqs/answer/7625886 > > As of yet, you have quoted passages that have little relevance to > interpreter devs, especially non-JIT interpreters, and you have linked > to entire articles for non-experts with little relevance to > interpreter devs. This doesn't show that you have any better of an > understanding than I have, which is less than the understanding that > some of the Python devs have, and much less than what Steve has. In > short, it LOOKS like you don't know what you're talking about. If you > have a different and deeper understanding of the problem, then you > need to show it, and say why there is a problem for CPython > specifically. Or find someone who can do that for you. > > > Here's one: > > https://github.com/Eugnis/spectre-attack/blob/master/Source.c > > > > Is this too slow in CP
Re: [Python-Dev] 3.7.1 and 3.6.7 Releases Coming Soon
On 19/09/2018 23.12, Ned Deily wrote: > Update: not surprisingly, there have been a number of issues that have popped > up during and since the sprint that we would like to ensure are addressed in > 3.7.1 and 3.6.7. In order to do so, I've been holding off on starting the > releases. I think we are now getting close to having the important ones > resolved so I'm going to plan on cutting off code for 3.7.1rc1 and 3.6.7rc1 > by the end of 2018-09-20 (23:59 AoE). That's roughly 38 hours from now. > > Thanks for all of your help in improving Python for everyone! Hi Ned, I'm really sorry, but would it be possible to delay the RCs until Sunday or Monday AoE? Some of the XML security fixes, OpenSSL 1.1.1 fixes (TLS 1.3 post-handshake authentication), and SSL module regression haven't landed yet. I'm confident that I can land most to all fixes during the weekend. Related PRs are: * https://github.com/python/cpython/pull/9468 * https://github.com/python/cpython/pull/9460 * https://github.com/python/cpython/pull/9217 * https://github.com/python/cpython/pull/9265 I'm also still collaborating with Sebastian Pipping (libexpat maintainer) on the DoS mitigations (CVE-2013-0340). My initial patch had some flaws. I might be able to get expat release 2.3.0 in time, too. https://github.com/libexpat/libexpat/pull/220 Christian ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.7.1 and 3.6.7 Releases Coming Soon
On Sep 21, 2018, at 05:37, Christian Heimes wrote: > On 19/09/2018 23.12, Ned Deily wrote: >> Update: not surprisingly, there have been a number of issues that have >> popped up during and since the sprint that we would like to ensure are >> addressed in 3.7.1 and 3.6.7. In order to do so, I've been holding off on >> starting the releases. I think we are now getting close to having the >> important ones resolved so I'm going to plan on cutting off code for >> 3.7.1rc1 and 3.6.7rc1 by the end of 2018-09-20 (23:59 AoE). That's roughly >> 38 hours from now. > I'm really sorry, but would it be possible to delay the RCs until Sunday > or Monday AoE? > > Some of the XML security fixes, OpenSSL 1.1.1 fixes (TLS 1.3 > post-handshake authentication), and SSL module regression haven't landed > yet. I'm confident that I can land most to all fixes during the weekend. > > Related PRs are: > > * https://github.com/python/cpython/pull/9468 > * https://github.com/python/cpython/pull/9460 > * https://github.com/python/cpython/pull/9217 > * https://github.com/python/cpython/pull/9265 > > I'm also still collaborating with Sebastian Pipping (libexpat > maintainer) on the DoS mitigations (CVE-2013-0340). My initial patch had > some flaws. I might be able to get expat release 2.3.0 in time, too. > > https://github.com/libexpat/libexpat/pull/220 I agree that it would be good to get the security-related and OpenSSL-related fixes in sooner than later and there has been a lot going on recently. Since you have asked so nicely, I have rescheduled the cutoffs for 3.7.1rc1 and 3.6.7rc1 to be by the end of 2018-09-24 (23:59 AoE) and the final releases now on 2018-10-04. Everyone else: here are a few more days to get important things in to these releases. -- Ned Deily [email protected] -- [] ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
> What about the small integers cache? I believe the small integers cache is only used to reduce the number of objects -- I don't think there's any code (in CPython itself) that just *assumes* that because an int is small it must be in the cache. So it should be fine. On Thu, Sep 20, 2018 at 11:23 PM Stefan Behnel wrote: > Larry Hastings schrieb am 14.09.2018 um 23:27: > > What the patch does: it takes all the Python modules that are loaded as > > part of interpreter startup and deserializes the marshalled .pyc file > into > > precreated objects stored as static C data. > > What about the small integers cache? The C serialisation generates several > PyLong objects that would normally reside in the cache. Is this handled > somewhere? I guess the cache could entirely be loaded from the data > segment. And the same would have to be done for interned strings. Basically > anything that CPython only wants to have one instance of. > > That would severely limit the application of this optimisation to external > modules, though. I don't see a way how they could load their data > structures from the data segment without duplicating all sorts of > "singletons". > > Stefan > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
On 21/09/2018 16.26, Guido van Rossum wrote: >> What about the small integers cache? > > I believe the small integers cache is only used to reduce the number of > objects -- I don't think there's any code (in CPython itself) that just > *assumes* that because an int is small it must be in the cache. So it > should be fine. Some places may assume that PyLong_FromLong() for a small int never fails. I certainly expect this in coverity scan modeling. Christian ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Late Python 3.7.1 changes to fix the C locale coercion (PEP 538) implementation
On Wed, Sep 19, 2018 at 4:26 PM Ned Deily wrote: > On Sep 19, 2018, at 13:30, Yury Selivanov wrote: [..] > > Currently it's designed to expose "PyContext*" and "PyContextVar*" > > pointers. I want to change that to "PyObject*" as using non-PyObject > > pointers turned out to be a very bad idea (interfacing with Cython is > > particularly challenging). > > > > Is it a good idea to change this in Python 3.7.1? > > It's hard to make an informed decision without a concrete PR to review. What > would be the impact on any user code that has already adopted it in 3.7.0? Ned, I've created an issue to track this: https://bugs.python.org/issue34762 Yury ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
> On Sep 21, 2018, at 06:53, Stefan Behnel wrote: > > Totally. This might actually be more relevant for Cython than for CPython > in the end, because it wouldn't be limited to the stdlib and its core modules. > > It's a bit more difficult for us, because this probably won't work easily > across Python releases (2.[67] and 3.[45678] for now) and also definitely > not for PyPy, but that just means some multiplication of the generated > code, and we have the dynamic part of it already. Supporting that for > Unicode strings will be fun, I'm sure. :) I’m glad to hear that this might be relevant to Cython. I believe it should be straightforward to parametrize the code generator to generate code targeted at specific cPython versions. While we originally targeted 3.6, Larry Hastings managed to quickly port it to 3.8. The two changes in cPython’s data structures between 3.6 and 3.8 that needed changes to the code-gen were [1] from 3.7 and [2] from 3.8. And internally, I’ve still got a task open to back-port this to support 2.7. -- Jeethu [1]: https://bugs.python.org/issue18896 [2]: https://bugs.python.org/issue33597 ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2018-09-14 - 2018-09-21) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open6795 (-24) closed 39723 (+111) total 46518 (+87) Open issues with patches: 2711 Issues opened (63) == #19756: test_nntplib: sporadic failures, network isses? server down? https://bugs.python.org/issue19756 reopened by berker.peksag #34682: Typo reports on docs@ https://bugs.python.org/issue34682 opened by zach.ware #34683: Caret positioned wrong for SyntaxError reported by ast.c https://bugs.python.org/issue34683 opened by gvanrossum #34686: Add `-r`, as opposed to `-R` to Python core interpreter https://bugs.python.org/issue34686 opened by lepaperwan #34687: asyncio: is it time to make ProactorEventLoop as the default e https://bugs.python.org/issue34687 opened by vstinner #34688: Segfault in pandas that works fine on 3.7 https://bugs.python.org/issue34688 opened by xtreak #34689: Lib/sysconfig.py expands non-variables https://bugs.python.org/issue34689 opened by lepaperwan #34690: Store startup modules as C structures for 20%+ startup speed i https://bugs.python.org/issue34690 opened by larry #34691: _contextvars missing in x64 master branch Windows build? https://bugs.python.org/issue34691 opened by tim.peters #34696: PyByteArray_FromObject() has undocumented (and untested) behav https://bugs.python.org/issue34696 opened by ZackerySpytz #34697: ctypes: Crash if manually-created CField instance is used https://bugs.python.org/issue34697 opened by izbyshev #34698: urllib.request.Request.set_proxy doesn't (necessarily) replace https://bugs.python.org/issue34698 opened by tburke #34699: allow path-like objects in program arguments in Windows https://bugs.python.org/issue34699 opened by guoci #34700: typing.get_type_hints doesn't know about typeshed https://bugs.python.org/issue34700 opened by Ben.Darnell #34701: Asyncio documentation for recursive coroutines is lacking https://bugs.python.org/issue34701 opened by azaria.zornberg #34702: urlopen doesn't handle query strings with "file" scheme https://bugs.python.org/issue34702 opened by liZe #34704: Do not access ob_type directly, introduce Py_TP https://bugs.python.org/issue34704 opened by nascheme #34705: Python 3.8 changes how returns through finally clauses are tra https://bugs.python.org/issue34705 opened by nedbat #34706: Signature.from_callable sometimes drops subclassing https://bugs.python.org/issue34706 opened by bukzor #34707: Python not reentrant https://bugs.python.org/issue34707 opened by skaller #34708: Odd crashes/freezes when sys.stdout.shell.console is typed https://bugs.python.org/issue34708 opened by Harrison Chudleigh #34709: Suggestion: make getuser.getpass() also look at SUDO_USER envi https://bugs.python.org/issue34709 opened by Amos S #34711: return ENOTDIR when open() accepts filenames with a trailing s https://bugs.python.org/issue34711 opened by Michael.Felt #34713: csvwriter.writerow()'s return type is undocumented https://bugs.python.org/issue34713 opened by nchammas #34714: timeout in test_multiprocessing_spawn x86 Windows7 3.x buildbo https://bugs.python.org/issue34714 opened by pablogsal #34716: MagicMock.__divmod__ should return a pair https://bugs.python.org/issue34716 opened by serhiy.storchaka #34720: Fix test_importlib.test_bad_traverse for AIX https://bugs.python.org/issue34720 opened by Michael.Felt #34722: Non-deterministic bytecode generation https://bugs.python.org/issue34722 opened by Peter Ebden #34724: argparse subparser help indent too short https://bugs.python.org/issue34724 opened by TakingItCasual #34725: Py_GetProgramFullPath() odd behaviour in Windows https://bugs.python.org/issue34725 opened by mariofutire #34726: Add support of checked hash-based pycs in zipimport https://bugs.python.org/issue34726 opened by serhiy.storchaka #34728: deprecate *loop* argument for asyncio.sleep https://bugs.python.org/issue34728 opened by yselivanov #34730: aclose() doesn't stop raise StopAsyncIteration / GeneratorExit https://bugs.python.org/issue34730 opened by dfee #34732: uuid returns version more than 5 https://bugs.python.org/issue34732 opened by jophine pranjal #34734: Azure linux buildbot failure https://bugs.python.org/issue34734 opened by xtreak #34736: Confusing base64.b64decode output https://bugs.python.org/issue34736 opened by mark.dickinson #34737: Python upgrade with SYSTEM account uninstalls python https://bugs.python.org/issue34737 opened by JatinGoel #34738: Distutils: ZIP files don't include directory entries https://bugs.python.org/issue34738 opened by serhiy.storchaka #34739: Get rid of tp_getattro in xml.etree.ElementTree.XMLParser https://bugs.python.org/issue34739 opened by serhiy.storchaka #34740: Get rid of tp_getattro in ossaudiodev.oss_audio_device https://bugs.pytho
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
On Fri, Sep 21, 2018 at 8:16 AM Christian Heimes wrote: > On 21/09/2018 16.26, Guido van Rossum wrote: > >> What about the small integers cache? > > > > I believe the small integers cache is only used to reduce the number of > > objects -- I don't think there's any code (in CPython itself) that just > > *assumes* that because an int is small it must be in the cache. So it > > should be fine. > > Some places may assume that PyLong_FromLong() for a small int never > fails. I certainly expect this in coverity scan modeling. > Ah, that goes in the other direction. That function will always return a value from the cache if it's in range for the cache. and nothing change there. I was talking about situations where code might assume that if an object's address is not that of the canonical cached zero-valued PyLong object, it couldn't be a PyLong with value zero (same for other values in range of the cache). I'd be very surprised if there was code assuming that, and I'd say it was always wrong. (It's like beginners' code using 'x is 0' instead of 'x == 0'.) Though now I start worrying about interned strings. That's a concept that's a little closer to being a feature. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] switch statement
Hi, A humble proposal for a switch-like statement syntax for Python: - - - switch blah in (100, 2, 30, 'bumm'): dosomething1() x = 88 case blah in (44, 55): otherstuff(9) case blah in (8): boo() else: wawa() - - - So, let's use, and allow only *tuples*. As early as possible, build a jump table, based on (foreknown) small integer values. As in other languages. Strings may have to be hashed (in "compile time"), to obtain small integer value. Some secondary checking may have to be done for exact content equality. (Alternative: do no allow strings at all.) For gaps in the integer range: maybe apply some very basic dividing/shifting to "compact" the range. (As compilers optimize in other languages, I guess -- but I may be totally wrong.) (For example find "unused bits" in the numbers (in 2-base representation). newnum = orignum >> 3 & 6 | orignum & ~6. newnum is smaller (roughly 1/8) than orignum.) The (achievable) goal is to be faster than hash table lookup. (A hash table with keys 100, 2, 30, 'bumm' etc.) And approach the speed of single array-index lookup. (Or even faster in some cases as there will be just jumps instead of calls?) (I am not an "expert"!) Let allow fallthrough or not? - To be decided. (Either is compatible with the above.) I know about PEP 3103 and https://docs.python.org/3.8/faq/design.html?highlight=switch#why-isn-t-there-a-switch-or-case-statement-in-python (I do not know how to comment on a PEP, where to discuss a PEP. If this is inappropriate place, please forward it.) -- ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
On Tue, Sep 18, 2018 at 3:00 PM, Neil Schemenauer wrote: > The users of Python are pretty diverse so it depends on who you ask. > Some would like a giant executable that includes everything they > need (so of like the Go approach). Other people want an executable > that has just importlib inside it and then mix-and-match different > shared libs for their different purposes. Some will not want work > "old school" and load from separate .py or .pyc files. > > I see no reason why we can't support all these options. > Supporting those options is possible if a some of our simplifying assumptions are revisited. Here are a few We know about all the objects being stored in the data segment. That makes it is easy to ensure that immutable objects are unique. Knowing anything less, that work would have to be done at load-time. We do not have to worry about the relocation cost of the pointers we add to the data segment. We are compiled into an executable that typically gets loaded at a fixed address. This could become a performance concern if we wrote our data into a shared library. Because we are compiled into the runtime, we do not have versioning issues. There is no possibility of PyObject_HEAD or any PyObject subclass being changed out from under us. The existing marshal format abstracts away from these details but our format is very sensitive to the host environment. All of these problems have technical solutions. They should be evaluated carefully to ensure that the added overhead does not wipe-out the performance wins or add lots of complexity to the runtime. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
Guido van Rossum schrieb am 21.09.2018 um 19:35: > Though now I start worrying about interned strings. That's a concept that's > a little closer to being a feature. True. While there's the general '"ab"+"cd" is (not) "abcd"' caveat, I'm sure quite a bit of code out there assumes that parsed identifiers in a module, such as the names of functions and classes, are interned, since this was often communicated. And in fact, explicitly interning the same name might return a different string object with this change than what's in the module/class dict. Stefan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] switch statement
First, this sounds like it belongs on python-ideas, not python-dev. Second, when you do send a message to python-ideas, it'll help to accompany it with a realistic example usage that motivates your proposal. On Fri, Sep 21, 2018 at 11:18 AM wrote: > > Hi, > > A humble proposal for a switch-like statement syntax for Python: > > - - - > switch blah in (100, 2, 30, 'bumm'): > dosomething1() > x = 88 > case blah in (44, 55): > otherstuff(9) > case blah in (8): > boo() > else: > wawa() > - - - > > So, let's use, and allow only *tuples*. > As early as possible, build a jump table, based on (foreknown) small integer > values. As in other languages. > Strings may have to be hashed (in "compile time"), to obtain small integer > value. Some secondary checking may > have to be done for exact content equality. (Alternative: do no allow strings > at all.) > For gaps in the integer range: maybe apply some very basic dividing/shifting > to "compact" the range. (As > compilers optimize in other languages, I guess -- but I may be totally > wrong.) (For example find "unused bits" > in the numbers (in 2-base representation). newnum = orignum >> 3 & 6 | > orignum & ~6. newnum is smaller (roughly > 1/8) than orignum.) > The (achievable) goal is to be faster than hash table lookup. (A hash table > with keys 100, 2, 30, 'bumm' etc.) > And approach the speed of single array-index lookup. (Or even faster in some > cases as there will be just jumps > instead of calls?) > (I am not an "expert"!) > > Let allow fallthrough or not? - To be decided. (Either is compatible with the > above.) > > > I know about PEP 3103 and > https://docs.python.org/3.8/faq/design.html?highlight=switch#why-isn-t-there-a-switch-or-case-statement-in-python > > (I do not know how to comment on a PEP, where to discuss a PEP. If this is > inappropriate place, please forward it.) > > -- > > > > > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mike%40selik.org ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] switch statement
As Michael said, this belongs on python-ideas, but it's interesting. I'd support it, though my input in that regard is worth approximately $0.00. ;) It's the core devs and especially the eventual BDFL replacement whom you would have to convince. Without getting into an extended discussion here on python-dev, I'd like to offer one brief comment to try to help improve the proposal: On Fri, Sep 21, 2018 at 2:17 PM wrote: > Let allow fallthrough or not? - To be decided. (Either is compatible with > the above.) > I would argue for non-fallthrough as the default with support for explicitly requesting fallthrough when desired by using "continue". I don't know of any prior art for doing it that way, but it makes sense to me. It eliminates hidden bugs from missing a "break" while still allowing it when needed. Good luck with the proposal! ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
On Thu, Sep 20, 2018 at 11:20 PM, Stefan Behnel wrote: > What about the small integers cache? The C serialisation generates several > PyLong objects that would normally reside in the cache. Is this handled > somewhere? I guess the cache could entirely be loaded from the data > segment. And the same would have to be done for interned strings. Basically > anything that CPython only wants to have one instance of. > Un-marshaled immutable objects are tracked in a table to ensure their uniqueness. Thanks for mentioning the small integer cache. It is not part of the change, but it could be brought under this framework. By doing so, we could store the small integer objects instances in the data segment and other data segment objects could reference those unique small integer instances. That would severely limit the application of this optimisation to external > modules, though. I don't see a way how they could load their data > structures from the data segment without duplicating all sorts of > "singletons". Yes, additional load-time work would have to be done to ensure the uniqueness of those objects. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register
On 21/09/18 08:11, Wes Turner wrote: I feel like you are actively undermining attempts to prevent exploitation of known vulnerabilities because the software in question is currently too slow. Did you write "How to Win Friends And Influence People"? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] switch statement
There's already a rejected PEP about a switch statement: https://www.python.org/dev/peps/pep-3103/. There's no point bringing this up again unless you have a new use case. There have been several promising posts to python-ideas about the much more powerful idea of a "match" statement. Please search for those before re-posting on python-ideas. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] switch statement
> There have been several promising posts to python-ideas about the much more powerful idea of a "match" statement I actually even started working on a PEP about this (pattern matching), but then decided to postpone it because it is unlikely that anything of this size can be discussed/accepted in current situation. We can return back to the idea when decision-making model will clarify. -- Ivan On Fri, 21 Sep 2018 at 22:12, Guido van Rossum wrote: > There's already a rejected PEP about a switch statement: > https://www.python.org/dev/peps/pep-3103/. There's no point bringing this > up again unless you have a new use case. > > There have been several promising posts to python-ideas about the much > more powerful idea of a "match" statement. Please search for those before > re-posting on python-ideas. > > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Questions about signal handling.
Hi all, I've got a pretty good sense of how signal handling works in the runtime (i.e. via a dance with the eval loop), but still have some questions: 1. Why do we restrict calls to signal.signal() to the main thread? 2. Why must signal handlers run in the main thread? 3. Why does signal handling operate via the "pending calls" machinery and not distinctly? More details are below. My interest in the topic relates to improving in-process interpreter isolation. #1 & #2 --- Toward the top of signalmodule.c we find the following comment [1] (written in 1994): /* NOTES ON THE INTERACTION BETWEEN SIGNALS AND THREADS When threads are supported, we want the following semantics: - only the main thread can set a signal handler - any thread can get a signal handler - signals are only delivered to the main thread I.e. we don't support "synchronous signals" like SIGFPE (catching this doesn't make much sense in Python anyway) nor do we support signals as a means of inter-thread communication, since not all thread implementations support that (at least our thread library doesn't). We still have the problem that in some implementations signals generated by the keyboard (e.g. SIGINT) are delivered to all threads (e.g. SGI), while in others (e.g. Solaris) such signals are delivered to one random thread (an intermediate possibility would be to deliver it to the main thread -- POSIX?). For now, we have a working implementation that works in all three cases -- the handler ignores signals if getpid() isn't the same as in the main thread. XXX This is a hack. */ At the very top of the file we see another relevant comment: /* XXX Signals should be recorded per thread, now we have thread state. */ That one was written in 1997, right after PyThreadState was introduced. So is the constraint about the main thread just a historical artifact? If not, what would be an appropriate explanation for why signals must be strictly bound to the main thread? #3 - Regarding the use of Py_MakePendingCalls() for signal handling, I can imagine the history there. However, is there any reason signal handling couldn't be separated from the "pending calls" machinery at this point? As far as I can tell there is no longer any strong relationship between the two. -eric [1] https://github.com/python/cpython/blob/master/Modules/signalmodule.c#L71 ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Questions about signal handling.
Le sam. 22 sept. 2018 à 01:05, Eric Snow a écrit : > 3. Why does signal handling operate via the "pending calls" machinery > and not distinctly? Signals can be received anytime, between two instructions at the machine code level. But the Python code base is rarely reentrant. Moreover, you can get the signal while you don't hold the GIL :-) Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register
On Friday, September 21, 2018, Mark Lawrence wrote: > On 21/09/18 08:11, Wes Turner wrote: > >> I feel like you are actively undermining attempts to prevent exploitation >> of known vulnerabilities because the software in question is currently too >> slow. >> >> > Did you write "How to Win Friends And Influence People"? "Andrew Carnegie - Rags to Riches Power to Peace" ... Vredespaleis "Think and Grow Rich" by Napoleon Hill Today is International Day of Peace. But, now, I've gone way OT, So, we'll just have to hope IBRS is enabled, On all of the Jupyter cloud hosts. > > -- > My fellow Pythonistas, ask not what our language can do for you, ask > what you can do for our language. > > Mark Lawrence > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes. > turner%40gmail.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
