[Python-Dev] Re: Why aren't we allowing the use of C11?
On Fri, 29 Jan 2021 18:15:37 +0100 Victor Stinner wrote: > Hi Mark, > > I tried to use C11 _Thread_local (Thread Local Storage, TLS) only with > GCC and clang and I got issues on macOS: > https://github.com/python/cpython/pull/23976 > > My PR uses __thread if _Thread_local is not available. > > I don't think that MSC (Visual Studio) implements C11 _Thread_local, > but it provides __declspec(thread). I tried it and I got issues with > DLL. I didn't dig into this issue. MSC loves C++ but seems stuck at > C99 (I'm not even sure if it's fully implemented). > > It seems like declaring a TLS in libpython and using it from an > extension module (another dynamic library) is causing issues depending > on the linker. It "should" work on macOS, but it doesn't. You can hide the access behind a function call. Slightly more costly, but shouldn't be that expensive on modern machines. Regards Antoine. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/DSRLQLP6CT4TKTKL5UKS4LGDDFAKNMQ6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Comments on PEP 558
Hi Nick, On 30/01/2021 4:44 am, Nick Coghlan wrote: On Sat, 30 Jan 2021 at 10:30, Nick Coghlan wrote: On Sat, 30 Jan 2021, 12:13 am Mark Shannon, wrote: With a direct proxy coherency is not an issue. For things in the frame, it *is* a direct proxy - reads pull from the frame object, and writes go to both the frame object and the mapping: https://github.com/python/cpython/pull/3640/files#diff-7b8cef249e5cca077d30de4e428a6bde6b9b803464e790e9cffa7e052e19efddR1315 Reviewing the code again, I'd misremembered how the draft implementation works - right now, reads are relying on the snapshot being up to date. Given the resulting C API compatibility break and the loss of code sharing, I don't think it's a net win to drop the extra storage. I'm not sure what you mean by "loss of code sharing", but I don't see how a mapping that is both a cache and a proxy can be easier to implement than a mapping that is just a proxy. "reads are relying on the snapshot being up to date", doesn't that just mean that reads can be wrong? There is no way to keep a snapshot reliably up to date in the presence of threads. For now, I have a PR adding this as an open question: https://github.com/python/peps/pull/1787/files Given the performance benefit of being able to more reasonably drop the implicit call to `PyFrame_LocalsToFast()`, I'm mostly convinced that switching reads to pull from the frame if possible is the right thing to do, even if it reduces the amount of code that can be inherited without modification from MappingProxyType. The API compatibility concerns would mean the extra mapping store still needed to stick around, but it would only be used for: > * providing backwards compatibility for the `PyEval_GetLocals()` and C-level `f_locals` interfaces * reading and writing names that don't have entries in the `fast_refs` mapping * writing shadow updates for names in the `fast_refs` mapping Given that f_locals is broken, why is keeping compatibility for this obscure, and probably unused case worthwhile? The break in compatibility with locals() seems much more intrusive, yet you are OK with that (as am I). Cheers, Mark. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/OEJKE6JB6V4KPFUQI7MP4Q35UTBKGTXF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] More comments on PEP 558
Hi Nick, A couple more issues with PEP 558, as I see it. Lack of specification - There is no specification section in PEP 558. Much of PEP 558 describes the differences between the current behavior and what PEP 558 proposes. These differences are often intermingled with a discussion of the flaws with the current behavior, which makes it difficult to understand what the proposed behavior is. I could not implement PEP 558 from the PEP alone. Cycles and the cost of maintaining the f_locals cache - PEP 558 proposes that f_locals acts as a proxy mapping to the locals in a frame *and* that the frame caches the proxy. This obviously creates a cycle, which needs to be broken by the cycle GC. Although the cycle only exists if f_locals is actually used, there is still some overhead for *every* call and return, as the f_locals field needs to be initialized on call and checked for non-NULL on return. Cheers, Mark. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/7TKPMD5LHCBXGFUIMKDAUZELRH6EX76S/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: More comments on PEP 558
On Sat, 30 Jan 2021, 10:22 pm Mark Shannon, wrote: > Hi Nick, > > A couple more issues with PEP 558, as I see it. > > > Lack of specification > - > > There is no specification section in PEP 558. > > Much of PEP 558 describes the differences between the current behavior > and what PEP 558 proposes. These differences are often intermingled with > a discussion of the flaws with the current behavior, which makes it > difficult to understand what the proposed behavior is. > > I could not implement PEP 558 from the PEP alone. > That's probably true, but as I wasn't trying to fully specify the implementation of the new C API, just describe the usage of it. > > Cycles and the cost of maintaining the f_locals cache > - > > PEP 558 proposes that f_locals acts as a proxy mapping to the locals in > a frame *and* that the frame caches the proxy. This obviously creates a > cycle, which needs to be broken by the cycle GC. > > Although the cycle only exists if f_locals is actually used, there is > still some overhead for *every* call and return, as the f_locals field > needs to be initialized on call and checked for non-NULL on return. > Where does it still propose caching the proxy? You convinced me to take that out ages ago, and just store a plain dict instead, the same way Python 3.9 (et al) do. There was one remaining reference to cycle breaking, but the latest PR removes that. (That PR is still open, as it may yet move the new "Open Issue" to an actual design change). Cheers, Nick. > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/D2C5UE47XJN2GU3HT22WR77WPDCDNYHX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Comments on PEP 558
On Sat, 30 Jan 2021, 10:18 pm Mark Shannon, wrote: > Hi Nick, > > On 30/01/2021 4:44 am, Nick Coghlan wrote: > > On Sat, 30 Jan 2021 at 10:30, Nick Coghlan wrote: > >> On Sat, 30 Jan 2021, 12:13 am Mark Shannon, wrote: > >>> With a direct proxy coherency is not an issue. > >> > >> For things in the frame, it *is* a direct proxy - reads pull from the > frame object, and writes go to both the frame object and the mapping: > https://github.com/python/cpython/pull/3640/files#diff-7b8cef249e5cca077d30de4e428a6bde6b9b803464e790e9cffa7e052e19efddR1315 > > > > Reviewing the code again, I'd misremembered how the draft > > implementation works - right now, reads are relying on the snapshot > > being up to date. > > > >> Given the resulting C API compatibility break and the loss of code > sharing, I don't think it's a net win to drop the extra storage. > > I'm not sure what you mean by "loss of code sharing", but I don't see > how a mapping that is both a cache and a proxy can be easier to > implement than a mapping that is just a proxy. > The read side is currently completely handled by the existing MappingProxyType, so the frame locals proxy subclass only implements the extra MutableMapping bits. "reads are relying on the snapshot being up to date", doesn't that just > mean that reads can be wrong? > > There is no way to keep a snapshot reliably up to date in the presence > of threads. > We don't support running the same *frame* from multiple threads, so only one thread can be updating the frame directly, and the proxy writes to both the frame and the snapshot. But avoiding the need to update the entire snapshot to read specific variables from the frame would definitely be a compelling technical benefit regardless. > > > > For now, I have a PR adding this as an open question: > > https://github.com/python/peps/pull/1787/files > > > > Given the performance benefit of being able to more reasonably drop > > the implicit call to `PyFrame_LocalsToFast()`, I'm mostly convinced > > that switching reads to pull from the frame if possible is the right > > thing to do, even if it reduces the amount of code that can be > > inherited without modification from MappingProxyType. > > > > The API compatibility concerns would mean the extra mapping store > > still needed to stick around, but it would only be used for: > > > * providing backwards compatibility for the `PyEval_GetLocals()` and > > C-level `f_locals` interfaces > > * reading and writing names that don't have entries in the `fast_refs` > mapping > > * writing shadow updates for names in the `fast_refs` mapping > > Given that f_locals is broken, why is keeping compatibility for this > obscure, and probably unused case worthwhile? > f_locals *isn't* broken a lot of the time. The PEP aims to keep that code working unchanged rather than forcing people to modify their code to handle problems they may not have. > The break in compatibility with locals() seems much more intrusive, yet > you are OK with that (as am I). > PyEval_GetLocals() is part of the stable ABI and returns a borrowed reference. That means there are a lot of implementation restrictions around keeping that API working. A follow-up PEP could propose deprecating and removing the API as intrinsically broken, but I don't want to go that far in PEP 558. By contrast, for the Python level API, Nathaniel was able to make a compelling case that most locals() usage would at worst suffer a performance degradation with the semantic change, and at least some use cases would see latent defects transparently fixed. Cheers, Nick. > Cheers, > Mark. > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/HLWVTPUF5GPSUDU322AHYRPCEPFZEYKM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Why aren't we allowing the use of C11?
On 29Jan2021 2227, Random832 wrote: On Thu, Jan 28, 2021, at 22:57, Emily Bowman wrote: On Thu, Jan 28, 2021 at 1:31 PM MRAB wrote: I have Microsoft Visual Studio Community 2019 Version 16.8.4 (it's free) and it supports C11 and C17. While an upgrade for Community is free, for Pro/Enterprise without an annual license it's not. Something that's hard for me to find a clear answer for in a few minutes of searching: Does the current version of the *Windows SDK* still come with a compiler, and can it be used without the Visual Studio IDE [the situation is confusing because it seems to install through the visual studio installer now] or at least without a license for Visual Studio Pro/Enterprise [i.e. by users who do not qualify for community edition] or with earlier versions of the Visual Studio IDE? The Windows SDK contains system headers/libs, no compilers. Visual Studio includes MSVC, and is one of only a few bundles that does (and if you have access to any of the others, you'll know by the massive hole in your bank account, so let's assume Visual Studio). It will also install the Windows SDK on your behalf if the box is checked, which it is by default when you select C++ support. *Anyone* is allow to use Visual Studio Community to work on open-source software. So the only way to not qualify is to refuse to share whatever you're working on. Obviously some people will be in that position, but it's really not that onerous an ask (almost like the GPL, if you squint ;) ). Anyone contributing to CPython is allowed to use VS Community to do so. [1] Additionally, you can install VS Community alongside any other edition or version and they will be kept separate (as much as is possible, which is on Microsoft's side to deal with and shouldn't impact licensing). Finally, the VS Build Tools are only licensed to you if you have a Visual Studio license, which means you need to have qualified for VS Community (or paid for a higher one) to use the build tools. Again, if you're building open-source software, you've qualified, so it's basically a non-issue for us. Hopefully that settles some concerns. IANAL, but I am on the Visual Studio team and have been involved in discussions around how VS Community licensing applies to open source ecosystems ever since it was first created. If you're still concerned, go pay a lawyer to give you more opinions, because you honestly aren't going to find a more informed opinion for free on the internet ;) Cheers, Steve 1: See page 8 of https://visualstudio.microsoft.com/wp-content/uploads/2020/03/Visual-Studio-Licensing-Whitepaper-Mar-2020.pdf ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/WBVMFEGGL4NOGDHUWZWM2C6PDYAIAIHD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Why aren't we allowing the use of C11?
On 29Jan2021 1715, Victor Stinner wrote: It seems like declaring a TLS in libpython and using it from an extension module (another dynamic library) is causing issues depending on the linker. It "should" work on macOS, but it doesn't. I'm pretty sure this is not defined in any calling convention that would be able to cross dynamic library boundaries, or at least I've never seen it shown how this information would be enecoded. In general, provided the features don't leak out of our compilation process (i.e. cross dynamic library boundaries or show up in public/implicit header files), I don't have any issue allowing them to be used. Though I don't want to speak for the people who maintain CPython distros on smaller/slower/more focused platforms. I'm not sure how best to reach them, but I'd rather do that than find out when they complain on the issue tracker. Only thing I'm certain of is that we shouldn't assume that we know everyone who ever builds CPython. Cheers, Steve ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/NWK3O5DPDA2NIP5EEE6JOLKLENKBCAOL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Comments on PEP 558
On Sat, 30 Jan 2021, 10:55 pm Nick Coghlan, wrote: > > > On Sat, 30 Jan 2021, 10:18 pm Mark Shannon, wrote: > >> >> > >> The break in compatibility with locals() seems much more intrusive, yet >> you are OK with that (as am I). >> > > PyEval_GetLocals() is part of the stable ABI and returns a borrowed > reference. That means there are a lot of implementation restrictions around > keeping that API working. A follow-up PEP could propose deprecating and > removing the API as intrinsically broken, but I don't want to go that far > in PEP 558. > After sleeping on this, I'm now convinced that you're right, and we can reasonably drop the "stash extra info in the frame locals snapshot" feature from the *new* optimised frame locals C API. What I realised is that even if we don't offer that feature on the new frame locals proxy type, the *old* C API can still support it, unless & until that API is deprecated, as PyEval_GetLocals() bypasses the new API and accesses the frame state directly. Cheers, Nick. >> ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/VED6ZQSXVXEEDBB4M3UBICZNONEHBASH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Comments on PEP 558
On Sun, 31 Jan 2021 at 14:09, Nick Coghlan wrote: > On Sat, 30 Jan 2021, 10:55 pm Nick Coghlan, wrote: >> On Sat, 30 Jan 2021, 10:18 pm Mark Shannon, wrote: >>> The break in compatibility with locals() seems much more intrusive, yet >>> you are OK with that (as am I). >> PyEval_GetLocals() is part of the stable ABI and returns a borrowed >> reference. That means there are a lot of implementation restrictions around >> keeping that API working. A follow-up PEP could propose deprecating and >> removing the API as intrinsically broken, but I don't want to go that far in >> PEP 558. > After sleeping on this, I'm now convinced that you're right, and we can > reasonably drop the "stash extra info in the frame locals snapshot" feature > from the *new* optimised frame locals C API. The PR at https://github.com/python/peps/pull/1787 has been updated to accept your suggestion into the PEP rather than listing it as an open question. It also cleans up a bunch more references to the old "store the proxy on the frame" idea that I missed when moving away from the idea. I haven't updated the reference implementation at https://github.com/python/cpython/pull/3640/files to match though, and don't know when I might get to doing that myself. If you're interested in tackling that, the offer of PEP co-authorship definitely still stands :) While I haven't listed it in the PR as an open issue, there is one potential point for simplification that I haven't resolved yet: whether the new APIs should *completely* ignore the shared dynamic snapshot that ``PyEval_GetLocals()`` uses. At the moment, most of the APIs (including the Python ``locals()`` builtin) will still see extra values written via that backwards compatibility interface, as they implicitly update it, and then use it to generate their final return value. The only APIs that will ignore it completely are the Python level frame.f_locals, and the C level PyLocals_GetView() and PyFrame_GetLocalsView(), as those rely on the new fast locals proxy, and bypass the dynamic snapshot. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/6XED5D7UAXSILC2NSM6C6D6H62Z6GRFL/ Code of Conduct: http://python.org/psf/codeofconduct/
