Re: [Python-Dev] Questions about signal handling.
On Sat, 22 Sep 2018 at 09:14, Victor Stinner wrote: > > Le sam. 22 sept. 2018 à 01:05, Eric Snow a > écrit : > > 3. Why does signal handling operate via the "pending calls" machinery > > and not distinctly? > > Signals can be received anytime, between two instructions at the > machine code level. But the Python code base is rarely reentrant. > Moreover, you can get the signal while you don't hold the GIL :-) This would actually be the main reason to keep the current behaviour: at least some folks are running their applications in subthreads as a workaround to avoid https://bugs.python.org/issue29988 and the general fact that Ctrl-C handling and deterministic resource cleanup generally don't get along overly well. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Questions about signal handling.
On Fri, Sep 21, 2018 at 5:11 PM Victor Stinner wrote: > Le sam. 22 sept. 2018 à 01:05, Eric Snow a > écrit : > > 3. Why does signal handling operate via the "pending calls" machinery > > and not distinctly? > > Signals can be received anytime, between two instructions at the > machine code level. But the Python code base is rarely reentrant. > Moreover, you can get the signal while you don't hold the GIL :-) Sorry, I wasn't clear. I'm not suggesting that signals should be handled outside the interpreter. Instead, why do we call PyErr_CheckSignals() in Py_MakePendingCalls() rather than distinctly, right before we call Py_MakePendingCalls()? -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
On Fri, Sep 14, 2018 at 6:08 PM Larry Hastings wrote: > I can suggest that, based on conversation from Carl, that adding the stat > calls back in costs you half the startup. So any mechanism where we're > talking to the disk _at all_ simply isn't going to be as fast. Is that cost for when the stat calls are done in parallel with the new loading mechanism? ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Questions about signal handling.
On Fri, Sep 21, 2018 at 7:04 PM Eric Snow wrote: > > Hi all, > > I've got a pretty good sense of how signal handling works in the > runtime (i.e. via a dance with the eval loop), but still have some > questions: > > 1. Why do we restrict calls to signal.signal() to the main thread? > 2. Why must signal handlers run in the main thread? > 3. Why does signal handling operate via the "pending calls" machinery > and not distinctly? Here's my take on this: Handling signals in a multi-threaded program is hard. Some signals can be delivered to an arbitrary thread, some to the one that caused them. Posix provides lots of mechanisms to tune how signals are received (or blocked) by individual threads, but (a) Python doesn't expose those APIs, (b) using those APIs correctly is insanely hard. By restricting that we can only receive signals in the main thread we remove all that complexity. Restricting that signal.signal() can only be called from the main thread just makes this API more consistent (and also IIRC avoids weird sigaction() behaviour when it is called from different threads within one program). Next, you can only call reentrant functions in your signal handlers. For instance, printf() function isn't safe to use. Therefore one common practice is to set a flag that a signal was received and check it later (exactly what we do with the pending calls machinery). Therefore, IMO, the current way we handle signals in Python is the safest, most predictable, and most cross-platform option there is. And changing how Python signals API works with threads in any way will actually break the world. Yury ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Questions about signal handling.
On Fri, Sep 21, 2018 at 6:10 PM, Victor Stinner wrote: > > Moreover, you can get the signal while you don't hold the GIL :-) Note that, in Windows, SIGINT and SIGBREAK are implemented in the C runtime and linked to the corresponding console control events in a console application, such as python.exe. Console control events are delivered on a new thread (i.e. no Python thread state) that starts at CtrlRoutine in kernelbase.dll. The session server (csrss.exe) creates this thread remotely upon request from the console host process (conhost.exe). ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Questions about signal handling.
On Mon, Sep 24, 2018 at 11:14 AM Yury Selivanov wrote: > On Fri, Sep 21, 2018 at 7:04 PM Eric Snow wrote: > > 1. Why do we restrict calls to signal.signal() to the main thread? > > 2. Why must signal handlers run in the main thread? > > 3. Why does signal handling operate via the "pending calls" machinery > > and not distinctly? > > Here's my take on this: > > Handling signals in a multi-threaded program is hard. Some signals can > be delivered to an arbitrary thread, some to the one that caused them. > Posix provides lots of mechanisms to tune how signals are received (or > blocked) by individual threads, but (a) Python doesn't expose those > APIs, (b) using those APIs correctly is insanely hard. By restricting > that we can only receive signals in the main thread we remove all that > complexity. Just to be clear, I'm *not* suggesting that we allow folks to specify in which Python (or kernel) thread a Python-level signal handler is called. The reason I've asked about signals is because of the semantics under subinterpreters (where there is no "main" thread). However, I don't have any plans to introduce per-interpreter signal handlers. Mostly I want to understand about the "main" thread restriction for the possible impact on subinterpreters. FWIW, I'm also mildly curious about the value of the "main" thread restriction currently. From what I can tell the restriction was made early on and there are hints in the C code that it's no longer needed. I suspect we still have the restriction solely because no one has bothered to change it. However, I wasn't sure so I figured I'd ask. :) > Restricting that signal.signal() can only be called from > the main thread just makes this API more consistent Yeah, that's what I thought. > (and also IIRC > avoids weird sigaction() behaviour when it is called from different > threads within one program). Is there a good place where this weirdness is documented? > Next, you can only call reentrant functions in your signal handlers. > For instance, printf() function isn't safe to use. Therefore one > common practice is to set a flag that a signal was received and check > it later (exactly what we do with the pending calls machinery). We don't actually use the pending calls machinery for signals though. The only thing we do is *always* call PyErr_CheckSignals() before making any pending calls. Wouldn't it be equivalent if we called PyErr_CheckSignals() at the beginning of the eval loop right before calling Py_MakePendingCalls()? This matters to me because I'd like to use "pending" calls for subinterpreters, which means dealing with signals *in* Py_MakePendingCalls() is problematic. Pulling the PyErr_CheckSignals() call out would eliminate that problem. > Therefore, IMO, the current way we handle signals in Python is the > safest, most predictable, and most cross-platform option there is. > And changing how Python signals API works with threads in any way will > actually break the world. I agree that the way we deal with signals (i.e. set a flag that is later handled in PyErr_CheckSignals(), protected by the GIL) shouldn't change. My original 3 questions do not relate to that. Looks like that wasn't terribly clear. :) -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Questions about signal handling.
On Mon, Sep 24, 2018 at 4:19 PM Eric Snow wrote: [..] > Is there a good place where this weirdness is documented? I'll need to look through uvloop & libuv commit log to remember that; will try to find time tonight/tomorrow. [..] > This matters to me because I'd like to use "pending" calls for > subinterpreters, which means dealing with signals *in* > Py_MakePendingCalls() is problematic. Pulling the > PyErr_CheckSignals() call out would eliminate that problem. Py_MakePendingCalls is a public API, even though it's not documented. If we change it to not call PyErr_CheckSignals and if there are C extensions that block pure Python code execution for long time (but call Py_MakePendingCalls explicitly), such extensions would stop reacting to ^C. Maybe a better workaround would be to introduce a concept of "main" sub-interpreter? We can then fix Py_MakePendingCalls to only check for signals when it's called from the main interpreter. Yury ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Questions about signal handling.
On Mon, Sep 24, 2018 at 3:10 PM Yury Selivanov wrote: > On Mon, Sep 24, 2018 at 4:19 PM Eric Snow wrote: > > This matters to me because I'd like to use "pending" calls for > > subinterpreters, which means dealing with signals *in* > > Py_MakePendingCalls() is problematic. Pulling the > > PyErr_CheckSignals() call out would eliminate that problem. > > Py_MakePendingCalls is a public API, even though it's not documented. > If we change it to not call PyErr_CheckSignals and if there are C > extensions that block pure Python code execution for long time (but > call Py_MakePendingCalls explicitly), such extensions would stop > reacting to ^C. > > Maybe a better workaround would be to introduce a concept of "main" > sub-interpreter? We can then fix Py_MakePendingCalls to only check for > signals when it's called from the main interpreter. I'm planning on making Py_MakePendingCalls() a backward-compatible wrapper around a new private _Py_MakePendingCalls() which supports per-interpreter operation. Then the eval loop will call the new internal function. So nothing would change for users. -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
