[Python-Dev] Signals, threads, blocking C functions
We have to resort to timeouts in pygtk in order to catch unix signals in threaded mode. The reason is this. We call gtk_main() (mainloop function) which blocks forever. Suppose there are threads in the program; then any thread can receive a signal (e.g. SIGINT). Python catches the signal, but doesn't do anything; it simply sets a flag in a global structure and calls Py_AddPendingCall(), and I guess it expects someone to call Py_MakePendingCalls(). However, the main thread is blocked calling a C function and has no way of being notified it needs to give control back to python to handle the signal. Hence, we use a 100ms timeout for polling. Unfortunately, timeouts needlessly consume CPU time and drain laptop batteries. According to [1], all python needs to do to avoid this problem is block all signals in all but the main thread; then we can guarantee signal handlers are always called from the main thread, and pygtk doesn't need a timeout. Another alternative would be to add a new API like Py_AddPendingCallNotification, which would let python notify extensions that new pending calls exist and need to be processed. But I would really prefer the first alternative, as it could be fixed within python 2.5; no need to wait for 2.6. Please, let's make Python ready for the enterprise! [2] [1] https://bugzilla.redhat.com/bugzilla/process_bug.cgi#c3 [2] http://perkypants.org/blog/2006/09/02/rfte-python/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
"Gustavo Carneiro" <[EMAIL PROTECTED]> wrote: > > We have to resort to timeouts in pygtk in order to catch unix signals > in threaded mode. A common defect of modern designs - TCP/IP is particularly objectionable in this respect, but that battle was lost and won over two decades ago :-( > The reason is this. We call gtk_main() (mainloop function) which > blocks forever. Suppose there are threads in the program; then any > thread can receive a signal (e.g. SIGINT). Python catches the signal, > but doesn't do anything; it simply sets a flag in a global structure > and calls Py_AddPendingCall(), and I guess it expects someone to call > Py_MakePendingCalls(). However, the main thread is blocked calling a > C function and has no way of being notified it needs to give control > back to python to handle the signal. Hence, we use a 100ms timeout > for polling. Unfortunately, timeouts needlessly consume CPU time and > drain laptop batteries. Yup. > According to [1], all python needs to do to avoid this problem is > block all signals in all but the main thread; then we can guarantee > signal handlers are always called from the main thread, and pygtk > doesn't need a timeout. 1) That page is password protected, so I can't see what it says, and am disinclined to register myself to yet another such site. 2) No way, Jose, anyway. The POSIX signal handling model was broken beyond redemption, even before threading was added, and the combination is evil almost beyond belief. That procedure is good practice, yes, but that is NOT all that you have to do - it may be all that you CAN do, but that is not the same. Come back MVS (or even VMS) - all is forgiven! That is only partly a joke. > Another alternative would be to add a new API like > Py_AddPendingCallNotification, which would let python notify > extensions that new pending calls exist and need to be processed. Nope. Sorry, but you can't solve a broken design by adding interfaces. > But I would really prefer the first alternative, as it could be > fixed within python 2.5; no need to wait for 2.6. It clearly should be done, assuming that Python's model is that it doesn't want to get involved with subthread signalling (and I really, but REALLY, recommend not doing so). The best that can be done is to say that all signal handling is the business of the main thread and that, when the system bypasses that, all bets are off. > Please, let's make Python ready for the enterprise! [2] Given that no Unix variant or Microsoft system is, isn't that rather an unreasonable demand? I am probably one of the last half-dozen people still employed in a technical capacity who has implemented run-time systems that supported user-level signal handling with threads/asynchronicity and allowing for signals received while in system calls. It would be possible to modify/extend POSIX or Microsoft designs to support this, but currently they don't make it possible. There is NOTHING that Python can do but to minimise the chaos. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.5 issue: decimal context manager misimplemented, misdesigned, and misdocumented
On Thu, 31 Aug 2006, Nick Coghlan wrote: [...] > I committed this fix as 51664 on the trunk (although the docstrings are still > example free because doctest doesn't understand __future__ statements). [...] Assuming doctest doesn't try to parse the Python code when SKIP is specified, I guess this would solve that little problem: http://docs.python.org/dev/lib/doctest-options.html """ SKIP When specified, do not run the example at all. This can be useful in contexts where doctest examples serve as both documentation and test cases, and an example should be included for documentation purposes, but should not be checked. E.g., the example's output might be random; or the example might depend on resources which would be unavailable to the test driver. The SKIP flag can also be used for temporarily "commenting out" examples. ... Changed in version 2.5: Constant SKIP was added. """ John ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.5 issue: decimal context manager misimplemented, misdesigned, and misdocumented
John J Lee wrote: > On Thu, 31 Aug 2006, Nick Coghlan wrote: > [...] >> I committed this fix as 51664 on the trunk (although the docstrings are still >> example free because doctest doesn't understand __future__ statements). > [...] > > Assuming doctest doesn't try to parse the Python code when SKIP is > specified, I guess this would solve that little problem: > > http://docs.python.org/dev/lib/doctest-options.html > > """ > SKIP A quick experiment suggests that using SKIP will solve the problem - fixing that can wait until 2.5.1 though. The localcontext() docstring does actually contain an example - it just isn't in a form that doctest will try to execute. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Windows build slave down until Tuesday-ish
The "x86 XP trunk" build slave will be down for a bit longer, unfortunately. Tropical storm Ernesto got in the way of my DSL installation - I don't have a new install date yet, but I'm assuming it's going to be Tuesday or later. Alan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
On 9/2/06, Nick Maclaren <[EMAIL PROTECTED]> wrote: > According to [1], all python needs to do to avoid this problem is > block all signals in all but the main thread; then we can guarantee > signal handlers are always called from the main thread, and pygtk > doesn't need a timeout. 1) That page is password protected, so I can't see what it says, and am disinclined to register myself to yet another such site. Oh, sorry, here's the comment: (coment by Arjan van de Ven): | afaik the kernel only sends signals to threads that don't have them blocked. | If python doesn't want anyone but the main thread to get signals, it should just | block signals on all but the main thread and then by nature, all signals will go | to the main thread 2) No way, Jose, anyway. The POSIX signal handling model was broken beyond redemption, even before threading was added, and the combination is evil almost beyond belief. That procedure is good practice, yes, but that is NOT all that you have to do - it may be all that you CAN do, but that is not the same. Nope. Sorry, but you can't solve a broken design by adding interfaces. Well, Python has a broken design too; it postpones tasks and expects to magically regain control in order to finish the job. That often doesn't happen! > But I would really prefer the first alternative, as it could be > fixed within python 2.5; no need to wait for 2.6. It clearly should be done, assuming that Python's model is that it doesn't want to get involved with subthread signalling (and I really, but REALLY, recommend not doing so). The best that can be done is to say that all signal handling is the business of the main thread and that, when the system bypasses that, all bets are off. Python is halfway there; it assumes signals are to be handled in the main thread. However, it _catches_ them in any thread, sets a flag, and just waits for the next opportunity when it runs again in the main thread. It is precisely this "split handling" of signals that is failing now. Anyway, attached a patch that should fix the problem in posix threads systems, in case anyone wants to review. Cheers. Index: Modules/threadmodule.c === --- Modules/threadmodule.c (revision 51687) +++ Modules/threadmodule.c (working copy) @@ -12,6 +12,17 @@ #include "pythread.h" +#if defined(_POSIX_THREADS) && defined(HAVE_PTHREAD_SIGMASK) && defined(HAVE_SIGNAL_H) +# include +# include +# if defined(HAVE_BROKEN_PTHREAD_SIGMASK) +#define SET_THREAD_SIGMASK sigprocmask +# else +#define SET_THREAD_SIGMASK pthread_sigmask +# endif +#endif + + static PyObject *ThreadError; @@ -417,6 +428,15 @@ PyThreadState *tstate; PyObject *res; + /* We want to block signals from all non-main threads, so + that we can guarantee they arrive always at the main + thread, where we can handle them */ +#ifdef SET_THREAD_SIGMASK +sigset_t set; +sigfillset(&set); +SET_THREAD_SIGMASK(SIG_BLOCK, &set, NULL); +#endif + tstate = PyThreadState_New(boot->interp); PyEval_AcquireThread(tstate); ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.5 issue: decimal context manager misimplemented, misdesigned, and misdocumented
[Neal] > Please review the patch and make a comment. I did a diff between HEAD > and 2.4 and am fine with this going in once you are happy. I fixed a couple of documentation nits in rev 51688. The patch is ready-to-go. Nick, please go ahead and backport. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
"Gustavo Carneiro" <[EMAIL PROTECTED]> wrote: > > Oh, sorry, here's the comment: > > (coment by Arjan van de Ven): > | afaik the kernel only sends signals to threads that don't have them blocked. > | If python doesn't want anyone but the main thread to get signals, it > should just > | block signals on all but the main thread and then by nature, all > signals will go > | to the main thread Well, THAT'S wrong, I am afraid! Things ain't that simple :-( Yes, POSIX implies that things work that way, but there are so many get-out clauses and problems with trying to implement that specification that such behaviour can't be relied on. > Well, Python has a broken design too; it postpones tasks and expects > to magically regain control in order to finish the job. That often > doesn't happen! Very true. And that is another problem with POSIX :-( > Python is halfway there; it assumes signals are to be handled in the > main thread. However, it _catches_ them in any thread, sets a flag, > and just waits for the next opportunity when it runs again in the main > thread. It is precisely this "split handling" of signals that is > failing now. I agree that is not how to do it, but that code should not be removed. Despite best attempts, there may well be circumstances under which signals are received in a subthread, despite all attempts of the program to ensure that the main thread gets them. > Anyway, attached a patch that should fix the problem in posix > threads systems, in case anyone wants to review. Not "fix" - "improve" :-) I haven't looked at it, but I agree that what you have said is the way to proceed. The best solution is to enable the main thread for all relevant signals, disable all subthreads, but to not rely on any of that working in all cases. It won't help with the problem where merely receiving a signal causes chaos, or where blocking them does so, but there is nothing that Python can do about that, in general. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.5 issue: decimal context manager misimplemented, misdesigned, and misdocumented
On Sunday 03 September 2006 03:11, Raymond Hettinger wrote: > [Neal] > > > Please review the patch and make a comment. I did a diff between HEAD > > and 2.4 and am fine with this going in once you are happy. > > I fixed a couple of documentation nits in rev 51688. > The patch is ready-to-go. > Nick, please go ahead and backport. I think this is suitable for 2.5. I'm thinking, though, that we need a second release candidate, given the number of changes since rc1. -- Anthony Baxter <[EMAIL PROTECTED]> It's never too late to have a happy childhood. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.5 issue: decimal context manager misimplemented, misdesigned, and misdocumented
On Sun, Sep 03, 2006, Anthony Baxter wrote: > > I think this is suitable for 2.5. I'm thinking, though, that we need > a second release candidate, given the number of changes since rc1. +1 -- Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncraft.com/ I support the RKAB ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.5 issue: decimal context manager misimplemented, misdesigned, and misdocumented
On Saturday 02 September 2006 23:58, Anthony Baxter wrote: > I think this is suitable for 2.5. I'm thinking, though, that we need a > second release candidate, given the number of changes since rc1. +1 -Fred -- Fred L. Drake, Jr. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
