[issue4293] Thread Safe Py_AddPendingCall

Mark Dickinson added the comment: That seems to have done the trick. Thanks! -- status: open -> closed ___ Python tracker ___ ___ Pyt

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson added the comment: checked in r68722, let's hope this fixes things. ___ Python tracker ___ ___ Python-bugs-list mailing

[issue4293] Thread Safe Py_AddPendingCall

Mark Dickinson added the comment: How about using a timer instead of the 'count += 1' loop: after starting the 32 self.pendingcalls_submit threads, set up a threading.Event and start yet another thread that simply does a time.sleep(5.0) (or whatever) and then sets that event. Then your wait

[issue4293] Thread Safe Py_AddPendingCall

Mark Dickinson added the comment: Kristján, The r68461 checkin seems to be causing a number of the buildbots to fail, typically with output like: test_capi test test_capi failed -- Traceback (most recent call last): File "/home/pybot/buildarea/trunk.klose-debian-ppc/build/Lib/test/test_capi

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson added the comment: Added MISC/News entry in revision: 68545 ___ Python tracker ___ ___ Python-bugs-list mailing list Uns

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson added the comment: I indeed forgot the unittests and docs. They are now in, in revision: 68461 ___ Python tracker ___ __

[issue4293] Thread Safe Py_AddPendingCall

Antoine Pitrou added the comment: > Kristján Valur Jónsson added the comment: > > Checked in as revision: 68460 Looks like you forgot the unit tests! ___ Python tracker ___ ___

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson added the comment: Checked in as revision: 68460 -- resolution: -> accepted status: open -> closed ___ Python tracker ___

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson added the comment: 1) volatile is not required when access is guarded by locks. 2) pendingcalls_to_do is initialized to 1, so the very first thing that the interpreter does (after executing one bytecode, I init _PyTicker to 0 now) is to initialize this lock. Creating a

[issue4293] Thread Safe Py_AddPendingCall

Antoine Pitrou added the comment: Sorry, forget what I said about the GIL, it looks fine. Perhaps some comments should be added to clarify that matter, however. ___ Python tracker ___ ___

[issue4293] Thread Safe Py_AddPendingCall

Antoine Pitrou added the comment: Ok, the test patch is fine! Now looking at the main patch, some comments: - you removed "volatile" from pendingfirst and pendinglast, is it really safe? - what if pending_lock is not initialized when Py_AddPendingCall is called? I know it's unlikely, but imagin

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson added the comment: Ok, I have addressed the issues raised about testcapi. A new file has been submitted. Added file: http://bugs.python.org/file12619/testcapi.patch ___ Python tracker _

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson added the comment: Thanks. What do you mean by making the test a method of TestCAPI? I have found no such class. Are you talking about the python part or the c part? Also, atomicity in the callback is not required, as the callback is never interrupted implicitly. In

[issue4293] Thread Safe Py_AddPendingCall

Antoine Pitrou added the comment: A few comments about the test: - it should be a method of TestCAPI; we try to unittest everywhere, although there is some old code which predates that - the implementation would be stressed better if each callback was added from a separate thread, rather than ad

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson added the comment: Ah, I forgot the test code in my patch. I submit a separate patch to add that code. Also, reworking things to have single function definitions is in my opinion too messy. I started doing things that way, but the code becomes hard to read and therefo

[issue4293] Thread Safe Py_AddPendingCall

Antoine Pitrou added the comment: Would it be possible to rework your code so as not to define two versions of Py_AddPendingCall() and Py_MakePendingCalls(), only putting `#ifdef WITH_THREAD` sections where it's necessary? Or would it make the code too quirky? Also, it would be nice to have a t

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson added the comment: Ok, here is a revised, patch. It now contains rudimentary documentation, which probably needs to be proofread by the official documentation lords. Added file: http://bugs.python.org/file12594/xmlrpc.patch ___ Python

[issue4293] Thread Safe Py_AddPendingCall

Amaury Forgeot d'Arc added the comment: Well, I could not find these functions documented anywhere :-( If you want to start to write some, a good place could be in http://docs.python.org/c-api/init.html (Doc/c-api/init.rst, maybe after the PyGILState_ functions) At the very least, a short sente

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson added the comment: Good points. Any suggestion where to document the new functionality? ___ Python tracker ___ ___ Pytho

[issue4293] Thread Safe Py_AddPendingCall

Amaury Forgeot d'Arc added the comment: - The things_to_do static variable should be declared as "volatile": it is read by the main loop without any lock. (by the way, could you rename it to something like "pendingcalls_to_do"?) - in the old Py_MakePendingCalls function, the "#ifdef WITH_THREAD

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson added the comment: Ping. Amaury, any thoughts? ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: ht

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson <[EMAIL PROTECTED]> added the comment: I turns out that for SIGINT, windows creates a new thread. So, the assumption that signals are delivered on the main thread are not valid in general. Therefore, we must initialize the lock pending_lock independently of PyEval_Init

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson <[EMAIL PROTECTED]> added the comment: I had forgotten that locks aren't initialized until PyEval_InitThreads () is called. Now we check for this case. Also reinitialize when fork () is called. Is there any suggested place where I should add documentation to this? Added

[issue4293] Thread Safe Py_AddPendingCall

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment: Two comments about the patch: - typo in a comment: "...change chat a signal..." - PyThread_acquire_lock should not be called with a NULL lock: on win32 this does not matter (there is a test in thread_nt.h), but Unix segfaults in thread

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson <[EMAIL PROTECTED]> added the comment: Small refinement: Deadlock avoidance is only needed on the main thread since we only install signal handlers for the main thread. Added file: http://bugs.python.org/file12031/pendingalls.patch __

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson <[EMAIL PROTECTED]> added the comment: Here is a revised version. Since there is no portable way to block signals, and no way at all on windows (apparently) the simplest way is to simply use NOWAIT_LOCK when adding a new pending call. While this does not guarantee that w

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson <[EMAIL PROTECTED]> added the comment: Right. Isn't the best way to avoid it to block signals just while we pop the queue? I'll see if python has a portable sigaction thing to do that. Otherwise we'd have to start to mess with a "volatile busy" flag again and possibly in

[issue4293] Thread Safe Py_AddPendingCall

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment: Py_AddPendingCall is used inside signal handlers. It seems that there is a (tiny) chance that a signal interrupts the program inside Py_MakePendingCalls, while the pendinglock is held. Py_AddPendingCall would try to acquire this lock aga

[issue4293] Thread Safe Py_AddPendingCall

Changes by Kristján Valur Jónsson <[EMAIL PROTECTED]>: Removed file: http://bugs.python.org/file11974/pendingalls.patch ___ Python tracker <[EMAIL PROTECTED]> ___ _

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson <[EMAIL PROTECTED]> added the comment: Okay, i add a new patch that includes tests. I am unsure where I should add documentation for the Py_AddPendingCall () function. Any suggestions? Added file: http://bugs.python.org/file11978/pendingalls.patch ___

[issue4293] Thread Safe Py_AddPendingCall

Kristján Valur Jónsson <[EMAIL PROTECTED]> added the comment: Good point. I'll make a test using ctypes and _testcapimodule.c to try to make it as non-platform-specific as possible. ___ Python tracker <[EMAIL PROTECTED]> __

[issue4293] Thread Safe Py_AddPendingCall

Christian Heimes <[EMAIL PROTECTED]> added the comment: Your patch sounds like a useful addition. However I can comment on the details because I haven't dealt with Py_AddPendingCall() yet. Documentation updates and a unit test (Modules/_testcapimodule.c), that shows the bug and verifies your pat

[issue4293] Thread Safe Py_AddPendingCall

New submission from Kristján Valur Jónsson <[EMAIL PROTECTED]>: At CCP We have started using the Py_AddPendingCall() mechanism to signal python about a completed IO operation. However, we noticed that the existing mechanism was hoplelessly un- thread safe. This is bad, since on Windows at least