Many thanks for the clarification. Also good idea to focus on lock ownership rather that thread activity in your diagram.
To be honest I was not actually experiencing deadlock issues. I had just deduced (incorrectly) that I might do so if I started using locks in my py code called from c-land. As I was suffering some horrible race-condtions that had suddenly appeared in code that had been perfectly stable for the past 6 months I didn't want to go down that route given my misconceptions; Even if introducing the locks worked I'd always feel it was a deadlock waiting to happen. Misunderstanding arose as for some reason I had decided that python code called from C did not allow any other python threads to run until control had returned to C and the gil had been released from C-land. I.e. once the gil was given to the C thread then python had to wait until it was given back. I have to say the docs could be a little more explicit on the mechanisms involved. They tend to be better at stating what individual functions do rather that giving overall explanations. Maybe I'll get around to submitting a chapter on it ;) On Nov 12, 4:54 pm, "Chris Mellon" <[EMAIL PROTECTED]> wrote: > On Nov 12, 2007 6:56 AM, <[EMAIL PROTECTED]> wrote: > > > > > Hi, > > > I have a native windows thread in a c python module which calls into > > python code and adds an item to a data structure (a home-grown > > circular buffer). At the same time my main python application is > > removing items from this data structure. > > > Unlike native python containers adding and removing items from the > > circular buffer is not atomic. So I need to put access to it in a > > critical section. Using threading.Lock will work for native python > > threads competing for access. But when access is from a windows thread > > this will not work. That is because my call to PyGilState_ensure will > > preempt my native python thread ***even when it is inside the critical > > section***. > > > What is going on looks something like this (I think). > > > Py Python Windows Py threading.Lock > > resource > > Sched Thread Thread Code | | > > | | | | | > > |Go (GIL)# | | | | > > | # | | | | > > | # | | | | > > | #...Doit.....|...........>| | | > > | # | |. acquire...>| | > > |<-PyGILState_Ensure--| | | | > > | ... ... ... ... > > |Stop # > > |-------`| > > | > > |----Ensure rtns-----># PyObject_ | | > > | : |CallMethod | | | > > | : |.(Doit)...> |. acquire...>| DEADLOCK | > > : > > : > > :.how does python thread tell > > PyScheduler not to give away > > Gil until we are done with > > critical section?? > > > So my question is how in python do I tell the scheduler not to prempt > > the current python thread. Especially how to tell it not to give the > > GIL to any calls to PyGILState_ensure until further notice (i.e. until > > I am outside my critical section?? It sounds like a reasonable request > > - surely this exists already but I can't find it. > > It took me some time to understand the problem you were describing > because you've got some terms backward - there's no Python scheduler, > and calling PyGILState_ensure doesn't preempt anything. The Python > interpreter *releases* the GIL every so often to allow other threads > looking for it to run, but calling the python GIL functions has no > effect on preemption. > > The problem is that the GIL is being released while your object lock > is held, a second thread (started from C) acquires the GIL and then > blocks on the object lock. What you seem to be seeing is that it > blocking on the object lock is preventing it from releasing the GIL, > which prevents the python thread from running and releasing the lock. > > This shouldn't happen - blocking on a lock releases the GIL, so the > python thread should run, release the GIL, and eventually your C > thread should be able to acquire both locks at the same time. Are you > sure that you're correctly acquiring the GIL in your C code? > > The data flow you *should* be seeing should look something like this: > > GIL object lock (P=Python, C=C, *=released) > ------------------- > P P Python holds both locks > * P Python releases the GIL, inside critical section > C P C thread acquires the GIL and starts executing Python code > * P C thread tries to acquire object lock and blocks, > releasing the GIL > P P Python thread re-acquires the GIL > P * Python thread exits critical section and releases > the object lock > * * Python thread releases the GIL (can be in any > order with next state) > * C The C thread acquires the object lock and blocks on the > GIL > C C C thread acquires the GIL and continues execution. > > > One thing that may work (though the documentation does not > > specifically say so) is using setcheckinterval() to set the check > > interval to a very large value, acessing my shared structure and then > > setting the check interval back to the previous value. Provided my > > access to the shared structure takes less byte codes than what I set > > checkinterval to I should be okay. However that would be very > > dependant on the exact fine detail of how the check interval works and > > may not be compatible with other Python releases > > > Maybe someone familiar with the python source code would know for > > sure? > > > I am using Python 2.4.3 on windows XP. > > > Thanks for any help/suggestions offered! > > BR, > > Billy. > > > -- > >http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
