Many thanks for the clarification. Also good idea to focus on lock
ownership rather that thread activity in your diagram.
To be honest I was not actually experiencing deadlock issues. I had
just deduced (incorrectly) that I might do so if I started using locks
in my py code called from c-land. As I was suffering some horrible
race-condtions that had suddenly appeared in code that had been
perfectly stable for the past 6 months I didn't want to go down that
route given my misconceptions; Even if introducing the locks worked
I'd always feel it was a deadlock waiting to happen.
Misunderstanding arose as for some reason I had decided that python
code called from C did not allow any other python threads to run until
control had returned to C and the gil had been released from C-land.
I.e. once the gil was given to the C thread then python had to wait
until it was given back.
I have to say the docs could be a little more explicit on the
mechanisms involved. They tend to be better at stating what individual
functions do rather that giving overall explanations. Maybe I'll get
around to submitting a chapter on it ;)
On Nov 12, 4:54 pm, "Chris Mellon" <[EMAIL PROTECTED]> wrote:
> On Nov 12, 2007 6:56 AM, <[EMAIL PROTECTED]> wrote:
>
>
>
> > Hi,
>
> > I have a native windows thread in a c python module which calls into
> > python code and adds an item to a data structure (a home-grown
> > circular buffer). At the same time my main python application is
> > removing items from this data structure.
>
> > Unlike native python containers adding and removing items from the
> > circular buffer is not atomic. So I need to put access to it in a
> > critical section. Using threading.Lock will work for native python
> > threads competing for access. But when access is from a windows thread
> > this will not work. That is because my call to PyGilState_ensure will
> > preempt my native python thread ***even when it is inside the critical
> > section***.
>
> > What is going on looks something like this (I think).
>
> >Py Python Windows Py threading.Lock
> > resource
> > SchedThread Thread Code| |
> > | || | |
> > |Go (GIL)#|| | |
> > |#|| | |
> > |#|| | |
> > |#...Doit.|...>| | |
> > |#||. acquire...>| |
> > |<-PyGILState_Ensure--|| | |
> > | ... ... ... ...
> > |Stop#
> > |---`|
> > |
> > |Ensure rtns-># PyObject_ ||
> > | :|CallMethod | | |
> > | :|.(Doit)...> |. acquire...>| DEADLOCK |
> > :
> > :
> > :.how does python thread tell
> > PyScheduler not to give away
> > Gil until we are done with
> > critical section??
>
> > So my question is how in python do I tell the scheduler not to prempt
> > the current python thread. Especially how to tell it not to give the
> > GIL to any calls to PyGILState_ensure until further notice (i.e. until
> > I am outside my critical section?? It sounds like a reasonable request
> > - surely this exists already but I can't find it.
>
> It took me some time to understand the problem you were describing
> because you've got some terms backward - there's no Python scheduler,
> and calling PyGILState_ensure doesn't preempt anything. The Python
> interpreter *releases* the GIL every so often to allow other threads
> looking for it to run, but calling the python GIL functions has no
> effect on preemption.
>
> The problem is that the GIL is being released while your object lock
> is held, a second thread (started from C) acquires the GIL and then
> blocks on the object lock. What you seem to be seeing is that it
> blocking on the object lock is preventing it from releasing the GIL,
> which prevents the python thread from running and releasing the lock.
>
> This shouldn't happen - blocking on a lock releases the GIL, so the
> python thread should run, release the GIL, and eventually your C
> thread should be able to acquire both locks at the same time. Are you
> sure that you're correctly acquiring the GIL in your C code?
>
> The data flow you *should* be seeing should look something like this:
>
> GIL object lock (P=Python, C=C, *=released)
> ---
> P P Python holds both locks
> * P Python releases the GIL, inside critical section
> C P C thread acquires the GIL and starts executing Python code
> * P
