Hiho,
could somebody please enlighten me about the mechanics of C callbacks to
Python? My domain is more specifically callbacks from the win32 API, but
I'm not sure that's where the problem lies. Here's a description...
I want a callback-based MIDI input/processing, so PortMidi was not an
alternative. I have written a C extension module that links to the mmsys
MIDI API. I separated the win32-dependant code from the Python extension
code, so a) the module interface is system-neutral, b) the
implementation can be tested (re-used) outside of Python. So, that code
is tested OK, but it might be useful to sketch it's design:
- as you may know, the MIDI input on win32 is already managed thru a
callback mechanism; the driver calls back your program with data buffers
- yet, I don't call directly into Python from the callback, since some
restrictions apply on what system calls you can make from there, and I
don't know what Python's interpreter / scripts might call.
- so, on callback, I create a new thread, after checking that the
previous one has returned already (WaitOnSingleObject(mythread)) so we
only have one thread involved.
- this is this thread that calls the user callback, yet this callback
isn't yet a Python callable, we're still in native C code.
- on the side of the extension module now, I merely copied the callback
example from the Python/C API doc, and added GIL management around the call:
static PyObject * my_callback = NULL; //this gets fixed by a
setCallback() func
static void external_callback(const MidiData * const data) {
if (my_callback && (my_callback != Py_None)) {
if (! data) {
PyErr_SetString(PyExc_IndexError, getLastErrorMessage());
} else {
PyObject * arglist = NULL;
PyObject * result = NULL;
arglist = Py_BuildValue("(i,i,s#)", data->deviceIndex,
data->timestamp, data->buffer, data->size);// 0, 0, "test", 4);//
PyGILState_STATE gil = PyGILState_Ensure();
result = PyEval_CallObject(my_callback, arglist);
PyGILState_Release(gil);
Py_DECREF(arglist);
Py_DECREF(result);
}
}
}
- this one above is what is actually passed as callback to the 'native'
C part. So, here's what (I presume) happens:
1. the driver calls into my C code
2. my C code spawns a thread that calls into the extension
3. the extension calls into Python, after acquiring the GIL
Now, here's the hiccup:
inside a Python script, anytime a Python object is accessed by both the
(Python) callback and the main program, I get a GPF :/
You bet I tried to use locks, Queues and whatnot, but nothing will do,
it seems I get protection faults on accessing... thread exclusion objects.
Yet, there is a way where it works seamlessly: it's when the __main__
actually spawns a threading.Thread to access the shared object. Hence I
am lost.
This works:
def input_callback(msg):
midiio.send(msg) ##MIDI thru, a call into my extension that wraps
the implementation call with BEGIN_ALLOW_THREADS and END_...
__main__:
raw_input()
This results in GPF as soon as the callback is fired:
tape = Queue.Queue()
def input_callback(msg):
tape.put(msg)
__main__:
while True:
print tape.get()
This works like a charm:
tape = Queue.Queue()
def input_callback(msg):
tape.put(msg)
def job():
while True:
print tape.get()
__main__:
t = threading.Thread(target = job)
t.start()
raw_input()
Note that I also get a GPF in the later case if I stop the program with
KeyInterrupt, but I guess it's something I'll look into afterwards...
While I can think of hiding away the added threading within a wrapper
module, I won't sleep well untill I figure out what's really happening.
Could someone save my precious and already sparse sleeping time, by
pointing me to what I'm missing here?
WinXP SP1
Python 2.4.1
MinGW GCC 3.4.2
TIA,
Francois De Serres
--
http://mail.python.org/mailman/listinfo/python-list
