Hi!
I hope I've got the right list here- there were a few to choose from. :}
I am trying to embed Python with multiple interpreters into an existing
application. I have things working fine with a single interpreter thus
far. I am running into problems when using multiple interpreters [1] and
I am presently trying to track down these issues. Can anyone familiar
with the process of embedding multiple interpreters have a skim of the
details below and let me know of any obvious problems? If I can get the
essentials right, then presumably it's just a matter of my tracking down
any problems with my code.
I am presently using Python 3.3.3.
What I am after:
- Each sub-interpreter will have its own dedicated thread. Each thread
will have no more than one sub-interpreter. Basically, there is a
one-to-one mapping between threads and interpreters (some threads are
unrelated to Python though).
- The default interpreter in the main thread will never be used,
although I can explicitly use it if it'll help in some way.
- Each thread is created and managed outside of Python. This can't be
readily changed.
- I have a single internal module I need to be able to use for each
interpreter.
- I load scripts into __main__ and create objects from it to bootstrap.
- I understand that for the most part only a single interpreter will be
running at a time due to the GIL. This is unfortunate but not a major
problem.
- I don't need to share objects between interpreters (if it is even
possible- I don't know).
- My fallback if I can't do this is to implement each instance in a
dedicated *process* rather than per-thread. However, there is a
significant cost to doing this that I would rather not incur.
Could I confirm:
- There is one GIL in a given process, shared amongst all (sub)
interpreters. There seems some disagreement on this one online, although
I'm fairly confident that there is only the one GIL.
- I am using the mod_wsgi source for inspiration. Is there a better
source for an example of embedding multiple interpreters?
A query re the doco:
http://docs.python.org/3/c-api/init.html#gilstate
"Python supports the creation of additional interpreters (using
Py_NewInterpreter()), but mixing multiple interpreters and the
PyGILState_*() API is unsupported."
Is this actually correct? mod_wsgi seems to do it. Have I misunderstood?
I've extracted what I have so far from my code into a form that can be
followed more easily. Hopefully I have not made any mistakes in doing
so. The essence of what my code calls should be as follows:
=== Global init, run once:
static PyThreadState *mtstate = NULL;
PyImport_AppendInittab("myinternalmodule", PyInit_myinternalmodule);
Py_SetProgramName((wchar_t *)"foo");
Pu_InitializeEx(0);
PyEval_InitThreads();
mtstate = PyThreadState_Get();
PyEval_ReleaseThread(mtstate);
=== Global shutdown, run once at end:
Py_Finalize();
=== Per-interpreter init in main thread before launching child thread:
(none thus far)
=== Init in dedicated thread for each interpreter:
// NB: Also protected by a single global non-Python mutex to be sure.
PyGILState_STATE gil = PyGILState_Ensure();
PyThreadState *save_tstate = PyThreadState_Swap(NULL);
state = Py_NewInterpreter();
PyThreadState_Swap(save_tstate);
PyObject *mmodule = PyImport_AddModule("__main__");
Py_INCREF(mmodule);
PyImport_ImportModule("myinternalmodule");
PyGILState_Release(gil);
=== Shutdown in dedicated thread for each interpreter:
// NB: Also protected by the same single global non-Python mutex as in
the init.
PyGILState_STATE gil = PyGILState_Ensure();
PyThreadState *save_tstate = PyThreadState_Swap(state);
Py_EndInterpreter(state);
PyThreadState_Swap(save_tstate);
PyGILState_Release(gil);
=== Placed at top of scope where calls made to Python C API:
SafeLock lock;
=== SafeLock implementation:
class SafeLock
{
public:
SafeLock() {gil = PyGILState_Ensure();}
~SafeLock() {PyGILState_Release(gil);}
private:
PyGILState_STATE gil;
};
===
Does this look roughly right? Have I got the global and per-interpreter
init and shutdown right? Am I locking correctly in SafeLock- is
PyGILState_Ensure() and PyGILState_Release() sufficient?
Is there an authoritative summary of the global and per-interpreter init
and shutdown somewhere that I have missed? Any resource I should be reading?
Cheers,
Garth
[1] It presently crashes in Py_EndInterpreter() after running through a
series of tests during the shutdown of the 32nd interpreter I create. I
don't know if this is significant, but the tests pass for the first 31
interpreters.
--
https://mail.python.org/mailman/listinfo/python-list
