Re: Py_NewInterpreter(), is this a bug in the python core?
freesteel wrote:
> Yes, I see that now in the documentation, which to me is quite
> confusing.
> So, how do you use python in a multithreaded environment, where for
> example you want to run some embeded python code from a number of
> different C threads?
>
> This article: http://www.linuxjournal.com/article/3641 is quite good,
> but must have been written before these PyGILState_* functions.
>
> I only used Py_NewInterpreter to have a fresh 'import sys'.
>
> Somebody enlighten me, please.
>
> Martin
If you try to replicate the code from the linux journal
(http://www.linuxjournal.com/article/3641) and compile with Py_DEBUG
defined, when running it you will find that you get an exception and a
fatal error message from the python core. The exception is thrown from
pystate.c, line 306:
Py_FatalError("Invalid thread state for this thread");
The exception is thrown in my understanding of the code because there
can only ever be one thread state. If this is true a function to swap
thread states seems rather pointless.
Now, reading up about how to call C API to python from a C thread I
find that with version 2.3 the function pair
PyGILState_Ensure/PyGILState_Release was introduced. This allows
'grabbing' the global interpreter lock, calling your embedded python
code and release the GIL at the end again.
Am I right in my understanding that the use of this PyGILState_* pair
is meant to 'replace' the prologue of creating a new thread state from
the main thread state, swapping it with the current state, doing your
Python/C API calls and then in an epilogue swap the previous thread
state back in, and deletie the now obsolete previously created thread
state?
At least that how I understand the motivation behind the introduction
of tPyGILState_*, read here:
http://www.python.org/dev/peps/pep-0311/
Now, I tried to use this mechanism, but I am not really successful with
it. Basically, in my C thread, I wrap a number of calls to embedded
python (a few PyRun_SimpleString calls, nothing really fancy) with
PyGILState_Ensure and PyGILState_Release. The first C thread also
initializes python as well as initializes threading:
Py_Initialize();
PyEval_InitThreads();
(If it helps I can post my whole code here).
When I test this I get deadlocks in most cases after the first thread
has called PyGILState_Release. All other threads at that point 'freeze'
and never return from whatever Python/C API function they are in. I
don't understand why. Moreover, this behavious is not always
reproducable, sometimes the code works as intended. Must be some kind
of race condition? Who can help?
I read that the problem of calling python from a multithreaded
application per thread is very difficult, but a few projects have
managed to solve it, but now we have these PyGILStates and all is much
easier?
Martin
--
http://mail.python.org/mailman/listinfo/python-list
embedded python and windows multi threading, can't get it to work
I am trying to run a python programme embedded from C++. I want to run
the same python code concurrently in several threads. I read the manual
on embedding, especially chapter 8, and searched for relevant info on
google all afternoon, but I can't get this to work. What am I doing
wrong? I use python2.4 and vc++7 (.net). The first thread seems to work
okay, the 2nd thread crashes, but the exception information is not very
useful:
(An unhandled exception of type 'System.NullReferenceException'
occurred in pyembed_test.exe
Additional information: Object reference not set to an instance of an
object.)
Here is a simplified example, it is heavily based on this article:
http://www.codeproject.com/cpp/embedpython_1.asp
#include
#include
#include
#include
#define USE_AFX 0
#if USE_AFX
UINT MyThread(LPVOID lpParam)
#else
void MyThread(void* lpParam)
#endif
{
Py_Initialize();
//#if 0
// PyEval_InitThreads();
PyThreadState *mainthreadstate = PyThreadState_Get();
PyInterpreterState *maininterpreterstate = mainthreadstate->interp;
PyThreadState *mythreadstate =
PyThreadState_New(maininterpreterstate);
PyEval_ReleaseLock();
PyEval_AcquireLock();
PyThreadState *tmpstate = PyThreadState_Swap(mythreadstate);
//#endif
int* num = (int *)lpParam;
int ret = 0;
ret = PyRun_SimpleString("x = []");
ret = PyRun_SimpleString("for i in range(10):\n x.append(i)");
char cmd[100];
sprintf(cmd, "f = open('test%d.txt', 'w')", *num);
ret = PyRun_SimpleString(cmd);
ret = PyRun_SimpleString("f.write('%s' % x.__str__())");
ret = PyRun_SimpleString("f.close()");
//#if 0
PyThreadState_Swap(tmpstate);
PyEval_ReleaseLock();
PyThreadState_Clear(mythreadstate);
PyThreadState_Delete(mythreadstate);
//#endif
Py_Finalize();
#if USE_AFX
return 0;
#else
_endthread();
#endif
}
class CMyWinApp : public CWinApp
{
public:
CMyWinApp() { }
BOOL InitInstance()
{
int num0 = 0, num1 = 1;
HANDLE hnd0, hnd1;
#if USE_AFX
CWinThread* pThread0 = AfxBeginThread(MyThread, &num0);
hnd0 = pThread0->m_hThread;
CWinThread* pThread1 = AfxBeginThread(MyThread, &num1);
hnd1 = pThread1->m_hThread;
#else
hnd0 = (HANDLE) _beginthread(MyThread, 0, &num0);
hnd1 = (HANDLE) _beginthread(MyThread, 0, &num1);
#endif
// wait for the worker thread to terminate.
WaitForSingleObject(hnd0, INFINITE);
WaitForSingleObject(hnd1, INFINITE);
return TRUE;
};
};
CMyWinApp app;
--
http://mail.python.org/mailman/listinfo/python-list
multithreading windows and embedding python
I have posted about this problem before. SInce then I found a much
better article to help with embedding python in a multithreaded
application:
http://www.linuxjournal.com/article/3641
I found this article very good and it clarified for me what needs
doing. Now I have an example application that almost works, but it is
far from reliable. If I run this several times I get crashes telling me
that the heap is modified after deallocation. Can anybody else
reproduce this?
At the bottom of this file I left a debug dump, and a stack dump.
Here is my application, compile in windows using a standard windows
application project.
#include
#include
#include
#include
#include
#include
static int threadnum = 0;
UINT MyThread(LPVOID lpParam)
{
ASSERT(Py_IsInitialized());
threadnum++;
PyThreadState* mainThreadState = (PyThreadState *)lpParam;
// get the global lock
PyEval_AcquireLock();
// get a reference to the PyInterpreterState
PyInterpreterState * mainInterpreterState = mainThreadState->interp;
PyThreadState_Swap(mainThreadState);
// create a thread state object for this thread
PyThreadState * myThreadState =
PyThreadState_New(mainInterpreterState);
// free the lock
PyEval_ReleaseLock();
// lock - swap in thread state - swap out thread state - unlock
PyEval_AcquireLock();
PyThreadState_Swap(myThreadState);
int num = 0;
int ret = 0;
ret = PyRun_SimpleString("x = []");
ret = PyRun_SimpleString("for i in range(10):\n x.append(i)");
char cmd[100];
sprintf(cmd, "f = open('c:/windows/temp/test%d.txt', 'w')",
threadnum);
ret = PyRun_SimpleString(cmd);
ret = PyRun_SimpleString("f.write('%s\\n' % x.__str__())");
sprintf(cmd, "f.write('0x%d\\n')", &myThreadState);
ret = PyRun_SimpleString(cmd);
ret = PyRun_SimpleString("f.close()");
PyThreadState_Swap(NULL);
PyEval_ReleaseLock();
// clean up
// grab the lock
PyEval_AcquireLock();
// swap my thread state out of the interpreter
PyThreadState_Swap(NULL);
// clear out any cruft from thread state object
PyThreadState_Clear(myThreadState);
// delete my thread state object
PyThreadState_Delete(myThreadState);
// release the lock
PyEval_ReleaseLock();
return 0;
}
class CMyWinApp : public CWinApp
{
public:
CMyWinApp() { }
BOOL InitInstance()
{
Py_Initialize();
PyEval_InitThreads();
// save a pointer to the main PyThreadState object
PyThreadState * mainThreadState = PyThreadState_Get();
// release the lock
PyEval_ReleaseLock();
const int nhandles = 100;
HANDLE hnd[nhandles];
CWinThread* pThread[nhandles];
for (int ih = 0; ih < nhandles; ih++)
{
pThread[ih] = AfxBeginThread(MyThread, mainThreadState,
THREAD_PRIORITY_NORMAL, CREATE_SUSPENDED);
pThread[ih]->m_bAutoDelete = false;
pThread[ih]->ResumeThread();
hnd[ih] = pThread[ih]->m_hThread;
}
int nwaits, nfails;
do
{
nwaits = 0;
nfails = 0;
for (int ih = 0; ih < nhandles; ih++)
{
DWORD ret = WaitForSingleObject(hnd[ih],
INFINITE);
switch (ret)
{
case WAIT_OBJECT_0:
printf("WAIT_OBJECT_0\n");
break;
case WAIT_TIMEOUT:
++nwaits;
printf("WAIT_TIMEOUT\n");
break;
case WAIT_FAILED:
++nfails;
printf("WAIT_FAILED\n");
break;
}
}
}
while (nwaits > 0);
ASSERT(nfails == 0);
// delete all windows threads
for (int ih = 0; ih < nhandles; ++ih)
delete pThread[ih];
PyEval_AcquireLock();
PyThreadState_Swap(mainThreadState);
Py_Finalize();
return TRUE;
};
};
CMyWinApp app;
Debug dump:
'pyembed_test.exe': Loaded
'C:\mdunschen\pyembed
Py_NewInterpreter(), is this a bug in the python core?
/*
Is this a bug in Py_NewInterpreter?
The function below "MyThread" is instantiated from a windows worker
thread, but I don't
think that is relevant.
(I can try this on a linux box, but I would have to compile a python
library with debugging
enabled.)
The following code fragment throws an exception in a debug version of
python:
*/
UINT MyThread(LPVOID lpParam)
{
{
cs.Lock(); // this is a CCriticalSection lock
if (!Py_IsInitialized())
{
Py_Initialize();
PyEval_InitThreads();
// global pointer to the main PyThreadState object
mainThreadState = PyThreadState_Get();
PyEval_ReleaseLock();
}
cs.Unlock();
}
ASSERT(Py_IsInitialized());
ASSERT(PyEval_ThreadsInitialized());
ASSERT(mainThreadState);
threadnum++;
// get the global lock
PyEval_AcquireLock();
PyGILState_STATE gstate;
gstate = PyGILState_Ensure(); // Is tis necessary?
PyThreadState_Swap(mainThreadState);
PyThreadState* nts = Py_NewInterpreter();
/*
The exception is thrown inside the above function call:
This statement tries to swap the new threadstate 'tstate' with the
current one
save_tstate = PyThreadState_Swap(tstate);
Inside PyThreadState_Swap the code uses another way
'PyGILState_GetThisThreadState()' to find the current thread state and
compares this with the newly set thread state.
Naturally you would expect the two to be equal but that test fails:
#if defined(Py_DEBUG) && defined(WITH_THREAD)
if (new) {
PyThreadState *check = PyGILState_GetThisThreadState();
if (check && check != new)
Py_FatalError("Invalid thread state for this thread");
}
#endif
The variable 'check' looks as if it is the 'previous' thread state, as
if changing the thread state
is not been done properly. Py_FatalError is called and that's the end.
Is there a mistake in my code, or is there something wrong in how
Py_NewInterpreter is implemented?
Thanks
Martin
PS: Below the rest of my simple test worker thread function.
*/
ASSERT(nts == PyThreadState_Get());
// lock (already locked) - swap in thread state - swap out thread
state - unlock
init_testclass();
int ret = 0;
ret = PyRun_SimpleString("import sys");
ret = PyRun_SimpleString("class redir:\n def __init__(self, id):\n
self.id = id\n def write(self, s):\nf = open('stdoutputs_%s.txt' %
self.id, 'a')\nf.write('%s: %s' % (self.id, s))\nf.close()\n");
char str[100];
sprintf(str,"r = redir('0x%x')", &nts);
ret = PyRun_SimpleString(str);
ret = PyRun_SimpleString("sys.stderr = r");
sprintf(str,"s = redir('0x%x')", &nts);
ret = PyRun_SimpleString(str);
ret = PyRun_SimpleString("sys.stdout = s");
ret = PyRun_SimpleString("import testclass");
ret = PyRun_SimpleString("t = testclass.testclass()");
sprintf(str,"print 't = ', t ");
ret = PyRun_SimpleString(str);
ret = PyRun_SimpleString("print t.run(10)");
Py_EndInterpreter(nts);
PyGILState_Release(gstate);
PyEval_ReleaseLock();
return 0;
}
--
http://mail.python.org/mailman/listinfo/python-list
Re: multithreading windows and embedding python
freesteel wrote: ... > pThread[ih] = AfxBeginThread(MyThread, mainThreadState, > THREAD_PRIORITY_NORMAL, CREATE_SUSPENDED); ... Here the call to AfxBeginThread is wrong, there is one argument missing, it should be: pThread[ih] = AfxBeginThread(MyThread, mainThreadState, THREAD_PRIORITY_NORMAL, 0, CREATE_SUSPENDED); Because there are so many default arguments of similar types the compiler did not notice that I passed 'CREATE_SUSPENDED' as a stack size, and use the default 'creation' state of 'start right away' for the thread. Don't you love default args and types like 'void*' ? Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Py_NewInterpreter(), is this a bug in the python core?
Yes, I see that now in the documentation, which to me is quite confusing. So, how do you use python in a multithreaded environment, where for example you want to run some embeded python code from a number of different C threads? This article: http://www.linuxjournal.com/article/3641 is quite good, but must have been written before these PyGILState_* functions. I only used Py_NewInterpreter to have a fresh 'import sys'. Somebody enlighten me, please. Martin -- http://mail.python.org/mailman/listinfo/python-list
