On Fri, 26 Feb 2010 12:08:58 +0000, Phil Thompson <p...@riverbankcomputing.com> wrote: > On Fri, 26 Feb 2010 11:05:44 +0100, Denis RIVIERE <denis.rivi...@cea.fr> > wrote: >> We have run into what we think to be more or less a bug in PyQt4, in >> QProcess, >> somewhat indirectly. >> QProcess sometimes hangsup in threaded PyQt applications, at least on > Linux >> >> systems. >> The situation is the following: >> - A PyQt program is using two or more threads (say, python threads) >> - one of the threads is using a QProcess (whatever thread), while the > other >> is >> using python (of course). >> >> We think we have understood what is going on: >> >> Internally, QProcess forks then probably uses a kind of exec(). >> But the child forked process sometimes gets locked on the python GIL > lock. >> Then the calling process waits for the child, and also gets hung on a >> select() >> call. >> >> Actually, when forking a multi-threaded process, only the forking thread > is >> >> duplicated: so the child process has only 1 thread. But the locks, > mutexes >> and other threading items have kept the state they had in their parent >> process: some are locked, but the other threads which would normally >> release >> them do not exist in the child. >> Here, the python GIL happens to be locked by another thread in the parent > >> process, and the child process seems to try to lock the GIL too. We guess > >> QProcess calls a virtual method in the forked process, which is > overloaded >> by >> SIP/PyQt, which in turn calls the Python API, and finally gets hung > trying >> to >> lock the GIL. >> We have never encountered this problem using PyQt3, maybe because > QProcess >> or >> its PyQt binding may be implemented differently. >> >> This threading + fork problem is a known difficulty in fork(), explained > in >> >> the linux manpages for instance. >> >> We have 2 possible solutions for this problem: >> 1. Using pthreads, it is possible to define handlers functions that can >> take >> care of locking/unlocking the needed locks whenever fork() is called, > both >> in >> the parent and child processes: this is done using the pthread_atfork() >> function. Typically PyQt may define such handlers to take care of the GIL > >> locking when fork() is used. However we don't know if such a mechanism >> exists >> on non-pthread systems (Windows?). We also don't really know if the > problem >> >> exists on Windows; possibly not because Windows doesn't use fork() but >> probably rather a spawn equivalent (?). >> 2. The problem actually happens because the python API gets called from > the >> >> forked process. Normally, in C++ Qt, the QProcess doesn't trigger such a
>> locking problem. However the SIP binding defines a subclass which traps > all >> >> virtual methods calls and redirects them to python. So if we are not > using >> a >> sip-inherited QProcess class, the problem should not happen. We have made >> the >> test, making a sip-bound factory function which instantiates a QProcess >> from >> C++, then returns it to the pyton layer. This way no inherited class is >> used, >> and the problem is avoided. And it actually works. >> >> Would it be difficult to have solution 1 included in the standard PyQt ? > > Unless you can provide me with a test case you'll have to experiment... > > Try adding... > > pthread_atfork(NULL, NULL, PyOS_AfterFork); > > ...after the call to PyEval_InitThreads() in siplib.c. An alternative (which I may be more comfortable with) is to add the /HoldGIL/ annotation to all QProcess.start() overloads. However the asynchronous nature of process starts may mean it has no effect. Phil _______________________________________________ PyQt mailing list PyQt@riverbankcomputing.com http://www.riverbankcomputing.com/mailman/listinfo/pyqt
