segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Christian Heimes
Hello, I've updated our software stack from Python 2.6.6 to Python 2.7.1. Since the update I'm seeing random segfaults all related to JCCEnv::deleteGlobalRef() and Python's GC. At first I thought the bug is an incompatibility between Python 2.7 and JCC 2.7. However an update to JCC 2.8 and Lucence

Re: segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Christian Heimes
Here is another backtrace with a debug build of JCC. #10 #11 0x2b05b2cf4c74 in JNIEnv_::DeleteGlobalRef (this=0x0, gref=0x2b05d02e0250) at /opt/vlspy27/lib/jdk1.6/include/jni.h:830 #12 0x2b05b2cf0ce5 in JCCEnv::deleteGlobalRef (this=0x360f650, obj=0x2b05d02e0250, id=1351575229) at jcc/sou

Re: segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Andi Vajda
On May 11, 2011, at 7:59, Christian Heimes wrote: > Here is another backtrace with a debug build of JCC. > > #10 > #11 0x2b05b2cf4c74 in JNIEnv_::DeleteGlobalRef (this=0x0, > gref=0x2b05d02e0250) at /opt/vlspy27/lib/jdk1.6/include/jni.h:830 > #12 0x2b05b2cf0ce5 in JCCEnv::deleteGlobalR

Re: segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Christian Heimes
Am 11.05.2011 17:36, schrieb Andi Vajda: >> As you can clearly see, the JNIEnv_ instance is a NULL pointer. Contrary >> to my initial assumption, the thread doesn't have a JCC thread local >> object. Since any thread may trigger a GC collect run, and not just >> threads, that use JCC, this looks li

Re: segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Christian Heimes
Am 11.05.2011 18:14, schrieb Christian Heimes: > --- > import lucene > import threading > import time > import gc > > lucene.initVM() > > def alloc(): > while 1: > gc.collect() > time.sleep(0.011) > > t = threading.Thread(target=alloc) > t.daemon = True > > t.start() > > wh

Re: segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Andi Vajda
On Wed, 11 May 2011, Christian Heimes wrote: Am 11.05.2011 17:36, schrieb Andi Vajda: As you can clearly see, the JNIEnv_ instance is a NULL pointer. Contrary to my initial assumption, the thread doesn't have a JCC thread local object. Since any thread may trigger a GC collect run, and not ju

Re: segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Andi Vajda
On Wed, 11 May 2011, Christian Heimes wrote: Am 11.05.2011 18:14, schrieb Christian Heimes: --- import lucene import threading import time import gc lucene.initVM() def alloc(): while 1: gc.collect() time.sleep(0.011) t = threading.Thread(target=alloc) t.daemon = True t

Re: segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Christian Heimes
> There shouldn't be any random threads. Threads don't just appear out of thin > air. You create them. If there is a chance that they call into the JVM, then > attachCurrentThread(). I've already made sure, that all our code and threads are calling a hook, which attaches the thread to the JVM. B

Re: segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Christian Heimes
Am 11.05.2011 18:27, schrieb Andi Vajda: > Does it crash as easily with Python 2.6 ? > If not, then that could be an answer as to why this wasn't noticed before. It's crashing with Python 2.6.6 and Python 2.7.1. Sometimes it takes less than a second, sometimes half a minute or more. I would have t

Re: segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Christian Heimes
Am 11.05.2011 18:27, schrieb Andi Vajda: > Does it crash as easily with Python 2.6 ? > If not, then that could be an answer as to why this wasn't noticed before. With 20 test samples, it seems like Python 2.6 survives 50% longer than Python 2.7. python2.6 0, 1.089 1, 2.688 2, 1.066 3, 6.416 4, 0.

Re: segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Christian Heimes
Am 11.05.2011 18:26, schrieb Andi Vajda: > There shouldn't be any random threads. Threads don't just appear out of thin > air. You create them. If there is a chance that they call into the JVM, then > attachCurrentThread(). I've already made sure, that all our code and threads are calling a hook

Re: segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Christian Heimes
Am 11.05.2011 19:03, schrieb Andi Vajda: > If these libraries use Python's Thread class you have some control. > > Create a subclass of Thread that runs your hook and insert it into the > threading module (threading.Thread = YourThreadSubclass) before anyone else > gets a chance to create thread

Re: segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Andi Vajda
On Wed, 11 May 2011, Christian Heimes wrote: Am 11.05.2011 19:03, schrieb Andi Vajda: If these libraries use Python's Thread class you have some control. Create a subclass of Thread that runs your hook and insert it into the threading module (threading.Thread = YourThreadSubclass) before anyo

Re: segfault in JCCEnv::deleteGlobalRef

2011-05-11 Thread Christian Heimes
Am 11.05.2011 19:41, schrieb Andi Vajda: > If these functions eventually instantiate a Thread class, even indirectly, > the monkey-patching may still work. Some of the code doesn't use the threading module at all, just thread or the internal C API. I'd have to patch the modules and C code. > Tha