Re: IndexWriter optimize() deadlock

2009-10-17 Thread Michael McCandless
OK then this is exactly what was happening with LUCENE-1573 -- Lucene was not properly handling the interrupt request, and instead of throwing an exception, entered a spin loop that starves the merge threads. Can you try 2.9.0 (and turn back on your Future.cancel(true) call) and see if it's fixed?

Re: IndexWriter optimize() deadlock

2009-10-16 Thread Christopher Tignor
the doWait() call is synchronized on IndexWriter but it is also, as you suggest in a loop in a block synchronized on IndexWriter. The doWait() call returns immediately, still holding the IndexWriter lock from the loop in the synchornized block as my stack trace shows, without blocking and giving t

Re: IndexWriter optimize() deadlock

2009-10-16 Thread Michael McCandless
I'm glad you worked around it! But I don't fully understand the issue. That doWait is inside a sync(writer) block... if the Future manages to interrupt it, then that thread will release the lock when it exits that sync block. Actually, if the thread was indeed interrupted, you may be hitting thi

Re: IndexWriter optimize() deadlock

2009-10-16 Thread Christopher Tignor
I discovered the problem and fixed it's effect on my code: Using the source for Lucene version 2.4.1, in IndexWriter.optimize() there is a call to doWait() on line 2283 this method attempts to wait for a second in order to give other threads it has spawned a chance to acquire it's mutex and comple

Re: IndexWriter optimize() deadlock

2009-10-16 Thread Christopher Tignor
Indeed it looks like the thread running MergerThread started (After passing off to ConcurentMergeScheduler) by the thread calling IndexWriter.optimize() is indeed waiting on the mutex for the IndexWriter to be free so it can use the object to call mergeInit(). The IndexWriter however has entered a

Re: IndexWriter optimize() deadlock

2009-10-16 Thread Christopher Tignor
After tracing through the lucene source more it seems that what is happening is after I call Future.cancel(true) on my parent thread, optimize() is called and this method launches it's own thread using a ConcurrentMergeScheduler$MergeThread to do the actual merging. When this Thread comes around t

Re: IndexWriter optimize() deadlock

2009-10-16 Thread Michael McCandless
But if the Future.cancel call turns out to be a no-op (simply waits instead of interrupting the thread), how could it be that the deadlock only happens when you call it? Weird. Are you really sure it's not actually calling Thread.interrupt? That stack trace looks like a normal "optimize is waiti

Re: IndexWriter optimize() deadlock

2009-10-16 Thread Christopher Tignor
It doesn't look like my Future.cancel(true) is actually interrupting the thread. It only does so "if necessary" and in this case seems to be letting the Thread finish gracefully without need for interruption. The stack trace leading up to the hanging IndexWriter.optimize() method is below, though

Re: IndexWriter optimize() deadlock

2009-10-16 Thread Michael McCandless
My guess is it's the invocation of Thread.interrupt (which Future.cancel(true) calls if the task is running) that lead to the deadlock. Is it possible to get the stack trace of the thrown exception when the thread was interrupted? Maybe indeed something in IW isn't cleaning up its state on being

Re: IndexWriter optimize() deadlock

2009-10-16 Thread Christopher Tignor
thanks for getting back. I do not lock on the IndexWriter object itself but all methods in my consumer class that use IndexWriter are synchronized (locking my singleton consumer object itself). The thread is waiting at IndexWriter.doWait(). What might cuase this? thanks - C>T> On Fri, Oct 16,

RE: IndexWriter optimize() deadlock

2009-10-16 Thread Uwe Schindler
Do you use the IndexWriter as mutex in a synchronized() block? This is not supported and may hang. Never lock on IndexWriter instances. IndexWriter itself is thread safe. - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Messag