Hi Guys, So I think I’ve figured it out… We use a common merge scheduler to handle merges across all of our indices. When the SearchManager starts up, it triggers a merge operation, this merge request pushes a task on the merge scheduling queue and we only allow one merge request per index writer in the queue at a time. What appears to be happening is that a closed index is being left in the merge queue and the call is failing due to the .equals() method on the closed index when inserting into our BlockingArrayQueue. Usually this shouldn’t happen and appears to be a timing issue combined with not protecting against this possibility.
Thanks for your assistance guys! -Brian > On Jan 6, 2015, at 10:22 AM, Brian Call <brian.c...@soterawireless.com> wrote: > > No exception at all… and that’s the crazy part. I create a new IndexWriter > and then immediately create a new SearcherManager using that freshly-created > IndexWriter. Granted, I don’t actually do anything with the IndexWriter > before passing it to the SearcherManager, but no exceptions are thrown > either. > > -Brian > > >> On Jan 6, 2015, at 9:55 AM, Ian Lea <ian....@gmail.com >> <mailto:ian....@gmail.com>> wrote: >> >> Presumably no exception is thrown from the new IndexWriter(....) call? >> I'd double check that, and try some harmless method call on the >> writer and make sure that works. And run CheckIndex against the >> index. >> >> >> -- >> Ian. >> >> >> On Tue, Jan 6, 2015 at 5:05 PM, Brian Call >> <brian.c...@soterawireless.com <mailto:brian.c...@soterawireless.com>> wrote: >>> Hi Tomoko, >>> >>> Thank you for your response! We’ve actually never seen this before in the >>> three+ years of developing using Lucene 3.6.x. The only time we’ve ever >>> seen this kind of exception is once recently in a running production system >>> and it caught me way off guard. We’re deploying on Suse linux (enterprise), >>> and jdk1.7. >>> >>> Our application creates and deletes indices within the context of a single >>> thread so I don’t think another thread is abruptly closing the index. All >>> index open/close operations are always done sequentially in the context of >>> one thread as operation requests are received. >>> >>> Blessings, >>> Brian >>> >>> >>> >>>> On Jan 6, 2015, at 2:16 AM, Tomoko Uchida <tomoko.uchida.1...@gmail.com >>>> <mailto:tomoko.uchida.1...@gmail.com>> wrote: >>>> >>>> Hi, >>>> >>>> How often does this error occur? >>>> You do not tell the lucene version, but I guess you use lucene 3.x >>>> according to the stack trace... >>>> IndexWriter would not be closed until IndexWriter.close() method is called >>>> explicitly. >>>> https://github.com/apache/lucene-solr/blob/lucene_solr_3_6_2/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java >>>> >>>> <https://github.com/apache/lucene-solr/blob/lucene_solr_3_6_2/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java> >>>> >>>> <https://github.com/apache/lucene-solr/blob/lucene_solr_3_6_2/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java >>>> >>>> <https://github.com/apache/lucene-solr/blob/lucene_solr_3_6_2/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java>> >>>> >>>> Do you have custom codes wrapping lucene objects? There can be any codes >>>> which call IndexWriter.close() unexpectedly? >>>> >>>> If your application seems to have no problem, you need to share more >>>> information including lucene version and system environments. >>>> >>>> Regards, >>>> Tomoko >>>> >>>> >>>> 2015-01-06 8:32 GMT+09:00 Brian Call <brian.c...@soterawireless.com >>>> <mailto:brian.c...@soterawireless.com> >>>> <mailto:brian.c...@soterawireless.com >>>> <mailto:brian.c...@soterawireless.com>>>: >>>> >>>>> Hi Guys, >>>>> >>>>> So I’m seeing a problem in production that is very bizarre and I wanted to >>>>> see if anyone else has encountered this issue and/or has a suggestion or >>>>> fix. Here goes: >>>>> >>>>> We create a wrapper around an index and searcher manager to encapsulate >>>>> both. First we create the IndexWriter and then immediately after create >>>>> the >>>>> SearcherManager, like this: >>>>> >>>>> indexWriter = new IndexWriter(indexDirectory, getWriterConfig()); >>>>> searcherManager = new SearcherManager(indexWriter, true, new >>>>> ExecutorSearcherFactory()); >>>>> >>>>> >>>>> During construction of the SearcherManager this is thrown: >>>>> >>>>> Caused by: org.apache.lucene.store.AlreadyClosedException: this >>>>> IndexWriter is closed >>>>> at >>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:766) >>>>> at >>>>> org.apache.lucene.index.IndexWriter.getDirectory(IndexWriter.java:1909) >>>>> at >>>>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.getIndexDirectory(LuceneIndexMergeScheduler.java:162) >>>>> at >>>>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.access$000(LuceneIndexMergeScheduler.java:31) >>>>> at >>>>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler$MergeTask.equals(LuceneIndexMergeScheduler.java:127) >>>>> at >>>>> java.util.concurrent.ArrayBlockingQueue.contains(ArrayBlockingQueue.java:497) >>>>> at >>>>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.merge(LuceneIndexMergeScheduler.java:148) >>>>> at >>>>> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2740) >>>>> at >>>>> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2734) >>>>> at >>>>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:457) >>>>> at >>>>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) >>>>> at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) >>>>> at >>>>> org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:82) >>>>> at >>>>> com.triagewireless.h1s.session.data.index.LuceneIndex.initNoCache(LuceneIndex.java:312) >>>>> at >>>>> com.triagewireless.h1s.session.data.index.LuceneIndex.init(LuceneIndex.java:270) >>>>> at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>> at >>>>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:346) >>>>> at >>>>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:299) >>>>> at >>>>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:132) >>>>> ... 57 more >>>>> >>>>> >>>>> Why? No other threads are acting on this index, etc. The index writer was >>>>> just created so how could it already be closed? >>>>> >>>>> I’m completely baffled on this one guys… so many thanks in advance for >>>>> your help! I’ll take any suggestions on a possible mitigation too if >>>>> anyone >>>>> thinks of any. >>>>> >>>>> >>>>> Blessings, >>>>> Brian Call >>>>> Manager, Systems Software Development >>>>> Work: +1 (619) 373-4840 | Cell: +1 (619) 344-1013 >>>>> >>>>> <http://soterawireless.com/ <http://soterawireless.com/> >>>>> <http://soterawireless.com/ <http://soterawireless.com/>>> >>>>> Sotera Wireless, Inc. >>>>> 10020 Huennekens Street San Diego, CA 92121, USA >>>>> Phone: +1 (858) 427-4620 | Fax: +1 (858) 999-2487 >>>>> >>>>> PRIVILEGED AND CONFIDENTIAL COMMUNICATION: This electronic transmission, >>>>> and any documents attached hereto, may contain confidential and/or legally >>>>> privileged information. The information is intended only for use by the >>>>> recipient named above. If you have received this electronic message in >>>>> error, please notify the sender and delete the electronic message. Any >>>>> disclosure, copying, distribution, or use of the contents of information >>>>> received in error is strictly prohibited. >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> <mailto:java-user-unsubscr...@lucene.apache.org> >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> <mailto:java-user-h...@lucene.apache.org>
