Re: AlreadyClosedException on new index

[Tomoko Uchida]

Hi,

How often does this error occur?
You do not tell the lucene version, but I guess you use lucene 3.x
according to the stack trace...
IndexWriter would not be closed until IndexWriter.close() method is called
explicitly.
https://github.com/apache/lucene-solr/blob/lucene_solr_3_6_2/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java

Do you have custom codes wrapping lucene objects? There can be any codes
which call IndexWriter.close() unexpectedly?

If your application seems to have no problem, you need to share more
information including lucene version and system environments.

Regards,
Tomoko


2015-01-06 8:32 GMT+09:00 Brian Call :

> Hi Guys,
>
> So I’m seeing a problem in production that is very bizarre and I wanted to
> see if anyone else has encountered this issue and/or has a suggestion or
> fix. Here goes:
>
> We create a wrapper around an index and searcher manager to encapsulate
> both. First we create the IndexWriter and then immediately after create the
> SearcherManager, like this:
>
> indexWriter = new IndexWriter(indexDirectory, getWriterConfig());
> searcherManager = new SearcherManager(indexWriter, true, new
> ExecutorSearcherFactory());
>
>
> During construction of the SearcherManager this is thrown:
>
> Caused by: org.apache.lucene.store.AlreadyClosedException: this
> IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:766)
> at
> org.apache.lucene.index.IndexWriter.getDirectory(IndexWriter.java:1909)
> at
> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.getIndexDirectory(LuceneIndexMergeScheduler.java:162)
> at
> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.access$000(LuceneIndexMergeScheduler.java:31)
> at
> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler$MergeTask.equals(LuceneIndexMergeScheduler.java:127)
> at
> java.util.concurrent.ArrayBlockingQueue.contains(ArrayBlockingQueue.java:497)
> at
> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.merge(LuceneIndexMergeScheduler.java:148)
> at
> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2740)
> at
> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2734)
> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:457)
> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:296)
> at
> org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
> at
> com.triagewireless.h1s.session.data.index.LuceneIndex.initNoCache(LuceneIndex.java:312)
> at
> com.triagewireless.h1s.session.data.index.LuceneIndex.init(LuceneIndex.java:270)
> at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:346)
> at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:299)
> at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:132)
> ... 57 more
>
>
> Why? No other threads are acting on this index, etc. The index writer was
> just created so how could it already be closed?
>
> I’m completely baffled on this one guys… so many thanks in advance for
> your help! I’ll take any suggestions on a possible mitigation too if anyone
> thinks of any.
>
>
> Blessings,
> Brian Call
> Manager, Systems Software Development
> Work: +1 (619) 373-4840 | Cell: +1 (619) 344-1013
>
>     Sotera Wireless, Inc.
> 10020 Huennekens Street San Diego, CA 92121, USA
> Phone: +1 (858) 427-4620 | Fax: +1 (858) 999-2487
>
> PRIVILEGED AND CONFIDENTIAL COMMUNICATION: This electronic transmission,
> and any documents attached hereto, may contain confidential and/or legally
> privileged information. The information is intended only for use by the
> recipient named above. If you have received this electronic message in
> error, please notify the sender and delete the electronic message. Any
> disclosure, copying, distribution, or use of the contents of information
> received in error is strictly prohibited.
>
>

Re: AlreadyClosedException on new index

[Brian Call]

Hi Tomoko,

Thank you for your response! We’ve actually never seen this before in the 
three+ years of developing using Lucene 3.6.x. The only time we’ve ever seen 
this kind of exception is once recently in a running production system and it 
caught me way off guard. We’re deploying on Suse linux (enterprise), and 
jdk1.7. 

Our application creates and deletes indices within the context of a single 
thread so I don’t think another thread is abruptly closing the index. All index 
open/close operations are always done sequentially in the context of one thread 
as operation requests are received.  

Blessings,
Brian 



> On Jan 6, 2015, at 2:16 AM, Tomoko Uchida  
> wrote:
> 
> Hi,
> 
> How often does this error occur?
> You do not tell the lucene version, but I guess you use lucene 3.x
> according to the stack trace...
> IndexWriter would not be closed until IndexWriter.close() method is called
> explicitly.
> https://github.com/apache/lucene-solr/blob/lucene_solr_3_6_2/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
>  
> 
> 
> Do you have custom codes wrapping lucene objects? There can be any codes
> which call IndexWriter.close() unexpectedly?
> 
> If your application seems to have no problem, you need to share more
> information including lucene version and system environments.
> 
> Regards,
> Tomoko
> 
> 
> 2015-01-06 8:32 GMT+09:00 Brian Call  >:
> 
>> Hi Guys,
>> 
>> So I’m seeing a problem in production that is very bizarre and I wanted to
>> see if anyone else has encountered this issue and/or has a suggestion or
>> fix. Here goes:
>> 
>> We create a wrapper around an index and searcher manager to encapsulate
>> both. First we create the IndexWriter and then immediately after create the
>> SearcherManager, like this:
>> 
>> indexWriter = new IndexWriter(indexDirectory, getWriterConfig());
>> searcherManager = new SearcherManager(indexWriter, true, new
>> ExecutorSearcherFactory());
>> 
>> 
>> During construction of the SearcherManager this is thrown:
>> 
>> Caused by: org.apache.lucene.store.AlreadyClosedException: this
>> IndexWriter is closed
>>at
>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:766)
>>at
>> org.apache.lucene.index.IndexWriter.getDirectory(IndexWriter.java:1909)
>>at
>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.getIndexDirectory(LuceneIndexMergeScheduler.java:162)
>>at
>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.access$000(LuceneIndexMergeScheduler.java:31)
>>at
>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler$MergeTask.equals(LuceneIndexMergeScheduler.java:127)
>>at
>> java.util.concurrent.ArrayBlockingQueue.contains(ArrayBlockingQueue.java:497)
>>at
>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.merge(LuceneIndexMergeScheduler.java:148)
>>at
>> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2740)
>>at
>> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2734)
>>at
>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:457)
>>at
>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
>>at org.apache.lucene.index.IndexReader.open(IndexReader.java:296)
>>at
>> org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
>>at
>> com.triagewireless.h1s.session.data.index.LuceneIndex.initNoCache(LuceneIndex.java:312)
>>at
>> com.triagewireless.h1s.session.data.index.LuceneIndex.init(LuceneIndex.java:270)
>>at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
>>at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>at java.lang.reflect.Method.invoke(Method.java:606)
>>at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:346)
>>at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:299)
>>at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:132)
>>... 57 more
>> 
>> 
>> Why? No other threads are acting on this index, etc. The index writer was
>> just created so how could it already be closed?
>> 
>> I’m completely baffled on this one guys… so many thanks in advance for
>> your help! I’ll take any suggestions on a possible mitigation too if anyone
>> thinks of any.
>> 
>> 
>> Blessings,
>> Brian Call
>> Manager, Systems Software Development
>> Work: +1 (619) 373-4840 | Cell: +1 (619) 344-

Re: AlreadyClosedException on new index

[Ian Lea]

Presumably no exception is thrown from the new IndexWriter() call?
 I'd double check that, and try some harmless method call on the
writer and make sure that works.  And run CheckIndex against the
index.


--
Ian.


On Tue, Jan 6, 2015 at 5:05 PM, Brian Call
 wrote:
> Hi Tomoko,
>
> Thank you for your response! We’ve actually never seen this before in the 
> three+ years of developing using Lucene 3.6.x. The only time we’ve ever seen 
> this kind of exception is once recently in a running production system and it 
> caught me way off guard. We’re deploying on Suse linux (enterprise), and 
> jdk1.7.
>
> Our application creates and deletes indices within the context of a single 
> thread so I don’t think another thread is abruptly closing the index. All 
> index open/close operations are always done sequentially in the context of 
> one thread as operation requests are received.
>
> Blessings,
> Brian
>
>
>
>> On Jan 6, 2015, at 2:16 AM, Tomoko Uchida  
>> wrote:
>>
>> Hi,
>>
>> How often does this error occur?
>> You do not tell the lucene version, but I guess you use lucene 3.x
>> according to the stack trace...
>> IndexWriter would not be closed until IndexWriter.close() method is called
>> explicitly.
>> https://github.com/apache/lucene-solr/blob/lucene_solr_3_6_2/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
>>  
>> 
>>
>> Do you have custom codes wrapping lucene objects? There can be any codes
>> which call IndexWriter.close() unexpectedly?
>>
>> If your application seems to have no problem, you need to share more
>> information including lucene version and system environments.
>>
>> Regards,
>> Tomoko
>>
>>
>> 2015-01-06 8:32 GMT+09:00 Brian Call > >:
>>
>>> Hi Guys,
>>>
>>> So I’m seeing a problem in production that is very bizarre and I wanted to
>>> see if anyone else has encountered this issue and/or has a suggestion or
>>> fix. Here goes:
>>>
>>> We create a wrapper around an index and searcher manager to encapsulate
>>> both. First we create the IndexWriter and then immediately after create the
>>> SearcherManager, like this:
>>>
>>> indexWriter = new IndexWriter(indexDirectory, getWriterConfig());
>>> searcherManager = new SearcherManager(indexWriter, true, new
>>> ExecutorSearcherFactory());
>>>
>>>
>>> During construction of the SearcherManager this is thrown:
>>>
>>> Caused by: org.apache.lucene.store.AlreadyClosedException: this
>>> IndexWriter is closed
>>>at
>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:766)
>>>at
>>> org.apache.lucene.index.IndexWriter.getDirectory(IndexWriter.java:1909)
>>>at
>>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.getIndexDirectory(LuceneIndexMergeScheduler.java:162)
>>>at
>>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.access$000(LuceneIndexMergeScheduler.java:31)
>>>at
>>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler$MergeTask.equals(LuceneIndexMergeScheduler.java:127)
>>>at
>>> java.util.concurrent.ArrayBlockingQueue.contains(ArrayBlockingQueue.java:497)
>>>at
>>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.merge(LuceneIndexMergeScheduler.java:148)
>>>at
>>> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2740)
>>>at
>>> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2734)
>>>at
>>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:457)
>>>at
>>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
>>>at org.apache.lucene.index.IndexReader.open(IndexReader.java:296)
>>>at
>>> org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
>>>at
>>> com.triagewireless.h1s.session.data.index.LuceneIndex.initNoCache(LuceneIndex.java:312)
>>>at
>>> com.triagewireless.h1s.session.data.index.LuceneIndex.init(LuceneIndex.java:270)
>>>at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
>>>at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>at java.lang.reflect.Method.invoke(Method.java:606)
>>>at
>>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:346)
>>>at
>>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:299)
>>>at
>>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:132)
>>>... 57 more
>>>
>>>
>>> Why? No other threads are acting on this index, etc. Th

Re: AlreadyClosedException on new index

[Brian Call]

No exception at all… and that’s the crazy part. I create a new IndexWriter and 
then immediately create a new SearcherManager using that freshly-created 
IndexWriter. Granted, I don’t actually do anything with the IndexWriter before 
passing it to the SearcherManager, but no exceptions are thrown either. 

-Brian


> On Jan 6, 2015, at 9:55 AM, Ian Lea  wrote:
> 
> Presumably no exception is thrown from the new IndexWriter() call?
> I'd double check that, and try some harmless method call on the
> writer and make sure that works.  And run CheckIndex against the
> index.
> 
> 
> --
> Ian.
> 
> 
> On Tue, Jan 6, 2015 at 5:05 PM, Brian Call
> mailto:brian.c...@soterawireless.com>> wrote:
>> Hi Tomoko,
>> 
>> Thank you for your response! We’ve actually never seen this before in the 
>> three+ years of developing using Lucene 3.6.x. The only time we’ve ever seen 
>> this kind of exception is once recently in a running production system and 
>> it caught me way off guard. We’re deploying on Suse linux (enterprise), and 
>> jdk1.7.
>> 
>> Our application creates and deletes indices within the context of a single 
>> thread so I don’t think another thread is abruptly closing the index. All 
>> index open/close operations are always done sequentially in the context of 
>> one thread as operation requests are received.
>> 
>> Blessings,
>> Brian
>> 
>> 
>> 
>>> On Jan 6, 2015, at 2:16 AM, Tomoko Uchida  
>>> wrote:
>>> 
>>> Hi,
>>> 
>>> How often does this error occur?
>>> You do not tell the lucene version, but I guess you use lucene 3.x
>>> according to the stack trace...
>>> IndexWriter would not be closed until IndexWriter.close() method is called
>>> explicitly.
>>> https://github.com/apache/lucene-solr/blob/lucene_solr_3_6_2/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
>>>  
>>> 
>>>  
>>> >>  
>>> >
>>> 
>>> Do you have custom codes wrapping lucene objects? There can be any codes
>>> which call IndexWriter.close() unexpectedly?
>>> 
>>> If your application seems to have no problem, you need to share more
>>> information including lucene version and system environments.
>>> 
>>> Regards,
>>> Tomoko
>>> 
>>> 
>>> 2015-01-06 8:32 GMT+09:00 Brian Call >>  
>>> >> >>:
>>> 
 Hi Guys,
 
 So I’m seeing a problem in production that is very bizarre and I wanted to
 see if anyone else has encountered this issue and/or has a suggestion or
 fix. Here goes:
 
 We create a wrapper around an index and searcher manager to encapsulate
 both. First we create the IndexWriter and then immediately after create the
 SearcherManager, like this:
 
 indexWriter = new IndexWriter(indexDirectory, getWriterConfig());
 searcherManager = new SearcherManager(indexWriter, true, new
 ExecutorSearcherFactory());
 
 
 During construction of the SearcherManager this is thrown:
 
 Caused by: org.apache.lucene.store.AlreadyClosedException: this
 IndexWriter is closed
   at
 org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:766)
   at
 org.apache.lucene.index.IndexWriter.getDirectory(IndexWriter.java:1909)
   at
 com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.getIndexDirectory(LuceneIndexMergeScheduler.java:162)
   at
 com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.access$000(LuceneIndexMergeScheduler.java:31)
   at
 com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler$MergeTask.equals(LuceneIndexMergeScheduler.java:127)
   at
 java.util.concurrent.ArrayBlockingQueue.contains(ArrayBlockingQueue.java:497)
   at
 com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.merge(LuceneIndexMergeScheduler.java:148)
   at
 org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2740)
   at
 org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2734)
   at
 org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:457)
   at
 org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
   at org.apache.lucene.index.IndexReader.open(IndexReader.java:296)
   at
 org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
   at
 com.triagewireless.h1s.session.data.index.LuceneIndex.initNoCache(LuceneIndex.java:312)
   at
 com.triagewireless.h1s.session.data.index.LuceneIndex.init(LuceneIndex.java

Re: AlreadyClosedException on new index

[Brian Call]

Hi Guys,

So I think I’ve figured it out… We use a common merge scheduler to handle 
merges across all of our indices. When the SearchManager starts up, it triggers 
a merge operation, this merge request pushes a task on the merge scheduling 
queue and we only allow one merge request per index writer in the queue at a 
time. What appears to be happening is that a closed index is being left in the 
merge queue and the call is failing due to the .equals() method on the closed 
index when inserting into our BlockingArrayQueue. Usually this shouldn’t happen 
and appears to be a timing issue combined with not protecting against this 
possibility. 

Thanks for your assistance guys!

-Brian  


> On Jan 6, 2015, at 10:22 AM, Brian Call  wrote:
> 
> No exception at all… and that’s the crazy part. I create a new IndexWriter 
> and then immediately create a new SearcherManager using that freshly-created 
> IndexWriter. Granted, I don’t actually do anything with the IndexWriter 
> before passing it to the SearcherManager, but no exceptions are thrown 
> either. 
> 
> -Brian
> 
> 
>> On Jan 6, 2015, at 9:55 AM, Ian Lea > > wrote:
>> 
>> Presumably no exception is thrown from the new IndexWriter() call?
>> I'd double check that, and try some harmless method call on the
>> writer and make sure that works.  And run CheckIndex against the
>> index.
>> 
>> 
>> --
>> Ian.
>> 
>> 
>> On Tue, Jan 6, 2015 at 5:05 PM, Brian Call
>> mailto:brian.c...@soterawireless.com>> wrote:
>>> Hi Tomoko,
>>> 
>>> Thank you for your response! We’ve actually never seen this before in the 
>>> three+ years of developing using Lucene 3.6.x. The only time we’ve ever 
>>> seen this kind of exception is once recently in a running production system 
>>> and it caught me way off guard. We’re deploying on Suse linux (enterprise), 
>>> and jdk1.7.
>>> 
>>> Our application creates and deletes indices within the context of a single 
>>> thread so I don’t think another thread is abruptly closing the index. All 
>>> index open/close operations are always done sequentially in the context of 
>>> one thread as operation requests are received.
>>> 
>>> Blessings,
>>> Brian
>>> 
>>> 
>>> 
 On Jan 6, 2015, at 2:16 AM, Tomoko Uchida >>> > wrote:
 
 Hi,
 
 How often does this error occur?
 You do not tell the lucene version, but I guess you use lucene 3.x
 according to the stack trace...
 IndexWriter would not be closed until IndexWriter.close() method is called
 explicitly.
 https://github.com/apache/lucene-solr/blob/lucene_solr_3_6_2/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
  
 
  
 >
 
 Do you have custom codes wrapping lucene objects? There can be any codes
 which call IndexWriter.close() unexpectedly?
 
 If your application seems to have no problem, you need to share more
 information including lucene version and system environments.
 
 Regards,
 Tomoko
 
 
 2015-01-06 8:32 GMT+09:00 Brian Call >>>  
 >>:
 
> Hi Guys,
> 
> So I’m seeing a problem in production that is very bizarre and I wanted to
> see if anyone else has encountered this issue and/or has a suggestion or
> fix. Here goes:
> 
> We create a wrapper around an index and searcher manager to encapsulate
> both. First we create the IndexWriter and then immediately after create 
> the
> SearcherManager, like this:
> 
> indexWriter = new IndexWriter(indexDirectory, getWriterConfig());
> searcherManager = new SearcherManager(indexWriter, true, new
> ExecutorSearcherFactory());
> 
> 
> During construction of the SearcherManager this is thrown:
> 
> Caused by: org.apache.lucene.store.AlreadyClosedException: this
> IndexWriter is closed
>   at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:766)
>   at
> org.apache.lucene.index.IndexWriter.getDirectory(IndexWriter.java:1909)
>   at
> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.getIndexDirectory(LuceneIndexMergeScheduler.java:162)
>   at
> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.access$000(LuceneIndexMergeScheduler.java:31)
>   at
> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler$MergeTask.equals(LuceneIndexMergeScheduler.java:127)
>