Hi Michael, Thanks for the quick response. I only have one IndexWriter, and there are no other processes accessing this particular index. I have tried deleting the entire index and reconstructing it, but the index corruption is repeatable. Incidentally, there are no new writes since the last commit when the merge happens. I have over-padded my code with ReadWrite locks to make sure that no writes/read are happening between the commits, optimizations, and reopening of the index.
Here is a snippet of the thread I use to maintain the Index ( I hope that I am not doing something terribly wrong): while (true) { try { getWriteLock(); indexWriter.commit(); if (shouldOptimize()) { indexWriter.optimize(); } IndexReader oldIR = indexSearcher.getIndexReader(); IndexReader ir = oldIR.reopen(); if (ir != oldIR) { IndexSearcher oldIS = indexSearcher; indexSearcher = new IndexSearcher(ir); oldIS.close(); oldIR.close(); } catch (Throwable t) { trace.error(t, t); } finally { releaseWriteLock(); } Regards, Khawaja On Thu, Apr 9, 2009 at 5:05 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > These are serious corruption exceptions. > > Is it at all possible two writers are accessing the index at the same time? > > Can you describe more about how you're using Lucene? > > Mike > > On Thu, Apr 9, 2009 at 7:59 PM, Khawaja Shams <kssh...@gmail.com> wrote: > > Hello, > > I am having a problem with reopening the IndexReader with Lucene 2.4 ( I > > updated to 2.4.1, but still no luck). The exception is preceded by an > > exception in optimizing the index. I am not reopening the reader while > the > > commit or optimization is going on in the writer (optimizing happens in > the > > same thread, but much less often). The issues go away once I turn off > > optimizations. I was also getting this problem before I turned off the > use > > of compound files. I would appreciate any guidance. > > > > Thanks! > > > > Regards, > > Khawaja > > > > > > 2009-04-09 15:57:47,033 (941820) [Index Maint Thread] ERROR > > gov.nasa.ensemble.core.indexer.Indexer - java.io.IOException: background > > merge hit exception: _8:C41258 _9:C11382 into _a [optimize] > > java.io.IOException: background merge hit exception: _8:C41258 _9:C11382 > > into _a [optimize] > > at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273) > > at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218) > > at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198) > > at > > > gov.nasa.ensemble.core.indexer.Indexer$IndexMaintThread.run(Indexer.java:102) > > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts > differ > > for segment _8: fieldsReader shows 30074 but segmentInfo shows 41258 > > at > > org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:362) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260) > > at > > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220) > > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884) > > at > > > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205) > > at > > > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260) > > Exception in thread "Lucene Merge Thread #0" > > org.apache.lucene.index.MergePolicy$MergeException: > > org.apache.lucene.index.CorruptIndexException: doc counts differ for > segment > > _8: fieldsReader shows 30074 but segmentInfo shows 41258 > > at > > > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309) > > at > > > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286) > > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts > differ > > for segment _8: fieldsReader shows 30074 but segmentInfo shows 41258 > > at > > org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:362) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260) > > at > > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220) > > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884) > > at > > > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205) > > at > > > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260) > > 2009-04-09 15:57:52,814 (947601) [Index Maint Thread] ERROR > > gov.nasa.ensemble.core.indexer.Indexer - java.io.FileNotFoundException: > > /opt/users/merops/index/_a.fnm (No such file or directory) > > java.io.FileNotFoundException: /opt/users/merops/index/_a.fnm (No such > file > > or directory) > > at java.io.RandomAccessFile.open(Native Method) > > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212) > > at > > > org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552) > > at > > > org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582) > > at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488) > > at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:482) > > at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:58) > > at > > org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:341) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:269) > > at > > > org.apache.lucene.index.MultiSegmentReader.doReopen(MultiSegmentReader.java:201) > > at > > > org.apache.lucene.index.DirectoryIndexReader$2.doBody(DirectoryIndexReader.java:157) > > at > > > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653) > > at > > > org.apache.lucene.index.DirectoryIndexReader.reopen(DirectoryIndexReader.java:179) > > at > > > gov.nasa.ensemble.core.indexer.Indexer$IndexMaintThread.run(Indexer.java:106) > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >