Hi Michael,
  Thanks for the quick response. I only have one IndexWriter, and there are
no other processes accessing this particular index. I have tried deleting
the entire index and reconstructing it, but the index corruption is
repeatable. Incidentally, there are no new writes since the last commit when
the merge happens. I have over-padded my code with ReadWrite locks to make
sure that no writes/read are happening between the commits, optimizations,
and reopening of the index.


Here is a snippet of the thread I use to maintain the Index ( I hope that I
am not doing something terribly wrong):
            while (true) {
                try {
                    getWriteLock();
                    indexWriter.commit();
                   if (shouldOptimize()) {
                       indexWriter.optimize();
                    }

                    IndexReader oldIR = indexSearcher.getIndexReader();
                    IndexReader ir = oldIR.reopen();
                    if (ir != oldIR) {
                        IndexSearcher oldIS = indexSearcher;
                        indexSearcher = new IndexSearcher(ir);
                        oldIS.close();
                        oldIR.close();
                } catch (Throwable t) {
                    trace.error(t, t);
                } finally {
                    releaseWriteLock();
                }


Regards,
Khawaja

On Thu, Apr 9, 2009 at 5:05 PM, Michael McCandless <
luc...@mikemccandless.com> wrote:

> These are serious corruption exceptions.
>
> Is it at all possible two writers are accessing the index at the same time?
>
> Can you describe more about how you're using Lucene?
>
> Mike
>
> On Thu, Apr 9, 2009 at 7:59 PM, Khawaja Shams <kssh...@gmail.com> wrote:
> > Hello,
> >  I am having a problem with reopening the IndexReader with Lucene 2.4 ( I
> > updated to 2.4.1, but still no luck). The exception is preceded by an
> > exception in optimizing the index. I am not reopening the reader while
> the
> > commit or optimization is going on in the writer (optimizing happens in
> the
> > same thread, but much less often). The issues go away once I turn off
> > optimizations. I was also getting this problem before I turned off the
> use
> > of compound files. I would appreciate any guidance.
> >
> > Thanks!
> >
> > Regards,
> > Khawaja
> >
> >
> > 2009-04-09 15:57:47,033 (941820) [Index Maint Thread] ERROR
> > gov.nasa.ensemble.core.indexer.Indexer  - java.io.IOException: background
> > merge hit exception: _8:C41258 _9:C11382 into _a [optimize]
> > java.io.IOException: background merge hit exception: _8:C41258 _9:C11382
> > into _a [optimize]
> >    at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
> >    at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
> >    at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
> >    at
> >
> gov.nasa.ensemble.core.indexer.Indexer$IndexMaintThread.run(Indexer.java:102)
> > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
> differ
> > for segment _8: fieldsReader shows 30074 but segmentInfo shows 41258
> >    at
> > org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:362)
> >    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
> >    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
> >    at
> > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
> >    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
> >    at
> >
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
> >    at
> >
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
> > Exception in thread "Lucene Merge Thread #0"
> > org.apache.lucene.index.MergePolicy$MergeException:
> > org.apache.lucene.index.CorruptIndexException: doc counts differ for
> segment
> > _8: fieldsReader shows 30074 but segmentInfo shows 41258
> >    at
> >
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
> >    at
> >
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
> > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
> differ
> > for segment _8: fieldsReader shows 30074 but segmentInfo shows 41258
> >    at
> > org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:362)
> >    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
> >    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
> >    at
> > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
> >    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
> >    at
> >
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
> >    at
> >
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
> > 2009-04-09 15:57:52,814 (947601) [Index Maint Thread] ERROR
> > gov.nasa.ensemble.core.indexer.Indexer  - java.io.FileNotFoundException:
> > /opt/users/merops/index/_a.fnm (No such file or directory)
> > java.io.FileNotFoundException: /opt/users/merops/index/_a.fnm (No such
> file
> > or directory)
> >    at java.io.RandomAccessFile.open(Native Method)
> >    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> >    at
> >
> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
> >    at
> >
> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
> >    at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
> >    at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:482)
> >    at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:58)
> >    at
> > org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:341)
> >    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
> >    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:269)
> >    at
> >
> org.apache.lucene.index.MultiSegmentReader.doReopen(MultiSegmentReader.java:201)
> >    at
> >
> org.apache.lucene.index.DirectoryIndexReader$2.doBody(DirectoryIndexReader.java:157)
> >    at
> >
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653)
> >    at
> >
> org.apache.lucene.index.DirectoryIndexReader.reopen(DirectoryIndexReader.java:179)
> >    at
> >
> gov.nasa.ensemble.core.indexer.Indexer$IndexMaintThread.run(Indexer.java:106)
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>

Reply via email to