Mike,
I am sorry for wasting your time :). There were indeed two threads that
were performing this operation. Out of curiosity, which part of this is not
thread safe? An indexreader reopening while a commit is going on? Thanks
again for your help.
Regards,
Khawaja
On Thu, Apr 9, 2009 at 5:44 PM, Michael McCandless <
luc...@mikemccandless.com> wrote:
> That code looks right. Are there multiple threads that may enter it?
>
> Can you show the code where you create the IndexWriter, add docs, etc?
>
> Can you call IndexWriter.setInfoStream for the entire life of the
> index, up until when the optimize error happens, and post back?
>
> Mike
>
> On Thu, Apr 9, 2009 at 8:33 PM, Khawaja Shams <kssh...@gmail.com> wrote:
> > Hi Michael,
> > Thanks for the quick response. I only have one IndexWriter, and there
> are
> > no other processes accessing this particular index. I have tried deleting
> > the entire index and reconstructing it, but the index corruption is
> > repeatable. Incidentally, there are no new writes since the last commit
> when
> > the merge happens. I have over-padded my code with ReadWrite locks to
> make
> > sure that no writes/read are happening between the commits,
> optimizations,
> > and reopening of the index.
> >
> >
> > Here is a snippet of the thread I use to maintain the Index ( I hope that
> I
> > am not doing something terribly wrong):
> > while (true) {
> > try {
> > getWriteLock();
> > indexWriter.commit();
> > if (shouldOptimize()) {
> > indexWriter.optimize();
> > }
> >
> > IndexReader oldIR = indexSearcher.getIndexReader();
> > IndexReader ir = oldIR.reopen();
> > if (ir != oldIR) {
> > IndexSearcher oldIS = indexSearcher;
> > indexSearcher = new IndexSearcher(ir);
> > oldIS.close();
> > oldIR.close();
> > } catch (Throwable t) {
> > trace.error(t, t);
> > } finally {
> > releaseWriteLock();
> > }
> >
> >
> > Regards,
> > Khawaja
> >
> > On Thu, Apr 9, 2009 at 5:05 PM, Michael McCandless <
> > luc...@mikemccandless.com> wrote:
> >
> >> These are serious corruption exceptions.
> >>
> >> Is it at all possible two writers are accessing the index at the same
> time?
> >>
> >> Can you describe more about how you're using Lucene?
> >>
> >> Mike
> >>
> >> On Thu, Apr 9, 2009 at 7:59 PM, Khawaja Shams <kssh...@gmail.com>
> wrote:
> >> > Hello,
> >> > I am having a problem with reopening the IndexReader with Lucene 2.4
> ( I
> >> > updated to 2.4.1, but still no luck). The exception is preceded by an
> >> > exception in optimizing the index. I am not reopening the reader while
> >> the
> >> > commit or optimization is going on in the writer (optimizing happens
> in
> >> the
> >> > same thread, but much less often). The issues go away once I turn off
> >> > optimizations. I was also getting this problem before I turned off the
> >> use
> >> > of compound files. I would appreciate any guidance.
> >> >
> >> > Thanks!
> >> >
> >> > Regards,
> >> > Khawaja
> >> >
> >> >
> >> > 2009-04-09 15:57:47,033 (941820) [Index Maint Thread] ERROR
> >> > gov.nasa.ensemble.core.indexer.Indexer - java.io.IOException:
> background
> >> > merge hit exception: _8:C41258 _9:C11382 into _a [optimize]
> >> > java.io.IOException: background merge hit exception: _8:C41258
> _9:C11382
> >> > into _a [optimize]
> >> > at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
> >> > at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
> >> > at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
> >> > at
> >> >
> >>
> gov.nasa.ensemble.core.indexer.Indexer$IndexMaintThread.run(Indexer.java:102)
> >> > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
> >> differ
> >> > for segment _8: fieldsReader shows 30074 but segmentInfo shows 41258
> >> > at
> >> >
> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:362)
> >> > at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
> >> > at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
> >> > at
> >> > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
> >> > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
> >> > at
> >> >
> >>
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
> >> > at
> >> >
> >>
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
> >> > Exception in thread "Lucene Merge Thread #0"
> >> > org.apache.lucene.index.MergePolicy$MergeException:
> >> > org.apache.lucene.index.CorruptIndexException: doc counts differ for
> >> segment
> >> > _8: fieldsReader shows 30074 but segmentInfo shows 41258
> >> > at
> >> >
> >>
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
> >> > at
> >> >
> >>
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
> >> > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
> >> differ
> >> > for segment _8: fieldsReader shows 30074 but segmentInfo shows 41258
> >> > at
> >> >
> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:362)
> >> > at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
> >> > at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
> >> > at
> >> > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
> >> > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
> >> > at
> >> >
> >>
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
> >> > at
> >> >
> >>
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
> >> > 2009-04-09 15:57:52,814 (947601) [Index Maint Thread] ERROR
> >> > gov.nasa.ensemble.core.indexer.Indexer -
> java.io.FileNotFoundException:
> >> > /opt/users/merops/index/_a.fnm (No such file or directory)
> >> > java.io.FileNotFoundException: /opt/users/merops/index/_a.fnm (No such
> >> file
> >> > or directory)
> >> > at java.io.RandomAccessFile.open(Native Method)
> >> > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> >> > at
> >> >
> >>
> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
> >> > at
> >> >
> >>
> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
> >> > at
> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
> >> > at
> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:482)
> >> > at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:58)
> >> > at
> >> >
> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:341)
> >> > at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
> >> > at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:269)
> >> > at
> >> >
> >>
> org.apache.lucene.index.MultiSegmentReader.doReopen(MultiSegmentReader.java:201)
> >> > at
> >> >
> >>
> org.apache.lucene.index.DirectoryIndexReader$2.doBody(DirectoryIndexReader.java:157)
> >> > at
> >> >
> >>
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653)
> >> > at
> >> >
> >>
> org.apache.lucene.index.DirectoryIndexReader.reopen(DirectoryIndexReader.java:179)
> >> > at
> >> >
> >>
> gov.nasa.ensemble.core.indexer.Indexer$IndexMaintThread.run(Indexer.java:106)
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> >> For additional commands, e-mail: java-user-h...@lucene.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>
