Thanks Mike.
I think it is something to do with the merge factor.
Modified the code to do optimize in the finally block the following error
message was thrown.
Code Snippet:
nameWriter.optimize(); // errors here
nameWriter.close();
valueWriter.optimize(); //I am using multiple writers
simultaneously. Writing to different directory.
valueWriter.close()
Index Directory Files:
_1o.cfs
_1p.cfs
_1q.cfs
_1r.cfs
_1s.cfs
_1t.fnm
segments_1
segments.gen
write.lock
Exception in thread "main" java.io.IOException: background merge hit
exception: _1o:c55 _1p:c1 _1q:c1 _1r:c1 _1s:c1 into _1t [optimize]
[mergeDocStores]
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2854)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2789)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2759)
at
com.hli.vis.index.VisualSoftIndexWriter.commitChanges(VSIndexWriter.java:46)
at com.hli.vis.index.Main.main(Main.java:39)
Caused by: java.io.FileNotFoundException:
/Users/Kumar/Dev/Cust/index/names/_1t.fdx (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:186)
at
org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:76)
at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:82)
at
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:334)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:153)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5029)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
Java Result: 1
Kumar _/|\_
www.saisk.com
ku...@saisk.com
"making a profound difference with knowledge and creativity..."
On Thu, Dec 31, 2009 at 6:45 AM, Michael McCandless <
luc...@mikemccandless.com> wrote:
> It sounds like you may be running out of file descriptors -- how many
> segments are in your index?
>
> The reopen logic looks correct (you are closing the old reader). Is
> there anything else that may be holding files open?
>
> Have you changed any of IW's settings, eg mergeFactor?
>
> Mike
>
> On Wed, Dec 30, 2009 at 3:21 PM, Kumaravel Kandasami
> <kumaravel.kandas...@gmail.com> wrote:
> > I am getting IOException when I am doing a "Real-time" search, i.e. I am
> > creating a Index using the Index Writer and also opening the Index using
> > Index Reader (writer.getReader()) to make sure the document does not
> exist
> > prior adding to the Index file.
> >
> > The code works perfect fine multiple time indexed across couple of GB
> data,
> > however, on a TB it fails with the exception listed below.
> > Any suggestions appreciated. Thanks. Kumar
> >
> > Code Snippet:
> >
> > IndexReader reader2 = reader.reopen();//get
> >
> > if (reader2 != reader) {
> >
> > reader.close();
> > reader = reader2;
> > searcher = new IndexSearcher(reader);
> > reader2 = null;
> > }
> >
> >
> > Exception:
> >
> > Exception in thread "main" java.io.IOException: directory
> > '/Users/Kumar/Dev/Cust/index/values' exists and is a directory, but
> cannot
> > be listed: list() returned null
> > at
> org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516)
> > at
> org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531)
> > at
> >
> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307)
> > at
> >
> org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300)
> > at
> > org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
> > at
> org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
> > at
> > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:470)
> > at
> > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:444)
> > at
> >
> org.apache.lucene.index.DirectoryReader.doReopen(DirectoryReader.java:384)
> > at
> > org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader.java:348)
> >
> > Kumar _/|\_
> > www.saisk.com
> > ku...@saisk.com
> > "making a profound difference with knowledge and creativity..."
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>
