Hi, NFS is not a filesystem that works reliable and correctly with Lucene. Please consider using another file system, preferably on local disks or (if it needs to be networked) using iSCSI or similar. MMap crushes the whole JVM easily if NFS connections drop. NIOFSDir works better, but the whole issue are those stale file handles, which makes commits in Lucene unreliable.
Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Shlomit Rosen [mailto:shlom...@il.ibm.com] > Sent: Monday, September 08, 2014 11:09 AM > To: java-user@lucene.apache.org > Subject: IOExceptions during search > > Hello :) > > We have a customer who's system keeps crashing at certain queries. > At default we use Mmap directory for search as it usually gives us the best > performance. > Although the heap usage seemed normal, we asked them to switch to > NIOFSDirectory to make sure it's not a memory issue... > The system stopped crashing, but now they see the following errors occur in > some of the searches: > > java.io.IOException: Input/output error > sun.nio.ch.FileDispatcher.pread0(Native Method) > sun.nio.ch.FileDispatcher.pread(FileDispatcher.java:43) > sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:207) > sun.nio.ch.IOUtil.read(IOUtil.java:183) > sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:642) > > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIO > FSDirectory.java:162) > > org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput > .java:156) > > org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal( > CompoundFileReader.java:275) > > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java: > 270) > > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput. > java:40) > org.apache.lucene.store.DataInput.readVInt(DataInput.java:107) > > org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.j > ava:217) > > org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:1 > 21) > > org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPosition > s.java:103) > org.apache.lucene.search.spans.TermSpans.next(TermSpans.java:49) > > AND - > > > java.io.IOException: Stale NFS file handle > sun.nio.ch.FileDispatcher.pread0(Native Method) > sun.nio.ch.FileDispatcher.pread(FileDispatcher.java:43) > sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:207) > sun.nio.ch.IOUtil.read(IOUtil.java:183) > sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:642) > > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIO > FSDirectory.java:162) > > org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput > .java:156) > > org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal( > CompoundFileReader.java:275) > > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java: > 270) > > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput. > java:40) > org.apache.lucene.store.DataInput.readVInt(DataInput.java:107) > > org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.j > ava:217) > org.apache.lucene.index.TermBuffer.read(TermBuffer.java:64) > > org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java: > 133) > > org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.ja > va:174) > org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:236) > > org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:304 > ) > > org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:473) > > org.apache.lucene.index.DirectoryReader$MultiTermEnum.<init>(Direc > toryReader.java:1086) > org.apache.lucene.index.DirectoryReader.terms(DirectoryReader.java:698) > > org.apache.lucene.search.TermRangeTermEnum.<init>(TermRangeTer > mEnum.java:93) > > org.apache.lucene.search.TermRangeQuery.getEnum(TermRangeQuery.java > :135) > > > The customer is running on Linux 64, OS version: 2.6.18-371.8.1.el5 I saw in > different forums that this might have to do with the lock factory, but we are > using SimpleFSLockFactory.... > Currently they are running on lucene 3.6.0 but previously they were on > lucene 3.0.2/ lucene 3.0.3, so the collections they are searching might have > been created with an older version. > > We have asked them to run checkIndex to rule out corruption, but we > haven't heard back yet. > > Any idea what might be wrong here, or how can we help them? > > Thanks in advance ! > Shlomit --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
