Hello again, A little background of our experiment. We are storing lucene (version 4.6.0) on top of cassandra. We are using the demo IndexFiles.java from the lucene with minor modification such that the directory used is reference to the CassandraDirectory.
With large dataset (that is, index more than 50000 of files), after index is done, and set forceMerge(1) and get the following exception. BufferedIndexInput readBytes [ERROR] bufferStart = '0' bufferPosition = '1024' len = '9252' after = '10276' BufferedIndexInput readBytes [ERROR] length = '8192' caught a class java.io.IOException with message: background merge hit exception: _1(4.6):c10250 _0(4.6):c10355 _2(4.6):c10297 _3(4.6):c10217 _4(4.6):c8882 into _5 [maxNumSegments=1] java.io.IOException: background merge hit exception: _1(4.6):c10250 _0(4.6):c10355 _2(4.6):c10297 _3(4.6):c10217 _4(4.6):c8882 into _5 [maxNumSegments=1] at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1755) at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1691) at org.apache.lucene.store.IndexFiles.main(IndexFiles.java:159) Caused by: java.io.IOException: read past EOF: CassandraSimpleFSIndexInput(_1.nvd in path="_1.cfs" slice=5557885:5566077) at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:186) at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:125) at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:230) at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:186) at org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:159) at org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:516) at org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:232) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:127) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4057) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3654) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482) We do not know what is wrong as our understanding on lucene is limited. Can someone give explanation on what is happening, or which might be the possible error source is? Thank you and any advice is appreciated. /Jason