Likely you have a limit set on maximum file size? Try running "ulimit -f" to see the limit?
Mike McCandless http://blog.mikemccandless.com On Sun, Sep 9, 2012 at 9:03 PM, Gully Burns <gullybu...@gmail.com> wrote: > Hey Everyone, > > I'm building a solr store on version 3.6.1 and I encounter the following > error when the system gets to about 1,000,000 documents. > > > commit{dir=/lfs1/code/apache-solr-3.6.1/vpdmfSolr/digitalLibrary/articlecitation/data/index,segFN=segments_3jki,version=1345856472880,generation=165330,filenames=[.... > LONG LIST OF INDEX FILES .... ] > Sep 8, 2012 9:30:11 PM org.apache.solr.core.SolrDeletionPolicy updateCommits > INFO: newest commit = 1345856472880 > Sep 8, 2012 9:30:11 PM org.apache.solr.update.processor.LogUpdateProcessor > finish > INFO: {add=[8343524, 8343525, 8343526, 8343527, 8343528, 8343529, 8343530, > 8343531, ... (100 adds)]} 0 60 > Sep 8, 2012 9:30:11 PM org.apache.solr.core.SolrCore execute > INFO: [ArticleCitation] webapp=/solr path=/update > params={wt=xml&version=2.2} status=0 QTime=60 > Sep 8, 2012 9:30:11 PM org.apache.solr.update.DirectUpdateHandler2 commit > INFO: start > commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false) > > Exception in thread "Lucene Merge Thread #0" > org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: > File too large > at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:509) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482) > Caused by: java.io.IOException: File too large > at java.io.RandomAccessFile.writeBytes(Native Method) > at java.io.RandomAccessFile.write(RandomAccessFile.java:499) > at > org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:448) > at > org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:99) > at > org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:88) > at > org.apache.lucene.store.BufferedIndexOutput.close(BufferedIndexOutput.java:113) > at > org.apache.lucene.store.FSDirectory$FSIndexOutput.close(FSDirectory.java:458) > at org.apache.lucene.util.IOUtils.close(IOUtils.java:141) > at org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:139) > at > org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:232) > at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:107) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4263) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908) > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456) > Sep 8, 2012 9:30:12 PM org.apache.solr.core.SolrDeletionPolicy onCommit > INFO: SolrDeletionPolicy.onCommit: commits:num=2 > > My solrconfig.xml file has the following indexing parameters: > > <indexConfig> > <maxFieldLength>10000</maxFieldLength> > <writeLockTimeout>1000</writeLockTimeout> > <ramBufferSizeMB>2048</ramBufferSizeMB> > <useCompoundFile>false</useCompoundFile> > <mergeFactor>25</mergeFactor> > </indexConfig> > > There are 451 separate index files, the largest of which is 60GB. > > The system is redhat linux simply running the jetty version of the solr > server. > > Note that the system seems to work fine, even with the error. The number of > files in the index seems very large. > > Having these exceptions worries me, but the system seems to operate fine > even then. > > Any help would be very appreciated. --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
