OK. There seems to be something going on within our system configuration. I can't run ulimit from within my home shell but when I run a bash shell, I can run ulimit and get the 'unlimited' result.
If I run Solr from within the bash shell, the error disappears. So, this looks like a configuration issue concerned with how my home shell is set up. I think this works fine now. Thank you for the help! Gully On Mon, Sep 10, 2012 at 11:16 AM, Gully Burns <gullybu...@gmail.com> wrote: > Thanks for the response. > > ulimit -f provides the value 'unlimited' > > Not sure what could be going on. > > Gully > > > On Mon, Sep 10, 2012 at 4:13 AM, Michael McCandless < > luc...@mikemccandless.com> wrote: > >> Likely you have a limit set on maximum file size? >> >> Try running "ulimit -f" to see the limit? >> >> Mike McCandless >> >> http://blog.mikemccandless.com >> >> On Sun, Sep 9, 2012 at 9:03 PM, Gully Burns <gullybu...@gmail.com> wrote: >> > Hey Everyone, >> > >> > I'm building a solr store on version 3.6.1 and I encounter the following >> > error when the system gets to about 1,000,000 documents. >> > >> > >> > >> commit{dir=/lfs1/code/apache-solr-3.6.1/vpdmfSolr/digitalLibrary/articlecitation/data/index,segFN=segments_3jki,version=1345856472880,generation=165330,filenames=[.... >> > LONG LIST OF INDEX FILES .... ] >> > Sep 8, 2012 9:30:11 PM org.apache.solr.core.SolrDeletionPolicy >> updateCommits >> > INFO: newest commit = 1345856472880 >> > Sep 8, 2012 9:30:11 PM >> org.apache.solr.update.processor.LogUpdateProcessor >> > finish >> > INFO: {add=[8343524, 8343525, 8343526, 8343527, 8343528, 8343529, >> 8343530, >> > 8343531, ... (100 adds)]} 0 60 >> > Sep 8, 2012 9:30:11 PM org.apache.solr.core.SolrCore execute >> > INFO: [ArticleCitation] webapp=/solr path=/update >> > params={wt=xml&version=2.2} status=0 QTime=60 >> > Sep 8, 2012 9:30:11 PM org.apache.solr.update.DirectUpdateHandler2 >> commit >> > INFO: start >> > >> commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false) >> > >> > Exception in thread "Lucene Merge Thread #0" >> > org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: >> > File too large >> > at >> > >> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:509) >> > at >> > >> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482) >> > Caused by: java.io.IOException: File too large >> > at java.io.RandomAccessFile.writeBytes(Native Method) >> > at java.io.RandomAccessFile.write(RandomAccessFile.java:499) >> > at >> > >> org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:448) >> > at >> > >> org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:99) >> > at >> > >> org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:88) >> > at >> > >> org.apache.lucene.store.BufferedIndexOutput.close(BufferedIndexOutput.java:113) >> > at >> > >> org.apache.lucene.store.FSDirectory$FSIndexOutput.close(FSDirectory.java:458) >> > at org.apache.lucene.util.IOUtils.close(IOUtils.java:141) >> > at >> org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:139) >> > at >> > >> org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:232) >> > at >> > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:107) >> > at >> > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4263) >> > at >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908) >> > at >> > >> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388) >> > at >> > >> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456) >> > Sep 8, 2012 9:30:12 PM org.apache.solr.core.SolrDeletionPolicy onCommit >> > INFO: SolrDeletionPolicy.onCommit: commits:num=2 >> > >> > My solrconfig.xml file has the following indexing parameters: >> > >> > <indexConfig> >> > <maxFieldLength>10000</maxFieldLength> >> > <writeLockTimeout>1000</writeLockTimeout> >> > <ramBufferSizeMB>2048</ramBufferSizeMB> >> > <useCompoundFile>false</useCompoundFile> >> > <mergeFactor>25</mergeFactor> >> > </indexConfig> >> > >> > There are 451 separate index files, the largest of which is 60GB. >> > >> > The system is redhat linux simply running the jetty version of the solr >> > server. >> > >> > Note that the system seems to work fine, even with the error. The >> number of >> > files in the index seems very large. >> > >> > Having these exceptions worries me, but the system seems to operate fine >> > even then. >> > >> > Any help would be very appreciated. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> >
