unable to start solr instance

[Niklas Langvig]

Hello,
We havet o solr instances running on linux/tomcat7
Both have been working fine, now only 1 works. The other seems to have crashed 
or something.

SolrCore Initialization Failures
*         collection1: 
org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: 
Error initializing QueryElevationComponent.

We havn't changed anything in the setup.

Earlier 4 days ago I could see in the logs
<response>
<lst name="responseHeader"><int name="status">500</int><int 
name="QTime">0</int></lst><lst name="error"><str 
name="msg">java.io.FileNotFoundException: 
/opt/solr410/document/collection1/data/tlog/tlog.0000000000002494137 (Too many 
open files)</str><str name="trace">org.apache.solr.common.SolrException: 
java.io.FileNotFoundException: 
/opt/solr410/document/collection1/data/tlog/tlog.0000000000002494137 (Too many 
open files)
                             at 
org.apache.solr.update.TransactionLog.&lt;init&gt;(TransactionLog.java:182)
                             at 
org.apache.solr.update.TransactionLog.&lt;init&gt;(TransactionLog.java:140)
                             at 
org.apache.solr.update.UpdateLog.ensureLog(UpdateLog.java:796)
                             at 
org.apache.solr.update.UpdateLog.delete(UpdateLog.java:409)
                             at 
org.apache.solr.update.DirectUpdateHandler2.delete(DirectUpdateHandler2.java:284)
                             at 
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:77)
                             at 
org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
                             at 
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:460)
                             at 
org.apache.solr.update.processor.DistributedUpdateProcessor.versionDelete(DistributedUpdateProcessor.java:1036)
                             at 
org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:721)
                             at 
org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
                             at 
org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
                             at 
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
                             at 
org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
                             at 
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
                             at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
                             at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
                             at 
org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
                             at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448)
                             at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
                             at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
                             at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
                             at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
                             at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
                             at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
                             at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
                             at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
                             at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
                             at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
                             at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
                             at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
                             at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.FileNotFoundException: 
/opt/solr410/document/collection1/data/tlog/tlog.0000000000002494137 (Too many 
open files)
                             at java.io.RandomAccessFile.open(Native Method)
                             at 
java.io.RandomAccessFile.&lt;init&gt;(RandomAccessFile.java:233)
                             at 
org.apache.solr.update.TransactionLog.&lt;init&gt;(TransactionLog.java:151)
                             ... 31 more
</str><int name="code">500</int></lst>
*         </response>

we tried to restart solr and now we get the error collection1: 
org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: 
Error initializing QueryElevationComponent.

We have been using solr to index word/pdf documents and we have had around 10 
000 000 documents, is that a problem? We have added a new document if not every 
second but very often, is it better to load a bunch of documents every hour for 
example than 1 document alone?

Does anyone have an idea what we should try to get this instance started again 
or should we remove the data and re-read everything?

thanks for any input

/Niklas