Hi, sporadically when using distributed searches on a single box (18 cores) using Solr/Lucene 3.1.0 via jetty I am getting a 'java.net.SocketException: Connection reset' exceptions (full stack trace below). The maximum number of shard arguments is 120. The StatsComponent is set to true with a stats.field and stats.facet arguments specified. Document IDs are indexed and stored. Jetty headerBufferSize is set to 10MB (attached the complete jetty.xml). When the exception occurs performance goes down significantly. After restart performance is restored until the exception occurs. Has anyone seen this before? Any ideas what may cause this ?
Here the full stack trace: SEVERE: org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: java.net.SocketException: Connection reset at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:281) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451) Caused by: org.apache.solr.client.solrj.SolrServerException: java.net.SocketException: Connection reset at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:484) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:245) at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:421) at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:393) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:168) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:428) ... 11 more Thanks, Johannes
<?xml version="1.0"?> <!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.mortbay.org/configure.dtd";> <!-- =============================================================== --> <!-- Configure the Jetty Server --> <!-- --> <!-- Documentation of this file format can be found at: --> <!-- http://docs.codehaus.org/display/JETTY/jetty.xml --> <!-- --> <!-- =============================================================== --> <Configure id="Server" class="org.mortbay.jetty.Server"> <!-- Increase the maximum POST size to 1 MB to be able to handle large shard requests --> <Call class="java.lang.System" name="setProperty"> <Arg>org.mortbay.jetty.Request.maxFormContentSize</Arg> <Arg>10000000</Arg> </Call> <!-- =========================================================== --> <!-- Server Thread Pool --> <!-- =========================================================== --> <Set name="ThreadPool"> <!-- Default bounded blocking threadpool --> <New class="org.mortbay.thread.BoundedThreadPool"> <Set name="minThreads">10</Set> <Set name="lowThreads">50</Set> <Set name="maxThreads">10000</Set> </New> <!-- Optional Java 5 bounded threadpool with job queue <New class="org.mortbay.thread.concurrent.ThreadPool"> <Arg type="int">0</Arg> <Set name="corePoolSize">10</Set> <Set name="maximumPoolSize">250</Set> </New> --> </Set> <!-- =========================================================== --> <!-- Set connectors --> <!-- =========================================================== --> <!-- One of each type! --> <!-- =========================================================== --> <!-- Use this connector for many frequently idle connections and for threadless continuations. <Call name="addConnector"> <Arg> <New class="org.mortbay.jetty.nio.SelectChannelConnector"> <Set name="port"><SystemProperty name="jetty.port" default="8983"/></Set> <Set name="maxIdleTime">30000</Set> <Set name="Acceptors">2</Set> <Set name="confidentialPort">8443</Set> <Set name='headerBufferSize'>1048576</Set> </New> </Arg> </Call> --> <!-- Use this connector if NIO is not available. --> <!-- This connector is currently being used for Solr because the nio.SelectChannelConnector showed poor performance under WindowsXP from a single client with non-persistent connections (35s vs ~3min) to complete 10,000 requests) --> <Call name="addConnector"> <Arg> <New class="org.mortbay.jetty.bio.SocketConnector"> <Set name="port"><SystemProperty name="jetty.port" default="8983"/></Set> <Set name="maxIdleTime">50000</Set> <Set name="lowResourceMaxIdleTime">1500</Set> <!--<Set name='headerBufferSize'>1048576</Set>--> <!--SET HEADER BUFFER SIZE TO 10MB --> <Set name='headerBufferSize'>10485760</Set> </New> </Arg> </Call> <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --> <!-- To add a HTTPS SSL listener --> <!-- see jetty-ssl.xml to add an ssl connector. use --> <!-- java -jar start.jar etc/jetty.xml etc/jetty-ssl.xml --> <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --> <!-- =========================================================== --> <!-- Set up global session ID manager --> <!-- =========================================================== --> <!-- <Set name="sessionIdManager"> <New class="org.mortbay.jetty.servlet.HashSessionIdManager"> <Set name="workerName">node1</Set> </New> </Set> --> <!-- =========================================================== --> <!-- Set handler Collection Structure --> <!-- =========================================================== --> <Set name="handler"> <New id="Handlers" class="org.mortbay.jetty.handler.HandlerCollection"> <Set name="handlers"> <Array type="org.mortbay.jetty.Handler"> <Item> <New id="Contexts" class="org.mortbay.jetty.handler.ContextHandlerCollection"/> </Item> <Item> <New id="DefaultHandler" class="org.mortbay.jetty.handler.DefaultHandler"/> </Item> <Item> <New id="RequestLog" class="org.mortbay.jetty.handler.RequestLogHandler"/> </Item> </Array> </Set> </New> </Set> <!-- =========================================================== --> <!-- Configure the context deployer --> <!-- A context deployer will deploy contexts described in --> <!-- configuration files discovered in a directory. --> <!-- The configuration directory can be scanned for hot --> <!-- deployments at the configured scanInterval. --> <!-- --> <!-- This deployer is configured to deploy contexts configured --> <!-- in the $JETTY_HOME/contexts directory --> <!-- --> <!-- =========================================================== --> <Call name="addLifeCycle"> <Arg> <New class="org.mortbay.jetty.deployer.ContextDeployer"> <Set name="contexts"><Ref id="Contexts"/></Set> <Set name="configurationDir"><SystemProperty name="jetty.home" default="."/>/contexts</Set> <Set name="scanInterval">1</Set> </New> </Arg> </Call> <!-- =========================================================== --> <!-- Configure the webapp deployer. --> <!-- A webapp deployer will deploy standard webapps discovered --> <!-- in a directory at startup, without the need for additional --> <!-- configuration files. It does not support hot deploy or --> <!-- non standard contexts (see ContextDeployer above). --> <!-- --> <!-- This deployer is configured to deploy webapps from the --> <!-- $JETTY_HOME/webapps directory --> <!-- --> <!-- Normally only one type of deployer need be used. --> <!-- --> <!-- =========================================================== --> <Call name="addLifeCycle"> <Arg> <New class="org.mortbay.jetty.deployer.WebAppDeployer"> <Set name="contexts"><Ref id="Contexts"/></Set> <Set name="webAppDir"><SystemProperty name="jetty.home" default="."/>/webapps</Set> <Set name="parentLoaderPriority">false</Set> <Set name="extract">true</Set> <Set name="allowDuplicates">false</Set> <Set name="defaultsDescriptor"><SystemProperty name="jetty.home" default="."/>/etc/webdefault.xml</Set> </New> </Arg> </Call> <!-- =========================================================== --> <!-- Configure Authentication Realms --> <!-- Realms may be configured for the entire server here, or --> <!-- they can be configured for a specific web app in a context --> <!-- configuration (see $(jetty.home)/contexts/test.xml for an --> <!-- example). --> <!-- =========================================================== --> <Set name="UserRealms"> <Array type="org.mortbay.jetty.security.UserRealm"> <!-- <Item> <New class="org.mortbay.jetty.security.HashUserRealm"> <Set name="name">Test Realm</Set> <Set name="config"><SystemProperty name="jetty.home" default="."/>/etc/realm.properties</Set> </New> </Item> --> </Array> </Set> <!-- =========================================================== --> <!-- Configure Request Log --> <!-- Request logs may be configured for the entire server here, --> <!-- or they can be configured for a specific web app in a --> <!-- contexts configuration (see $(jetty.home)/contexts/test.xml --> <!-- for an example). --> <!-- =========================================================== --> <Ref id="RequestLog"> <Set name="requestLog"> <New id="RequestLogImpl" class="org.mortbay.jetty.NCSARequestLog"> <Arg><SystemProperty name="jetty.logs" default="./logs"/>/yyyy_mm_dd.request.log</Arg> <Set name="retainDays">20</Set> <Set name="append">true</Set> <Set name="extended">true</Set> <Set name="LogTimeZone">EST</Set> </New> </Set> </Ref> <!-- =========================================================== --> <!-- extra options --> <!-- =========================================================== --> <Set name="stopAtShutdown">true</Set> <!-- ensure/prevent Server: header being sent to browsers --> <Set name="sendServerVersion">true</Set> </Configure>
--------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
