Solr/Lucene 3.1 | apache ab benchmark | org.mortbay.jetty.EofException | java.net.SocketException: Broken pipe

[Johannes Goll]

Hi,

I am running ab, Apache's HTTP server benchmarking tool
to evaluate the query performance of a Solr/Lucene 3.1  instance.

The test index has 12 million documents. The search returns the first
10 rows of 8 stored fields that match a standard query (q=field:value).

The box has 256G RAM and 32 cores, the VM is run with -Xms512M -Xmx100G

Up to a ab concurrency level of 4 (-c4), everything works fine, above that
I am getting timeout exceptions from ab and org.mortbay.jetty.EofException
(Caused by: java.net.SocketException: Broken pipe) exceptions in the Solr
log file. I ran the same test on a smaller box using Solr/Lucene 1.4.1 with
a similar index (8 million docs) in the past and I got results up to a
concurrency level of 512.

Here are the details for the Solr/Lucene 3.1 run:

AB setting:

concurrency: 2
complete requests: 2000 [ab -n2000 -c2]
works fine

concurrency: 4
complete requests: 4000 [ab -n4000 -c4]
works fine

concurrency: 8
complete requests: 8000 [ab -n8000 -c8]
apr_poll: The timeout specified has expired (70007)

solr-log:

May 4, 2011 10:24:27 AM org.apache.solr.common.SolrException log
SEVERE: org.mortbay.jetty.EofException
        at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
        at
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
        at
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
        at org.apache.solr.common.util.FastWriter.flush(FastWriter.java:115)
        at
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:344)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265)
        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

Any explanations for this ? Do you recommend to switch to Tomcat instead of
JETTY ?

Thanks a lot,
Johannes