Hi Deepak, thanks for reaching out! Prod has 8 servers with 8x Xeon CPU @ 2.00GHz and 64GB RAM Dev has 2 servers with 2x Xeon CPU @ 2.00GHz and 16GB RAM
Is there a way to monitor the MultiplexConnectionPool? And is it CPU or RAM that is maxed out, you think? I really appreciate any input :) Cheers, Henrik On Mon, May 16, 2022 at 10:41 AM Deepak Goel <deic...@gmail.com> wrote: > Looks like *MultiplexConnectionPool *is getting maxed out. > > Could you please tell us the server configuration of dev and prod? Looks > like the dev environment is getting maxed out. > > Deepak > "The greatness of a nation can be judged by the way its animals are treated > - Mahatma Gandhi" > > +91 73500 12833 > deic...@gmail.com > > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.makeinindia.com/home > > > On Mon, May 16, 2022 at 12:53 PM Henrik Brautaset Aronsen < > henrik.aron...@gmail.com> wrote: > > > Hi. > > > > We keep getting "SolrServerException: Max requests queued per destination > > 3000 exceeded for HttpDestination" for several different collections and > > instances in our SolrCloud dev environment. The instances don't reply to > > requests and need to be restarted to function again. > > > > We're not getting those errors in the prod environment even though those > > are set up similarly. > > > > Any ideas what could cause this? > > > > We're on Solr 8.11.1 with connTimeout=60000, socketTimeout=60000, > > zkClientTimeout=30000, solr.jetty.http.idleTimeout=60000. Full stacktrace > > below. Thanks for any help! > > > > Cheers, > > Henrik > > > > java.util.concurrent.RejectedExecutionException: Max requests queued per > > destination 3000 exceeded for > > HttpDestination[http://dev-foo3672.bar.no:12621 > > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62 > > [c=0/4/4,a=0,i=0,q=3000] > > at > org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289) > > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262) > > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] > > at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600) > > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] > > at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778) > > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] > > at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765) > > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409) > > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - > > janhoy - 2021-12-14 13:50:57] > > at org.apache.solr.client.solrj.SolrServerException: Max requests queued > > per destination 3000 exceeded for > > HttpDestination[http://dev-foo3672.bar.no:12621 > > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62 > > [c=0/4/4,a=0,i=0,q=3000] > > at > > > > > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:439) > > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - > > janhoy - 2021-12-14 13:50:57] > > at > > > > > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776) > > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - > > janhoy - 2021-12-14 13:50:57] > > at > > > > > org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370) > > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - > > janhoy - 2021-12-14 13:50:57] > > at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) > > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - > > janhoy - 2021-12-14 13:50:57] > > at > > > > > org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345) > > ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - > > janhoy - 2021-12-14 13:50:55] > > at > > > > > org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334) > > ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - > > janhoy - 2021-12-14 13:50:55] > > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) > > ~[?:?] > > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] > > at > > > > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) > > ~[metrics-core-4.1.5.jar:4.1.5] > > at > > > > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) > > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - > > janhoy - 2021-12-14 13:50:57] > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > > [?:?] > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > > [?:?] > > at java.lang.Thread.run(Thread.java:833) [?:?] Caused by: > > java.util.concurrent.RejectedExecutionException: Max requests queued per > > destination 3000 exceeded for > > HttpDestination[http://dev-foo3672.bar.no:12621 > > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62 > > [c=0/4/4,a=0,i=0,q=3000] > > at > org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289) > > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262) > > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] > > at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600) > > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] > > at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778) > > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] > > at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765) > > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409) > > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - > > janhoy - 2021-12-14 13:50:57] > > >
