Looks like the memory is getting maxed out. However there must be some way to monitor the dev environment and confirm the same (something like a top command).
Deepak "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" Make In India : http://www.makeinindia.com/home On Mon, May 23, 2022 at 1:20 PM Henrik Brautaset Aronsen < henrik.aron...@gmail.com> wrote: > Hi Deepak, thanks for reaching out! > > Prod has 8 servers with 8x Xeon CPU @ 2.00GHz and 64GB RAM > Dev has 2 servers with 2x Xeon CPU @ 2.00GHz and 16GB RAM > > Is there a way to monitor the MultiplexConnectionPool? > > And is it CPU or RAM that is maxed out, you think? > > I really appreciate any input :) > > Cheers, > Henrik > > On Mon, May 16, 2022 at 10:41 AM Deepak Goel <deic...@gmail.com> wrote: > >> Looks like *MultiplexConnectionPool *is getting maxed out. >> >> Could you please tell us the server configuration of dev and prod? Looks >> like the dev environment is getting maxed out. >> >> Deepak >> "The greatness of a nation can be judged by the way its animals are >> treated >> - Mahatma Gandhi" >> >> +91 73500 12833 >> deic...@gmail.com >> >> Facebook: https://www.facebook.com/deicool >> LinkedIn: www.linkedin.com/in/deicool >> >> "Plant a Tree, Go Green" >> >> Make In India : http://www.makeinindia.com/home >> >> >> On Mon, May 16, 2022 at 12:53 PM Henrik Brautaset Aronsen < >> henrik.aron...@gmail.com> wrote: >> >> > Hi. >> > >> > We keep getting "SolrServerException: Max requests queued per >> destination >> > 3000 exceeded for HttpDestination" for several different collections and >> > instances in our SolrCloud dev environment. The instances don't reply to >> > requests and need to be restarted to function again. >> > >> > We're not getting those errors in the prod environment even though those >> > are set up similarly. >> > >> > Any ideas what could cause this? >> > >> > We're on Solr 8.11.1 with connTimeout=60000, socketTimeout=60000, >> > zkClientTimeout=30000, solr.jetty.http.idleTimeout=60000. Full >> stacktrace >> > below. Thanks for any help! >> > >> > Cheers, >> > Henrik >> > >> > java.util.concurrent.RejectedExecutionException: Max requests queued per >> > destination 3000 exceeded for >> > HttpDestination[http://dev-foo3672.bar.no:12621 >> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62 >> > [c=0/4/4,a=0,i=0,q=3000] >> > at >> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289) >> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >> > at >> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262) >> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >> > at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600) >> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >> > at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778) >> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >> > at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765) >> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >> > at >> > >> > >> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409) >> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 >> - >> > janhoy - 2021-12-14 13:50:57] >> > at org.apache.solr.client.solrj.SolrServerException: Max requests >> queued >> > per destination 3000 exceeded for >> > HttpDestination[http://dev-foo3672.bar.no:12621 >> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62 >> > [c=0/4/4,a=0,i=0,q=3000] >> > at >> > >> > >> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:439) >> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 >> - >> > janhoy - 2021-12-14 13:50:57] >> > at >> > >> > >> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776) >> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 >> - >> > janhoy - 2021-12-14 13:50:57] >> > at >> > >> > >> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370) >> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 >> - >> > janhoy - 2021-12-14 13:50:57] >> > at >> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) >> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 >> - >> > janhoy - 2021-12-14 13:50:57] >> > at >> > >> > >> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345) >> > ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - >> > janhoy - 2021-12-14 13:50:55] >> > at >> > >> > >> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334) >> > ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - >> > janhoy - 2021-12-14 13:50:55] >> > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] >> > at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) >> > ~[?:?] >> > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] >> > at >> > >> > >> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) >> > ~[metrics-core-4.1.5.jar:4.1.5] >> > at >> > >> > >> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) >> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 >> - >> > janhoy - 2021-12-14 13:50:57] >> > at >> > >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) >> > [?:?] >> > at >> > >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >> > [?:?] >> > at java.lang.Thread.run(Thread.java:833) [?:?] Caused by: >> > java.util.concurrent.RejectedExecutionException: Max requests queued per >> > destination 3000 exceeded for >> > HttpDestination[http://dev-foo3672.bar.no:12621 >> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62 >> > [c=0/4/4,a=0,i=0,q=3000] >> > at >> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289) >> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >> > at >> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262) >> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >> > at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600) >> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >> > at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778) >> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >> > at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765) >> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >> > at >> > >> > >> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409) >> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 >> - >> > janhoy - 2021-12-14 13:50:57] >> > >> >
