Thanks! I'll try doubling the JVM memory and see what happens. The dev instances only had "-Xms1024M -Xmx1024M -Xss256k" set.
Cheers, Henrik On Mon, May 23, 2022 at 9:56 AM Deepak Goel <deic...@gmail.com> wrote: > Looks like the memory is getting maxed out. However there must be some way > to monitor the dev environment and confirm the same (something like a top > command). > > Deepak > "The greatness of a nation can be judged by the way its animals are > treated - Mahatma Gandhi" > > +91 73500 12833 > deic...@gmail.com > > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.makeinindia.com/home > > > On Mon, May 23, 2022 at 1:20 PM Henrik Brautaset Aronsen < > henrik.aron...@gmail.com> wrote: > >> Hi Deepak, thanks for reaching out! >> >> Prod has 8 servers with 8x Xeon CPU @ 2.00GHz and 64GB RAM >> Dev has 2 servers with 2x Xeon CPU @ 2.00GHz and 16GB RAM >> >> Is there a way to monitor the MultiplexConnectionPool? >> >> And is it CPU or RAM that is maxed out, you think? >> >> I really appreciate any input :) >> >> Cheers, >> Henrik >> >> On Mon, May 16, 2022 at 10:41 AM Deepak Goel <deic...@gmail.com> wrote: >> >>> Looks like *MultiplexConnectionPool *is getting maxed out. >>> >>> Could you please tell us the server configuration of dev and prod? Looks >>> like the dev environment is getting maxed out. >>> >>> Deepak >>> "The greatness of a nation can be judged by the way its animals are >>> treated >>> - Mahatma Gandhi" >>> >>> +91 73500 12833 >>> deic...@gmail.com >>> >>> Facebook: https://www.facebook.com/deicool >>> LinkedIn: www.linkedin.com/in/deicool >>> >>> "Plant a Tree, Go Green" >>> >>> Make In India : http://www.makeinindia.com/home >>> >>> >>> On Mon, May 16, 2022 at 12:53 PM Henrik Brautaset Aronsen < >>> henrik.aron...@gmail.com> wrote: >>> >>> > Hi. >>> > >>> > We keep getting "SolrServerException: Max requests queued per >>> destination >>> > 3000 exceeded for HttpDestination" for several different collections >>> and >>> > instances in our SolrCloud dev environment. The instances don't reply >>> to >>> > requests and need to be restarted to function again. >>> > >>> > We're not getting those errors in the prod environment even though >>> those >>> > are set up similarly. >>> > >>> > Any ideas what could cause this? >>> > >>> > We're on Solr 8.11.1 with connTimeout=60000, socketTimeout=60000, >>> > zkClientTimeout=30000, solr.jetty.http.idleTimeout=60000. Full >>> stacktrace >>> > below. Thanks for any help! >>> > >>> > Cheers, >>> > Henrik >>> > >>> > java.util.concurrent.RejectedExecutionException: Max requests queued >>> per >>> > destination 3000 exceeded for >>> > HttpDestination[http://dev-foo3672.bar.no:12621 >>> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62 >>> > [c=0/4/4,a=0,i=0,q=3000] >>> > at >>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289) >>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>> > at >>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262) >>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>> > at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600) >>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>> > at >>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778) >>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>> > at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765) >>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>> > at >>> > >>> > >>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409) >>> > ~[solr-solrj-8.11.1.jar:8.11.1 >>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>> > janhoy - 2021-12-14 13:50:57] >>> > at org.apache.solr.client.solrj.SolrServerException: Max requests >>> queued >>> > per destination 3000 exceeded for >>> > HttpDestination[http://dev-foo3672.bar.no:12621 >>> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62 >>> > [c=0/4/4,a=0,i=0,q=3000] >>> > at >>> > >>> > >>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:439) >>> > ~[solr-solrj-8.11.1.jar:8.11.1 >>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>> > janhoy - 2021-12-14 13:50:57] >>> > at >>> > >>> > >>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776) >>> > ~[solr-solrj-8.11.1.jar:8.11.1 >>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>> > janhoy - 2021-12-14 13:50:57] >>> > at >>> > >>> > >>> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370) >>> > ~[solr-solrj-8.11.1.jar:8.11.1 >>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>> > janhoy - 2021-12-14 13:50:57] >>> > at >>> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) >>> > ~[solr-solrj-8.11.1.jar:8.11.1 >>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>> > janhoy - 2021-12-14 13:50:57] >>> > at >>> > >>> > >>> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345) >>> > ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 >>> - >>> > janhoy - 2021-12-14 13:50:55] >>> > at >>> > >>> > >>> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334) >>> > ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 >>> - >>> > janhoy - 2021-12-14 13:50:55] >>> > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] >>> > at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) >>> > ~[?:?] >>> > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] >>> > at >>> > >>> > >>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) >>> > ~[metrics-core-4.1.5.jar:4.1.5] >>> > at >>> > >>> > >>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) >>> > ~[solr-solrj-8.11.1.jar:8.11.1 >>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>> > janhoy - 2021-12-14 13:50:57] >>> > at >>> > >>> > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) >>> > [?:?] >>> > at >>> > >>> > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >>> > [?:?] >>> > at java.lang.Thread.run(Thread.java:833) [?:?] Caused by: >>> > java.util.concurrent.RejectedExecutionException: Max requests queued >>> per >>> > destination 3000 exceeded for >>> > HttpDestination[http://dev-foo3672.bar.no:12621 >>> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62 >>> > [c=0/4/4,a=0,i=0,q=3000] >>> > at >>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289) >>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>> > at >>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262) >>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>> > at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600) >>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>> > at >>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778) >>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>> > at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765) >>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>> > at >>> > >>> > >>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409) >>> > ~[solr-solrj-8.11.1.jar:8.11.1 >>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>> > janhoy - 2021-12-14 13:50:57] >>> > >>> >>
