Hi, I suspect that you have multiple shards, and that UPDATE requests in your cluster expereience errors, causing DistributedUpdateProcessor / ConcurrentUpdateHttp2SolrClient to auto retry the request. According to https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L89 <https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L89> there may be up to 25 retries on forward to leader.
I'm just guessing here, that for some reason the indexing operation just gets overloaded (perhaps due to low RAM or other factors) and errors then propell out of control due to retries? 1Gb is not a huge heap for heavy indexing. - How fast are you indexing? - Do you have any kind of indexing back-off/throttling strategy? Perhaps circuit breakers <https://solr.apache.org/guide/solr/latest/deployment-guide/circuit-breakers.html> and/or rate limiters <https://solr.apache.org/guide/solr/latest/deployment-guide/rate-limiters.html> could be considered to help throttle update requests? I have not tried them myself.. Rate limiters may be a 9.0 only feature. Jan > 23. mai 2022 kl. 13:40 skrev Henrik Brautaset Aronsen > <henrik.aron...@gmail.com>: > > Thanks! > > I'll try doubling the JVM memory and see what happens. The dev instances > only had "-Xms1024M -Xmx1024M -Xss256k" set. > > Cheers, > Henrik > > On Mon, May 23, 2022 at 9:56 AM Deepak Goel <deic...@gmail.com> wrote: > >> Looks like the memory is getting maxed out. However there must be some way >> to monitor the dev environment and confirm the same (something like a top >> command). >> >> Deepak >> "The greatness of a nation can be judged by the way its animals are >> treated - Mahatma Gandhi" >> >> +91 73500 12833 >> deic...@gmail.com >> >> Facebook: https://www.facebook.com/deicool >> LinkedIn: www.linkedin.com/in/deicool >> >> "Plant a Tree, Go Green" >> >> Make In India : http://www.makeinindia.com/home >> >> >> On Mon, May 23, 2022 at 1:20 PM Henrik Brautaset Aronsen < >> henrik.aron...@gmail.com> wrote: >> >>> Hi Deepak, thanks for reaching out! >>> >>> Prod has 8 servers with 8x Xeon CPU @ 2.00GHz and 64GB RAM >>> Dev has 2 servers with 2x Xeon CPU @ 2.00GHz and 16GB RAM >>> >>> Is there a way to monitor the MultiplexConnectionPool? >>> >>> And is it CPU or RAM that is maxed out, you think? >>> >>> I really appreciate any input :) >>> >>> Cheers, >>> Henrik >>> >>> On Mon, May 16, 2022 at 10:41 AM Deepak Goel <deic...@gmail.com> wrote: >>> >>>> Looks like *MultiplexConnectionPool *is getting maxed out. >>>> >>>> Could you please tell us the server configuration of dev and prod? Looks >>>> like the dev environment is getting maxed out. >>>> >>>> Deepak >>>> "The greatness of a nation can be judged by the way its animals are >>>> treated >>>> - Mahatma Gandhi" >>>> >>>> +91 73500 12833 >>>> deic...@gmail.com >>>> >>>> Facebook: https://www.facebook.com/deicool >>>> LinkedIn: www.linkedin.com/in/deicool >>>> >>>> "Plant a Tree, Go Green" >>>> >>>> Make In India : http://www.makeinindia.com/home >>>> >>>> >>>> On Mon, May 16, 2022 at 12:53 PM Henrik Brautaset Aronsen < >>>> henrik.aron...@gmail.com> wrote: >>>> >>>>> Hi. >>>>> >>>>> We keep getting "SolrServerException: Max requests queued per >>>> destination >>>>> 3000 exceeded for HttpDestination" for several different collections >>>> and >>>>> instances in our SolrCloud dev environment. The instances don't reply >>>> to >>>>> requests and need to be restarted to function again. >>>>> >>>>> We're not getting those errors in the prod environment even though >>>> those >>>>> are set up similarly. >>>>> >>>>> Any ideas what could cause this? >>>>> >>>>> We're on Solr 8.11.1 with connTimeout=60000, socketTimeout=60000, >>>>> zkClientTimeout=30000, solr.jetty.http.idleTimeout=60000. Full >>>> stacktrace >>>>> below. Thanks for any help! >>>>> >>>>> Cheers, >>>>> Henrik >>>>> >>>>> java.util.concurrent.RejectedExecutionException: Max requests queued >>>> per >>>>> destination 3000 exceeded for >>>>> HttpDestination[http://dev-foo3672.bar.no:12621 >>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62 >>>>> [c=0/4/4,a=0,i=0,q=3000] >>>>> at >>>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289) >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>>>> at >>>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262) >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>>>> at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600) >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>>>> at >>>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778) >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>>>> at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765) >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>>>> at >>>>> >>>>> >>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409) >>>>> ~[solr-solrj-8.11.1.jar:8.11.1 >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>>>> janhoy - 2021-12-14 13:50:57] >>>>> at org.apache.solr.client.solrj.SolrServerException: Max requests >>>> queued >>>>> per destination 3000 exceeded for >>>>> HttpDestination[http://dev-foo3672.bar.no:12621 >>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62 >>>>> [c=0/4/4,a=0,i=0,q=3000] >>>>> at >>>>> >>>>> >>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:439) >>>>> ~[solr-solrj-8.11.1.jar:8.11.1 >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>>>> janhoy - 2021-12-14 13:50:57] >>>>> at >>>>> >>>>> >>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776) >>>>> ~[solr-solrj-8.11.1.jar:8.11.1 >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>>>> janhoy - 2021-12-14 13:50:57] >>>>> at >>>>> >>>>> >>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370) >>>>> ~[solr-solrj-8.11.1.jar:8.11.1 >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>>>> janhoy - 2021-12-14 13:50:57] >>>>> at >>>> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) >>>>> ~[solr-solrj-8.11.1.jar:8.11.1 >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>>>> janhoy - 2021-12-14 13:50:57] >>>>> at >>>>> >>>>> >>>> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345) >>>>> ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 >>>> - >>>>> janhoy - 2021-12-14 13:50:55] >>>>> at >>>>> >>>>> >>>> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334) >>>>> ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 >>>> - >>>>> janhoy - 2021-12-14 13:50:55] >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] >>>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) >>>>> ~[?:?] >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] >>>>> at >>>>> >>>>> >>>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) >>>>> ~[metrics-core-4.1.5.jar:4.1.5] >>>>> at >>>>> >>>>> >>>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) >>>>> ~[solr-solrj-8.11.1.jar:8.11.1 >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>>>> janhoy - 2021-12-14 13:50:57] >>>>> at >>>>> >>>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) >>>>> [?:?] >>>>> at >>>>> >>>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >>>>> [?:?] >>>>> at java.lang.Thread.run(Thread.java:833) [?:?] Caused by: >>>>> java.util.concurrent.RejectedExecutionException: Max requests queued >>>> per >>>>> destination 3000 exceeded for >>>>> HttpDestination[http://dev-foo3672.bar.no:12621 >>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62 >>>>> [c=0/4/4,a=0,i=0,q=3000] >>>>> at >>>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289) >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>>>> at >>>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262) >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>>>> at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600) >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>>>> at >>>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778) >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>>>> at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765) >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] >>>>> at >>>>> >>>>> >>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409) >>>>> ~[solr-solrj-8.11.1.jar:8.11.1 >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 - >>>>> janhoy - 2021-12-14 13:50:57] >>>>> >>>> >>>
