Interesting! So a workaround would be to just switch back to http1? That
would totally be an acceptable solution for us. I guess the http1 transport
won't disappear any time soon?
Finding the real root cause would be nice too, of course :)
Henrik
On Wed, May 25, 2022, 18:39 Tomás Fernández Löbbe <tomasflo...@gmail.com>
wrote:
> I have seen the same exception raised with search requests too. Not related
> to traffic, since I've seen it in very low query rate cases. Also, not
> related to resource utilization, since I've seen it in very under-utilized
> clusters. An interesting part of the error is this:
>
> > MultiplexConnectionPool@310bfd62
> [c=0/4/4,a=0,i=0,q=3000]
>
> This is the toString() of the MultiplexConnectionPool[1]:
>
> public String toString()
> {
> return String.format("%s@%x[c=%d/%d/%d,a=%d,i=%d,q=%d]",
> getClass().getSimpleName(),
> hashCode(),
> getPendingConnectionCount(),
> getConnectionCount(),
> getMaxConnectionCount(),
> getActiveConnectionCount(),
> getIdleConnectionCount(),
> destination.getQueuedRequestCount());
> }
>
> Meaning, there are 0 pending connections, 4 connections out of a max of 4,
> but then 0 active, and 0 idle. I don't know what state the other
> connections are in. I didn't have time to dive into this further, so I
> ended up switching back to http1. I'm not sure if this is something we are
> doing wrong in Solr or if it's a bug with the Jetty client.
>
> In my case, the client would not recover from this situation until it was
> restarted.
>
> Tomás
>
> [1]
>
> https://github.com/eclipse/jetty.project/blob/d262ef0d62f65f5e0f5f61462617e1a4562ea566/jetty-client/src/main/java/org/eclipse/jetty/client/AbstractConnectionPool.java#L517-L528
>
> On Mon, May 23, 2022 at 1:36 PM Jan Høydahl <jan....@cominvent.com> wrote:
>
> > Hi,
> >
> > I suspect that you have multiple shards, and that UPDATE requests in your
> > cluster expereience errors, causing
> > DistributedUpdateProcessor / ConcurrentUpdateHttp2SolrClient to auto
> retry
> > the request.
> > According to
> >
> https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L89
> > <
> >
> https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L89
> >
> >
> > there may be up to 25 retries on forward to leader.
> >
> > I'm just guessing here, that for some reason the indexing operation just
> > gets overloaded (perhaps due to low RAM or other factors)
> > and errors then propell out of control due to retries? 1Gb is not a huge
> > heap for heavy indexing.
> > - How fast are you indexing?
> > - Do you have any kind of indexing back-off/throttling strategy?
> >
> > Perhaps circuit breakers <
> >
> https://solr.apache.org/guide/solr/latest/deployment-guide/circuit-breakers.html
> >
> > and/or rate limiters <
> >
> https://solr.apache.org/guide/solr/latest/deployment-guide/rate-limiters.html
> >
> > could be considered to help throttle update requests? I have not tried
> them
> > myself.. Rate limiters may be a 9.0 only feature.
> >
> > Jan
> >
> >
> > > 23. mai 2022 kl. 13:40 skrev Henrik Brautaset Aronsen <
> > henrik.aron...@gmail.com>:
> > >
> > > Thanks!
> > >
> > > I'll try doubling the JVM memory and see what happens. The dev
> instances
> > > only had "-Xms1024M -Xmx1024M -Xss256k" set.
> > >
> > > Cheers,
> > > Henrik
> > >
> > > On Mon, May 23, 2022 at 9:56 AM Deepak Goel <deic...@gmail.com> wrote:
> > >
> > >> Looks like the memory is getting maxed out. However there must be some
> > way
> > >> to monitor the dev environment and confirm the same (something like a
> > top
> > >> command).
> > >>
> > >> Deepak
> > >> "The greatness of a nation can be judged by the way its animals are
> > >> treated - Mahatma Gandhi"
> > >>
> > >> +91 73500 12833
> > >> deic...@gmail.com
> > >>
> > >> Facebook: https://www.facebook.com/deicool
> > >> LinkedIn: www.linkedin.com/in/deicool
> > >>
> > >> "Plant a Tree, Go Green"
> > >>
> > >> Make In India : http://www.makeinindia.com/home
> > >>
> > >>
> > >> On Mon, May 23, 2022 at 1:20 PM Henrik Brautaset Aronsen <
> > >> henrik.aron...@gmail.com> wrote:
> > >>
> > >>> Hi Deepak, thanks for reaching out!
> > >>>
> > >>> Prod has 8 servers with 8x Xeon CPU @ 2.00GHz and 64GB RAM
> > >>> Dev has 2 servers with 2x Xeon CPU @ 2.00GHz and 16GB RAM
> > >>>
> > >>> Is there a way to monitor the MultiplexConnectionPool?
> > >>>
> > >>> And is it CPU or RAM that is maxed out, you think?
> > >>>
> > >>> I really appreciate any input :)
> > >>>
> > >>> Cheers,
> > >>> Henrik
> > >>>
> > >>> On Mon, May 16, 2022 at 10:41 AM Deepak Goel <deic...@gmail.com>
> > wrote:
> > >>>
> > >>>> Looks like *MultiplexConnectionPool *is getting maxed out.
> > >>>>
> > >>>> Could you please tell us the server configuration of dev and prod?
> > Looks
> > >>>> like the dev environment is getting maxed out.
> > >>>>
> > >>>> Deepak
> > >>>> "The greatness of a nation can be judged by the way its animals are
> > >>>> treated
> > >>>> - Mahatma Gandhi"
> > >>>>
> > >>>> +91 73500 12833
> > >>>> deic...@gmail.com
> > >>>>
> > >>>> Facebook: https://www.facebook.com/deicool
> > >>>> LinkedIn: www.linkedin.com/in/deicool
> > >>>>
> > >>>> "Plant a Tree, Go Green"
> > >>>>
> > >>>> Make In India : http://www.makeinindia.com/home
> > >>>>
> > >>>>
> > >>>> On Mon, May 16, 2022 at 12:53 PM Henrik Brautaset Aronsen <
> > >>>> henrik.aron...@gmail.com> wrote:
> > >>>>
> > >>>>> Hi.
> > >>>>>
> > >>>>> We keep getting "SolrServerException: Max requests queued per
> > >>>> destination
> > >>>>> 3000 exceeded for HttpDestination" for several different
> collections
> > >>>> and
> > >>>>> instances in our SolrCloud dev environment. The instances don't
> reply
> > >>>> to
> > >>>>> requests and need to be restarted to function again.
> > >>>>>
> > >>>>> We're not getting those errors in the prod environment even though
> > >>>> those
> > >>>>> are set up similarly.
> > >>>>>
> > >>>>> Any ideas what could cause this?
> > >>>>>
> > >>>>> We're on Solr 8.11.1 with connTimeout=60000, socketTimeout=60000,
> > >>>>> zkClientTimeout=30000, solr.jetty.http.idleTimeout=60000. Full
> > >>>> stacktrace
> > >>>>> below. Thanks for any help!
> > >>>>>
> > >>>>> Cheers,
> > >>>>> Henrik
> > >>>>>
> > >>>>> java.util.concurrent.RejectedExecutionException: Max requests
> queued
> > >>>> per
> > >>>>> destination 3000 exceeded for
> > >>>>> HttpDestination[http://dev-foo3672.bar.no:12621
> > >>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> > >>>>> [c=0/4/4,a=0,i=0,q=3000]
> > >>>>> at
> > >>>>
> > org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at
> > >>>>
> > org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at
> > >>>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>> at org.apache.solr.client.solrj.SolrServerException: Max requests
> > >>>> queued
> > >>>>> per destination 3000 exceeded for
> > >>>>> HttpDestination[http://dev-foo3672.bar.no:12621
> > >>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> > >>>>> [c=0/4/4,a=0,i=0,q=3000]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:439)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>> at
> > >>>>
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
> > >>>>> ~[solr-core-8.11.1.jar:8.11.1
> > 0b002b11819df70783e83ef36b42ed1223c14b50
> > >>>> -
> > >>>>> janhoy - 2021-12-14 13:50:55]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334)
> > >>>>> ~[solr-core-8.11.1.jar:8.11.1
> > 0b002b11819df70783e83ef36b42ed1223c14b50
> > >>>> -
> > >>>>> janhoy - 2021-12-14 13:50:55]
> > >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> > >>>>> at
> > >>>>
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> > >>>>> ~[?:?]
> > >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> > >>>>> ~[metrics-core-4.1.5.jar:4.1.5]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> > >>>>> [?:?]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> > >>>>> [?:?]
> > >>>>> at java.lang.Thread.run(Thread.java:833) [?:?] Caused by:
> > >>>>> java.util.concurrent.RejectedExecutionException: Max requests
> queued
> > >>>> per
> > >>>>> destination 3000 exceeded for
> > >>>>> HttpDestination[http://dev-foo3672.bar.no:12621
> > >>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> > >>>>> [c=0/4/4,a=0,i=0,q=3000]
> > >>>>> at
> > >>>>
> > org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at
> > >>>>
> > org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at
> > >>>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>>
> > >>>>
> > >>>
> >
> >
>
