Not a solution to your problem, but I'm just curious if starting all the nodes with -Dsolr.http1=true results in better performance? I'm observing some performance problems in Solr that manifest itself after a sustained query load, and was wondering if that's relevant. https://issues.apache.org/jira/browse/SOLR-15840
On Thu, Dec 9, 2021 at 1:50 PM Christopher Harden <christopher.har...@gm.com> wrote: > Hi all, > > > > I was doing some performance testing on a Solr cloud when I > noticed that at about 7 and a half hours into the test, I saw the > collection leader started to slow down real bad with p95 values of over 13 > seconds per request. The heap crawled upward and then the whole node > failed. When I checked the logs, I saw this message repeatedly for various > servers in the cluster: > > > > request: > https://dcmitavbrd0149.epga.nam.gm.com:8984/solr/gma-live_shard1_replica_n17/ > at > org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:275) > ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - > mayyasharipova - 2021-06-10 > 17:54:42] at > org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181) > ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - > mayyasharipova - 2021-06-10 > 17:54:42] at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) > ~[metrics-core-4.1.5.jar:4.1.5] > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) > ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - > mayyasharipova - 2021-06-10 17:54:42] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > ~[?:?] at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > ~[?:?] at > java.lang.Thread.run(Thread.java:834) > [?:?] > 2021-12-09 03:10:57.302 WARN > (updateExecutor-9-thread-2771-processing-x:gma-live_shard1_replica_n15 > r:core_node16 null n:dcmitavbrd0148.epga.nam.gm.com:8984_solr c:gma-live > s:shard1) [c:gma-live s:shard1 r:core_node16 x:gma-live_shard1_replica_n15] > o.a.s.c.s.i.ConcurrentUpdateHttp2SolrClient Failed to parse error response > from > https://dcmitavbrd0910.epg.nam.gm.com:8984/solr/gma-live_shard1_replica_n4/ > due to: => org.apache.solr.common.SolrException: parsing error > > at > org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:53) > org.apache.solr.common.SolrException: parsing > error > at > org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:53) > ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - > mayyasharipova - 2021-06-10 17:54:42] > at > org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:261) > ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - > mayyasharipova - 2021-06-10 > 17:54:42] at > org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181) > ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - > mayyasharipova - 2021-06-10 > 17:54:42] at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) > ~[metrics-core-4.1.5.jar:4.1.5] > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) > ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - > mayyasharipova - 2021-06-10 > 17:54:42] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > ~[?:?] at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > ~[?:?] at > java.lang.Thread.run(Thread.java:834) > [?:?] > Caused by: java.io.IOException: > org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: > Authentication challenge without WWW-Authenticate header > > at > org.eclipse.jetty.client.util.InputStreamResponseListener$Input.toIOException(InputStreamResponseListener.java:345) > ~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516] > at > org.eclipse.jetty.client.util.InputStreamResponseListener$Input.read(InputStreamResponseListener.java:313) > ~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516] > > at > org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:90) > ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - > mayyasharipova - 2021-06-10 17:54:42] > > at > org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:99) > ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - > mayyasharipova - 2021-06-10 > 17:54:42] > at > org.apache.solr.common.util.FastInputStream.readByte(FastInputStream.java:217) > ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - > mayyasharipova - 2021-06-10 17:54:42] > > > > We are using the Solr basic authenticator. What is particularly strange > here is that at the beginning of the test, this message is nowhere to be > seen. I cannot think of a circumstance which would cause the Solr cloud to > simply stop sending the Authenticate headers in the middle of normal use. > The security.json used in this cloud is included below: > > > > { > > "authentication":{ > > "blockUnknown": true, > > "class":"solr.BasicAuthPlugin", > > "credentials":{"solr":"q68bwnAC5MqVYGnRa8i6s+4D7Q/Kc1i3EgfiVHTdMKU= > hmVFwTNZYeObcnsCyKAN4mnC9+7EZF+Oq9eM0AHmMyY=", > > "searchapi":"4N4QX8m68/cdQhbXlv6cDdunIes9lkv1MMm2HRxd194= > NK4OzuYwxOcM0xPAbERVAK172ALGcfKYg3Q+S5+3SBU="}, > > "realm":"My Solr users", > > "forwardCredentials": false > > }, > > "authorization":{ > > "class":"solr.RuleBasedAuthorizationPlugin", > > "permissions":[{"name":"all", > > "role":"admin"}], > > "user-role":{"solr":"admin"} > > }} > > > > Has anyone seen this before? Does anyone know how to fix it? Is it a bug? > > > > > > > > [image: Icon Description automatically generated] > > Christopher Harden > > Solution Architect, Global Sales, Marketing, and Customer Experience > > christopher.har...@gm.com > > > > > > > > > Nothing in this message is intended to constitute an electronic signature > unless a specific statement to the contrary is included in this message. > > Confidentiality Note: This message is intended only for the person or > entity to which it is addressed. It may contain confidential and/or > privileged material. Any review, transmission, dissemination or other use, > or taking of any action in reliance upon this message by persons or > entities other than the intended recipient is prohibited and may be > unlawful. If you received this message in error, please contact the sender > and delete it from your computer. >
