Hi all,
I was doing some performance testing on a Solr cloud when I
noticed that at about 7 and a half hours into the test, I saw the collection
leader started to slow down real bad with p95 values of over 13 seconds per
request. The heap crawled upward and then the whole node failed. When I checked
the logs, I saw this message repeatedly for various servers in the cluster:
request:
https://dcmitavbrd0149.epga.nam.gm.com:8984/solr/gma-live_shard1_replica_n17/
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:275)
~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 -
mayyasharipova - 2021-06-10 17:54:42]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181)
~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 -
mayyasharipova - 2021-06-10 17:54:42]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.5.jar:4.1.5]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 -
mayyasharipova - 2021-06-10 17:54:42]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?] at
java.lang.Thread.run(Thread.java:834) [?:?]
2021-12-09 03:10:57.302
WARN (updateExecutor-9-thread-2771-processing-x:gma-live_shard1_replica_n15
r:core_node16 null n:dcmitavbrd0148.epga.nam.gm.com:8984_solr c:gma-live
s:shard1) [c:gma-live s:shard1 r:core_node16 x:gma-live_shard1_replica_n15]
o.a.s.c.s.i.ConcurrentUpdateHttp2SolrClient Failed to parse error response from
https://dcmitavbrd0910.epg.nam.gm.com:8984/solr/gma-live_shard1_replica_n4/ due
to: => org.apache.solr.common.SolrException: parsing error
at
org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:53)
org.apache.solr.common.SolrException: parsing
error
at
org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:53)
~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 -
mayyasharipova - 2021-06-10 17:54:42]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:261)
~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 -
mayyasharipova - 2021-06-10 17:54:42]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181)
~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 -
mayyasharipova - 2021-06-10 17:54:42]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.5.jar:4.1.5]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 -
mayyasharipova - 2021-06-10 17:54:42]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?] at
java.lang.Thread.run(Thread.java:834) [?:?]
Caused by:
java.io.IOException: org.eclipse.jetty.client.HttpResponseException: HTTP
protocol violation: Authentication challenge without WWW-Authenticate header
at
org.eclipse.jetty.client.util.InputStreamResponseListener$Input.toIOException(InputStreamResponseListener.java:345)
~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516]
at
org.eclipse.jetty.client.util.InputStreamResponseListener$Input.read(InputStreamResponseListener.java:313)
~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516]
at
org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:90)
~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 -
mayyasharipova - 2021-06-10 17:54:42]
at
org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:99)
~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 -
mayyasharipova - 2021-06-10 17:54:42]
at
org.apache.solr.common.util.FastInputStream.readByte(FastInputStream.java:217)
~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 -
mayyasharipova - 2021-06-10 17:54:42]
We are using the Solr basic authenticator. What is particularly strange here is
that at the beginning of the test, this message is nowhere to be seen. I cannot
think of a circumstance which would cause the Solr cloud to simply stop sending
the Authenticate headers in the middle of normal use. The security.json used in
this cloud is included below:
{
"authentication":{
"blockUnknown": true,
"class":"solr.BasicAuthPlugin",
"credentials":{"solr":"q68bwnAC5MqVYGnRa8i6s+4D7Q/Kc1i3EgfiVHTdMKU=
hmVFwTNZYeObcnsCyKAN4mnC9+7EZF+Oq9eM0AHmMyY=",
"searchapi":"4N4QX8m68/cdQhbXlv6cDdunIes9lkv1MMm2HRxd194=
NK4OzuYwxOcM0xPAbERVAK172ALGcfKYg3Q+S5+3SBU="},
"realm":"My Solr users",
"forwardCredentials": false
},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"permissions":[{"name":"all",
"role":"admin"}],
"user-role":{"solr":"admin"}
}}
Has anyone seen this before? Does anyone know how to fix it? Is it a bug?
[Icon Description automatically generated]
Christopher Harden
Solution Architect, Global Sales, Marketing, and Customer Experience
christopher.har...@gm.com<mailto:christopher.har...@gm.com>
Nothing in this message is intended to constitute an electronic signature
unless a specific statement to the contrary is included in this message.
Confidentiality Note: This message is intended only for the person or entity to
which it is addressed. It may contain confidential and/or privileged material.
Any review, transmission, dissemination or other use, or taking of any action
in reliance upon this message by persons or entities other than the intended
recipient is prohibited and may be unlawful. If you received this message in
error, please contact the sender and delete it from your computer.
