[
https://issues.apache.org/jira/browse/SOLR-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17517539#comment-17517539
]
Nicholas Kumia commented on SOLR-13510:
---------------------------------------
Hi again [~anshum], I did a deep dive on all of our solr configuration and have
verified that we are not actually defining a custom shardHandlerFactory on the
/select requestHandler. I created a new instance of our solr setup and am
currently re-indexing it. While it is not the same issue I was seeing before,
it seems vaguely related since it has to do with HTTP Authentication. Here's
the backtrace from the logs,
{code:java}
2022-04-05 16:07:22.471 ERROR
(updateExecutor-9-thread-7-processing-n:default-solr:80_solr
x:ckan_shard1_replica_n1 c:ckan s:shard1 r:core_node3) [c:ckan s:shard1
r:core_node3 x:ckan_shard1_replica_n1] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error
from server at null: Could not parse response with encoding ISO-8859-1 =>
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error
from server at null: Could not parse response with encoding ISO-8859-1
at
org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:703)
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error
from server at null: Could not parse response with encoding ISO-8859-1
at
org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:703)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:421)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
~[?:?]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370)
~[?:?]
at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334)
~[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.5.jar:4.1.5]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
~[?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.io.IOException: org.eclipse.jetty.client.HttpResponseException:
HTTP protocol violation: Authentication challenge without WWW-Authenticate
header
at
org.eclipse.jetty.client.util.InputStreamResponseListener$Input.toIOException(InputStreamResponseListener.java:345)
~[?:?]
at
org.eclipse.jetty.client.util.InputStreamResponseListener$Input.read(InputStreamResponseListener.java:313)
~[?:?]
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source) ~[?:?]
at sun.nio.cs.StreamDecoder.implRead(Unknown Source) ~[?:?]
at sun.nio.cs.StreamDecoder.read(Unknown Source) ~[?:?]
at java.io.InputStreamReader.read(Unknown Source) ~[?:?]
at java.io.Reader.read(Unknown Source) ~[?:?]
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1282) ~[?:?]
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1261) ~[?:?]
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1108) ~[?:?]
at org.apache.commons.io.IOUtils.copy(IOUtils.java:922) ~[?:?]
at org.apache.commons.io.IOUtils.toString(IOUtils.java:2681) ~[?:?]
at org.apache.commons.io.IOUtils.toString(IOUtils.java:2707) ~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:701)
~[?:?]
... 14 more
Caused by: org.eclipse.jetty.client.HttpResponseException: HTTP protocol
violation: Authentication challenge without WWW-Authenticate header
at
org.eclipse.jetty.client.AuthenticationProtocolHandler$AuthenticationListener.onComplete(AuthenticationProtocolHandler.java:164)
~[?:?]
at
org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:218)
~[?:?]
at
org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:210)
~[?:?]
at
org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:481)
~[?:?]
at
org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:461)
~[?:?]
at
org.eclipse.jetty.client.HttpReceiver.responseSuccess(HttpReceiver.java:424)
~[?:?]
at
org.eclipse.jetty.http2.client.http.HttpReceiverOverHTTP2.access$1400(HttpReceiverOverHTTP2.java:49)
~[?:?]
at
org.eclipse.jetty.http2.client.http.HttpReceiverOverHTTP2$ContentNotifier.process(HttpReceiverOverHTTP2.java:273)
~[?:?]
at
org.eclipse.jetty.http2.client.http.HttpReceiverOverHTTP2$ContentNotifier.offer(HttpReceiverOverHTTP2.java:234)
~[?:?]
at
org.eclipse.jetty.http2.client.http.HttpReceiverOverHTTP2$ContentNotifier.access$300(HttpReceiverOverHTTP2.java:214)
~[?:?]
at
org.eclipse.jetty.http2.client.http.HttpReceiverOverHTTP2.notifyContent(HttpReceiverOverHTTP2.java:211)
~[?:?]
at
org.eclipse.jetty.http2.client.http.HttpReceiverOverHTTP2.onData(HttpReceiverOverHTTP2.java:179)
~[?:?]
at org.eclipse.jetty.http2.HTTP2Stream.notifyData(HTTP2Stream.java:645)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.http2.HTTP2Stream.onData(HTTP2Stream.java:387)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:306)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.http2.HTTP2Session.onData(HTTP2Session.java:260)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.http2.HTTP2Connection$ParserListener.onData(HTTP2Connection.java:390)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.http2.parser.BodyParser.notifyData(BodyParser.java:108)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.http2.parser.DataBodyParser.onData(DataBodyParser.java:150)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.http2.parser.DataBodyParser.onData(DataBodyParser.java:145)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.http2.parser.DataBodyParser.parse(DataBodyParser.java:111)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:198)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:127)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:261)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:183)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:138)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:361)
~[http2-common-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
~[jetty-io-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
~[jetty-io-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:69)
~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:350)
~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:305)
~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927]
... 4 more {code}
The following log trace is the one I was originally seeing and the one that
replicated the error on this issue and the one you posted. I was getting the
same error on RecoveryStrategy (even though this one is just a
MetricsHistoryHandler).
{code:java}
2022-04-05 16:09:41.318 WARN (MetricsHistoryHandler-22-thread-1) [ ]
o.a.s.c.s.i.SolrClientNodeStateProvider could not get tags from node
default-solr:80_solr =>
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://default-solr:80/solr: Expected mime type
application/octet-stream but got text/html. <html>
<head>
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://default-solr:80/solr: Expected mime type
application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Authentication failed, Response code: 401</title>
</head>
<body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
<table>
<tr><th>URI:</th><td>/solr/admin/metrics</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Authentication failed, Response code: 401</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table></body>
</html> at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:635)
~[?:?]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
~[?:?]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
~[?:?]
at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[?:?]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.invoke(SolrClientNodeStateProvider.java:389)
~[?:?]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.invokeWithRetry(SolrClientNodeStateProvider.java:354)
~[?:?]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:197)
~[?:?]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:187)
~[?:?]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getReplicaInfo(SolrClientNodeStateProvider.java:170)
~[?:?]
at
org.apache.solr.handler.admin.MetricsHistoryHandler.collectGlobalMetrics(MetricsHistoryHandler.java:506)
~[?:?]
at
org.apache.solr.handler.admin.MetricsHistoryHandler.collectMetrics(MetricsHistoryHandler.java:394)
~[?:?]
at
org.apache.solr.handler.admin.MetricsHistoryHandler.lambda$new$0(MetricsHistoryHandler.java:241)
~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[?:?]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[?:?]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
~[?:?]
at java.lang.Thread.run(Unknown Source) [?:?] {code}
I know you have other tasks, but if you have any additional information on how
to fix the error I'm seeing, it would be greatly appreciated. Also, if you
think this issue would be resolved in the Solr 9.0 release, that information
would be useful as well.
> Intermittent 401's for internode requests with basicauth enabled
> ----------------------------------------------------------------
>
> Key: SOLR-13510
> URL: https://issues.apache.org/jira/browse/SOLR-13510
> Project: Solr
> Issue Type: Bug
> Components: Authentication
> Affects Versions: 9.0
> Reporter: Jason Gerlowski
> Assignee: Cao Manh Dat
> Priority: Major
> Fix For: 8.1.2, 8.2, 9.0
>
> Attachments: SOLR-13510.patch
>
>
> We recently got a bug report on the mailing list:
> {quote}
> On Solr 8.1.1, using our previously working security.json, running queries
> (through the admin UI currently) I non-deterministically get 401 responses
> on queries when a collection has more than 1 shard. Increasing the number
> of shards in the collection makes the errors more likely.
> {
> "responseHeader":{
> "zkConnected":true,
> "status":401,
> "QTime":30,
> "params":{
> "q":"*:*",
> "_":"1559474550365"}},
> "error":{
> "metadata":[
> "error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException",
> "root-error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException"],
> "msg":"Error from server at null: Expected mime type
> application/octet-stream but got text/html. <html>\n<head>\n<meta
> http-equiv=\"Content-Type\"
> content=\"text/html;charset=utf-8\"/>\n<title>Error 401 require
> authentication</title>\n</head>\n<body><h2>HTTP ERROR 401</h2>\n<p>Problem
> accessing /solr/gettingstarted_shard4_replica_n6/select. Reason:\n<pre>
> require authentication</pre></p>\n</body>\n</html>\n",
> "code":401}}
> {quote}
> The reporter (credit to Colvin Cowie) also gives reproduction steps:
> {quote}
> # Extract solr 8.1.1.
> # bin\solr start -e cloud
> 1 node / [default port] / [default collection name] / 4 shards / 1
> replica / [_default configuration]
> # server\scripts\cloud-scripts\zkcli -zkhost localhost:9983 -cmd putfile
> /security.json <path-to-security-json-file-with-content-below>
> {
> "authentication": {
> "blockUnknown": true,
> "class": "solr.BasicAuthPlugin",
> "credentials": {
> "solradmin": "PIWZwkGnEKxKnqUs3X08xmbmYBaYyAeP3FiKp7fmeHc=
> Lnbp6bEbE7Ap8lXvQDKkUX2Xw53QDgP6Ae8QRT0P5/A="
> }
> },
> "authorization": {
> "class": "solr.RuleBasedAuthorizationPlugin",
> "permissions": [{ "name": "all", "role": "admin"} ],
> "user-role": {"solradmin": "admin"}
> }
> }
> {quote}
> (Minor edits for conciseness)
> I'm able to reproduce this bug as well. Other auth issues (SOLR-13472) look
> like they're impacted by the topography of the collection and cluster. But
> this doesn't seem affected by that at all (401's occur on inter-node requests
> regardless of the recipient of the initial request, and even when all nodes
> have a shard replica).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
