[
https://issues.apache.org/jira/browse/SOLR-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855427#comment-16855427
]
Jan Høydahl commented on SOLR-13510:
------------------------------------
I started {{bin/solr -e cloud}} with 1 node, 4 shards, 1 replica and auth
enabled through {{bin/solr auth enable -credentials solr:solr -blockUnknown
true -z localhost:9983}}
{{}}Here's the difference in logs between successful request and failed (with
DEBUG enabled for o.a.s.security)
SUCCESSFUL:
{noformat}
2019-06-04 07:31:06.515 DEBUG
(httpShardExecutor-5-thread-22-processing-x:gettingstarted_shard2_replica_n2
r:core_node5
http:////192.168.127.248:8983//solr//gettingstarted_shard3_replica_n4//
n:192.168.127.248:8983_solr c:gettingstarted s:shard2
[http:////192.168.127.248:8983//solr//gettingstarted_shard3_replica_n4//])
[c:gettingstarted s:shard2 r:core_node5 x:gettingstarted_shard2_replica_n2]
o.a.s.s.PKIAuthenticationPlugin secures this internode request
2019-06-04 07:31:06.515 DEBUG
(httpShardExecutor-5-thread-21-processing-x:gettingstarted_shard2_replica_n2
r:core_node5
http:////192.168.127.248:8983//solr//gettingstarted_shard2_replica_n2//
n:192.168.127.248:8983_solr c:gettingstarted s:shard2
[http:////192.168.127.248:8983//solr//gettingstarted_shard2_replica_n2//])
[c:gettingstarted s:shard2 r:core_node5 x:gettingstarted_shard2_replica_n2]
o.a.s.s.PKIAuthenticationPlugin secures this internode request
2019-06-04 07:31:06.515 DEBUG
(httpShardExecutor-5-thread-23-processing-x:gettingstarted_shard2_replica_n2
r:core_node5
http:////192.168.127.248:8983//solr//gettingstarted_shard4_replica_n6//
n:192.168.127.248:8983_solr c:gettingstarted s:shard2
[http:////192.168.127.248:8983//solr//gettingstarted_shard4_replica_n6//])
[c:gettingstarted s:shard2 r:core_node5 x:gettingstarted_shard2_replica_n2]
o.a.s.s.PKIAuthenticationPlugin secures this internode request
2019-06-04 07:31:06.516 DEBUG
(httpShardExecutor-5-thread-24-processing-x:gettingstarted_shard2_replica_n2
r:core_node5
http:////192.168.127.248:8983//solr//gettingstarted_shard1_replica_n1//
n:192.168.127.248:8983_solr c:gettingstarted s:shard2
[http:////192.168.127.248:8983//solr//gettingstarted_shard1_replica_n1//])
[c:gettingstarted s:shard2 r:core_node5 x:gettingstarted_shard2_replica_n2]
o.a.s.s.PKIAuthenticationPlugin secures this internode request
2019-06-04 07:31:06.521 DEBUG (qtp67730604-26) [c:gettingstarted s:shard1
r:core_node3 x:gettingstarted_shard1_replica_n1]
o.a.s.s.PKIAuthenticationPlugin Successfully decrypted header solr 1559633466515
2019-06-04 07:31:06.523 DEBUG (qtp67730604-27) [c:gettingstarted s:shard2
r:core_node5 x:gettingstarted_shard2_replica_n2]
o.a.s.s.PKIAuthenticationPlugin Successfully decrypted header solr 1559633466515
2019-06-04 07:31:06.523 DEBUG (qtp67730604-113) [c:gettingstarted s:shard4
r:core_node8 x:gettingstarted_shard4_replica_n6]
o.a.s.s.PKIAuthenticationPlugin Successfully decrypted header solr 1559633466515
2019-06-04 07:31:06.523 INFO (qtp67730604-26) [c:gettingstarted s:shard2
r:core_node5 x:gettingstarted_shard2_replica_n2] o.a.s.c.S.Request
[gettingstarted_shard2_replica_n2] webapp=/solr path=/select
params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://192.168.127.248:8983/solr/gettingstarted_shard2_replica_n2/&rows=10&version=2&q=*:*&NOW=1559633466513&isShard=true&wt=javabin&_=1559633303203}
hits=0 status=0 QTime=1
2019-06-04 07:31:06.523 INFO (qtp67730604-27) [c:gettingstarted s:shard3
r:core_node7 x:gettingstarted_shard3_replica_n4] o.a.s.c.S.Request
[gettingstarted_shard3_replica_n4] webapp=/solr path=/select
params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://192.168.127.248:8983/solr/gettingstarted_shard3_replica_n4/&rows=10&version=2&q=*:*&NOW=1559633466513&isShard=true&wt=javabin&_=1559633303203}
hits=0 status=0 QTime=0
2019-06-04 07:31:06.523 INFO (qtp67730604-113) [c:gettingstarted s:shard4
r:core_node8 x:gettingstarted_shard4_replica_n6] o.a.s.c.S.Request
[gettingstarted_shard4_replica_n6] webapp=/solr path=/select
params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://192.168.127.248:8983/solr/gettingstarted_shard4_replica_n6/&rows=10&version=2&q=*:*&NOW=1559633466513&isShard=true&wt=javabin&_=1559633303203}
hits=0 status=0 QTime=0
2019-06-04 07:31:06.524 DEBUG (qtp67730604-23) [c:gettingstarted s:shard4
r:core_node8 x:gettingstarted_shard4_replica_n6]
o.a.s.s.PKIAuthenticationPlugin Successfully decrypted header solr 1559633466516
2019-06-04 07:31:06.527 INFO (qtp67730604-23) [c:gettingstarted s:shard1
r:core_node3 x:gettingstarted_shard1_replica_n1] o.a.s.c.S.Request
[gettingstarted_shard1_replica_n1] webapp=/solr path=/select
params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://192.168.127.248:8983/solr/gettingstarted_shard1_replica_n1/&rows=10&version=2&q=*:*&NOW=1559633466513&isShard=true&wt=javabin&_=1559633303203}
hits=0 status=0 QTime=2
2019-06-04 07:31:06.529 INFO (qtp67730604-20) [c:gettingstarted s:shard2
r:core_node5 x:gettingstarted_shard2_replica_n2] o.a.s.c.S.Request
[gettingstarted_shard2_replica_n2] webapp=/solr path=/select
params={q=*:*&_=1559633303203} hits=0 status=0 QTime=15{noformat}
FAILED:
{noformat}
2019-06-04 07:31:00.217 DEBUG
(httpShardExecutor-5-thread-17-processing-x:gettingstarted_shard1_replica_n1
r:core_node3
http:////192.168.127.248:8983//solr//gettingstarted_shard2_replica_n2//
n:192.168.127.248:8983_solr c:gettingstarted s:shard1
[http:////192.168.127.248:8983//solr//gettingstarted_shard2_replica_n2//])
[c:gettingstarted s:shard1 r:core_node3 x:gettingstarted_shard1_replica_n1]
o.a.s.s.PKIAuthenticationPlugin secures this internode request
2019-06-04 07:31:00.217 DEBUG
(httpShardExecutor-5-thread-18-processing-x:gettingstarted_shard1_replica_n1
r:core_node3
http:////192.168.127.248:8983//solr//gettingstarted_shard3_replica_n4//
n:192.168.127.248:8983_solr c:gettingstarted s:shard1
[http:////192.168.127.248:8983//solr//gettingstarted_shard3_replica_n4//])
[c:gettingstarted s:shard1 r:core_node3 x:gettingstarted_shard1_replica_n1]
o.a.s.s.PKIAuthenticationPlugin secures this internode request
2019-06-04 07:31:00.217 DEBUG
(httpShardExecutor-5-thread-19-processing-x:gettingstarted_shard1_replica_n1
r:core_node3
http:////192.168.127.248:8983//solr//gettingstarted_shard4_replica_n6//
n:192.168.127.248:8983_solr c:gettingstarted s:shard1
[http:////192.168.127.248:8983//solr//gettingstarted_shard4_replica_n6//])
[c:gettingstarted s:shard1 r:core_node3 x:gettingstarted_shard1_replica_n1]
o.a.s.s.PKIAuthenticationPlugin secures this internode request
2019-06-04 07:31:00.217 DEBUG
(httpShardExecutor-5-thread-20-processing-x:gettingstarted_shard1_replica_n1
r:core_node3
http:////192.168.127.248:8983//solr//gettingstarted_shard1_replica_n1//
n:192.168.127.248:8983_solr c:gettingstarted s:shard1
[http:////192.168.127.248:8983//solr//gettingstarted_shard1_replica_n1//])
[c:gettingstarted s:shard1 r:core_node3 x:gettingstarted_shard1_replica_n1]
o.a.s.s.PKIAuthenticationPlugin secures this internode request
2019-06-04 07:31:00.223 DEBUG (qtp67730604-27) [c:gettingstarted s:shard1
r:core_node3 x:gettingstarted_shard1_replica_n1]
o.a.s.s.PKIAuthenticationPlugin Successfully decrypted header solr 1559633460217
2019-06-04 07:31:00.224 INFO (qtp67730604-27) [c:gettingstarted s:shard2
r:core_node5 x:gettingstarted_shard2_replica_n2] o.a.s.c.S.Request
[gettingstarted_shard2_replica_n2] webapp=/solr path=/select
params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://192.168.127.248:8983/solr/gettingstarted_shard2_replica_n2/&rows=10&version=2&q=*:*&NOW=1559633460216&isShard=true&wt=javabin&_=1559633303203}
hits=0 status=0 QTime=0
2019-06-04 07:31:00.224 DEBUG (qtp67730604-113) [c:gettingstarted s:shard2
r:core_node5 x:gettingstarted_shard2_replica_n2]
o.a.s.s.PKIAuthenticationPlugin Successfully decrypted header solr 1559633460217
2019-06-04 07:31:00.224 DEBUG (qtp67730604-86) [c:gettingstarted s:shard3
r:core_node7 x:gettingstarted_shard3_replica_n4]
o.a.s.s.PKIAuthenticationPlugin Successfully decrypted header solr 1559633460217
2019-06-04 07:31:00.225 INFO (qtp67730604-86) [c:gettingstarted s:shard1
r:core_node3 x:gettingstarted_shard1_replica_n1] o.a.s.c.S.Request
[gettingstarted_shard1_replica_n1] webapp=/solr path=/select
params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://192.168.127.248:8983/solr/gettingstarted_shard1_replica_n1/&rows=10&version=2&q=*:*&NOW=1559633460216&isShard=true&wt=javabin&_=1559633303203}
hits=0 status=0 QTime=0
2019-06-04 07:31:00.225 INFO (qtp67730604-113) [c:gettingstarted s:shard4
r:core_node8 x:gettingstarted_shard4_replica_n6] o.a.s.c.S.Request
[gettingstarted_shard4_replica_n6] webapp=/solr path=/select
params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://192.168.127.248:8983/solr/gettingstarted_shard4_replica_n6/&rows=10&version=2&q=*:*&NOW=1559633460216&isShard=true&wt=javabin&_=1559633303203}
hits=0 status=0 QTime=1
2019-06-04 07:31:00.227 ERROR (qtp67730604-24) [c:gettingstarted s:shard1
r:core_node3 x:gettingstarted_shard1_replica_n1] o.a.s.h.RequestHandlerBase
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error
from server at null: Expected mime type application/octet-stream but got
text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 require authentication</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /solr/gettingstarted_shard3_replica_n4/select. Reason:
<pre> require authentication</pre></p>
</body>
</html>
at
org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:681)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:400)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:739)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274)
at
org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:227)
at
org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:188)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2019-06-04 07:31:00.227 INFO (qtp67730604-24) [c:gettingstarted s:shard1
r:core_node3 x:gettingstarted_shard1_replica_n1] o.a.s.c.S.Request
[gettingstarted_shard1_replica_n1] webapp=/solr path=/select
params={q=*:*&_=1559633303203} status=401 QTime=11{noformat}
Interestingly enough, all four sub-requests report {{PKIAuthenticationPlugin
secures this internode request}}, but only three reports {{Successfully
decrypted header solr 1559633460217}}.
> Intermittent 401's for internode requests with basicauth enabled
> ----------------------------------------------------------------
>
> Key: SOLR-13510
> URL: https://issues.apache.org/jira/browse/SOLR-13510
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Authentication
> Affects Versions: master (9.0)
> Reporter: Jason Gerlowski
> Priority: Major
>
> We recently got a bug report on the mailing list:
> {quote}
> On Solr 8.1.1, using our previously working security.json, running queries
> (through the admin UI currently) I non-deterministically get 401 responses
> on queries when a collection has more than 1 shard. Increasing the number
> of shards in the collection makes the errors more likely.
> {
> "responseHeader":{
> "zkConnected":true,
> "status":401,
> "QTime":30,
> "params":{
> "q":"*:*",
> "_":"1559474550365"}},
> "error":{
> "metadata":[
> "error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException",
> "root-error-class","org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException"],
> "msg":"Error from server at null: Expected mime type
> application/octet-stream but got text/html. <html>\n<head>\n<meta
> http-equiv=\"Content-Type\"
> content=\"text/html;charset=utf-8\"/>\n<title>Error 401 require
> authentication</title>\n</head>\n<body><h2>HTTP ERROR 401</h2>\n<p>Problem
> accessing /solr/gettingstarted_shard4_replica_n6/select. Reason:\n<pre>
> require authentication</pre></p>\n</body>\n</html>\n",
> "code":401}}
> {quote}
> The reporter (credit to Colvin Cowie) also gives reproduction steps:
> {quote}
> # Extract solr 8.1.1.
> # bin\solr start -e cloud
> 1 node / [default port] / [default collection name] / 4 shards / 1
> replica / [_default configuration]
> # server\scripts\cloud-scripts\zkcli -zkhost localhost:9983 -cmd putfile
> /security.json <path-to-security-json-file-with-content-below>
> {
> "authentication": {
> "blockUnknown": true,
> "class": "solr.BasicAuthPlugin",
> "credentials": {
> "solradmin": "PIWZwkGnEKxKnqUs3X08xmbmYBaYyAeP3FiKp7fmeHc=
> Lnbp6bEbE7Ap8lXvQDKkUX2Xw53QDgP6Ae8QRT0P5/A="
> }
> },
> "authorization": {
> "class": "solr.RuleBasedAuthorizationPlugin",
> "permissions": [{ "name": "all", "role": "admin"} ],
> "user-role": {"solradmin": "admin"}
> }
> }
> {quote}
> (Minor edits for conciseness)
> I'm able to reproduce this bug as well. Other auth issues (SOLR-13472) look
> like they're impacted by the topography of the collection and cluster. But
> this doesn't seem affected by that at all (401's occur on inter-node requests
> regardless of the recipient of the initial request, and even when all nodes
> have a shard replica).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
