Hey Timothy,
Thanks for looking into this.
I will try taking a clean Solr 8.9.0 and adding my solrconfig.xml and schema to
see if it reproduces there as well (no custom plugins this time, just to try
and take part out and see where the issue is at).
In the meanwhile,
I have decided to take the security.json out of the question, entirely and have
replaced it with {}, and yet, I am seeing the same error.
Only this time, I get the same error for both async and non-async collection
creation.
Is it related to the SOLR-15575 you have created?
Please find below the (quite longer) log:
2021-08-03 19:18:00.903 INFO (qtp431570856-23) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={replicationFactor=1&async=a21&collection.configName=documents&maxShardsPerNode=1&name=documents&action=CREATE&numShards=1&wt=xml}
status=0 QTime=20
2021-08-03 19:18:00.915 INFO
(OverseerThreadFactory-20-thread-1-processing-n:BL003-SOLR1:8080_solr) [ ]
o.a.s.c.a.c.CreateCollectionCmd Create collection documents
2021-08-03 19:18:01.073 INFO
(OverseerStateUpdate-72059578009518088-BL003-SOLR1:8080_solr-n_0000000017) [
] o.a.s.c.o.SliceMutator createReplica() {
"core":"documents_shard1_replica_n1",
"node_name":"BL003-SOLR1:8080_solr",
"base_url":"http://BL003-SOLR1:8080/solr";,
"collection":"documents",
"shard":"shard1",
"state":"down",
"type":"NRT",
"operation":"ADDREPLICA",
"waitForFinalState":"false"}
2021-08-03 19:18:01.353 ERROR
(OverseerThreadFactory-20-thread-1-processing-n:BL003-SOLR1:8080_solr) [ ]
o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard:
http://BL003-SOLR1:8080/solr =>
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://BL003-SOLR1:8080/solr: Expected mime type
application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Unauthorized</title>
</head>
<body><h2>HTTP ERROR 401 Unauthorized</h2>
<table>
<tr><th>URI:</th><td>/solr/admin/cores</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Unauthorized</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:635)
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://BL003-SOLR1:8080/solr: Expected mime type
application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Unauthorized</title>
</head>
<body><h2>HTTP ERROR 401 Unauthorized</h2>
<table>
<tr><th>URI:</th><td>/solr/admin/cores</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Unauthorized</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:635)
~[?:?]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
~[?:?]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
~[?:?]
at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[?:?]
at
org.apache.solr.handler.component.HttpShardHandlerFactory$1.request(HttpShardHandlerFactory.java:169)
~[?:?]
at
org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:130)
~[?:?]
at
org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:41)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_231]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_231]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_231]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.5.jar:4.1.5]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_231]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_231]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
2021-08-03 19:18:01.547 ERROR
(OverseerThreadFactory-20-thread-1-processing-n:BL003-SOLR1:8080_solr) [ ]
o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard:
http://BL003-SOLR1:8080/solr =>
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error
from server at null: Expected mime type application/octet-stream but got
text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Unauthorized</title>
</head>
<body><h2>HTTP ERROR 401 Unauthorized</h2>
<table>
<tr><th>URI:</th><td>/solr/admin/cores</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Unauthorized</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
at
org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:705)
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error
from server at null: Expected mime type application/octet-stream but got
text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Unauthorized</title>
</head>
<body><h2>HTTP ERROR 401 Unauthorized</h2>
<table>
<tr><th>URI:</th><td>/solr/admin/cores</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Unauthorized</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
at
org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:705)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:421)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
~[?:?]
at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[?:?]
at
org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:80)
~[?:?]
at
org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:130)
~[?:?]
at
org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:41)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_231]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_231]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_231]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.5.jar:4.1.5]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_231]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_231]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
2021-08-03 19:18:01.547 ERROR
(OverseerThreadFactory-20-thread-1-processing-n:BL003-SOLR1:8080_solr) [ ]
o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard
BL003-SOLR1:8080_solr: {STATUS=failed}
2021-08-03 19:18:01.547 ERROR
(OverseerThreadFactory-20-thread-1-processing-n:BL003-SOLR1:8080_solr) [ ]
o.a.s.c.a.c.OverseerCollectionMessageHandler Cleaning up collection [documents].
2021-08-03 19:18:01.555 INFO
(OverseerThreadFactory-20-thread-1-processing-n:BL003-SOLR1:8080_solr) [ ]
o.a.s.c.a.c.OverseerCollectionMessageHandler Executing Collection
Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true&deleteMetricsHistory=true,
asyncId=null
2021-08-03 19:18:01.560 ERROR
(OverseerThreadFactory-20-thread-1-processing-n:BL003-SOLR1:8080_solr) [ ]
o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard:
http://BL003-SOLR1:8080/solr =>
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://BL003-SOLR1:8080/solr: Expected mime type
application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Unauthorized</title>
</head>
<body><h2>HTTP ERROR 401 Unauthorized</h2>
<table>
<tr><th>URI:</th><td>/solr/admin/cores</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Unauthorized</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:635)
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://BL003-SOLR1:8080/solr: Expected mime type
application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Unauthorized</title>
</head>
<body><h2>HTTP ERROR 401 Unauthorized</h2>
<table>
<tr><th>URI:</th><td>/solr/admin/cores</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Unauthorized</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:635)
~[?:?]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
~[?:?]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
~[?:?]
at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[?:?]
at
org.apache.solr.handler.component.HttpShardHandlerFactory$1.request(HttpShardHandlerFactory.java:169)
~[?:?]
at
org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:130)
~[?:?]
at
org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:41)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_231]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_231]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_231]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.5.jar:4.1.5]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_231]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_231]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
2021-08-03 19:18:01.675 INFO (zkCallback-12-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDeleted path:/collections/documents/state.json]
for collection [documents] has occurred - updating... (live nodes size: [1])
2021-08-03 19:18:01.687 INFO
(OverseerThreadFactory-20-thread-1-processing-n:BL003-SOLR1:8080_solr) [ ]
o.a.s.c.a.c.CreateCollectionCmd Cleaned up artifacts for failed create
collection for [documents]
2021-08-03 19:18:01.689 ERROR
(OverseerThreadFactory-20-thread-1-processing-n:BL003-SOLR1:8080_solr) [ ]
o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: documents operation:
create failed:org.apache.solr.common.SolrException: Underlying core creation
failed while creating collection: documents
at
org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:371)
at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:270)
at
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:524)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thanks,
Michael
-----Original Message-----
From: Timothy Potter <thelabd...@gmail.com>
Sent: Tuesday, August 3, 2021 6:38 PM
To: users@solr.apache.org
Subject: Re: Getting 401 Unauthorized HTTP errors when trying to create
collection with async flag
WARNING - External email, exercise caution.
I tried reproducing this with a checkout of 8.9.0 to no avail but found another
issue (SOLR-15575). From the stack trace you supplied, the PKI Authentication
Plugin should be supplying the internal server identity for the ShardRequestor
call and in my env (once I fixed
15575) my quick hack test case passed.
Personally, I would disable forwardCredentials and rely on the PKI
Authentication mechanism for this request.
On Mon, Aug 2, 2021 at 8:19 AM Michael Erlikhman
<merlikh...@varonis.com.invalid> wrote:
>
> Hello,
> I am seeing a weird behavior with my Solr 8.9.0 environment.
> I am trying to create a collection in async mode and the process fails with
> 401 Unauthorized errors, which seem to occur during the internode requests
> (according to my debug).
> When I am trying to create the same collection with the EXACT same API
> request, omitting only the async flag, it works flawlessly.
> It seems to be similar to the issue observed here:
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SOLR
> -13510__;!!PowQjtoeD4s!1H_PKCBPhrPvttTvEXSkneCj2JsaQ9dt2l510ejPKUQWa87
> E8cm1L9jGwB91F795$
>
> I have tried configuring the and playing around with the forwardCredentials
> property in the security.json file, but to no avail.
>
> The request API I use is:
> https://urldefense.com/v3/__http://1.2.3.4:8080/solr/admin/collections
> ?action=CREATE&name=documents&numShards=16&replicationFactor=1&collect
> ion.configName=documents&maxShardsPerNode=16&createNodeSet=BL003-Solr1
> :8080_solr&wt=xml&async=123__;!!PowQjtoeD4s!1H_PKCBPhrPvttTvEXSkneCj2J
> saQ9dt2l510ejPKUQWa87E8cm1L9jGwBOXH-8D$
> (It should be noted that this has worked just fine previously, in Solr 7.3.1).
>
> I am seeing the below error reoccurring in the log:
>
> 2021-08-02 16:23:09.245 WARN
> (parallelCoreAdminExecutor-21-thread-16-processing-n:BL003-Solr1:8080_solr
> x:documents_shard6_replica_n10 2218689320227981 CREATE) [c:documents s:shard6
> r:core_node13 x:documents_shard6_replica_n10] o.a.s.c.SolrConfig
> solrconfig.xml: <jmx> is no longer supported, use solr.xml:/metrics/reporter
> section instead
> 2021-08-02 16:23:09.266 ERROR
> (OverseerThreadFactory-29-thread-1-processing-n:BL003-Solr1:8080_solr) [ ]
> o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard:
> https://urldefense.com/v3/__http://BL003-Solr1:8080/solr__;!!PowQjtoeD4s!1H_PKCBPhrPvttTvEXSkneCj2JsaQ9dt2l510ejPKUQWa87E8cm1L9jGwJQa-ZUm$
> =>
> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
> Error from server at null: Expected mime type application/octet-stream but
> got text/html. <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
> <title>Error 401 Unauthorized</title>
> </head>
> <body><h2>HTTP ERROR 401 Unauthorized</h2> <table>
> <tr><th>URI:</th><td>/solr/admin/cores</td></tr>
> <tr><th>STATUS:</th><td>401</td></tr>
> <tr><th>MESSAGE:</th><td>Unauthorized</td></tr>
> <tr><th>SERVLET:</th><td>default</td></tr>
> </table>
>
> </body>
> </html>
>
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResp
> onse(Http2SolrClient.java:705)
> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrExcepti
> on: Error from server at null: Expected mime type
> application/octet-stream but got text/html. <html> <head> <meta
> http-equiv="Content-Type" content="text/html;charset=utf-8"/>
> <title>Error 401 Unauthorized</title>
> </head>
> <body><h2>HTTP ERROR 401 Unauthorized</h2> <table>
> <tr><th>URI:</th><td>/solr/admin/cores</td></tr>
> <tr><th>STATUS:</th><td>401</td></tr>
> <tr><th>MESSAGE:</th><td>Unauthorized</td></tr>
> <tr><th>SERVLET:</th><td>default</td></tr>
> </table>
>
> </body>
> </html>
>
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:705)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:421)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
> ~[?:?]
> at
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[?:?]
> at
> org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:80)
> ~[?:?]
> at
> org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:130)
> ~[?:?]
> at
> org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:41)
> ~[?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_231]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_231]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_231]
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> ~[metrics-core-4.1.5.jar:4.1.5]
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_231]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_231]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
> 2021-08-02 16:23:09.266 ERROR
> (OverseerThreadFactory-29-thread-1-processing-n:BL003-Solr1:8080_solr) [ ]
> o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard
> BL003-Solr1:8080_solr: {STATUS=failed}
>
>
> Some background information:
> We use custom plugins in this Solr environment (but these do not interfere
> with the collection creation process), we have a custom security.jar file
> with 2 custom Authentication and Authorization classes, which again have not
> been changed since Solr 7.3.1 and worked just fine.
> (It should be noted that during my debug session I pass through these classes
> just fine during the internode communications).
>
> I would be glad for any assistance.
>
> Thank you,
> Michael Erlikhman
>
>
> ________________________________
> This email and any attachments thereto may contain private, confidential, and
> privileged material for the sole use of the intended recipient. Any review,
> copying, or distribution of this email (or any attachments thereto) by others
> is strictly prohibited. If you are not the intended recipient, please contact
> the sender immediately and permanently delete the original and any copies of
> this email and any attachments thereto.
________________________________
This email and any attachments thereto may contain private, confidential, and
privileged material for the sole use of the intended recipient. Any review,
copying, or distribution of this email (or any attachments thereto) by others
is strictly prohibited. If you are not the intended recipient, please contact
the sender immediately and permanently delete the original and any copies of
this email and any attachments thereto.
