I tried reproducing this with a checkout of 8.9.0 to no avail but found another issue (SOLR-15575). From the stack trace you supplied, the PKI Authentication Plugin should be supplying the internal server identity for the ShardRequestor call and in my env (once I fixed 15575) my quick hack test case passed.
Personally, I would disable forwardCredentials and rely on the PKI Authentication mechanism for this request. On Mon, Aug 2, 2021 at 8:19 AM Michael Erlikhman <merlikh...@varonis.com.invalid> wrote: > > Hello, > I am seeing a weird behavior with my Solr 8.9.0 environment. > I am trying to create a collection in async mode and the process fails with > 401 Unauthorized errors, which seem to occur during the internode requests > (according to my debug). > When I am trying to create the same collection with the EXACT same API > request, omitting only the async flag, it works flawlessly. > It seems to be similar to the issue observed here: > https://issues.apache.org/jira/browse/SOLR-13510 > > I have tried configuring the and playing around with the forwardCredentials > property in the security.json file, but to no avail. > > The request API I use is: > http://1.2.3.4:8080/solr/admin/collections?action=CREATE&name=documents&numShards=16&replicationFactor=1&collection.configName=documents&maxShardsPerNode=16&createNodeSet=BL003-Solr1:8080_solr&wt=xml&async=123 > (It should be noted that this has worked just fine previously, in Solr 7.3.1). > > I am seeing the below error reoccurring in the log: > > 2021-08-02 16:23:09.245 WARN > (parallelCoreAdminExecutor-21-thread-16-processing-n:BL003-Solr1:8080_solr > x:documents_shard6_replica_n10 2218689320227981 CREATE) [c:documents s:shard6 > r:core_node13 x:documents_shard6_replica_n10] o.a.s.c.SolrConfig > solrconfig.xml: <jmx> is no longer supported, use solr.xml:/metrics/reporter > section instead > 2021-08-02 16:23:09.266 ERROR > (OverseerThreadFactory-29-thread-1-processing-n:BL003-Solr1:8080_solr) [ ] > o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard: > http://BL003-Solr1:8080/solr => > org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: > Error from server at null: Expected mime type application/octet-stream but > got text/html. <html> > <head> > <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> > <title>Error 401 Unauthorized</title> > </head> > <body><h2>HTTP ERROR 401 Unauthorized</h2> > <table> > <tr><th>URI:</th><td>/solr/admin/cores</td></tr> > <tr><th>STATUS:</th><td>401</td></tr> > <tr><th>MESSAGE:</th><td>Unauthorized</td></tr> > <tr><th>SERVLET:</th><td>default</td></tr> > </table> > > </body> > </html> > > at > org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:705) > org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: > Error from server at null: Expected mime type application/octet-stream but > got text/html. <html> > <head> > <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> > <title>Error 401 Unauthorized</title> > </head> > <body><h2>HTTP ERROR 401 Unauthorized</h2> > <table> > <tr><th>URI:</th><td>/solr/admin/cores</td></tr> > <tr><th>STATUS:</th><td>401</td></tr> > <tr><th>MESSAGE:</th><td>Unauthorized</td></tr> > <tr><th>SERVLET:</th><td>default</td></tr> > </table> > > </body> > </html> > > at > org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:705) > ~[?:?] > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:421) > ~[?:?] > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776) > ~[?:?] > at > org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[?:?] > at > org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:80) > ~[?:?] > at > org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:130) > ~[?:?] > at > org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:41) > ~[?:?] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[?:1.8.0_231] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[?:1.8.0_231] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[?:1.8.0_231] > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) > ~[metrics-core-4.1.5.jar:4.1.5] > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) > ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > ~[?:1.8.0_231] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > ~[?:1.8.0_231] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231] > 2021-08-02 16:23:09.266 ERROR > (OverseerThreadFactory-29-thread-1-processing-n:BL003-Solr1:8080_solr) [ ] > o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard > BL003-Solr1:8080_solr: {STATUS=failed} > > > Some background information: > We use custom plugins in this Solr environment (but these do not interfere > with the collection creation process), we have a custom security.jar file > with 2 custom Authentication and Authorization classes, which again have not > been changed since Solr 7.3.1 and worked just fine. > (It should be noted that during my debug session I pass through these classes > just fine during the internode communications). > > I would be glad for any assistance. > > Thank you, > Michael Erlikhman > > > ________________________________ > This email and any attachments thereto may contain private, confidential, and > privileged material for the sole use of the intended recipient. Any review, > copying, or distribution of this email (or any attachments thereto) by others > is strictly prohibited. If you are not the intended recipient, please contact > the sender immediately and permanently delete the original and any copies of > this email and any attachments thereto.
