Hello, I am seeing a weird behavior with my Solr 8.9.0 environment. I am trying to create a collection in async mode and the process fails with 401 Unauthorized errors, which seem to occur during the internode requests (according to my debug). When I am trying to create the same collection with the EXACT same API request, omitting only the async flag, it works flawlessly. It seems to be similar to the issue observed here: https://issues.apache.org/jira/browse/SOLR-13510
I have tried configuring the and playing around with the forwardCredentials property in the security.json file, but to no avail. The request API I use is: http://1.2.3.4:8080/solr/admin/collections?action=CREATE&name=documents&numShards=16&replicationFactor=1&collection.configName=documents&maxShardsPerNode=16&createNodeSet=BL003-Solr1:8080_solr&wt=xml&async=123 (It should be noted that this has worked just fine previously, in Solr 7.3.1). I am seeing the below error reoccurring in the log: 2021-08-02 16:23:09.245 WARN (parallelCoreAdminExecutor-21-thread-16-processing-n:BL003-Solr1:8080_solr x:documents_shard6_replica_n10 2218689320227981 CREATE) [c:documents s:shard6 r:core_node13 x:documents_shard6_replica_n10] o.a.s.c.SolrConfig solrconfig.xml: <jmx> is no longer supported, use solr.xml:/metrics/reporter section instead 2021-08-02 16:23:09.266 ERROR (OverseerThreadFactory-29-thread-1-processing-n:BL003-Solr1:8080_solr) [ ] o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard: http://BL003-Solr1:8080/solr => org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at null: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Unauthorized</title> </head> <body><h2>HTTP ERROR 401 Unauthorized</h2> <table> <tr><th>URI:</th><td>/solr/admin/cores</td></tr> <tr><th>STATUS:</th><td>401</td></tr> <tr><th>MESSAGE:</th><td>Unauthorized</td></tr> <tr><th>SERVLET:</th><td>default</td></tr> </table> </body> </html> at org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:705) org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at null: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Unauthorized</title> </head> <body><h2>HTTP ERROR 401 Unauthorized</h2> <table> <tr><th>URI:</th><td>/solr/admin/cores</td></tr> <tr><th>STATUS:</th><td>401</td></tr> <tr><th>MESSAGE:</th><td>Unauthorized</td></tr> <tr><th>SERVLET:</th><td>default</td></tr> </table> </body> </html> at org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:705) ~[?:?] at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:421) ~[?:?] at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776) ~[?:?] at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[?:?] at org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:80) ~[?:?] at org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:130) ~[?:?] at org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:41) ~[?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_231] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_231] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_231] at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) ~[metrics-core-4.1.5.jar:4.1.5] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_231] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_231] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231] 2021-08-02 16:23:09.266 ERROR (OverseerThreadFactory-29-thread-1-processing-n:BL003-Solr1:8080_solr) [ ] o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard BL003-Solr1:8080_solr: {STATUS=failed} Some background information: We use custom plugins in this Solr environment (but these do not interfere with the collection creation process), we have a custom security.jar file with 2 custom Authentication and Authorization classes, which again have not been changed since Solr 7.3.1 and worked just fine. (It should be noted that during my debug session I pass through these classes just fine during the internode communications). I would be glad for any assistance. Thank you, Michael Erlikhman ________________________________ This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.
