Hey,
In continuation to this,
I have downloaded Solr 8.9.0 source code and copie the changes from SOLR-15575
over, created a jar and tried the flow again.
Unfortunately, the fix doesn't help.
I tried debugging the code, and no breakpoints are being hit in the
"processAndWait" function inside CollectionAdminRequest.
In the Solr log I am getting the same exception, despite that,
I tried creating a collection while debugging and I believe there are
additional insights:
Below is the exception that I have managed to log while the collection occurs:
Exception 'sun.nio.fs.WindowsException' occurred in thread
'parallelCoreAdminExecutor-21-thread-5-processing-n:BL003-SOLR1:8080_solr
x:documents_shard1_replica_n1 11814894494572756 CREATE' at
sun.nio.fs.WindowsNativeDispatcher.CreateFile(WindowsNativeDispatcher.java:79)
Breakpoint reached
at
sun.nio.fs.WindowsNativeDispatcher.CreateFile(WindowsNativeDispatcher.java:79)
at sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:302)
at
sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:162)
at
sun.nio.fs.WindowsFileSystemProvider.checkReadAccess(WindowsFileSystemProvider.java:325)
at
sun.nio.fs.WindowsFileSystemProvider.checkAccess(WindowsFileSystemProvider.java:362)
at java.nio.file.Files.exists(Files.java:2385)
at
org.apache.solr.core.CoreDescriptor.loadExtraProperties(CoreDescriptor.java:235)
at org.apache.solr.core.CoreDescriptor.<init>(CoreDescriptor.java:209)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1287)
at
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:95)
at
org.apache.solr.handler.admin.CoreAdminOperation$$Lambda$271.1612539426.execute(Unknown
Source:-1)
at
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:367)
at
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
at
org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:189)
at
org.apache.solr.handler.admin.CoreAdminHandler$$Lambda$401.860752875.run(Unknown
Source:-1)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$81.1696975231.run(Unknown
Source:-1)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exception
'org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException'
occurred in thread
'httpShardExecutor-7-thread-5-processing-n:BL003-SOLR1:8080_solr
http:////BL003-SOLR1:8080//solr [http:////BL003-SOLR1:8080//solr]' at
org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:771)
Breakpoint reached
at
org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:771)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:421)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at
org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:80)
at
org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:130)
at
org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:41)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$81.1696975231.run(Unknown
Source:-1)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exception 'com.lmax.disruptor.TimeoutException' occurred in thread
'Log4j2-TF-3-AsyncLoggerConfig-1' at
com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:47)
Breakpoint reached
at
com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:47)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
at
com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.lang.Thread.run(Thread.java:748)
Exception 'sun.nio.fs.WindowsException' occurred in thread
'parallelCoreAdminExecutor-21-thread-5-processing-n:BL003-SOLR1:8080_solr
x:documents_shard1_replica_n1 11814894494572756 CREATE' at
sun.nio.fs.WindowsNativeDispatcher.FindFirstFile(WindowsNativeDispatcher.java:185)
Breakpoint reached
I can see the collection is indeed being created, but the cores aren't.
I suspect the 401 unauthorized issue I am seeing in the log is not the actual
issue I am facing.
Is there any chance you could offer additional support?
Thanks,
Michael
-----Original Message-----
From: Timothy Potter <thelabd...@gmail.com>
Sent: Tuesday, August 3, 2021 6:38 PM
To: users@solr.apache.org
Subject: Re: Getting 401 Unauthorized HTTP errors when trying to create
collection with async flag
WARNING - External email, exercise caution.
I tried reproducing this with a checkout of 8.9.0 to no avail but found another
issue (SOLR-15575). From the stack trace you supplied, the PKI Authentication
Plugin should be supplying the internal server identity for the ShardRequestor
call and in my env (once I fixed
15575) my quick hack test case passed.
Personally, I would disable forwardCredentials and rely on the PKI
Authentication mechanism for this request.
On Mon, Aug 2, 2021 at 8:19 AM Michael Erlikhman
<merlikh...@varonis.com.invalid> wrote:
>
> Hello,
> I am seeing a weird behavior with my Solr 8.9.0 environment.
> I am trying to create a collection in async mode and the process fails with
> 401 Unauthorized errors, which seem to occur during the internode requests
> (according to my debug).
> When I am trying to create the same collection with the EXACT same API
> request, omitting only the async flag, it works flawlessly.
> It seems to be similar to the issue observed here:
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SOLR
> -13510__;!!PowQjtoeD4s!1H_PKCBPhrPvttTvEXSkneCj2JsaQ9dt2l510ejPKUQWa87
> E8cm1L9jGwB91F795$
>
> I have tried configuring the and playing around with the forwardCredentials
> property in the security.json file, but to no avail.
>
> The request API I use is:
> https://urldefense.com/v3/__http://1.2.3.4:8080/solr/admin/collections
> ?action=CREATE&name=documents&numShards=16&replicationFactor=1&collect
> ion.configName=documents&maxShardsPerNode=16&createNodeSet=BL003-Solr1
> :8080_solr&wt=xml&async=123__;!!PowQjtoeD4s!1H_PKCBPhrPvttTvEXSkneCj2J
> saQ9dt2l510ejPKUQWa87E8cm1L9jGwBOXH-8D$
> (It should be noted that this has worked just fine previously, in Solr 7.3.1).
>
> I am seeing the below error reoccurring in the log:
>
> 2021-08-02 16:23:09.245 WARN
> (parallelCoreAdminExecutor-21-thread-16-processing-n:BL003-Solr1:8080_solr
> x:documents_shard6_replica_n10 2218689320227981 CREATE) [c:documents s:shard6
> r:core_node13 x:documents_shard6_replica_n10] o.a.s.c.SolrConfig
> solrconfig.xml: <jmx> is no longer supported, use solr.xml:/metrics/reporter
> section instead
> 2021-08-02 16:23:09.266 ERROR
> (OverseerThreadFactory-29-thread-1-processing-n:BL003-Solr1:8080_solr) [ ]
> o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard:
> https://urldefense.com/v3/__http://BL003-Solr1:8080/solr__;!!PowQjtoeD4s!1H_PKCBPhrPvttTvEXSkneCj2JsaQ9dt2l510ejPKUQWa87E8cm1L9jGwJQa-ZUm$
> =>
> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
> Error from server at null: Expected mime type application/octet-stream but
> got text/html. <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
> <title>Error 401 Unauthorized</title>
> </head>
> <body><h2>HTTP ERROR 401 Unauthorized</h2> <table>
> <tr><th>URI:</th><td>/solr/admin/cores</td></tr>
> <tr><th>STATUS:</th><td>401</td></tr>
> <tr><th>MESSAGE:</th><td>Unauthorized</td></tr>
> <tr><th>SERVLET:</th><td>default</td></tr>
> </table>
>
> </body>
> </html>
>
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResp
> onse(Http2SolrClient.java:705)
> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrExcepti
> on: Error from server at null: Expected mime type
> application/octet-stream but got text/html. <html> <head> <meta
> http-equiv="Content-Type" content="text/html;charset=utf-8"/>
> <title>Error 401 Unauthorized</title>
> </head>
> <body><h2>HTTP ERROR 401 Unauthorized</h2> <table>
> <tr><th>URI:</th><td>/solr/admin/cores</td></tr>
> <tr><th>STATUS:</th><td>401</td></tr>
> <tr><th>MESSAGE:</th><td>Unauthorized</td></tr>
> <tr><th>SERVLET:</th><td>default</td></tr>
> </table>
>
> </body>
> </html>
>
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:705)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:421)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
> ~[?:?]
> at
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[?:?]
> at
> org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:80)
> ~[?:?]
> at
> org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:130)
> ~[?:?]
> at
> org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:41)
> ~[?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_231]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_231]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_231]
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> ~[metrics-core-4.1.5.jar:4.1.5]
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_231]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_231]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
> 2021-08-02 16:23:09.266 ERROR
> (OverseerThreadFactory-29-thread-1-processing-n:BL003-Solr1:8080_solr) [ ]
> o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard
> BL003-Solr1:8080_solr: {STATUS=failed}
>
>
> Some background information:
> We use custom plugins in this Solr environment (but these do not interfere
> with the collection creation process), we have a custom security.jar file
> with 2 custom Authentication and Authorization classes, which again have not
> been changed since Solr 7.3.1 and worked just fine.
> (It should be noted that during my debug session I pass through these classes
> just fine during the internode communications).
>
> I would be glad for any assistance.
>
> Thank you,
> Michael Erlikhman
>
>
> ________________________________
> This email and any attachments thereto may contain private, confidential, and
> privileged material for the sole use of the intended recipient. Any review,
> copying, or distribution of this email (or any attachments thereto) by others
> is strictly prohibited. If you are not the intended recipient, please contact
> the sender immediately and permanently delete the original and any copies of
> this email and any attachments thereto.
________________________________
This email and any attachments thereto may contain private, confidential, and
privileged material for the sole use of the intended recipient. Any review,
copying, or distribution of this email (or any attachments thereto) by others
is strictly prohibited. If you are not the intended recipient, please contact
the sender immediately and permanently delete the original and any copies of
this email and any attachments thereto.
