Hi, thanks for looking into this. In my case, solr is running with basic auth and without forwardCredentials (using the default). No SSL enabled and no customisations to HttpClient creation. (aside from "-Dsolr.httpclient.config=/var/solr/data/basicAuth.properties" to pass the basic auth credentials).
Regards, Endika On Thu, 24 Oct 2024 at 21:21, Jason Gerlowski <gerlowsk...@gmail.com> wrote: > Hi guys, > > Thanks for sharing your experience; sorry you've run into trouble here! > > I'm going to take a look at reproducing if I can - can you share some > details about your cluster setup? > > - what authc/authz plugins are enabled on your cluster? If basicAuth > is in use (as the stack suggests), is "forwardCredentials" setup? > - is SSL/TLS enabled on these clusters? If so, can you share the > controlling sysprops? > - have you made any customizations to Jetty HttpClient creation? (Solr > exposes sysprop-based hooks for influencing HttpClient settings) > > Best, > > Jason > > > On Thu, Oct 24, 2024 at 12:51 PM Endika Posadas <endikas...@gmail.com> > wrote: > > > > Hi all, > > > > Unfortunately, I am also getting the same error. In my case, I am running > > Solr 9.4.1 and upgrading to 9.7.0. When a node gets upgraded to 9.7.0 and > > tries to recover from the leader, the above errors happen. Going back to > > 9.4.1 allows them to recover, but I am now stuck in an old version. Any > > help would be appreciated. > > > > Thanks, > > > > Endika > > > > On Wed, 16 Oct 2024 at 15:08, Patrik Peng <patrik.p...@hostpoint.ch > .invalid> > > wrote: > > > > > Hi all > > > > > > After updating one of our Solrcloud instances to 9.7.0, the according > > > replicas are unable to recover after service startup. > > > The following exception is logged repeatedly: > > > > > > 2024-09-18 09:36:31.238 ERROR > (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr > dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) > [c:dovecot_fts s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:] > o.a.s.c.RecoveryStrategy Error while trying to recover. > core=dovecot_fts_shard5_replica_n61 => java.lang.NullPointerException: > Cannot invoke > "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)" > because "this.authenticationStore" is null > > > at > org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318) > > > java.lang.NullPointerException: Cannot invoke > "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)" > because "this.authenticationStore" is null > > > at > org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318) > ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > anshum - 2024-09-03 15:05:20] > > > at > org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:97) > ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > anshum - 2024-09-03 15:05:20] > > > at > org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:85) > ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > anshum - 2024-09-03 15:05:20] > > > at > org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.httpClientBuilderSetup(Http2SolrClient.java:1093) > ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > anshum - 2024-09-03 15:05:20] > > > at > org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1062) > ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > anshum - 2024-09-03 15:05:20] > > > at > org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:907) > ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > anshum - 2024-09-03 15:05:20] > > > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:633) > ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > anshum - 2024-09-03 15:05:20] > > > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333) > ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > anshum - 2024-09-03 15:05:20] > > > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309) > ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > anshum - 2024-09-03 15:05:20] > > > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:212) > ~[metrics-core-4.2.26.jar:4.2.26] > > > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) > ~[?:?] > > > at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] > > > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449) > ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > anshum - 2024-09-03 15:05:20] > > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > ~[?:?] > > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > ~[?:?] > > > at java.base/java.lang.Thread.run(Thread.java:840) [?:?] > > > 2024-09-18 09:36:31.238 ERROR > (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr > dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) > [c:dovecot_fts s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:] > o.a.s.c.RecoveryStrategy Recovery failed - trying again... (0) > > > 2024-09-18 09:36:31.238 INFO > (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr > dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) > [c:dovecot_fts s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:] > o.a.s.c.RecoveryStrategy Wait [4] seconds before trying to recover again > (attempt=1) > > > > > > Reverting back to 9.6.1 allows the replicas to recover again. > > > > > > Is this an issue when running 9.7.0 alongside 9.6.1 instances or does > > > anyone have a clue? > > > > > > > > > Regards, > > > Patrik > > > >
