Hi guys, Thanks for sharing your experience; sorry you've run into trouble here!
I'm going to take a look at reproducing if I can - can you share some details about your cluster setup? - what authc/authz plugins are enabled on your cluster? If basicAuth is in use (as the stack suggests), is "forwardCredentials" setup? - is SSL/TLS enabled on these clusters? If so, can you share the controlling sysprops? - have you made any customizations to Jetty HttpClient creation? (Solr exposes sysprop-based hooks for influencing HttpClient settings) Best, Jason On Thu, Oct 24, 2024 at 12:51 PM Endika Posadas <endikas...@gmail.com> wrote: > > Hi all, > > Unfortunately, I am also getting the same error. In my case, I am running > Solr 9.4.1 and upgrading to 9.7.0. When a node gets upgraded to 9.7.0 and > tries to recover from the leader, the above errors happen. Going back to > 9.4.1 allows them to recover, but I am now stuck in an old version. Any > help would be appreciated. > > Thanks, > > Endika > > On Wed, 16 Oct 2024 at 15:08, Patrik Peng <patrik.p...@hostpoint.ch.invalid> > wrote: > > > Hi all > > > > After updating one of our Solrcloud instances to 9.7.0, the according > > replicas are unable to recover after service startup. > > The following exception is logged repeatedly: > > > > 2024-09-18 09:36:31.238 ERROR > > (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr > > dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) > > [c:dovecot_fts s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:] > > o.a.s.c.RecoveryStrategy Error while trying to recover. > > core=dovecot_fts_shard5_replica_n61 => java.lang.NullPointerException: > > Cannot invoke > > "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)" > > because "this.authenticationStore" is null > > at > > org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318) > > java.lang.NullPointerException: Cannot invoke > > "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)" > > because "this.authenticationStore" is null > > at > > org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318) > > ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > > anshum - 2024-09-03 15:05:20] > > at > > org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:97) > > ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > > anshum - 2024-09-03 15:05:20] > > at > > org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:85) > > ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > > anshum - 2024-09-03 15:05:20] > > at > > org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.httpClientBuilderSetup(Http2SolrClient.java:1093) > > ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > > anshum - 2024-09-03 15:05:20] > > at > > org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1062) > > ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > > anshum - 2024-09-03 15:05:20] > > at > > org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:907) > > ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > > anshum - 2024-09-03 15:05:20] > > at > > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:633) > > ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > > anshum - 2024-09-03 15:05:20] > > at > > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333) > > ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > > anshum - 2024-09-03 15:05:20] > > at > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309) > > ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > > anshum - 2024-09-03 15:05:20] > > at > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:212) > > ~[metrics-core-4.2.26.jar:4.2.26] > > at > > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) > > ~[?:?] > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > ~[?:?] > > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449) > > ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - > > anshum - 2024-09-03 15:05:20] > > at > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > > ~[?:?] > > at > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > > ~[?:?] > > at java.base/java.lang.Thread.run(Thread.java:840) [?:?] > > 2024-09-18 09:36:31.238 ERROR > > (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr > > dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) > > [c:dovecot_fts s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:] > > o.a.s.c.RecoveryStrategy Recovery failed - trying again... (0) > > 2024-09-18 09:36:31.238 INFO > > (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr > > dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) > > [c:dovecot_fts s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:] > > o.a.s.c.RecoveryStrategy Wait [4] seconds before trying to recover again > > (attempt=1) > > > > Reverting back to 9.6.1 allows the replicas to recover again. > > > > Is this an issue when running 9.7.0 alongside 9.6.1 instances or does > > anyone have a clue? > > > > > > Regards, > > Patrik > >
