We have solr8.9 cluster running on 5 nodes. We were in the middle of
patching and rebooting the nodes. Couple of collections failed to recover
after the restart.
Below is the exception in the logs
lica_n23/data/index.20221119040221652
lockFactory=org.apache.lucene.store.NativeFSLockFactory@7342a407
2022-11-19 04:02:21.680 ERROR
(recoveryExecutor-11-thread-1-processing-x:c_facilities_a_shard1_replica_n23
3e118721a6dbea9d r:core_node24 n:10.151.17.82:8085_solr c:c_facilities_a
s:shard1) [c:c_facilities_a s:shar
d1 r:core_node24 x:c_facilities_a_shard1_replica_n23] o.a.s.h.IndexFetcher
Error fetching file, doing one
retry...:org.apache.solr.common.SolrException: Unable to download _ptc9.fdx
completely. Downloaded 0!=910
at
org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1863)
at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1743)
at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1713)
at
org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1109)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:619)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:384)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:458)
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:252)
at
org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:683)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:339)
at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:318)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-11-19 04:02:21.692 ERROR
(recoveryExecutor-11-thread-1-processing-x:c_facilities_a_shard1_replica_n23
3e118721a6dbea9d r:core_node24 n:10.151.17.82:8085_solr c:c_facilities_a
s:shard1) [c:c_facilities_a s:shard1 r:core_node24
x:c_facilities_a_shard1_replica_n23] o.a.s.h.IndexFetcher Error deleting
file: _ptc9.fdx => java.nio.file.NoSuchFileException:
/opt/copart/solr6-data/data/c_facilities_a_shard1_replica_n23/data/index.20221119040221652/_ptc9.fdx
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
java.nio.file.NoSuchFileException:
/opt/copart/solr6-data/data/c_facilities_a_shard1_replica_n23/data/index.20221119040221652/_ptc9.fdx
