Hi FYI. Below is the exact error. I don't understand why the solr url is referred to with http in the logs below instead of https, though the SOLR-UI is accessible fine with https and the certs are valid. Please assist.
2024-04-22 11:28:31.190 INFO (qtp1119242936-20-solr_hostname.corp.abc.com-122) [c: s: r: x: t:solr_hostname.corp.abc.com-122] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&_=1713776790070&since=0} status=0 QTime=0 2024-04-22 11:28:50.529 INFO (OverseerThreadFactory-17-thread-3) [c:workdayUserStore s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection workdayUserStore 2024-04-22 11:28:50.658 WARN (OverseerThreadFactory-17-thread-3) [c:workdayUserStore s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Specified number of replicas of 2 on collection workdayUserStore is higher than the number of Solr instances currently live or live and part of your createNodeSet(1). It's unusual to run two replica of the same slice on the same Solr-instance. 2024-04-22 11:28:50.676 INFO (OverseerStateUpdate-216275577626820608-solr_hostname.corp.abc.com:8985_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { "core":"workdayUserStore_shard1_replica_n1", "node_name":"solr_hostname.corp.abc.com:8985_solr", "base_url":"http://solr_hostname.corp.abc.com:8985/solr", "collection":"workdayUserStore", "shard":"shard1", "state":"down", "type":"NRT", "operation":"ADDREPLICA", "waitForFinalState":"false"} 2024-04-22 11:28:50.689 INFO (OverseerStateUpdate-216275577626820608-solr_hostname.corp.abc.com:8985_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { "core":"workdayUserStore_shard1_replica_n2", "node_name":"solr_hostname.corp.abc.com:8985_solr", "base_url":"http://solr_hostname.corp.abc.com:8985/solr", "collection":"workdayUserStore", "shard":"shard1", "state":"down", "type":"NRT", "operation":"ADDREPLICA", "waitForFinalState":"false"} 2024-04-22 11:28:50.801 INFO (zkCallback-11-thread-9) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/workdayUserStore/state.json zxid: 12884902255] for collection [workdayUserStore] has occurred - updating... (live nodes size: [1]) 2024-04-22 11:28:50.810 ERROR (OverseerThreadFactory-17-thread-3) [c:workdayUserStore s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Error from shard: http://solr_hostname.corp.abc.com:8985/solr => org.apache.solr.client.solrj.SolrServerException at org.apache.solr.client.solrj.impl.Http2SolrClient$1.onFailure(Http2SolrClient.java:542) org.apache.solr.client.solrj.SolrServerException: null at org.apache.solr.client.solrj.impl.Http2SolrClient$1.onFailure(Http2SolrClient.java:542) ~[solr-solrj-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.eclipse.jetty.client.ResponseNotifier.notifyFailure(ResponseNotifier.java:192) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.ResponseNotifier.notifyFailure(ResponseNotifier.java:184) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpExchange.notifyFailureComplete(HttpExchange.java:284) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpExchange.abort(HttpExchange.java:256) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpConversation.abort(HttpConversation.java:159) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpRequest.abort(HttpRequest.java:925) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpDestination.abort(HttpDestination.java:530) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpDestination.failed(HttpDestination.java:294) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.AbstractConnectionPool$FutureConnection.failed(AbstractConnectionPool.java:574) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.Promise$Wrapper.failed(Promise.java:169) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpClient$1$1.failed(HttpClient.java:578) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.client.http.HTTPSessionListenerPromise.failConnectionPromise(HTTPSessionListenerPromise.java:136) ~[http2-http-client-transport-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.client.http.HTTPSessionListenerPromise.onFailure(HTTPSessionListenerPromise.java:125) ~[http2-http-client-transport-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.api.Session$Listener.onFailure(Session.java:290) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.notifyFailure(HTTP2Session.java:1222) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.onFailure(HTTP2Session.java:605) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.abort(HTTP2Session.java:590) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:1904) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:1036) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:385) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:155) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:450) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:151) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.invokeAsNonBlocking(AdaptiveExecutionStrategy.java:438) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:380) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) ~[jetty-util-10.0.19.jar:10.0.19] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:299) ~[solr-solrj-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:834) [?:?] Caused by: java.nio.channels.ClosedChannelException at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:1865) ~[http2-common-10.0.19.jar:10.0.19] ... 19 more 2024-04-22 11:28:50.810 ERROR (OverseerThreadFactory-17-thread-3) [c:workdayUserStore s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Error from shard: http://solr_hostname.corp.abc.com:8985/solr => org.apache.solr.client.solrj.SolrServerException at org.apache.solr.client.solrj.impl.Http2SolrClient$1.onFailure(Http2SolrClient.java:542) org.apache.solr.client.solrj.SolrServerException: null at org.apache.solr.client.solrj.impl.Http2SolrClient$1.onFailure(Http2SolrClient.java:542) ~[solr-solrj-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.eclipse.jetty.client.ResponseNotifier.notifyFailure(ResponseNotifier.java:192) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.ResponseNotifier.notifyFailure(ResponseNotifier.java:184) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpExchange.notifyFailureComplete(HttpExchange.java:284) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpExchange.abort(HttpExchange.java:256) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpConversation.abort(HttpConversation.java:159) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpRequest.abort(HttpRequest.java:925) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpDestination.abort(HttpDestination.java:530) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpDestination.failed(HttpDestination.java:294) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.AbstractConnectionPool$FutureConnection.failed(AbstractConnectionPool.java:574) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.Promise$Wrapper.failed(Promise.java:169) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpClient$1$1.failed(HttpClient.java:578) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.client.http.HTTPSessionListenerPromise.failConnectionPromise(HTTPSessionListenerPromise.java:136) ~[http2-http-client-transport-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.client.http.HTTPSessionListenerPromise.onFailure(HTTPSessionListenerPromise.java:125) ~[http2-http-client-transport-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.api.Session$Listener.onFailure(Session.java:290) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.notifyFailure(HTTP2Session.java:1222) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.onFailure(HTTP2Session.java:605) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.abort(HTTP2Session.java:590) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:1904) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:1036) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:385) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:155) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:450) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:151) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.invokeAsNonBlocking(AdaptiveExecutionStrategy.java:438) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:380) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) ~[jetty-util-10.0.19.jar:10.0.19] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:299) ~[solr-solrj-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:834) [?:?] Caused by: java.nio.channels.ClosedChannelException at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:1865) ~[http2-common-10.0.19.jar:10.0.19] ... 19 more 2024-04-22 11:28:50.810 ERROR (OverseerThreadFactory-17-thread-3) [c:workdayUserStore s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Cleaning up collection [workdayUserStore]. 2024-04-22 11:28:50.815 INFO (OverseerThreadFactory-17-thread-3) [c:workdayUserStore s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2024-04-22 11:28:50.819 ERROR (OverseerThreadFactory-17-thread-3) [c:workdayUserStore s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Error from shard: http://solr_hostname.corp.abc.com:8985/solr => org.apache.solr.client.solrj.SolrServerException at org.apache.solr.client.solrj.impl.Http2SolrClient$1.onFailure(Http2SolrClient.java:542) org.apache.solr.client.solrj.SolrServerException: null at org.apache.solr.client.solrj.impl.Http2SolrClient$1.onFailure(Http2SolrClient.java:542) ~[solr-solrj-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.eclipse.jetty.client.ResponseNotifier.notifyFailure(ResponseNotifier.java:192) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.ResponseNotifier.notifyFailure(ResponseNotifier.java:184) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpExchange.notifyFailureComplete(HttpExchange.java:284) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpExchange.abort(HttpExchange.java:256) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpConversation.abort(HttpConversation.java:159) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpRequest.abort(HttpRequest.java:925) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpDestination.abort(HttpDestination.java:530) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpDestination.failed(HttpDestination.java:294) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.AbstractConnectionPool$FutureConnection.failed(AbstractConnectionPool.java:574) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.Promise$Wrapper.failed(Promise.java:169) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpClient$1$1.failed(HttpClient.java:578) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.client.http.HTTPSessionListenerPromise.failConnectionPromise(HTTPSessionListenerPromise.java:136) ~[http2-http-client-transport-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.client.http.HTTPSessionListenerPromise.onFailure(HTTPSessionListenerPromise.java:125) ~[http2-http-client-transport-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.api.Session$Listener.onFailure(Session.java:290) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.notifyFailure(HTTP2Session.java:1222) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.onFailure(HTTP2Session.java:605) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.abort(HTTP2Session.java:590) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:1904) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:1036) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:385) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:155) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:450) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:151) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.invokeAsNonBlocking(AdaptiveExecutionStrategy.java:438) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:380) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) ~[jetty-util-10.0.19.jar:10.0.19] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:299) ~[solr-solrj-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:834) [?:?] Caused by: java.nio.channels.ClosedChannelException at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:1865) ~[http2-common-10.0.19.jar:10.0.19] ... 19 more 2024-04-22 11:28:50.820 ERROR (OverseerThreadFactory-17-thread-3) [c:workdayUserStore s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Error from shard: http://solr_hostname.corp.abc.com:8985/solr => org.apache.solr.client.solrj.SolrServerException at org.apache.solr.client.solrj.impl.Http2SolrClient$1.onFailure(Http2SolrClient.java:542) org.apache.solr.client.solrj.SolrServerException: null at org.apache.solr.client.solrj.impl.Http2SolrClient$1.onFailure(Http2SolrClient.java:542) ~[solr-solrj-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.eclipse.jetty.client.ResponseNotifier.notifyFailure(ResponseNotifier.java:192) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.ResponseNotifier.notifyFailure(ResponseNotifier.java:184) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpExchange.notifyFailureComplete(HttpExchange.java:284) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpExchange.abort(HttpExchange.java:256) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpConversation.abort(HttpConversation.java:159) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpRequest.abort(HttpRequest.java:925) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpDestination.abort(HttpDestination.java:530) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpDestination.failed(HttpDestination.java:294) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.AbstractConnectionPool$FutureConnection.failed(AbstractConnectionPool.java:574) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.Promise$Wrapper.failed(Promise.java:169) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.client.HttpClient$1$1.failed(HttpClient.java:578) ~[jetty-client-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.client.http.HTTPSessionListenerPromise.failConnectionPromise(HTTPSessionListenerPromise.java:136) ~[http2-http-client-transport-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.client.http.HTTPSessionListenerPromise.onFailure(HTTPSessionListenerPromise.java:125) ~[http2-http-client-transport-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.api.Session$Listener.onFailure(Session.java:290) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.notifyFailure(HTTP2Session.java:1222) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.onFailure(HTTP2Session.java:605) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.abort(HTTP2Session.java:590) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:1904) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:1036) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:385) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:155) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:450) ~[http2-common-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:151) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.invokeAsNonBlocking(AdaptiveExecutionStrategy.java:438) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:380) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) ~[jetty-util-10.0.19.jar:10.0.19] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:299) ~[solr-solrj-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:834) [?:?] Caused by: java.nio.channels.ClosedChannelException at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:1865) ~[http2-common-10.0.19.jar:10.0.19] ... 19 more 2024-04-22 11:28:50.932 INFO (zkCallback-11-thread-9) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/workdayUserStore/state.json zxid: 12884902258] for collection [workdayUserStore] has occurred - updating... (live nodes size: [1]) 2024-04-22 11:28:50.932 INFO (zkCallback-11-thread-8) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/workdayUserStore/state.json zxid: 12884902258] for collection [workdayUserStore] has occurred - updating... (live nodes size: [1]) 2024-04-22 11:28:50.945 INFO (OverseerThreadFactory-17-thread-3) [c:workdayUserStore s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [workdayUserStore] 2024-04-22 11:28:50.945 ERROR (OverseerThreadFactory-17-thread-3) [c:workdayUserStore s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Collection workdayUserStore, operation create failed => org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: workdayUserStore at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:447) org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: workdayUserStore at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:447) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.cloud.api.collections.CollApiCmds$TraceAwareCommand.call(CollApiCmds.java:225) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:130) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:564) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:299) ~[solr-solrj-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:834) [?:?] 2024-04-22 11:28:50.964 ERROR (qtp1119242936-20-solr_hostname.corp.abc.com-123) [c:workdayUserStore s: r: x: t:solr_hostname.corp.abc.com-123] o.a.s.h.RequestHandlerBase Client exception => org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: workdayUserStore at org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:56) org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: workdayUserStore at org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:56) ~[solr-solrj-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.handler.admin.api.CreateCollection.createCollection(CreateCollection.java:133) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$0(CollectionsHandler.java:517) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1264) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:319) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:297) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:226) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:932) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:880) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:541) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:262) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:219) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:249) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:215) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.19.jar:10.0.19] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.19.jar:10.0.19] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) ~[jetty-security-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) ~[jetty-rewrite-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) ~[jetty-server-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) ~[jetty-util-10.0.19.jar:10.0.19] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) ~[jetty-util-10.0.19.jar:10.0.19] at java.base/java.lang.Thread.run(Thread.java:834) [?:?] 2024-04-22 11:28:50.964 INFO (qtp1119242936-20-solr_hostname.corp.abc.com-123) [c:workdayUserStore s: r: x: t:solr_hostname.corp.abc.com-123] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&maxShardsPerNode=1&collection.configName=workdayUserStoreConfig&name=workdayUserStore&action=CREATE&numShards=1} status=400 QTime=451 2024-04-22 11:28:52.535 INFO (OverseerCollectionConfigSetProcessor-216275577626820608-solr_hostname.corp.abc.com:8985_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist. Requestor may have disconnected from ZooKeeper On Mon, Apr 22, 2024 at 9:18 AM HariBabu kuruva <hari2708.kur...@gmail.com> wrote: > Hi All, > > Could you please help on the above issue. > > On Wed, Apr 17, 2024 at 2:39 PM HariBabu kuruva <hari2708.kur...@gmail.com> > wrote: > >> Hi Jan, >> >> Thanks for the response. >> >> Please find the details below: >> >> *Exactly how did you install Solr, and what is your ZK_HOST connection >> string?:* >> I got the solr-9.5.0.tg.zip file from Solr website and unzipped it. >> ZK_HOST = "ZK_host1.corp.abc.com:3181,ZK_Host2.corp.abc.com:3182, >> ZK_Host3.corp.equinbcax.com:3183" >> - *Exactly how did you install ZK, and what is your zoo.cfg, on each of >> the three nodes.* >> I have got the zip file and uninstalled it. >> *zoo.cfg:* >> >> *---------------*tickTime=2000 >> initLimit=10 >> syncLimit=5 >> 4lw.commands.whitelist=* >> dataDir=/gdev/zk1/zoo/data/1 >> clientPort=3181 >> autopurge.snapRetainCount=3 >> autopurge.purgeInterval=12 >> server.1=ZKHost1.corp.equinix.com:4888:5888 >> server.2=ZKHost2.corp.equinix.com:4889:5889 >> server.3=ZKHost3.corp.equinix.com:4890:5890 >> admin.serverPort=8081 >> ----------------------------------- >> *- Which command did you use to create the collection?* >> ------- Commands: ------------------------- >> update_command="/gdev/opt/eisp/solr-9.5.0/bin/solr zk -upconfig -z >> $zoo_keeper_host -n $config_name -d >> /gdev/opt/eisp/solr/configsets/$store_name" >> create_command="curl -k --user solr:SolrRocks >> '$solr_admin_url/solr/admin/collections? >> action=CREATE&name=$store_name&numShards=1&replicationFactor=2&maxShardsPerNode=1&collection.configName=$config_name'" >> >> *Note: *Configsets folder , i am using the same files of current running >> version solr-8.11.1 >> --------------------------------------- >> -* Does this happen every time you try to create a collection, even if >> you create a test collection with _default configset from the UI?* >> *Yes, i can see the same error from UI also and also with _default >> configset as well* >> >> *- Did the Solr node crash or recover during this process?* >> No >> *- Are there ERRORs in Zookeeper's log files?* >> No errors in ZK logs. >> >> Please let me know if any other info is required. >> >> >> >> On Wed, Apr 17, 2024 at 12:39 PM Jan Høydahl <jan....@cominvent.com> >> wrote: >> >>> Hi, >>> >>> Do you mind sharing the details with us? >>> >>> - Exactly how did you install Solr, and what is your ZK_HOST connection >>> string? >>> - Exactly how did you install ZK, and what is your zoo.cfg, on each of >>> the three nodes. >>> - Which command did you use to create the collection? >>> - Does this happen every time you try to create a collection, even if >>> you create a test collection with _default configset from the UI? >>> - Did the Solr node crash or recover during this process? >>> - Are there ERRORs in Zookeeper's log files? >>> >>> Jan >>> >>> > 17. apr. 2024 kl. 02:37 skrev HariBabu kuruva < >>> hari2708.kur...@gmail.com>: >>> > >>> > HI , I am working on a POC, to set up solr-9.5.0. Currently I have >>> > Solr-8.11.1 installed and running. >>> > >>> > As part of the POC. I have installed Solr-9.5.0(on one node) and >>> Zookeeper >>> > 3.9.2 (on 3 nodes). This setup is done on the same server where I have >>> > solr-8.5 running but with different ports. >>> > >>> > Solr and Zookeeper are running on separate nodes and are running fine. >>> > >>> > When I try to create the collection using the below command. I get this >>> > error. Please help me. >>> > >>> > >>> > >>> > 2024-04-16 05:17:09.045 ERROR (OverseerThreadFactory-17-thread-5) >>> [c:test1 >>> > s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Collect >>> > ion test1, operation create failed => >>> org.apache.solr.common.SolrException: >>> > Underlying core creation failed while creating collecti >>> > on: test1 >>> > at >>> > >>> org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:447) >>> > org.apache.solr.common.SolrException: Underlying core creation failed >>> while >>> > creating collection: test1 >>> > at >>> > >>> org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:447) >>> > ~[solr-core-9.5.0.jar:9.5.0 >>> > cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 >>> > 15:10:39] >>> > at >>> > >>> org.apache.solr.cloud.api.collections.CollApiCmds$TraceAwareCommand.call(CollApiCmds.java:225) >>> > ~[solr-core-9.5.0.jar:9.5 >>> > .0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 >>> > 15:10:39] >>> > at >>> > >>> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.j >>> > ava:130) ~[solr-core-9.5.0.jar:9.5.0 >>> > cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 >>> > 15:10:39] >>> > at >>> > >>> org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:564) >>> > ~[solr-core-9.5.0.jar:9.5.0 cdd27 >>> > dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 >>> 15:10:39] >>> > at >>> > >>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:299) >>> > ~[solr-solrj >>> > -9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - >>> gerlowskija - >>> > 2024-02-07 15:10:39] >>> > at >>> > >>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>> > ~[?:?] >>> > at >>> > >>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>> > ~[?:?] >>> > at java.base/java.lang.Thread.run(Thread.java:834) [?:?] >>> > 2024-04-16 05:17:09.054 ERROR >>> > (qtp1119242936-23-vmclxeissldi101.corp.equinix.com-50) [c:test1 s: r: >>> x: >>> > t:vmclxeissldi101.corp.equin >>> > ix.com-50] o.a.s.h.RequestHandlerBase Client exception => >>> > org.apache.solr.common.SolrException: Underlying core creation failed >>> whi >>> > le creating collection: test1 >>> > at >>> > >>> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:56) >>> > org.apache.solr.common.SolrException: Underlying core creation failed >>> while >>> > creating collection: test1 >>> > at >>> > >>> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:56) >>> > ~[solr-solrj-9.5.0.jar:9.5.0 cdd27dd15c3a65 >>> > 74032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] >>> > at >>> > >>> org.apache.solr.handler.admin.api.CreateCollection.createCollection(CreateCollection.java:133) >>> > ~[solr-core-9.5.0.jar:9.5 >>> > .0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 >>> > 15:10:39] >>> > at >>> > >>> org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$0(CollectionsHandler.java:517) >>> > ~[solr >>> > -core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - >>> > gerlowskija - 2024-02-07 15:10:39] >>> > at >>> > >>> org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1264) >>> > ~[solr-core-9 >>> > .5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - >>> > 2024-02-07 15:10:39] >>> > at >>> > >>> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:319) >>> > ~[solr-core-9.5.0.jar:9.5.0 c >>> > dd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 >>> 15:10:39] >>> > at >>> > >>> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:297) >>> > ~[solr-core-9.5.0.jar:9. >>> > 5.0 cdd27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 >>> > 15:10:39] >>> > at >>> > >>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:226) >>> > ~[solr-core-9.5.0.jar:9.5.0 cdd27d >>> > d15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] >>> > at >>> > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:932) >>> > ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e >>> > 9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] >>> > at >>> > >>> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:880) >>> > ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6 >>> > 574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] >>> > at >>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:541) >>> > ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a6574032e9b1b92b >>> > 148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] >>> > at >>> > >>> org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:262) >>> > ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3 >>> > a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] >>> > at >>> > >>> org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:219) >>> > ~[solr-core-9.5.0.jar:9.5.0 cd >>> > d27dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 >>> 15:10:39] >>> > at >>> > >>> org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:249) >>> > ~[solr-core-9.5.0.jar:9.5.0 cdd27 >>> > dd15c3a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 >>> 15:10:39] >>> > at >>> > >>> org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:215) >>> > ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3a657 >>> > 4032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] >>> > at >>> > >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213) >>> > ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3 >>> > a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] >>> > at >>> > >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) >>> > ~[solr-core-9.5.0.jar:9.5.0 cdd27dd15c3 >>> > a6574032e9b1b92b148ab4e383599 - gerlowskija - 2024-02-07 15:10:39] >>> > at >>> > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) >>> > ~[jetty-servlet-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) >>> > ~[jetty-servlet-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) >>> > ~[jetty-servlet-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) >>> > ~[jetty-security-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) >>> > ~[jetty-servlet-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) >>> > ~[jetty-server-10.0. >>> > 19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) >>> > ~[jetty-server-10.0.19.jar:10.0.19 >>> > ] >>> > at >>> > >>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) >>> > ~[jetty-server-10.0.19.jar:10.0.19 >>> > ] >>> > at >>> > >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) >>> > ~[jetty-rewrite-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at org.eclipse.jetty.server.Server.handle(Server.java:563) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) >>> > ~[jetty-server-10.0.19.jar:10.0 >>> > .19] >>> > at >>> > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461) >>> > ~[jetty-server-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) >>> > ~[jetty-uti >>> > l-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) >>> > ~[jetty >>> > -util-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) >>> > ~[jetty- >>> > util-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) >>> > ~[jetty-uti >>> > l-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208) >>> > ~[http2-common-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:155) >>> > ~[http2-common-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:450) >>> > ~[http2-common-10.0.19.jar: >>> > 10.0.19] >>> > at >>> > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) >>> > ~[jetty-io-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558) >>> > ~[jetty-io-10.0.19.jar:10.0. >>> > 19] >>> > at >>> > >>> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379) >>> > ~[jetty-io-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146) >>> > ~[jetty-io-10.0.19.jar:10.0.19] >>> > at >>> > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) >>> > ~[jetty-io-10.0.19.jar:10.0.19] >>> > at >>> > org.eclipse.jetty.io >>> .SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) >>> > ~[jetty-io-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) >>> > ~[jetty-uti >>> > l-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) >>> > ~[jetty >>> > -util-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) >>> > ~[jetty- >>> > util-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) >>> > ~[jetty-util-10 >>> > .0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) >>> > ~[jetty-util-10 >>> > .0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) >>> > ~[jetty-util-10.0.19.jar:10.0.19] >>> > at >>> > >>> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) >>> > ~[jetty-util-10.0.19.jar:10.0 >>> > .19] >>> > at >>> > >>> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) >>> > ~[jetty-util-10.0.19.jar:10.0.19] >>> > at java.base/java.lang.Thread.run(Thread.java:834) [?:?] >>> > 2024-04-16 05:17:09.054 INFO >>> > (qtp1119242936-23-vmclxeissldi101.corp.equinix.com-50) [c:test1 s: r: >>> x: >>> > t:vmclxeissldi101.corp.equin >>> > ix.com-50] o.a.s.s.HttpSolrCall [admin] webapp=null >>> path=/admin/collections >>> > params={replicationFactor=1&collection.configName=_defa >>> > ult&name=test1&router.name >>> =compositeId&action=CREATE&numShards=1&wt=json&_=1713244112747} >>> > status=400 QTime=405 >>> > 2024-04-16 05:17:10.667 INFO >>> > ( >>> OverseerCollectionConfigSetProcessor-72062388420345865-vmclxeissldi101.corp.equinix.com:8985 >>> > _solr-n_ >>> > 0000000008) [c: s: r: x: t:] o.a.s.c.OverseerTaskQueue Response ZK >>> path: >>> > /overseer/collection-queue-work/qnr-0000000046 doesn't exi >>> > st. Requestor may have disconnected from ZooKeeper >>> > >>> > -- >>> > >>> > Thanks and Regards, >>> > Hari >>> > Mobile:9790756568 >>> >>> >> >> -- >> >> Thanks and Regards, >> Hari >> Mobile:9790756568 >> > > > -- > > Thanks and Regards, > Hari > Mobile:9790756568 > -- Thanks and Regards, Hari Mobile:9790756568