Is there an OOM anywhere in that log? I've definitely seen java lose track
of things during a slow-moving oom.
Also definitely check dmesg for anything in that same timeframe if you
still have the logs.
On Fri, Mar 25, 2022 at 10:58 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<craig.oak...@nih.gov.invalid> wrote:
> I have a core which ceased responding either to select or to admin/core:
> restarting the Solr instance resolved the problem, but I am wondering
> whether there is some configuration which may need to be tweaked. Below is
> a portion of solr.log from the time when the problem began. Please note
> that the directory
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> does indeed exist
>
> 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.S.Request
> [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> hits=94 status=0 QTime=2610
> 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.SolrCore
> [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> org.apache.solr.core.SolrCore@59c22833<mailto:
> org.apache.solr.core.SolrCore@59c22833>
> 2022-03-25 08:47:58.527 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> 2022-03-25 08:47:58.531 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.m.r.SolrJmxReporter Closing reporter
> [org.apache.solr.metrics.reporters.SolrJmxReporter@2b7f5d5e: rootName =
> null, domain = solr.core.run_sel_cache.shard1.replica_n3, service url =
> null, agent id = null] for registry
> solr.core.run_sel_cache.shard1.replica_n3 /
> com.codahale.metrics.MetricRegistry@5211d667<mailto:
> com.codahale.metrics.MetricRegistry@5211d667>
> 2022-03-25 08:47:58.568 INFO (qtp1847637306-38090) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.S.Request
> [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> hits=94 status=0 QTime=593
> 2022-03-25 08:47:58.593 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
> 2022-03-25 08:47:58.596 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED
> (unnecessary).
> 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ] o.a.s.c.SolrCore
> [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> org.apache.solr.core.SolrCore@59c22833<mailto:
> org.apache.solr.core.SolrCore@59c22833>
> 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
> 2022-03-25 08:47:59.258 WARN (qtp1847637306-38404) [ ]
> o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> listeners, but a live one attempted to unregister!
> 2022-03-25 08:47:59.259 ERROR (qtp1847637306-38404) [ ] o.a.s.c.SolrCore
> java.lang.IllegalArgumentException: Unknown directory:
> NRTCachingDirectory(MMapDirectory@/data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92<mailto:
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
> maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
> at
> org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
> at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
> at org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
> at
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:500)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at
> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> at org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io
> .FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io
> .ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> at java.lang.Thread.run(Thread.java:748)
>
> 2022-03-25 08:47:59.260 INFO (qtp1847637306-38404) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> params={action=STATUS&indexInfo=false&wt=json} status=0 QTime=3
> 2022-03-25 08:47:59.262 INFO (qtp1847637306-37771) [ ]
> o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with
> params action=clusterstatus&wt=json and sendToOCPQueue=true
> 2022-03-25 08:47:59.275 INFO (qtp1847637306-37771) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> params={action=clusterstatus&wt=json} status=0 QTime=13
> 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ] o.a.s.c.SolrCore
> [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> org.apache.solr.core.SolrCore@59c22833<mailto:
> org.apache.solr.core.SolrCore@59c22833>
> 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
> 2022-03-25 08:48:09.579 WARN (qtp1847637306-38348) [ ]
> o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> listeners, but a live one attempted to unregister!
>
