My only other guess (and I mean wild guess) is that because the error is
for a *lock* that it is actually a timeout or similar locking error with a
bad error message.
On Fri, Mar 25, 2022 at 11:11 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<craig.oak...@nih.gov.invalid> wrote:
> Thanks for the quick reply
>
> grep -i -c OutOfMemory solr.log.20220* shows zero
>
> Nothing new in /var/log/dmesg since reboot of the host a couple weeks ago
>
> Let me know if you have any other suggestions
>
> Thanks again
>
> -----Original Message-----
> From: matthew sporleder <msporle...@gmail.com>
> Sent: Friday, March 25, 2022 11:00 AM
> To: users@solr.apache.org
> Subject: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
>
>
> Is there an OOM anywhere in that log? I've definitely seen java lose track
> of things during a slow-moving oom.
>
> Also definitely check dmesg for anything in that same timeframe if you
> still have the logs.
>
> On Fri, Mar 25, 2022 at 10:58 AM Oakley, Craig (NIH/NLM/NCBI) [C]
> <craig.oak...@nih.gov.invalid> wrote:
>
> > I have a core which ceased responding either to select or to admin/core:
> > restarting the Solr instance resolved the problem, but I am wondering
> > whether there is some configuration which may need to be tweaked. Below
> is
> > a portion of solr.log from the time when the problem began. Please note
> > that the directory
> >
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > does indeed exist
> >
> > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.S.Request
> > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > hits=94 status=0 QTime=2610
> > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:47:58.527 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:47:58.531 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.r.SolrJmxReporter Closing reporter
> > [org.apache.solr.metrics.reporters.SolrJmxReporter@2b7f5d5e: rootName =
> > null, domain = solr.core.run_sel_cache.shard1.replica_n3, service url =
> > null, agent id = null] for registry
> > solr.core.run_sel_cache.shard1.replica_n3 /
> > com.codahale.metrics.MetricRegistry@5211d667<mailto:
> > com.codahale.metrics.MetricRegistry@5211d667>
> > 2022-03-25 08:47:58.568 INFO (qtp1847637306-38090) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.S.Request
> > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > hits=94 status=0 QTime=593
> > 2022-03-25 08:47:58.593 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:47:58.596 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ...
> SKIPPED
> > (unnecessary).
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:47:59.258 WARN (qtp1847637306-38404) [ ]
> > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > listeners, but a live one attempted to unregister!
> > 2022-03-25 08:47:59.259 ERROR (qtp1847637306-38404) [ ]
> o.a.s.c.SolrCore
> > java.lang.IllegalArgumentException: Unknown directory:
> > NRTCachingDirectory(MMapDirectory@
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92<mailto:
> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
> > maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
> > at
> >
> org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
> > at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
> > at
> org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> > at
> > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
> > at
> >
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
> > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> > at
> >
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at
> >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at org.eclipse.jetty.server.Server.handle(Server.java:500)
> > at
> >
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> > at
> > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> > at
> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> > at
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> > at org.eclipse.jetty.io
> > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> > at org.eclipse.jetty.io
> > .FillInterest.fillable(FillInterest.java:103)
> > at org.eclipse.jetty.io
> > .ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> > at
> >
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2022-03-25 08:47:59.260 INFO (qtp1847637306-38404) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> > params={action=STATUS&indexInfo=false&wt=json} status=0 QTime=3
> > 2022-03-25 08:47:59.262 INFO (qtp1847637306-37771) [ ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus
> with
> > params action=clusterstatus&wt=json and sendToOCPQueue=true
> > 2022-03-25 08:47:59.275 INFO (qtp1847637306-37771) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> > params={action=clusterstatus&wt=json} status=0 QTime=13
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:48:09.579 WARN (qtp1847637306-38348) [ ]
> > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > listeners, but a live one attempted to unregister!
> >
>
