This error has happened again. Does anyone yet have any explanation or
suggestion?
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C] <craig.oak...@nih.gov.INVALID>
Sent: Friday, August 05, 2022 10:47 PM
To: users@solr.apache.org
Subject: RE: IllegalArgumentException: Unknown directory
This error has happened again. Does anyone yet have any explanation or
suggestion?
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C]
Sent: Monday, May 02, 2022 2:29 PM
To: users@solr.apache.org
Subject: Re: IllegalArgumentException: Unknown directory
This has happened several more times, and I have notice something else which
might be a clue:
The problem happened over the weekend; and the first erroneous complaint about
/data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
(as though it did not exist) occurred 17 minutes after
/data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/replication.properties
was updated. Last month, the problem occurred just 40 seconds after
replication.properties was updated.
What known connections might there be between replication.properties and
snapshot_metadata?
-----Original Message-----
From: matthew sporleder <msporle...@gmail.com>
Sent: Monday, March 28, 2022 9:41 AM
To: users@solr.apache.org
Subject: Re: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and are confident the
content is safe.
My only other guess (and I mean wild guess) is that because the error is
for a *lock* that it is actually a timeout or similar locking error with a
bad error message.
On Fri, Mar 25, 2022 at 11:11 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<craig.oak...@nih.gov.invalid> wrote:
> Thanks for the quick reply
>
> grep -i -c OutOfMemory solr.log.20220* shows zero
>
> Nothing new in /var/log/dmesg since reboot of the host a couple weeks ago
>
> Let me know if you have any other suggestions
>
> Thanks again
>
> -----Original Message-----
> From: matthew sporleder <msporle...@gmail.com>
> Sent: Friday, March 25, 2022 11:00 AM
> To: users@solr.apache.org
> Subject: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
>
>
> Is there an OOM anywhere in that log? I've definitely seen java lose track
> of things during a slow-moving oom.
>
> Also definitely check dmesg for anything in that same timeframe if you
> still have the logs.
>
> On Fri, Mar 25, 2022 at 10:58 AM Oakley, Craig (NIH/NLM/NCBI) [C]
> <craig.oak...@nih.gov.invalid> wrote:
>
> > I have a core which ceased responding either to select or to admin/core:
> > restarting the Solr instance resolved the problem, but I am wondering
> > whether there is some configuration which may need to be tweaked. Below
> is
> > a portion of solr.log from the time when the problem began. Please note
> > that the directory
> >
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > does indeed exist
> >
> > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.S.Request
> > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > hits=94 status=0 QTime=2610
> > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:47:58.527 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:47:58.531 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.r.SolrJmxReporter Closing reporter
> > [org.apache.solr.metrics.reporters.SolrJmxReporter@2b7f5d5e: rootName =
> > null, domain = solr.core.run_sel_cache.shard1.replica_n3, service url =
> > null, agent id = null] for registry
> > solr.core.run_sel_cache.shard1.replica_n3 /
> > com.codahale.metrics.MetricRegistry@5211d667<mailto:
> > com.codahale.metrics.MetricRegistry@5211d667>
> > 2022-03-25 08:47:58.568 INFO (qtp1847637306-38090) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.S.Request
> > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > hits=94 status=0 QTime=593
> > 2022-03-25 08:47:58.593 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:47:58.596 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ...
> SKIPPED
> > (unnecessary).
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:47:59.258 WARN (qtp1847637306-38404) [ ]
> > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > listeners, but a live one attempted to unregister!
> > 2022-03-25 08:47:59.259 ERROR (qtp1847637306-38404) [ ]
> o.a.s.c.SolrCore
> > java.lang.IllegalArgumentException: Unknown directory:
> > NRTCachingDirectory(MMapDirectory@
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92<mailto:
> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
> > maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
> > at
> >
> org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
> > at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
> > at
> org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> > at
> > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
> > at
> >
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
> > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> > at
> >
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at
> >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at org.eclipse.jetty.server.Server.handle(Server.java:500)
> > at
> >
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> > at
> > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> > at
> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> > at
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> > at org.eclipse.jetty.io
> > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> > at org.eclipse.jetty.io
> > .FillInterest.fillable(FillInterest.java:103)
> > at org.eclipse.jetty.io
> > .ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> > at
> >
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2022-03-25 08:47:59.260 INFO (qtp1847637306-38404) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> > params={action=STATUS&indexInfo=false&wt=json} status=0 QTime=3
> > 2022-03-25 08:47:59.262 INFO (qtp1847637306-37771) [ ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus
> with
> > params action=clusterstatus&wt=json and sendToOCPQueue=true
> > 2022-03-25 08:47:59.275 INFO (qtp1847637306-37771) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> > params={action=clusterstatus&wt=json} status=0 QTime=13
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:48:09.579 WARN (qtp1847637306-38348) [ ]
> > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > listeners, but a live one attempted to unregister!
> >
>
