Hi Shawn,
The solr version I'm running is 9.2.1 and it is not customized (not
compiled from source).
I scanned the whole stack trace for that object reference, but could only
find threads that state that they are waiting for that lock as well. What
exactly is the pattern I have to look for?
The only stack trace that differs from that I posted before is this one
(all other threads waiting for that lock have the "same" stacktrace as it
was the same metrics call that caused their creation):
{
"id": 65,
"name": "coreLoadExecutor-18-thread-1-processing-192.168.0.223:8983_solr",
"state": "WAITING",
"lock": "java.lang.Object@6665e80c",
"lock-waiting": {
"name": "java.lang.Object@6665e80c",
"owner": null
},
"synchronizers-locked": [
"java.util.concurrent.ThreadPoolExecutor$Worker@17d83fe4"
],
"cpuTime": "359.5291ms",
"userTime": "330.0000ms",
"stackTrace": [
"java.base@17.0.6/java.lang.Object.wait(Native Method)",
"java.base@17.0.6/java.lang.Object.wait(Object.java:338)",
"org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2528)",
"org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1283)",
"org.apache.solr.core.SolrCore.<init>(SolrCore.java:1168)",
"org.apache.solr.core.SolrCore.<init>(SolrCore.java:1051)",
"org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1666)",
"org.apache.solr.core.CoreContainer.lambda$load$12(CoreContainer.java:1015)",
"org.apache.solr.core.CoreContainer$$Lambda$545/0x00000008015bf2f0.call(Unknown
Source)",
"com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:234)",
"java.base@17.0.6
/java.util.concurrent.FutureTask.run(FutureTask.java:264)",
"org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)",
"org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$256/0x0000000800e4fd48.run(Unknown
Source)",
"java.base@17.0.6
/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)",
"java.base@17.0.6
/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)",
"java.base@17.0.6/java.lang.Thread.run(Thread.java:833)"
]
}
Is there anything else I can provide?
Thanks,
Michael
On Wed, Jul 12, 2023 at 1:13 AM Shawn Heisey <apa...@elyograg.org> wrote:
> On 7/11/23 06:24, michael dürr wrote:
> > I recently observed that calls to the metrics api get stuck (thread state
> > waiting) when we call it for an index that is built from scratch. This
> > became a problem when we used the solr metrics exporter and unexpectedly
> > ran out of memory due to the accumulating threads.
> >
> > This is one of the waiting thread's stacktrace (collected by the
> > solr/admin/info/threads
> > endpoint):
> >
> > {
> > "id": 28,
> > "name": "qtp2085745483-28",
> > "state": "WAITING",
> > "lock": "java.lang.Object@6665e80c",
>
> This thread is waiting for the lock on 'java.lang.Object@6665e80c' to be
> released. You would need to look through the rest of the threads to see
> what thread is holding that lock, then we need the exact version of Solr
> you're running and the stacktrace of that thread so we can look at the
> source code to see what might be wrong.
>
> If you are running a customized version of Solr that was compiled from
> source, it might be necessary to have access to the customized source code.
>
> Thanks,
> Shawn
>
