[
https://issues.apache.org/jira/browse/SOLR-18564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18131636#comment-18131636
]
Nick Shanin commented on SOLR-18564:
------------------------------------
A sighting of this failure shape in the Solr 10.1.0 RC3 smoke tests, reported
here because the counts and the trigger match this ticket.
In one of five completed RC3 smoke runs (JDK 21.0.12.1), {{TestTaskManagement}}
failed at teardown with an error shutting down MiniSolrCloudCluster, its
suppressed cause a timeout waiting for the {{closeThreadPool}} to shut down. In
the same run, {{TestTermsQParserPlugin}} was charged at class teardown with 16
unreleased {{SolrMetricsContext}} objects, created through
{{CoreAdminHandler.initializeMetrics}} during {{CoreContainer.loadInternal}}.
That is the signature this ticket describes: the close-pool timeout throw in
{{CoreContainer.shutdown()}} skips the trailing closes, including the root
metrics-context close, and the JVM-global tracker charges the leak to a later
test. I am not claiming the charged test is the source; on this ticket's own
analysis it usually is not. I am recording that the shape, the count of 16, and
the co-occurring close-pool timeout all appeared in the RC3 artifacts.
🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
> CoreContainer.shutdown: a client close pool timeout skips the remaining
> closes and leaks metrics contexts
> ---------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18564
> URL: https://issues.apache.org/jira/browse/SOLR-18564
> Project: Solr
> Issue Type: Test
> Reporter: Nick Shanin
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> 🤖 _AI text below_ 🤖 _(posted on behalf of Nick Shanin)_
> *Summary: when the client close pool in CoreContainer.shutdown() times out,
> its throw skips the handler close and the root metrics context close that
> follow it, so the container's metrics contexts leak and are charged to an
> unrelated test suite.*
> This surfaced as two class teardown failures in one Crave CI run ([PR #5030
> run|https://github.com/apache/solr/actions/runs/39699700526/job/119565807999],
> seed {{A35B8669BA78E261}}). Every test passed. {{CloudHttp2SolrClientTest}}
> failed in {{tearDownAfterClass}} with "Error shutting down
> MiniSolrCloudCluster", and {{TestJavaBinCodec}} was charged by the
> ObjectReleaseTracker with 16 unreleased {{SolrMetricsContext}} objects that
> the first suite's cluster had created. The two failures are one event.
> The mechanism, from the run log and the code: {{CoreContainer.shutdown()}}
> waits on the close pool through {{ExecutorUtil.awaitTermination}}
> ({{CoreContainer.java:1314}}). When the waits expire, the utility force-stops
> the pool and throws {{RuntimeException: Timeout waiting for pool to
> shutdown}}. The throw aborts {{shutdown()}} before
> {{containerHandlers.close()}} (line 1362) and the root metrics context close
> (line 1366), which are the only release calls for those contexts. The leaked
> contexts stay in the JVM-global tracker and land on the next suite in the
> fork whose teardown checks it, which is why the charge appears on a suite
> that created none of them.
> The failure is intermittent in CI (the full solrj suite passed twice locally
> under the same seed), but it reproduces deterministically by blocking a close
> on a latch and letting the real timeout budget run out: on current code that
> fails with the same throw and an ObjectTracker leak report. A fix is prepared
> on [this
> branch|https://github.com/nick-boss-tech/solr/tree/corecontainer-shutdown-closes]:
> catch and log the timeout throw at the call site so the trailing closes
> always run, with a regression test that fails without the fix and passes with
> it. {{ExecutorUtil}} is unchanged; its throw is a contract other callers rely
> on.
> Out of scope here: why the close exceeded its budget under CI load in the
> first place. This ticket covers the skipped closes, not the slowness.
> _AI-generated description. Nick Shanin directed the work and takes
> responsibility for this contribution._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
