[
https://issues.apache.org/jira/browse/SOLR-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley updated SOLR-18544:
--------------------------------
Affects Version/s: 10.0
> PlacementPluginIntegrationTest.testAttributeFetcherImpl fails intermittently:
> jvm_system_cpu_utilization absent when getCpuLoad() returns NaN
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18544
> URL: https://issues.apache.org/jira/browse/SOLR-18544
> Project: Solr
> Issue Type: Bug
> Components: metrics, Tests
> Affects Versions: 10.0, 10.1
> Reporter: David Smiley
> Priority: Major
> Labels: flaky-test
>
> testAttributeFetcherImpl fails with "systemLoadAverage 2". It failed during
> the 10.1.0 RC smoke test on macOS and reproduces on branch_10_1 with:
> {noformat}
> ./gradlew :solr:core:test --tests
> "org.apache.solr.cluster.placement.impl.PlacementPluginIntegrationTest.testAttributeFetcherImpl"
> -Ptests.seed=FE4CEBEDABF758B2
> {noformat}
> *Cause*
> NodeMetricImpl.SYSLOAD_AVG maps to jvm_system_cpu_utilization. That metric is
> emitted by OTel runtime-telemetry's JMX SystemCpu gauge, which only records a
> value when OperatingSystemMXBean.getCpuLoad() is >= 0. On macOS, getCpuLoad()
> returns NaN when it is called again before the host CPU tick counters
> advance. A standalone probe showed:
> * 199,999 of 200,000 back-to-back calls returned NaN
> * 3 threads calling concurrently (like 3 nodes in one test JVM scraping
> metrics): 595 of 600 calls returned NaN
> So any node whose scrape happens right after another call reports no
> jvm_system_cpu_utilization, and the assertion fails. Other consumers of this
> metric (CPUCircuitBreaker, SolrJ NodeValueFetcher) read the same gauge. I
> have not checked whether Linux can return NaN this way.
> PR #4711 (metric rename) addressed earlier failures of this test, but not
> this case.
> *Possible fixes*
> * Test: tolerate an absent SYSLOAD_AVG, or retry the fetch until present.
> * Production: have OtelRuntimeJvmMetrics prefer the JFR source
> (OverallCpuLoadHandler, jdk.CPULoad event) for jvm.system.cpu.utilization. It
> always records a value (updated every 1s, initially 0).
> 🤖 This issue was drafted with AI assistance (Claude Code).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
