David Smiley created SOLR-18544:
-----------------------------------
Summary: PlacementPluginIntegrationTest.testAttributeFetcherImpl
fails intermittently: jvm_system_cpu_utilization absent when getCpuLoad()
returns NaN
Key: SOLR-18544
URL: https://issues.apache.org/jira/browse/SOLR-18544
Project: Solr
Issue Type: Bug
Components: Tests, metrics
Affects Versions: 10.1
Reporter: David Smiley
testAttributeFetcherImpl fails with "systemLoadAverage 2". It failed during the
10.1.0 RC smoke test on macOS and reproduces on branch_10_1 with:
{noformat}
./gradlew :solr:core:test --tests
"org.apache.solr.cluster.placement.impl.PlacementPluginIntegrationTest.testAttributeFetcherImpl"
-Ptests.seed=FE4CEBEDABF758B2
{noformat}
*Cause*
NodeMetricImpl.SYSLOAD_AVG maps to jvm_system_cpu_utilization. That metric is
emitted by OTel runtime-telemetry's JMX SystemCpu gauge, which only records a
value when OperatingSystemMXBean.getCpuLoad() is >= 0. On macOS, getCpuLoad()
returns NaN when it is called again before the host CPU tick counters advance.
A standalone probe showed:
* 199,999 of 200,000 back-to-back calls returned NaN
* 3 threads calling concurrently (like 3 nodes in one test JVM scraping
metrics): 595 of 600 calls returned NaN
So any node whose scrape happens right after another call reports no
jvm_system_cpu_utilization, and the assertion fails. Other consumers of this
metric (CPUCircuitBreaker, SolrJ NodeValueFetcher) read the same gauge. I have
not checked whether Linux can return NaN this way.
PR #4711 (metric rename) addressed earlier failures of this test, but not this
case.
*Possible fixes*
* Test: tolerate an absent SYSLOAD_AVG, or retry the fetch until present.
* Production: have OtelRuntimeJvmMetrics prefer the JFR source
(OverallCpuLoadHandler, jdk.CPULoad event) for jvm.system.cpu.utilization. It
always records a value (updated every 1s, initially 0).
🤖 This issue was drafted with AI assistance (Claude Code).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
