[
https://issues.apache.org/jira/browse/SOLR-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127222#comment-18127222
]
Matthew Biscocho commented on SOLR-18544:
-----------------------------------------
My immediate thought was this was due to the OTEL upgrade that was committed
but that was only partially the problem and I couldn't reproduce with that
seed. I was surprised the tests always passed on my end for the OTEL upgrade
but after looking deeper, it actually might also be a JDK bug. Specifically
running the test with 21.0.6-tem fails but running with 21.0.9-tem passes. JVM
metrics initialization changed with the upgrade and the way these metrics are
fetched was previously JFR but then after the upgrade it got fetched with JMX.
There is a setting in OTEL to have it re-fetch from JFR which should be the
fix. I'll try and post a PR soon.
> PlacementPluginIntegrationTest.testAttributeFetcherImpl fails intermittently:
> jvm_system_cpu_utilization absent when getCpuLoad() returns NaN
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18544
> URL: https://issues.apache.org/jira/browse/SOLR-18544
> Project: Solr
> Issue Type: Bug
> Components: metrics, Tests
> Affects Versions: 10.0, 10.1
> Reporter: David Smiley
> Priority: Major
> Labels: flaky-test
>
> testAttributeFetcherImpl fails with "systemLoadAverage 2". It failed during
> the 10.1.0 RC smoke test on macOS and reproduces on branch_10_1 with:
> {noformat}
> ./gradlew :solr:core:test --tests
> "org.apache.solr.cluster.placement.impl.PlacementPluginIntegrationTest.testAttributeFetcherImpl"
> -Ptests.seed=FE4CEBEDABF758B2
> {noformat}
> *Cause*
> NodeMetricImpl.SYSLOAD_AVG maps to jvm_system_cpu_utilization. That metric is
> emitted by OTel runtime-telemetry's JMX SystemCpu gauge, which only records a
> value when OperatingSystemMXBean.getCpuLoad() is >= 0. On macOS, getCpuLoad()
> returns NaN when it is called again before the host CPU tick counters
> advance. A standalone probe showed:
> * 199,999 of 200,000 back-to-back calls returned NaN
> * 3 threads calling concurrently (like 3 nodes in one test JVM scraping
> metrics): 595 of 600 calls returned NaN
> So any node whose scrape happens right after another call reports no
> jvm_system_cpu_utilization, and the assertion fails. Other consumers of this
> metric (CPUCircuitBreaker, SolrJ NodeValueFetcher) read the same gauge. I
> have not checked whether Linux can return NaN this way.
> PR #4711 (metric rename) addressed earlier failures of this test, but not
> this case.
> *Possible fixes*
> * Test: tolerate an absent SYSLOAD_AVG, or retry the fetch until present.
> * Production: have OtelRuntimeJvmMetrics prefer the JFR source
> (OverallCpuLoadHandler, jdk.CPULoad event) for jvm.system.cpu.utilization. It
> always records a value (updated every 1s, initially 0).
> 🤖 This issue was drafted with AI assistance (Claude Code).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
