rjgoyln opened a new pull request, #11877: URL: https://github.com/apache/ozone/pull/11877
## What changes are proposed in this pull request? SCM skips node health checks after a long gap between checker runs (HDDS-1454), so that an SCM pause such as a long GC does not mark live datanodes stale. Three cases still get through: - Only one 3s checker run is skipped, but datanodes heartbeat every 30s, so one whose heartbeat was lost during the pause cannot report in time. - A pause just under the stale interval is not detected, although it can already make a datanode that heartbeats on time look stale. - The pause check and the health deadlines read the clock separately, so a pause between the two readings is missed. The datanode is then marked STALE, which closes its pipelines, or after a long enough pause DEAD, which also closes its containers and removes its replicas. This patch changes `NodeStateManager` to: - read the clock once per run for the pause check, the deadlines and the next run's gap - treat a gap of at least the stale interval minus one heartbeat interval as a pause - hold off checks after a pause for one heartbeat interval plus the heartbeat RPC timeout (35s by default) There is no new configuration. After a pause, a datanode that really died is marked STALE about 35s after SCM resumes instead of 3s. Recon runs the same checker with SCM's heartbeat interval although datanodes heartbeat it every 60s, so for Recon this narrows the gaps without closing them. ## What is the link to the Apache JIRA? https://issues.apache.org/jira/browse/HDDS-17519 ## How was this patch tested? - `testScmHandleJvmPause` now covers a pause just under and at the stale interval; both fail on master, and it passed 20 repeated local runs. - New `testPauseDuringHealthCheckRunDoesNotMarkNodesStale` advances the clock inside one run; it fails on master with the node STALE. - The JIRA's reproductions, which assert the bug, pass on master and fail with this patch; their controls pass on both. - SCM node package tests (177), checkstyle and PMD pass. Generated-by: Claude Code (Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
