[
https://issues.apache.org/jira/browse/HDDS-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129270#comment-18129270
]
Siyao Meng commented on HDDS-17180:
-----------------------------------
Root cause: the removed follower only learns it was removed when its election
timeout fires and the pre-vote returns NOT_IN_CONF. Ratis
{{FollowerState.runImpl}} sleeps a full random election timeout (5000 to 5200
ms in this test) per iteration, and the last appendEntries from the leader
usually arrives during the first sleep, so the check only passes on the second
iteration. Detection therefore takes up to about 10.4 s plus the close, which
overruns the 10 s {{waitFor}} added in HDDS-16922.
>From the failed run's log:
{noformat}
10:17:45,497 SCMRatisServerImpl - Successfully removed SCM: follower.
10:17:55,628 FollowerState - follower@group-81FCC4A722A8-FollowerState: change
to CANDIDATE, lastRpcElapsedTime:10137ms, electionTimeout:5130ms
10:17:55,631 LeaderElection - PRE_VOTE round 0: result NOT_IN_CONF (term=2)
10:17:55,635 SCMStateMachine - Terminating with exit status 0: SCM is
decommissioned, terminate SCM
{noformat}
Locally, {{testRemoveSCM}} took 10.2 to 10.4 s in each of 3 runs, so it is
right at the limit.
Suggested fix: raise the wait in {{TestSCMHAManagerImpl#testRemoveSCM}} well
above that bound, e.g.
{code:java}
GenericTestUtils.waitFor(ExitUtils::isTerminated, 100, 30_000);
{code}
With that change the class passed 3/3 locally.
- https://github.com/apache/ozone/actions/runs/39403744911/job/118571825900
> Interrmittent timeout in TestSCMHAManagerImpl#testRemoveSCM
> -----------------------------------------------------------
>
> Key: HDDS-17180
> URL: https://issues.apache.org/jira/browse/HDDS-17180
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test
> Affects Versions: 2.3.0
> Reporter: Attila Doroszlai
> Assignee: Andy Huang
> Priority: Major
>
> {code}
> [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.65
> s <<< FAILURE! -- in org.apache.hadoop.hdds.scm.ha.TestSCMHAManagerImpl
> [ERROR] org.apache.hadoop.hdds.scm.ha.TestSCMHAManagerImpl.testRemoveSCM --
> Time elapsed: 10.18 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> ...
> at
> app//org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
> at
> app//org.apache.hadoop.hdds.scm.ha.TestSCMHAManagerImpl.testRemoveSCM(TestSCMHAManagerImpl.java:158)
> {code}
> -
> https://github.com/adoroszlai/ozone-build-results/blob/master/2026/10/19/55352/integration-hdds/hadoop-hdds/server-scm/org.apache.hadoop.hdds.scm.ha.TestSCMHAManagerImpl.txt
> -
> https://github.com/adoroszlai/ozone-build-results/blob/master/2026/10/20/55380/integration-hdds/hadoop-hdds/server-scm/org.apache.hadoop.hdds.scm.ha.TestSCMHAManagerImpl.txt
> -
> https://github.com/adoroszlai/ozone-build-results/blob/master/2026/10/20/55384/integration-hdds/hadoop-hdds/server-scm/org.apache.hadoop.hdds.scm.ha.TestSCMHAManagerImpl.txt
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
