mazhengxuan created HBASE-30344:
-----------------------------------
Summary: [Flaky Test] TestSyncReplicationStandbyKillMaster may
verify before RegionServer sync state converges
Key: HBASE-30344
URL: https://issues.apache.org/jira/browse/HBASE-30344
Project: HBase
Issue Type: Bug
Components: Replication, test
Environment: Apache HBase master
GitHub Actions:
JDK17 Hadoop3 Unit Check, large-wave-3
Failed workflow job:
https://github.com/apache/hbase/actions/runs/33056801754/job/98468827602
Reporter: mazhengxuan
Assignee: mazhengxuan
TestSyncReplicationStandbyKillMaster is still flaky on master after HBASE-30249.
It failed in the JDK17 Hadoop3 Unit Check for apache/hbase#8483:
https://github.com/apache/hbase/actions/runs/33056801754/job/98468827602
The first run failed in
TestSyncReplicationStandbyKillMaster.testStandbyKillMaster. The test waits for
the peer state reported by Admin to become DOWNGRADE_ACTIVE and then
immediately verifies the replicated data:
await().atMost(Duration.ofMinutes(3))
.untilAsserted(() -> assertEquals(
SyncReplicationState.DOWNGRADE_ACTIVE,
UTIL2.getAdmin().getReplicationPeerSyncReplicationState(PEER_ID)));
verify(UTIL2, 0, COUNT);
The verification request was rejected by the RegionServer because its local
sync replication state was still STANDBY:
org.apache.hadoop.hbase.DoNotRetryIOException:
SyncRep,... is in STANDBY state.
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.rejectIfInStandByState(...)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(...)
The Admin result reflects the peer state on the Master side. The RegionServer
learns about this state asynchronously, so observing DOWNGRADE_ACTIVE through
Admin does not mean that every RegionServer has already applied the transition.
The Surefire retries then failed during setup with:
1. java.io.IOException: Shutting down
2. java.lang.IllegalStateException: A mini-cluster is already running
These look like follow-up failures caused by the cluster cleanup after the
first run failed.
HBASE-30249 made the Master failure injection deterministic, but the test still
does not wait for the RegionServer-side state transition before calling
verify().
The test should wait, with a timeout, until the RegionServer serving the table
no longer rejects requests because of the STANDBY state. This should be handled
as test synchronization unless further investigation shows that the
RegionServer state never converges.
The failure can be reproduced with Surefire retries disabled:
mvn -pl hbase-server -am \
-Dtest=TestSyncReplicationStandbyKillMaster \
-Dsurefire.failIfNoSpecifiedTests=false \
-Dsurefire.rerunFailingTestsCount=0 \
-DskipITs \
test
Related:
* HBASE-30249
* https://github.com/apache/hbase/pull/8393
* https://github.com/apache/hbase/pull/8483
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
