Siyao Meng created HDDS-16780:
---------------------------------
Summary: Intermittent failure in
TestPerVolumePushReplication#testDecommissionWithPerVolumePools
Key: HDDS-16780
URL: https://issues.apache.org/jira/browse/HDDS-16780
Project: Apache Ozone
Issue Type: Sub-task
Components: SCM, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 82.46 s
<<< FAILURE! -- in
org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication
[ERROR]
org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testDecommissionWithPerVolumePools
-- Time elapsed: 31.25 s <<< ERROR!
java.util.concurrent.TimeoutException:
Timed out waiting for condition.
at
org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
at
org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testDecommissionWithPerVolumePools(TestPerVolumePushReplication.java:271)
{code}
SCM log during the decommission (197 occurrences):
{code}
ERROR replication.UnhealthyReplicationProcessor
(UnhealthyReplicationProcessor.java:processAll(125)) - Error processing Health
result of class: class
org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult
...
java.lang.IllegalArgumentException: Capacity cannot be negative.
at
org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat.<init>(SCMNodeStat.java:45)
at
org.apache.hadoop.hdds.scm.node.SCMNodeManager.getNodeStatInternal(SCMNodeManager.java:1172)
at
org.apache.hadoop.hdds.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:1147)
at
org.apache.hadoop.hdds.scm.container.replication.ReplicationManagerUtil.excludeFullNodes(ReplicationManagerUtil.java:245)
at
org.apache.hadoop.hdds.scm.container.replication.ReplicationManagerUtil.getExcludedAndUsedNodes(ReplicationManagerUtil.java:223)
at
org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.processAndSendCommands(ECUnderReplicationHandler.java:130)
{code}
h3. Root cause
The integration test {{ozone-site.xml}} sets
{{hdds.datanode.du.factory.classname}} to {{MockSpaceUsageCheckFactory$None}},
which reports {{Long.MAX_VALUE}} capacity for every volume.
{{TestPerVolumePushReplication}} starts datanodes with 2 data volumes, so
{{SCMNodeManager#getNodeStatInternal}} overflows when it sums the storage
reports of a node, and {{SCMNodeStat}} throws {{Capacity cannot be negative}}.
{{ReplicationManagerUtil#excludeFullNodes}} calls {{getNodeStat}} for every
datanode with a pending replica ADD. So while any replication is in flight,
processing of every other under replicated container fails, and the
decommission effectively replicates one container per ReplicationManager cycle
(about 3s). The decommissioning node held 12 containers; it had reached 9 of
them when the 30s {{waitForDnToReachOpState}} wait expired.
h3. Fix
Use {{MockSpaceUsageCheckFactory$HalfTera}} (fixed 512 GB per volume) in this
test, so the per node sum stays in range.
- https://github.com/apache/ozone/actions/runs/37686266652/job/113019002301
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
