[
https://issues.apache.org/jira/browse/HDDS-17190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17190:
------------------------------
Status: Patch Available (was: Open)
> Intermittent failure in
> TestDecommissionAndMaintenance#testSingleNodeWithOpenPipelineCanGotoMaintenance
> -------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17190
> URL: https://issues.apache.org/jira/browse/HDDS-17190
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: SCM, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> [ERROR] Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 320.6
> s <<< FAILURE! -- in
> org.apache.hadoop.hdds.scm.node.TestDecommissionAndMaintenance
> [ERROR]
> org.apache.hadoop.hdds.scm.node.TestDecommissionAndMaintenance.testSingleNodeWithOpenPipelineCanGotoMaintenance
> -- Time elapsed: 59.78 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition.
> at
> org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
> at
> org.apache.hadoop.hdds.scm.node.NodeTestUtil.waitForDnToReachPersistedOpState(NodeTestUtil.java:98)
> at
> org.apache.hadoop.hdds.scm.node.TestDecommissionAndMaintenance.testSingleNodeWithOpenPipelineCanGotoMaintenance(TestDecommissionAndMaintenance.java:487)
> {code}
> h3. Root cause
> {{waitForDnToReachPersistedOpState}} polls {{getPersistedOpState()}} on the
> {{DatanodeDetails}} object it is given. The test passes {{dn}}, captured from
> {{nm.getNode(dnID)}} before the datanode restarts. When a restarted datanode
> re-registers with a different port set, {{SCMNodeManager#register}} replaces
> the stored {{DatanodeInfo}} (the {{portsChanged}} path added by HDDS-12991).
> In the failing run the HTTP port was missing from the initial registration
> and present on re-registration:
> {noformat}
> SCMNodeManager - Updating ports for registered datanode ca3c16ee-...:
> [STANDALONE=15350, ..., CLIENT_RPC=15349] -> [STANDALONE=15350, ...,
> HTTP=15348, CLIENT_RPC=15349]
> {noformat}
> After that SCM updates the persisted state on the new object (the log shows
> {{IN_MAINTENANCE -> IN_SERVICE}}), while the test keeps polling the stale
> {{dn}} until it times out. The neighbouring assertions already use {{newDn}}.
> {{testNodeWithOpenPipelineCanBeDecommissionedAndRecommissioned}} has the same
> stale reference ({{toDecommission}}) after a restart.
> h3. Fix
> Look up the node from {{NodeManager}} after the restart ({{newDn}} /
> {{nm.getNode(dnID)}}) before waiting for the persisted state.
> - https://github.com/apache/ozone/actions/runs/39445520577/job/118801747190
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
