Siyao Meng created HDDS-17217:
---------------------------------
Summary: A failed pipeline close at the start of decommission or
maintenance is never retried, so the datanode waits for its pipeline
indefinitely
Key: HDDS-17217
URL: https://issues.apache.org/jira/browse/HDDS-17217
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-11-retry-pipeline-close-in-admin-monitor.patch,
TestBugCR11PipelineCloseNotRetried.java
h3. Mechanism
{{DatanodeAdminMonitorImpl}} fires {{START_ADMIN_ON_NODE}} once, when it starts
tracking the node. {{StartDatanodeAdminHandler}} calls
{{PipelineManager.closePipeline}} once for each pipeline of the node, and an
{{IOException}} is logged at INFO without its cause. After that the monitor
only waits in {{checkPipelinesClosedOnNode}} until {{NodeManager.getPipelines}}
is empty. No other SCM path closes an OPEN Ratis pipeline whose datanodes are
healthy: the scrubber closes only ALLOCATED, CLOSED, and OPEN pipelines with
unregistered nodes, and datanodes ask for a close only when the Ratis group
fails. (An EC pipeline is closed by the EC container provider once its
container is closed or full.)
h3. Trigger
# An operator decommissions a healthy datanode that is in an OPEN Ratis
pipeline (or puts it into maintenance).
# In {{closePipeline}}, the container FINALIZE in
{{closeContainersForPipeline}} is a Ratis request. If it times out on an SCM
that stays leader, {{ScmInvoker}} throws {{SCMException(TIMEOUT)}} before the
pipeline state write is submitted, and the handler swallows it.
# The pipeline stays OPEN and the node stays DECOMMISSIONING. Running
{{decommission}} again is a no-op for a node already decommissioning.
A {{NotLeaderException}} does not cause this, because the new leader starts
tracking the node again. The node recovers only on a leader change, a
re-registration or address update of the datanode, the node going stale or
dead, a manual {{ozone admin pipeline close}}, or recommission followed by
decommission after a monitor run. The Ratis timeout itself was not produced in
a test; the reproduction injects the exception {{ScmInvoker}} builds for one.
h3. Impact
* Decommission, and maintenance without an end, never complete while the leader
stays. The monitor logs "Waiting for pipelines to close" on every run. With an
end time, maintenance ends when the end passes: the node goes back to
IN_SERVICE without ever entering maintenance.
* The pipeline stays OPEN on a node that is being drained. Container allocation
picks OPEN pipelines without looking at the op state of their nodes, so it can
keep receiving writes (argued from code, not run).
h3. Reproduction
PASS on the unmodified source, deterministic.
[^TestBugCR11PipelineCloseNotRetried.java] ({{hadoop-hdds/server-scm}})
connects the real {{StartDatanodeAdminHandler}} and
{{DatanodeAdminMonitorImpl}} through a real {{EventQueue}}, as
{{StorageContainerManager}} does, using the {{SimpleMockNodeManager}} of
{{TestDatanodeAdminMonitor}}. {{PipelineManager}} is a mock whose first
{{closePipeline}} throws {{SCMException(TIMEOUT)}}, and whose successful call
removes the pipeline from the node in place of the close and the later scrubber
delete. Output on the pin:
{noformat}
after 20 monitor runs: state=DECOMMISSIONING closePipeline calls=1 pipelines on
node=[Pipeline-bf82742c-2c85-47c8-93bc-561a9fe20066] tracked=1
after 3 monitor runs: state=DECOMMISSIONED closePipeline calls=1 pipelines on
node=null tracked=0
{noformat}
The second line is the control without the failure. A passing test means the
defect is present. With the patch the first test fails, because the close is
retried on the next run and the node becomes DECOMMISSIONED, and the control
passes.
h3. Patch
[^CR-11-retry-pipeline-close-in-admin-monitor.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
20d95ed37fe.
{{START_ADMIN_ON_NODE}} is now fired from {{checkPipelinesClosedOnNode}} on
every monitor run in which a DECOMMISSIONING or ENTERING_MAINTENANCE node still
has pipelines, instead of once from {{startTrackingNode}}, so a failed close is
retried at the next run. A pipeline closed normally stays on the node until the
scrubber deletes it (minutes with the default timeouts). For that reason
{{StartDatanodeAdminHandler}} now skips pipelines that are already CLOSED, and
logs its INFO line only for a pipeline it actually closes. A failure is logged
at WARN with the exception. The event is no longer fired for a node that starts
tracking already IN_MAINTENANCE, or for a node without pipelines. Neither can
have an OPEN pipeline in the normal flow.
Covered by {{testFailedPipelineCloseIsRetried}} in the existing
{{TestDatanodeAdminMonitor}}. It runs the real handler with a pipeline manager
whose first close fails, and checks that the close is retried once, not
repeated for the CLOSED pipeline, and that decommission completes once the
pipeline is removed. Without the change it fails with {{Wanted 2 times ... But
was 1 time}}. With it {{TestDatanodeAdminMonitor}},
{{TestNodeDecommissionManager}}, {{TestNodeDecommissionMetrics}} and
{{TestDeadNodeHandler}} (46 tests) pass, {{TestDecommissionAndMaintenance}} (9
tests, mini cluster) passes with this patch alone, and checkstyle is clean.
HDDS-16905 shows the same symptom from another cause (a deleted pipeline left
in the datanode to pipeline map). This patch does not fix it, because there the
pipeline manager does not know the pipeline. HDDS-16906 is the same one shot
close pattern in {{StaleNodeHandler}} and {{DeadNodeHandler}}.
Found by code review of the SCM datanode decommission and maintenance admin
paths, as part of the TLA+ verification effort under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
