[
https://issues.apache.org/jira/browse/HDDS-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17217:
------------------------------
Attachment: TestBugCR11PipelineCloseNotRetried.java
> A failed pipeline close at the start of decommission or maintenance is never
> retried, so the datanode waits for its pipeline indefinitely
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17217
> URL: https://issues.apache.org/jira/browse/HDDS-17217
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: CR-11-retry-pipeline-close-in-admin-monitor.patch,
> TestBugCR11PipelineCloseNotRetried.java
>
>
> h3. Mechanism
> {{DatanodeAdminMonitorImpl}} fires {{START_ADMIN_ON_NODE}} once, when it
> starts tracking the node. {{StartDatanodeAdminHandler}} calls
> {{PipelineManager.closePipeline}} once for each pipeline of the node, and an
> {{IOException}} is logged at INFO without its cause. After that the monitor
> only waits in {{checkPipelinesClosedOnNode}} until
> {{NodeManager.getPipelines}} is empty. No other SCM path closes an OPEN Ratis
> pipeline whose datanodes are healthy: the scrubber closes only ALLOCATED,
> CLOSED, and OPEN pipelines with unregistered nodes, and datanodes ask for a
> close only when the Ratis group fails. (An EC pipeline is closed by the EC
> container provider once its container is closed or full.)
> h3. Trigger
> # An operator decommissions a healthy datanode that is in an OPEN Ratis
> pipeline (or puts it into maintenance).
> # In {{closePipeline}}, the container FINALIZE in
> {{closeContainersForPipeline}} is a Ratis request. If it times out on an SCM
> that stays leader, {{ScmInvoker}} throws {{SCMException(TIMEOUT)}} before the
> pipeline state write is submitted, and the handler swallows it.
> # The pipeline stays OPEN and the node stays DECOMMISSIONING. Running
> {{decommission}} again is a no-op for a node already decommissioning.
> A {{NotLeaderException}} does not cause this, because the new leader starts
> tracking the node again. The node recovers only on a leader change, a
> re-registration or address update of the datanode, the node going stale or
> dead, a manual {{ozone admin pipeline close}}, or recommission followed by
> decommission after a monitor run. The Ratis timeout itself was not produced
> in a test; the reproduction injects the exception {{ScmInvoker}} builds for
> one.
> h3. Impact
> * Decommission, and maintenance without an end, never complete while the
> leader stays. The monitor logs "Waiting for pipelines to close" on every run.
> With an end time, maintenance ends when the end passes: the node goes back to
> IN_SERVICE without ever entering maintenance.
> * The pipeline stays OPEN on a node that is being drained. Container
> allocation picks OPEN pipelines without looking at the op state of their
> nodes, so it can keep receiving writes (argued from code, not run).
> h3. Reproduction
> PASS on the unmodified source, deterministic.
> [^TestBugCR11PipelineCloseNotRetried.java] ({{hadoop-hdds/server-scm}})
> connects the real {{StartDatanodeAdminHandler}} and
> {{DatanodeAdminMonitorImpl}} through a real {{EventQueue}}, as
> {{StorageContainerManager}} does, using the {{SimpleMockNodeManager}} of
> {{TestDatanodeAdminMonitor}}. {{PipelineManager}} is a mock whose first
> {{closePipeline}} throws {{SCMException(TIMEOUT)}}, and whose successful call
> removes the pipeline from the node in place of the close and the later
> scrubber delete. Output on the pin:
> {noformat}
> after 20 monitor runs: state=DECOMMISSIONING closePipeline calls=1 pipelines
> on node=[Pipeline-bf82742c-2c85-47c8-93bc-561a9fe20066] tracked=1
> after 3 monitor runs: state=DECOMMISSIONED closePipeline calls=1 pipelines on
> node=null tracked=0
> {noformat}
> The second line is the control without the failure. A passing test means the
> defect is present. With the patch the first test fails, because the close is
> retried on the next run and the node becomes DECOMMISSIONED, and the control
> passes.
> h3. Patch
> [^CR-11-retry-pipeline-close-in-admin-monitor.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 20d95ed37fe.
> {{START_ADMIN_ON_NODE}} is now fired from {{checkPipelinesClosedOnNode}} on
> every monitor run in which a DECOMMISSIONING or ENTERING_MAINTENANCE node
> still has pipelines, instead of once from {{startTrackingNode}}, so a failed
> close is retried at the next run. A pipeline closed normally stays on the
> node until the scrubber deletes it (minutes with the default timeouts). For
> that reason {{StartDatanodeAdminHandler}} now skips pipelines that are
> already CLOSED, and logs its INFO line only for a pipeline it actually
> closes. A failure is logged at WARN with the exception. The event is no
> longer fired for a node that starts tracking already IN_MAINTENANCE, or for a
> node without pipelines. Neither can have an OPEN pipeline in the normal flow.
> Covered by {{testFailedPipelineCloseIsRetried}} in the existing
> {{TestDatanodeAdminMonitor}}. It runs the real handler with a pipeline
> manager whose first close fails, and checks that the close is retried once,
> not repeated for the CLOSED pipeline, and that decommission completes once
> the pipeline is removed. Without the change it fails with {{Wanted 2 times
> ... But was 1 time}}. With it {{TestDatanodeAdminMonitor}},
> {{TestNodeDecommissionManager}}, {{TestNodeDecommissionMetrics}} and
> {{TestDeadNodeHandler}} (46 tests) pass, {{TestDecommissionAndMaintenance}}
> (9 tests, mini cluster) passes with this patch alone, and checkstyle is clean.
> HDDS-16905 shows the same symptom from another cause (a deleted pipeline left
> in the datanode to pipeline map). This patch does not fix it, because there
> the pipeline manager does not know the pipeline. HDDS-16906 is the same one
> shot close pattern in {{StaleNodeHandler}} and {{DeadNodeHandler}}.
> Found by code review of the SCM datanode decommission and maintenance admin
> paths, as part of the TLA+ verification effort under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
