[
https://issues.apache.org/jira/browse/HDDS-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17267:
------------------------------
Attachment: MC-2-put-back-container-actions-not-processed-by-scm.patch
> Datanode drops container CLOSE actions from heartbeats that SCM did not
> process, so a full container can stay OPEN after an SCM restart
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17267
> URL: https://issues.apache.org/jira/browse/HDDS-17267
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 0.3.0, 0.4.0, 0.4.1, 0.5.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0,
> 1.2.1, 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-2-put-back-container-actions-not-processed-by-scm.patch,
> TestBugMC2ContainerActionsNotPutBack.java
>
>
> h3. Mechanism
> {{HeartbeatEndpointTask.call}} takes the queued container actions for the SCM
> out of the per SCM queue ({{StateContext.getPendingContainerAction}} polls
> them) when it builds the heartbeat. Nothing returns them when SCM does not
> process that heartbeat:
> * *The RPC fails with an IOException.* {{putBackIncrementalReports}} returns
> only command status reports and incremental container reports to the queue.
> The container actions are dropped.
> * *SCM does not know the datanode*, which is the state of an SCM that has
> just restarted. {{SCMDatanodeHeartbeatDispatcher.dispatch}} replies with a
> re-register command and fires no event for the payload (intended since
> HDDS-601). The RPC succeeded, so the datanode puts nothing back.
> The only producer of a CLOSE container action is the write path
> ({{HddsDispatcher.sendCloseContainerActionIfNeeded}}), which raises it again
> on each later write to the container. If no write reaches the container after
> the action was dropped, SCM never receives it. Pipeline actions are not
> affected: the only pipeline action is CLOSE, and
> {{StateContext.PipelineActionMap}} keeps it queued until the pipeline is
> removed from the datanode.
> h3. Trigger
> An SCM restart while a container is being filled past the datanode close
> threshold, with the last write to that container landing in one of two
> windows:
> # while SCM is down: the heartbeats that fail drain the action (IOException
> path);
> # after SCM is back but before the datanode's first heartbeat to it: that
> heartbeat gets the re-register reply.
> After re-registration the full container report gives SCM the real size of
> the container, and {{ContainerStateManagerImpl.findContainerWithSpace}} only
> places a block in a container while used bytes plus block size fit in the
> container size. Once the container is past that point no block, and so no
> write, goes there again. With default sizes (5 GB container, 256 MB block,
> close threshold 0.9) the datanode asks to close from 4.5 GB and SCM stops
> placing blocks above 4.75 GB, so the container stays OPEN only if its last
> write left it above about 4.75 GB; below that SCM may place another block
> there and the next write raises CLOSE again (argued from the code, not run
> with default sizes).
> With SCM HA the datanode keeps a separate queue per SCM and only the leader
> acts on a CLOSE action ({{CloseContainerEventHandler}} skips it on
> followers). A restarted follower therefore loses nothing that matters. The
> loss matters with a single SCM, or when the restarted SCM is the leader again
> by the time the datanode re-registers, for example after all SCMs were
> restarted (argued, not run with SCM HA).
> The IOException path alone, without a restart (for example one heartbeat RPC
> timeout), is usually hidden: SCM learns the size of an open container only
> from container reports, so it keeps placing blocks there and the next write
> raises CLOSE again. It can strand the container only if writes stop until the
> next full container report shows the container past the point above (argued,
> not run).
> h3. Impact
> The container stays OPEN in SCM until its pipeline is closed or an
> administrator closes it. While it is OPEN:
> * {{DeletedBlockLogImpl}} skips delete transactions for open containers, so
> the space of deleted keys in it is not reclaimed;
> * ReplicationManager stops at {{OpenContainerHandler}}, which only closes an
> open container that has no healthy pipeline or a replica in another state;
> * the balancer does not move it and the datanode data scanner does not scan
> it.
> No data is lost and writes continue in other containers.
> h3. Reproduction
> [^TestBugMC2ContainerActionsNotPutBack.java] ({{hadoop-hdds/server-scm}}),
> deterministic, unmodified source. It connects a real {{StateContext}},
> {{EndpointStateMachine}} and {{HeartbeatEndpointTask}} to a real
> {{SCMNodeManager}} and {{SCMDatanodeHeartbeatDispatcher}} through a mocked
> RPC proxy that builds the reply like
> {{SCMDatanodeProtocolServer.sendHeartbeat}}. An SCM restart is a new node
> manager and dispatcher. The first test shows a CLOSE action sent to a
> registered SCM arriving, then a CLOSE action in the heartbeat to the
> restarted SCM never arriving, also after re-registration and three more
> heartbeats. The second shows the same after one heartbeat that fails with an
> IOException. A passing test means the defect is present.
> A mini cluster run (one datanode, RATIS/ONE, container size equal to block
> size of 1 MB, 20 s heartbeat interval, public client API, real SCM restart,
> not attached) showed the end state for both windows: with the last write
> after the restart, and with the last write while SCM was stopped, the full
> container was still OPEN in SCM 80 s after re-registration while SCM placed
> the next key in a new container. Without a restart, and with the write after
> re-registration, the same container went to CLOSING within seconds. The
> second window (last write after the restart) depends on restarting SCM right
> after a heartbeat so that the datanode has not yet contacted the new SCM.
> h3. Patch
> [^MC-2-put-back-container-actions-not-processed-by-scm.patch], against
> b10a7d427bdf83807a7e48a1be507d3b02d5a503. It also applies to master at
> e0de543cbe9.
> {{HeartbeatEndpointTask}} returns the container actions of the heartbeat to
> the queue of that SCM when the RPC fails with an IOException and when the
> reply contains a re-register command. The new
> {{StateContext.putBackContainerActions}} adds them only to that SCM's queue
> and skips an action that is already queued again, like
> {{addContainerActionIfAbsent}}. The datanode then sends the action in its
> next heartbeat, after re-registration in the second case. Incremental and
> command status reports are left as they are on the re-register path: the full
> container report sent with the registration supersedes the incremental
> reports, and putting them back there would send them after it (compare
> HDDS-17062). A CLOSE action that SCM did process before the RPC failed is
> sent again, and so is one in a heartbeat that Recon processed and answered
> with a re-register command (Recon does that for a datanode it has not heard
> from recently, including after its own restart); SCM already receives
> repeated CLOSE actions for the same container from repeated writes, and
> {{CloseContainerEventHandler}} does not change the state of a container that
> is no longer OPEN. While an SCM or Recon is unreachable the actions now stay
> queued for it instead of being drained, at most one per container and reason.
> Covered by {{testContainerActionsPutBackOnHeartbeatFailure}} (which also
> checks that the queue of a second SCM is left unchanged) and
> {{testContainerActionsPutBackOnReregisterCommand}} in the existing
> {{TestHeartbeatEndpointTask}}, and {{testheartbeatWithContainerActions}} now
> also checks that a processed heartbeat leaves nothing queued. Without the
> change both new tests fail. With it {{TestHeartbeatEndpointTask}},
> {{TestStateContext}}, {{TestHddsDispatcher}}, {{TestEndPoint}} and
> {{TestSCMDatanodeHeartbeatDispatcher}} pass (64 tests) and checkstyle is
> clean. With the patch both tests of the reproduction fail, because SCM
> receives the CLOSE action after re-registration and after the failed
> heartbeat, and in the mini cluster runs above the container went to CLOSING
> in both windows. Containers already left OPEN are not repaired by the patch;
> {{ozone admin container close}} closes them.
> Found by TLA+ model checking and code review of the datanode registration and
> heartbeat paths under HDDS-15926, on commit
> b10a7d427bdf83807a7e48a1be507d3b02d5a503. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-16723 (incremental reports queued for a re-registering SCM
> make the heartbeat too large), HDDS-17062 (incremental reports put back after
> a failed heartbeat are sent after the newer full report) and HDDS-17064 (a
> full container report in a failed heartbeat is not resent). The attached
> patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
