Siyao Meng created HDDS-17267:
---------------------------------
Summary: Datanode drops container CLOSE actions from heartbeats
that SCM did not process, so a full container can stay OPEN after an SCM restart
Key: HDDS-17267
URL: https://issues.apache.org/jira/browse/HDDS-17267
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-2-put-back-container-actions-not-processed-by-scm.patch,
TestBugMC2ContainerActionsNotPutBack.java
h3. Mechanism
{{HeartbeatEndpointTask.call}} takes the queued container actions for the SCM
out of the per SCM queue ({{StateContext.getPendingContainerAction}} polls
them) when it builds the heartbeat. Nothing returns them when SCM does not
process that heartbeat:
* *The RPC fails with an IOException.* {{putBackIncrementalReports}} returns
only command status reports and incremental container reports to the queue. The
container actions are dropped.
* *SCM does not know the datanode*, which is the state of an SCM that has just
restarted. {{SCMDatanodeHeartbeatDispatcher.dispatch}} replies with a
re-register command and fires no event for the payload (intended since
HDDS-601). The RPC succeeded, so the datanode puts nothing back.
The only producer of a CLOSE container action is the write path
({{HddsDispatcher.sendCloseContainerActionIfNeeded}}), which raises it again on
each later write to the container. If no write reaches the container after the
action was dropped, SCM never receives it. Pipeline actions are not affected:
the only pipeline action is CLOSE, and {{StateContext.PipelineActionMap}} keeps
it queued until the pipeline is removed from the datanode.
h3. Trigger
An SCM restart while a container is being filled past the datanode close
threshold, with the last write to that container landing in one of two windows:
# while SCM is down: the heartbeats that fail drain the action (IOException
path);
# after SCM is back but before the datanode's first heartbeat to it: that
heartbeat gets the re-register reply.
After re-registration the full container report gives SCM the real size of the
container, and {{ContainerStateManagerImpl.findContainerWithSpace}} only places
a block in a container while used bytes plus block size fit in the container
size. Once the container is past that point no block, and so no write, goes
there again. With default sizes (5 GB container, 256 MB block, close threshold
0.9) the datanode asks to close from 4.5 GB and SCM stops placing blocks above
4.75 GB, so the container stays OPEN only if its last write left it above about
4.75 GB; below that SCM may place another block there and the next write raises
CLOSE again (argued from the code, not run with default sizes).
With SCM HA the datanode keeps a separate queue per SCM and only the leader
acts on a CLOSE action ({{CloseContainerEventHandler}} skips it on followers).
A restarted follower therefore loses nothing that matters. The loss matters
with a single SCM, or when the restarted SCM is the leader again by the time
the datanode re-registers, for example after all SCMs were restarted (argued,
not run with SCM HA).
The IOException path alone, without a restart (for example one heartbeat RPC
timeout), is usually hidden: SCM learns the size of an open container only from
container reports, so it keeps placing blocks there and the next write raises
CLOSE again. It can strand the container only if writes stop until the next
full container report shows the container past the point above (argued, not
run).
h3. Impact
The container stays OPEN in SCM until its pipeline is closed or an
administrator closes it. While it is OPEN:
* {{DeletedBlockLogImpl}} skips delete transactions for open containers, so the
space of deleted keys in it is not reclaimed;
* ReplicationManager stops at {{OpenContainerHandler}}, which only closes an
open container that has no healthy pipeline or a replica in another state;
* the balancer does not move it and the datanode data scanner does not scan it.
No data is lost and writes continue in other containers.
h3. Reproduction
[^TestBugMC2ContainerActionsNotPutBack.java] ({{hadoop-hdds/server-scm}}),
deterministic, unmodified source. It connects a real {{StateContext}},
{{EndpointStateMachine}} and {{HeartbeatEndpointTask}} to a real
{{SCMNodeManager}} and {{SCMDatanodeHeartbeatDispatcher}} through a mocked RPC
proxy that builds the reply like {{SCMDatanodeProtocolServer.sendHeartbeat}}.
An SCM restart is a new node manager and dispatcher. The first test shows a
CLOSE action sent to a registered SCM arriving, then a CLOSE action in the
heartbeat to the restarted SCM never arriving, also after re-registration and
three more heartbeats. The second shows the same after one heartbeat that fails
with an IOException. A passing test means the defect is present.
A mini cluster run (one datanode, RATIS/ONE, container size equal to block size
of 1 MB, 20 s heartbeat interval, public client API, real SCM restart, not
attached) showed the end state for both windows: with the last write after the
restart, and with the last write while SCM was stopped, the full container was
still OPEN in SCM 80 s after re-registration while SCM placed the next key in a
new container. Without a restart, and with the write after re-registration, the
same container went to CLOSING within seconds. The second window (last write
after the restart) depends on restarting SCM right after a heartbeat so that
the datanode has not yet contacted the new SCM.
h3. Patch
[^MC-2-put-back-container-actions-not-processed-by-scm.patch], against
b10a7d427bdf83807a7e48a1be507d3b02d5a503. It also applies to master at
e0de543cbe9.
{{HeartbeatEndpointTask}} returns the container actions of the heartbeat to the
queue of that SCM when the RPC fails with an IOException and when the reply
contains a re-register command. The new
{{StateContext.putBackContainerActions}} adds them only to that SCM's queue and
skips an action that is already queued again, like
{{addContainerActionIfAbsent}}. The datanode then sends the action in its next
heartbeat, after re-registration in the second case. Incremental and command
status reports are left as they are on the re-register path: the full container
report sent with the registration supersedes the incremental reports, and
putting them back there would send them after it (compare HDDS-17062). A CLOSE
action that SCM did process before the RPC failed is sent again, and so is one
in a heartbeat that Recon processed and answered with a re-register command
(Recon does that for a datanode it has not heard from recently, including after
its own restart); SCM already receives repeated CLOSE actions for the same
container from repeated writes, and {{CloseContainerEventHandler}} does not
change the state of a container that is no longer OPEN. While an SCM or Recon
is unreachable the actions now stay queued for it instead of being drained, at
most one per container and reason.
Covered by {{testContainerActionsPutBackOnHeartbeatFailure}} (which also checks
that the queue of a second SCM is left unchanged) and
{{testContainerActionsPutBackOnReregisterCommand}} in the existing
{{TestHeartbeatEndpointTask}}, and {{testheartbeatWithContainerActions}} now
also checks that a processed heartbeat leaves nothing queued. Without the
change both new tests fail. With it {{TestHeartbeatEndpointTask}},
{{TestStateContext}}, {{TestHddsDispatcher}}, {{TestEndPoint}} and
{{TestSCMDatanodeHeartbeatDispatcher}} pass (64 tests) and checkstyle is clean.
With the patch both tests of the reproduction fail, because SCM receives the
CLOSE action after re-registration and after the failed heartbeat, and in the
mini cluster runs above the container went to CLOSING in both windows.
Containers already left OPEN are not repaired by the patch; {{ozone admin
container close}} closes them.
Found by TLA+ model checking and code review of the datanode registration and
heartbeat paths under HDDS-15926, on commit
b10a7d427bdf83807a7e48a1be507d3b02d5a503. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-16723 (incremental reports queued for a re-registering SCM
make the heartbeat too large), HDDS-17062 (incremental reports put back after a
failed heartbeat are sent after the newer full report) and HDDS-17064 (a full
container report in a failed heartbeat is not resent). The attached patch is a
proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
