Siyao Meng created HDDS-17266:
---------------------------------
Summary: Reconcile container command is not resent when it is lost
before the datanode runs it (lost heartbeat reply, SCM leader change, datanode
restart)
Key: HDDS-17266
URL: https://issues.apache.org/jira/browse/HDDS-17266
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: TestBugMC3LostHeartbeatReplyDropsCommand.java
h3. Mechanism
{{CommandQueue.getCommand}} removes the commands queued for a datanode when
{{SCMNodeManager.processHeartbeat}} builds the heartbeat response, and
{{SCMDatanodeProtocolServer.sendHeartbeat}} returns them in the reply. Nothing
acknowledges the reply and SCM keeps no copy. If the reply is lost after SCM
processed the heartbeat, the datanode gets an {{IOException}}, does not resend
that heartbeat, and the commands exist nowhere. Hadoop RPC does not resend the
call either.
Delivery is therefore at most once, and the other command types are built for
that: their issuers repair a loss on their own. From reading the code, not
tested:
* DeleteBlocks: the SENT timeout in {{SCMDeletedBlockTransactionStatusManager}}.
* Replicate, delete container, EC reconstruct: ReplicationManager pending op
expiry and re-evaluation.
* Close container: sent again while the container is CLOSING.
* Pipelines: {{BackgroundPipelineScrubber}} and {{PipelineReportHandler}}.
* Operational state, finalize and reregister: regenerated from every heartbeat.
Refresh volume usage: sent on every balancer iteration.
{{ReconcileContainerCommand}} is the exception.
{{SCMClientProtocolServer.reconcileContainer}} fires the event,
{{ReconcileContainerEventHandler}} queues one command per replica, and nothing
tracks or resends them. This is unchanged on master at e0de543cbe9.
h3. Trigger
An admin runs {{ozone admin container reconcile}}, and the command for one
replica is lost before that datanode runs it:
* the heartbeat reply carrying it is lost after SCM built it: the connection is
reset, or SCM answers after the heartbeat RPC timeout
({{ozone.scm.heartbeat.rpc-timeout}}, default 5s).
* the SCM leader changes first. The per datanode {{CommandQueue}} is in the old
leader's memory only, so the new leader has no copy; if the old leader hands
the command out on a later heartbeat, it carries the old term and the datanode
drops it once it knows the new term. A reconcile that already reached the
datanode runs through {{ReplicationSupervisor}} ({{ReconcileContainerTask}}
takes the command's term), which skips queued tasks whose term is lower than
the current leader term ("Ignoring ... since SCM leader has new term"). With a
replication backlog on the datanode, a failover long after the admin command
still discards it.
* the datanode restarts before it runs the command, or its command queue is
full.
For every other command type the new leader's ReplicationManager or report
handlers send the command again after a failover.
h3. Impact
Reconciliation runs per replica: each replica repairs itself from its peers. If
the replica that lost its command is one that needed repair, it stays
unrepaired. The admin command reports that reconciliation was triggered and
points to {{ozone admin container reconcile --status}}, which then keeps
showing that replica's checksum differing from its peers; re-running the
command repairs it. If the replica that lost its command was a healthy one, the
others still reconcile against it and the result converges. No data is lost or
changed. Not exercised: the full admin command on a cluster, the failover
routes (code reading only), and the datanode not reconciling afterwards (it
learns commands only from heartbeat replies).
h3. Reproduction
[^TestBugMC3LostHeartbeatReplyDropsCommand.java] ({{hadoop-hdds/server-scm}},
target path and Maven command in the class comment) passes on unmodified source
when the bug is present. It starts a real SCM with {{HddsTestUtils.getScm}} and
a datanode client built by the real {{SCMConnectionManager}}, connected through
a TCP relay in the test. A reconcile command is queued with the
{{DATANODE_COMMAND}} event that {{ReconcileContainerEventHandler}} fires, and
the relay loses the reply to the next heartbeat, by resetting the connection or
by swallowing it until the client times out. In both cases SCM holds no copy
afterwards and five later heartbeats carry no command. The datanode RPC server
is started explicitly because the HA stub of the test has no Ratis log whose
catch up starts it. The failover routes are not reproduced.
h3. Suggested fix
The fix belongs with the issuer, as for the other command types. The container
reconciliation design doc already plans this for Phase II (HDDS-12987):
"Automate container reconciliation requests as part of SCM's replication
manager". If ReplicationManager triggers reconciliation on a checksum mismatch,
a lost command gets the same repair as every other command type. Any issuer
side tracking has to survive an SCM failover, so it must be re-derivable on the
new leader from replica state (as ReplicationManager state is) or persisted;
tracking kept only in the leader's memory would not cover the failover routes.
There is no completion signal to stop on today (checksums that still differ can
also mean that reconciliation ran and could not repair), so this needs a
decision on what counts as done and how often to retry. Acknowledging heartbeat
replies and redelivering in SCM would be a protocol change for every command
type and would make non-idempotent commands at least once. Documenting that a
reconcile may need to be re-run is the minimal alternative.
Found by TLA+ model checking and code review of the datanode registration and
heartbeat paths under HDDS-15926, on commit
b10a7d427bdf83807a7e48a1be507d3b02d5a503. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-10372 (added the reconcile command), HDDS-13086 (blocks
duplicate reconcile requests on the datanode), HDDS-10714 (restrict
reconciliation requests by datanode status), HDDS-8882 (status tracking for
DeleteBlocks) and HDDS-17064 (a full container report lost on a failed
heartbeat, the other direction). Only the reproduction is attached, there is no
fix patch. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
