Siyao Meng created HDDS-17643:
---------------------------------
Summary: Pipeline leader recreates a Ratis container replica that
SCM force deleted, and SCM keeps the new copy over the real replicas
Key: HDDS-17643
URL: https://issues.apache.org/jira/browse/HDDS-17643
Project: Apache Ozone
Issue Type: Bug
Components: SCM, Ozone Datanode
Affects Versions: 2.2.2
Reporter: Siyao Meng
h3. Mechanism
SCM can delete a CLOSED replica on a datanode that is still the Ratis leader of
the pipeline the container was written on, for example the source of a balancer
move ({{MoveManager}} sends a forced delete) or an excess replica. Pipelines
outlive the containers on them, so writes for that container can still reach
the leader after the delete.
{{KeyValueHandler}} deletes the replica through
{{ContainerSet.removeContainer(containerId)}}, which also removes the ID from
the container ID table (HDDS-11650) and does not add it to the missing
container set. Nothing on the datanode remembers that the container existed.
When a WriteChunk for that container then reaches the leader:
# {{ContainerStateMachine.startTransaction}} calls
{{HddsDispatcher.validateContainerCommand}}, which returns without any check
when the container is not in the container set, so the leader accepts the write.
# In the WRITE_DATA stage {{HddsDispatcher.dispatchRequest}} auto creates a
missing container for WriteChunk and PutBlock. HDDS-15791 added
{{containerAutoCreate}} to turn this off, but only {{ECBlockOutputStream}} sets
it to false; {{BlockOutputStream}} for Ratis always allows it. The leader
creates an empty OPEN replica.
# The followers still have the container CLOSED and return CLOSED_CONTAINER_IO,
which {{ContainerStateMachine}} ignores in writeStateMachineData and
applyTransaction. The entry commits.
Two outcomes follow:
* If the block's earlier chunks were written to the deleted replica, PutBlock
fails with UNABLE_TO_FIND_CHUNK, the new replica becomes UNHEALTHY and the
leader closes the whole pipeline.
* If all chunks of a block are written after the recreate, PutBlock succeeds on
the new replica only. SCM then closes the mismatched OPEN replica, and it ends
CLOSED with a BCSID higher than the real replicas, but holding only the blocks
written after the recreate.
SCM cannot tell the new replica from the real ones and prefers it:
* {{AbstractContainerReportHandler}} raises the CLOSED container's sequence ID
to the new replica's BCSID (it only logs an error first, see HDDS-16448).
* {{RatisContainerReplicaCount}} counts any CLOSED replica as healthy
regardless of its BCSID.
* {{RatisOverReplicationHandler}} picks excess CLOSED replicas by
{{ContainerReplica.hashCode}}, so with four replicas it may delete a real one.
* {{RatisUnderReplicationHandler}} and the balancer copy from the replica with
the highest sequence ID, which is the new one.
Repeated over moves and replica losses, the real replicas are replaced by
copies of the new one. Reads of existing keys then fail with NO_SUCH_BLOCK.
Once the blocks written after the recreate are deleted, every replica is empty,
{{EmptyContainerHandler}} deletes the container, and reads fail with
NO_REPLICA_FOUND while OM still has the keys.
HDDS-17435 is the same datanode gap reached through StreamInit replay after a
restart. This one needs no restart and no data stream.
h3. Trigger
* Container balancer enabled (or any other forced or excess replica delete).
* A container closes and SCM deletes its replica on the pipeline leader while
the pipeline stays open.
* A client keeps writing to a block in that container afterwards (a writer that
allocated the block before the close).
h3. Observed
Seen on a production cluster running a release that contains HDDS-11650. One
datanode logged 103 containers that were deleted and then OPEN again with BCSID
0, over about 2.5 months. A typical sequence for container C on leader D1
(followers D2, D3):
{noformat}
00:00 C closes on D1, D2, D3 at BCSID N
00:04 balancer moves C from D1 to D4, then sends a forced delete to D1
00:06 D1: C is OPEN, BCSID 0 (recreated)
00:07 D1: GetBlock for an existing block of C returns NO_SUCH_BLOCK
00:09 D1: C CLOSED at BCSID N + 824
{noformat}
For containers in a window where SCM logs were available, SCM then found four
replicas and sent a forced delete 7 to 10 minutes after the recreate. In all
three sampled cases the over replication delete removed a real replica (the
balancer target, or an original pipeline member) and kept the recreated one. In
one case all real replicas were eventually removed and the container was later
deleted as empty while OM still referenced its keys.
h3. Reproduction
Not run yet. A mini cluster test should be able to show it: write a key so that
the pipeline leader holds the container, close the container, force delete the
replica on the leader with a delete container command while keeping the
pipeline, then write another block into the same container ID through the same
pipeline and check that the leader recreates the container and that the write
succeeds.
h3. Suggested fix
Datanode: when a replica is deleted, keep its ID as a tombstone (for example in
the container ID table with a deleted flag, or in the missing container set)
instead of removing it. Reject a tombstoned ID with CONTAINER_NOT_FOUND both in
{{validateContainerCommand}} on the leader and in the auto create path of
{{dispatchRequest}}, so the client gets an error and allocates a new block.
Container import and EC RECOVERING create must clear the tombstone. This also
closes the replay path in HDDS-17435.
SCM, as defense in depth: do not raise the sequence ID of a CLOSED container
(HDDS-16448), and treat a CLOSED replica whose BCSID differs from the
container's sequence ID as mismatched instead of healthy.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
