[
https://issues.apache.org/jira/browse/HDDS-17643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18136274#comment-18136274
]
Sumit Agrawal commented on HDDS-17643:
--------------------------------------
[~smeng]
Container write is only possible over ratis via pipeline. So if we keep *cache
of closed container over the pipeline* till its closed, this can avoid double
open container as above scenario and avoid book keeping of deleted container
over DB.
But not sure about impact of EC pipeline as its having different
implementation, and not sure if there is a similar problem over EC also.
> Pipeline leader recreates a Ratis container replica that SCM force deleted,
> and SCM keeps the new copy over the real replicas
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17643
> URL: https://issues.apache.org/jira/browse/HDDS-17643
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode, SCM
> Affects Versions: 2.2.2
> Reporter: Siyao Meng
> Priority: Critical
>
> h3. Mechanism
> SCM can delete a CLOSED replica on a datanode that is still the Ratis leader
> of the pipeline the container was written on, for example the source of a
> balancer move ({{MoveManager}} sends a forced delete) or an excess replica.
> Pipelines outlive the containers on them, so writes for that container can
> still reach the leader after the delete.
> {{KeyValueHandler}} deletes the replica through
> {{ContainerSet.removeContainer(containerId)}}, which also removes the ID from
> the container ID table (HDDS-11650) and does not add it to the missing
> container set. Nothing on the datanode remembers that the container existed.
> When a WriteChunk for that container then reaches the leader:
> # {{ContainerStateMachine.startTransaction}} calls
> {{HddsDispatcher.validateContainerCommand}}, which returns without any check
> when the container is not in the container set, so the leader accepts the
> write.
> # In the WRITE_DATA stage {{HddsDispatcher.dispatchRequest}} auto creates a
> missing container for WriteChunk and PutBlock. HDDS-15791 added
> {{containerAutoCreate}} to turn this off, but only {{ECBlockOutputStream}}
> sets it to false; {{BlockOutputStream}} for Ratis always allows it. The
> leader creates an empty OPEN replica.
> # The followers still have the container CLOSED and return
> CLOSED_CONTAINER_IO, which {{ContainerStateMachine}} ignores in
> writeStateMachineData and applyTransaction. The entry commits.
> Two outcomes follow:
> * If the block's earlier chunks were written to the deleted replica, PutBlock
> fails with UNABLE_TO_FIND_CHUNK, the new replica becomes UNHEALTHY and the
> leader closes the whole pipeline.
> * If all chunks of a block are written after the recreate, PutBlock succeeds
> on the new replica only. SCM then closes the mismatched OPEN replica, and it
> ends CLOSED with a BCSID higher than the real replicas, but holding only the
> blocks written after the recreate.
> SCM cannot tell the new replica from the real ones and prefers it:
> * {{AbstractContainerReportHandler}} raises the CLOSED container's sequence
> ID to the new replica's BCSID (it only logs an error first, see HDDS-16448).
> * {{RatisContainerReplicaCount}} counts any CLOSED replica as healthy
> regardless of its BCSID.
> * {{RatisOverReplicationHandler}} picks excess CLOSED replicas by
> {{ContainerReplica.hashCode}}, so with four replicas it may delete a real one.
> * {{RatisUnderReplicationHandler}} and the balancer copy from the replica
> with the highest sequence ID, which is the new one.
> Repeated over moves and replica losses, the real replicas are replaced by
> copies of the new one. Reads of existing keys then fail with NO_SUCH_BLOCK.
> Once the blocks written after the recreate are deleted, every replica is
> empty, {{EmptyContainerHandler}} deletes the container, and reads fail with
> NO_REPLICA_FOUND while OM still has the keys.
> HDDS-17435 is the same datanode gap reached through StreamInit replay after a
> restart. This one needs no restart and no data stream.
> h3. Trigger
> * Container balancer enabled (or any other forced or excess replica delete).
> * A container closes and SCM deletes its replica on the pipeline leader while
> the pipeline stays open.
> * A client keeps writing to a block in that container afterwards (a writer
> that allocated the block before the close).
> h3. Observed
> Seen on a production cluster running a release that contains HDDS-11650. One
> datanode logged 103 containers that were deleted and then OPEN again with
> BCSID 0, over about 2.5 months. A typical sequence for container C on leader
> D1 (followers D2, D3):
> {noformat}
> 00:00 C closes on D1, D2, D3 at BCSID N
> 00:04 balancer moves C from D1 to D4, then sends a forced delete to D1
> 00:06 D1: C is OPEN, BCSID 0 (recreated)
> 00:07 D1: GetBlock for an existing block of C returns NO_SUCH_BLOCK
> 00:09 D1: C CLOSED at BCSID N + 824
> {noformat}
> For containers in a window where SCM logs were available, SCM then found four
> replicas and sent a forced delete 7 to 10 minutes after the recreate. In all
> three sampled cases the over replication delete removed a real replica (the
> balancer target, or an original pipeline member) and kept the recreated one.
> In one case all real replicas were eventually removed and the container was
> later deleted as empty while OM still referenced its keys.
> h3. Reproduction
> Not run yet. A mini cluster test should be able to show it: write a key so
> that the pipeline leader holds the container, close the container, force
> delete the replica on the leader with a delete container command while
> keeping the pipeline, then write another block into the same container ID
> through the same pipeline and check that the leader recreates the container
> and that the write succeeds.
> h3. Suggested fix
> Datanode: when a replica is deleted, keep its ID as a tombstone (for example
> in the container ID table with a deleted flag, or in the missing container
> set) instead of removing it. Reject a tombstoned ID with CONTAINER_NOT_FOUND
> both in {{validateContainerCommand}} on the leader and in the auto create
> path of {{dispatchRequest}}, so the client gets an error and allocates a new
> block. Container import and EC RECOVERING create must clear the tombstone.
> This also closes the replay path in HDDS-17435.
> SCM, as defense in depth: do not raise the sequence ID of a CLOSED container
> (HDDS-16448), and treat a CLOSED replica whose BCSID differs from the
> container's sequence ID as mismatched instead of healthy.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
