Siyao Meng created HDDS-16755:
---------------------------------
Summary: Uncommitted block cleanup can delete a block that a
snapshot of an hsync'ed key still references
Key: HDDS-16755
URL: https://issues.apache.org/jira/browse/HDDS-16755
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Manager, Snapshot
Reporter: Siyao Meng
Assignee: Siyao Meng
h3. Mechanism
When a key commit leaves out a block that was allocated to the open key,
{{OMKeyRequest.wrapUncommittedBlocksAsPseudoKey}} ({{OMKeyRequest.java:1227}})
wraps the block in a pseudo key with object ID {{OBJECT_ID_RECLAIM_BLOCKS}} (0)
and the commit puts it in {{deletedTable}}. The marker means "reclaim
unconditionally", on the assumption that such a block was never visible to a
snapshot.
That assumption does not hold for an hsync'ed key. An hsync publishes the block
list to {{keyTable}}, so a snapshot taken after the hsync lists those blocks.
If a later hsync or the final commit leaves one of them out, it is released as
an uncommitted block when the key is closed.
{{ReclaimableKeyFilter.getPreviousSnapshotKeyInfo}}
({{ReclaimableKeyFilter.java:166}}) treats an object ID mismatch with the
previous snapshot's key as "not the same key" and returns reclaimable. The
pseudo key always mismatches, so {{KeyDeletingService}} sends the block to SCM
for deletion while a snapshot still references it.
h3. Trigger
# A writer hsyncs key {{k}} with blocks {{[A, B]}}.
# Snapshot {{S1}} is created. {{S1}} lists {{k}} with {{[A, B]}}.
# The writer allocates block {{C}} and commits {{k}} with {{[A, C]}}.
# The commit finds {{B}} uncommitted and writes a {{deletedTable}} row for it
with object ID 0.
# {{KeyDeletingService}} reclaims the row. {{B}} is deleted from datanodes and
reads of {{k}} through {{S1}} fail.
Variant with two snapshots: between steps 2 and 3 the writer hsyncs {{[A, C]}}
and snapshot {{S2}} is created. The previous snapshot {{S2}} then lists {{[A,
C]}} and shares no block with the pseudo key, while {{S1}} still references
{{B}}.
The same helper is used by {{OMKeyCommitRequestWithFSO}} ({{:314}}) and
{{S3MultipartUploadCommitPartRequest}} ({{:305}}).
h3. Impact
Data loss for snapshot reads. The active key is not affected.
Reproduced on master with a {{TestKeyDeletingService}} case that issues
{{openKey}}, {{allocateBlock}}, {{hsyncKey}}, {{createSnapshot}} and
{{commitKey}} against OM directly. Whether an unmodified Ozone client can drop
a block it has already hsync'ed has not been established, so this is currently
an OM side hazard: OM accepts the commit and deletes snapshot data.
h3. Proposed fix
The pseudo key must keep both the object ID and the hsync metadata of the key.
* Object ID: stop overwriting it with 0, so the filter finds the key's version
in the previous snapshot.
* Hsync metadata: the pseudo key is already built before {{HSYNC_CLIENT_ID}} is
removed ({{OMKeyCommitRequest.java:380}} and {{:394}},
{{OMKeyCommitRequestWithFSO.java:314}} and {{:338}}). That ordering must be
kept and documented at the helper.
With both, {{SnapshotUtils.isBlockLocationInfoSame}}
({{SnapshotUtils.java:303}}) retains the row while the previous snapshot holds
an hsync'ed version of the key with the same object ID. This also covers the
two snapshot variant, where the previous snapshot does not list the block
itself. The row becomes reclaimable once no previous snapshot holds an hsync'ed
version of the key.
The {{deletedTable}} row name is derived from the transaction index, not the
object ID, so rows do not collide.
Known side effects:
* While the previous snapshot holds an hsync'ed version of the key, the whole
pseudo row is retained, including blocks no snapshot references.
* A retained pseudo row adds the snapshot key's size to that snapshot's
exclusive size even though the active key still shares blocks with it.
The then unused {{OBJECT_ID_RECLAIM_BLOCKS}} constant and its exemption in
{{WithObjectID.validate}} can be removed in a follow up.
h3. Test
* New
{{TestKeyDeletingService.testUncommittedBlockOfHsyncedKeyRetainedBySnapshot}}
runs the two snapshot variant. The {{deletedTable}} row for {{B}} must stay
while {{S2}} is the previous snapshot and after {{S2}} is deleted, and must be
purged only after {{S1}} is deleted.
*
{{TestOMKeyCommitRequest.testValidateAndUpdateCacheOnOverwriteWithUncommittedBlocks}}
asserts that the pseudo key keeps a non zero object ID. It runs for OBS and
FSO.
Both fail on master without the fix. The new {{TestKeyDeletingService}} case
also fails with the object ID kept but the hsync metadata removed from the
pseudo key.
Line numbers are from master at e62603cf3ca.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
