[
https://issues.apache.org/jira/browse/HDDS-17287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17287:
------------------------------
Affects Version/s: 2.2.1
2.0.0
1.4.0
2.1.2
0.4.1
0.2.1
1.1.0
0.5.0
0.3.0
0.4.0
1.0.0
1.2.1
1.2.0
2.1.0
1.4.1
2.1.1
1.3.0
2.2.0
> On a versioned bucket, a key commit writes back older versions whose blocks
> the key committed at that time no longer references
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17287
> URL: https://issues.apache.org/jira/browse/HDDS-17287
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 0.2.1, 0.3.0, 0.4.0, 0.4.1, 0.5.0, 1.0.0, 1.1.0, 1.2.0,
> 1.3.0, 1.2.1, 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Critical
> Attachments:
> MC-2-versioned-commit-keeps-older-blocks-still-in-use.patch
>
>
> h3. Mechanism
> With bucket versioning enabled, {{OMKeyRequest.prepareFileInfo}} builds the
> open key of an overwrite from the committed key with
> {{OmKeyInfo.addNewVersion(locations, false, true)}}, so the open key carries
> the version groups the key had at open time. {{OMKeyCommitRequest}} and
> {{OMKeyCommitRequestWithFSO}} write the open key with all these groups to the
> key table, and on a versioned bucket they never compare the older groups with
> the key committed at that time. If that key was deleted after the open,
> {{OMKeyDeleteRequest}} has already moved its blocks to {{deletedTable}}. If
> it was renamed, the blocks are the current version of the renamed key. If the
> key was replaced in between, for example by the commit of another overwrite
> after a delete or a rename, the key committed at that time does not reference
> them either, although it has the same object ID. In each case the commit
> writes references to blocks the committed key does not own.
> h3. Impact
> * After a delete, the key references blocks that {{KeyDeletingService}} sends
> to SCM for deletion while the key still lists them. Deleting the key later
> queues them a second time and subtracts their size from the bucket used bytes
> a second time.
> * After a rename, the key references the current blocks of the renamed key.
> Deleting the key puts them in {{deletedTable}}, so the renamed key loses its
> current data once the datanodes delete the blocks (the datanode deletion was
> not run).
> * The latest version of the committed key stays readable, because reads use
> only the latest version group.
> h3. Reproduction
> {{TestOMKeyCommitRequest#testCommitOnVersionedBucketAfterCommittedKeyChanged}}
> in the attached patch (inherited by {{TestOMKeyCommitRequestWithFSO}}) fails
> on an unmodified checkout. On a versioned bucket it commits a key, builds the
> open key of an overwrite as {{prepareFileInfo}} does, then leaves the
> committed key alone, seeds a deleted key in the key table cache, seeds a new
> object under the same name in the cache, seeds a deleted key and commits a
> second overwrite of the same object, or renames the key with
> {{OMKeyRenameRequest}} and commits a second overwrite, and finally commits
> the first overwrite. Every case but the first fails for both layouts:
> {noformat}
> Expecting actual:
> [conID: 1000 locID: 100 bcsId: 0 replicaIndex: null,
> conID: 1001 locID: 101 bcsId: 0 replicaIndex: null]
> to contain exactly (and in same order):
> [conID: 1001 locID: 101 bcsId: 0 replicaIndex: null]
> but some elements were not expected:
> [conID: 1000 locID: 100 bcsId: 0 replicaIndex: null]
> {noformat}
> Also run on one OM with its background services ({{OmTestManagers}}) and an
> SCM stub that records block deletions. With a delete before the commit, the
> block of the deleted version was sent to SCM while the key still listed it,
> and deleting the key again queued it a second time. With a rename before the
> commit, deleting the overwritten key put the current block of the renamed key
> in {{deletedTable}}. Not exercised: block deletion on datanodes, the S3
> gateway and an OM HA ring.
> h3. Patch
> [^MC-2-versioned-commit-keeps-older-blocks-still-in-use.patch], against
> b10a7d427bdf83807a7e48a1be507d3b02d5a503. It also applies to master at
> 26388a80b37.
> A new {{OMKeyRequest.dropOlderBlocksNotInUse}} keeps a block of an older
> version of the open key only if the key committed at that time still
> references it (compared by {{ContainerBlockID}}, the inverse of
> {{filterOutBlocksStillInUse}}), and drops older versions that are left empty.
> The latest version holds the blocks of the open key itself and is kept as is.
> {{OMKeyCommitRequest}} and {{OMKeyCommitRequestWithFSO}} apply it to the open
> key before building the committed key. A normal versioned overwrite keeps all
> versions, because the committed key still references them, and an open key
> with a single version (no versioning, or a new key) is returned unchanged.
> Kept versions keep their version numbers. Dropped blocks are not queued for
> deletion: a delete has already queued them, a rename has moved them to
> another key, and a block that an earlier versioned commit already left
> unreferenced stays unreferenced and is not reclaimed (HDDS-16733).
> With the change the new test passes, and so do all key request suites under
> {{om.request.key}} with the commit response suites (413 tests) and
> {{TestOmBlockVersioning}} on a mini cluster (2 tests). Checkstyle is clean.
> An alternative is to reject enabling versioning on a bucket until the new
> versioning design (HDDS-15728) lands.
> Related: HDDS-16733 (concurrent open of an absent key on a versioned bucket
> drops a version) changes the same commit branch for a different case, and the
> two changes do not overlap. HDDS-16732 (open key cleanup on a versioned
> bucket) has the same cause, the open key carrying the committed versions, on
> the expiry path. HDDS-16792 covers a delete after a stale overwrite commit on
> a bucket without versioning.
> Found by TLA+ model checking and code review of the S3 gateway object put and
> key commit paths under HDDS-15926, on commit
> b10a7d427bdf83807a7e48a1be507d3b02d5a503. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
