Siyao Meng created HDDS-17287:
---------------------------------
Summary: On a versioned bucket, a key commit writes back older
versions whose blocks the key committed at that time no longer references
Key: HDDS-17287
URL: https://issues.apache.org/jira/browse/HDDS-17287
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-2-versioned-commit-keeps-older-blocks-still-in-use.patch
h3. Mechanism
With bucket versioning enabled, {{OMKeyRequest.prepareFileInfo}} builds the
open key of an overwrite from the committed key with
{{OmKeyInfo.addNewVersion(locations, false, true)}}, so the open key carries
the version groups the key had at open time. {{OMKeyCommitRequest}} and
{{OMKeyCommitRequestWithFSO}} write the open key with all these groups to the
key table, and on a versioned bucket they never compare the older groups with
the key committed at that time. If that key was deleted after the open,
{{OMKeyDeleteRequest}} has already moved its blocks to {{deletedTable}}. If it
was renamed, the blocks are the current version of the renamed key. If the key
was replaced in between, for example by the commit of another overwrite after a
delete or a rename, the key committed at that time does not reference them
either, although it has the same object ID. In each case the commit writes
references to blocks the committed key does not own.
h3. Impact
* After a delete, the key references blocks that {{KeyDeletingService}} sends
to SCM for deletion while the key still lists them. Deleting the key later
queues them a second time and subtracts their size from the bucket used bytes a
second time.
* After a rename, the key references the current blocks of the renamed key.
Deleting the key puts them in {{deletedTable}}, so the renamed key loses its
current data once the datanodes delete the blocks (the datanode deletion was
not run).
* The latest version of the committed key stays readable, because reads use
only the latest version group.
h3. Reproduction
{{TestOMKeyCommitRequest#testCommitOnVersionedBucketAfterCommittedKeyChanged}}
in the attached patch (inherited by {{TestOMKeyCommitRequestWithFSO}}) fails on
an unmodified checkout. On a versioned bucket it commits a key, builds the open
key of an overwrite as {{prepareFileInfo}} does, then leaves the committed key
alone, seeds a deleted key in the key table cache, seeds a new object under the
same name in the cache, seeds a deleted key and commits a second overwrite of
the same object, or renames the key with {{OMKeyRenameRequest}} and commits a
second overwrite, and finally commits the first overwrite. Every case but the
first fails for both layouts:
{noformat}
Expecting actual:
[conID: 1000 locID: 100 bcsId: 0 replicaIndex: null,
conID: 1001 locID: 101 bcsId: 0 replicaIndex: null]
to contain exactly (and in same order):
[conID: 1001 locID: 101 bcsId: 0 replicaIndex: null]
but some elements were not expected:
[conID: 1000 locID: 100 bcsId: 0 replicaIndex: null]
{noformat}
Also run on one OM with its background services ({{OmTestManagers}}) and an SCM
stub that records block deletions. With a delete before the commit, the block
of the deleted version was sent to SCM while the key still listed it, and
deleting the key again queued it a second time. With a rename before the
commit, deleting the overwritten key put the current block of the renamed key
in {{deletedTable}}. Not exercised: block deletion on datanodes, the S3 gateway
and an OM HA ring.
h3. Patch
[^MC-2-versioned-commit-keeps-older-blocks-still-in-use.patch], against
b10a7d427bdf83807a7e48a1be507d3b02d5a503. It also applies to master at
26388a80b37.
A new {{OMKeyRequest.dropOlderBlocksNotInUse}} keeps a block of an older
version of the open key only if the key committed at that time still references
it (compared by {{ContainerBlockID}}, the inverse of
{{filterOutBlocksStillInUse}}), and drops older versions that are left empty.
The latest version holds the blocks of the open key itself and is kept as is.
{{OMKeyCommitRequest}} and {{OMKeyCommitRequestWithFSO}} apply it to the open
key before building the committed key. A normal versioned overwrite keeps all
versions, because the committed key still references them, and an open key with
a single version (no versioning, or a new key) is returned unchanged. Kept
versions keep their version numbers. Dropped blocks are not queued for
deletion: a delete has already queued them, a rename has moved them to another
key, and a block that an earlier versioned commit already left unreferenced
stays unreferenced and is not reclaimed (HDDS-16733).
With the change the new test passes, and so do all key request suites under
{{om.request.key}} with the commit response suites (413 tests) and
{{TestOmBlockVersioning}} on a mini cluster (2 tests). Checkstyle is clean.
An alternative is to reject enabling versioning on a bucket until the new
versioning design (HDDS-15728) lands.
Related: HDDS-16733 (concurrent open of an absent key on a versioned bucket
drops a version) changes the same commit branch for a different case, and the
two changes do not overlap. HDDS-16732 (open key cleanup on a versioned bucket)
has the same cause, the open key carrying the committed versions, on the expiry
path. HDDS-16792 covers a delete after a stale overwrite commit on a bucket
without versioning.
Found by TLA+ model checking and code review of the S3 gateway object put and
key commit paths under HDDS-15926, on commit
b10a7d427bdf83807a7e48a1be507d3b02d5a503. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
