Siyao Meng created HDDS-17248:
---------------------------------
Summary: Deleted keys moved out of a deleted snapshot lose their
bucket ID, so their purge reduces the usage of a later bucket of the same name
Key: HDDS-17248
URL: https://issues.apache.org/jira/browse/HDDS-17248
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-2-keep-bucket-id-of-deleted-keys-moved-from-snapshot.patch,
TestBugMC2SnapshotMoveBucketIdAfterRecreate.java
h3. Mechanism
HDDS-13733 tags each {{deletedTable}} row with the objectID of the bucket the
key was deleted in, and HDDS-13764 lets {{OMKeyPurgeRequest}} subtract the
purged size from that bucket's {{snapshotUsedBytes}} and
{{snapshotUsedNamespace}} only when the tag equals the current bucket's
objectID. That keeps the purge of a deleted bucket's rows away from a later
bucket of the same name, except on one path:
# Bucket delete does not wait for the bucket's {{deletedTable}} rows to be
purged, so rows of the deleted bucket can still be in the active DB when a
bucket of the same name is created.
# Snapshot create on the new bucket moves the active {{deletedTable}} rows by
the {{/volume/bucket/}} name prefix into the snapshot ({{OmSnapshotManager}}),
the earlier bucket's rows included, with their tags.
# When that snapshot is deleted, {{SnapshotDeletingService}} moves the rows to
the next snapshot or the active DB. {{OMSnapshotMoveTableKeysRequest}} passes
the current bucket's objectID, and
{{OMSnapshotMoveTableKeysResponse.addKeysToNextSnapshot}} rebuilds each row
with {{SnapshotUtils.createMergedRepeatedOmKeyInfoFromDeletedTableEntry}} as
{{new RepeatedOmKeyInfo(keyInfoList, bucketId)}} when the destination has no
row of that name. The tag the row carried in the snapshot is dropped.
# {{KeyDeletingService}} purges the rows, the tag now matches the new bucket,
and {{purgeSnapshotUsedBytes}} / {{purgeSnapshotUsedNamespace}} subtract usage
that bucket never had. There is no lower bound.
Bucket quota checks in {{OMKeyRequest}} use {{getTotalBucketSpace}} and
{{getTotalBucketNamespace}} (used plus snapshot used), so the negative value
lets the bucket accept that much more.
h3. Trigger
A bucket is deleted while some of its deleted keys are not yet purged (a
{{KeyDeletingService}} backlog), a bucket of the same name is created, and a
snapshot of the new bucket is created and later deleted before those rows are
purged.
h3. Impact
Reproduced on unmodified source with three deleted 1 KB keys (replication ONE)
in an OBJECT_STORE bucket and a new bucket with a namespace quota of 2:
* The rows come back from the snapshot tagged with the new bucket's objectID
instead of the deleted bucket's.
* After the purge the new bucket has {{snapshotUsedBytes=-3072}} and
{{snapshotUsedNamespace=-3}}.
* The new bucket then accepted 5 keys under its namespace quota of 2 and
rejected the 6th ("namespace consumed: 3").
* Without the snapshot create and delete (same sequence otherwise) the new
bucket stays at 0.
The wrong values are persisted and nothing in normal operation corrects them; a
quota repair recomputes {{snapshotUsed*}} from the tags and should reset them
(argued, not run). Shown for OBJECT_STORE keys. FSO deleted files are keyed by
name in {{deletedTable}} and take the same path (not run); FSO deleted
directories are keyed by object IDs and are not moved by the name prefix.
h3. Reproduction
[^TestBugMC2SnapshotMoveBucketIdAfterRecreate.java] runs one OM on a mini
cluster in {{hadoop-ozone/integration-test}} with public client calls only
(target path and Maven command are in the class comment). The only aid is the
existing {{KeyDeletingService.suspend()}} / {{resume()}} test hook, which
stands for the purge backlog. {{deletedKeysOfEarlierBucketPassThroughSnapshot}}
passes when the bug is present; {{deletedKeysOfEarlierBucketWithoutSnapshot}}
is the control and passes either way.
h3. Patch
[^MC-2-keep-bucket-id-of-deleted-keys-moved-from-snapshot.patch], against
20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
64d4a791502.
{{OMSnapshotMoveTableKeysResponse.addKeysToNextSnapshot}} reads the from
snapshot's {{deletedTable}} row of each moved key and passes that row's bucket
ID to {{createMergedRepeatedOmKeyInfoFromDeletedTableEntry}}. Only when the row
is missing (for example on a replayed flush) does it use the current bucket's
objectID. Rows deleted in the current bucket keep the same tag as today; rows
of an earlier bucket keep that bucket's ID, so the existing purge check skips
them. Rows without a tag keep 0 instead of taking the current bucket's ID: they
were written before 2.1.0, which added both the tag and {{snapshotUsed*}}, so
they were never added to {{snapshotUsed*}}, and the purge check and quota
repair already skip tag 0 in the active DB. The legacy
{{OMSnapshotMoveDeletedKeysResponse}} re-tags the same way; nothing at this
commit submits it, and it is not changed. No request or proto change. The
unused {{bucketId}} field of {{SnapshotMoveKeyInfos}} could carry the tag
instead, but filling it needs {{KeyManager.getDeletedKeyEntries}} to return it,
and the response would still have to use it. The cost is one point lookup per
moved key in the from snapshot's {{deletedTable}}. The patch does not repair
buckets whose usage is already negative.
The change alters what an OM writes for such a move. With the current upgrade,
which goes through OM prepare, OMs of two versions do not apply the same
entries; under zero downtime upgrade this apply change, like others, would need
a version flag.
Covered by the existing
{{TestOMSnapshotMoveTableKeysResponse.testMoveTableKeysToNextSnapshot}}, now
run with rows tagged with an earlier bucket's ID and with untagged rows, with
and without a next snapshot, and asserting the tag of every row at the
destination, merged rows included. Without the change all four cases fail (the
moved rows carry the current bucket's ID, for example "expected: <0> but was:
<1792590778128>"). With it {{TestSnapshotUtils}},
{{TestOMSnapshotMoveTableKeysResponse}},
{{TestOMSnapshotMoveTableKeysRequest}},
{{TestOMSnapshotPurgeRequestAndResponse}},
{{TestOMKeyPurgeRequestAndResponse}}, {{TestSnapshotDeletingService}} and
{{TestKeyDeletingService}} pass (107 tests), as does
{{TestSnapshotDeletingServiceIntegrationTest}}, checkstyle is clean, and the
attached reproduction no longer shows the bug ({{snapshotUsedNamespace}} stays
0).
Found by TLA+ model checking and code review of the OM volume, bucket and
snapshot lifecycle paths under HDDS-15926, on commit
20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-13733 and HDDS-13764 (added the tag and the purge check this
path bypasses), HDDS-16126 (fences quota repair deltas by bucket objectID) and
HDDS-7565 (FSO directory purge of an old bucket updating the new one). The
attached patch is a proposal for review. Generated with Specula (Claude Opus
5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
