Siyao Meng created HDDS-16773:
---------------------------------
Summary: DeleteKeys with a key listed twice releases bucket quota
twice
Key: HDDS-16773
URL: https://issues.apache.org/jira/browse/HDDS-16773
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-8-delete-keys-release-quota-once-for-repeated-key.patch
h3. Mechanism
{{OMKeysDeleteRequest.validateAndUpdateCache}} resolves every listed name
before any cache tombstone is written and does not remove duplicate names, so a
repeated name is appended to {{omKeyInfoList}} twice and
{{markKeysAsDeletedInCache}} counts it twice. For an empty key the namespace is
released through {{decrUsedNamespace(emptyKeys, false)}}, which has no pending
delete counterpart and is never corrected by the purge.
No client, S3 gateway or OM layer removes duplicates from the request. On FSO
buckets the repeat does not have to be the same string: {{dir/file}},
{{dir//file}} and {{/dir/file}} resolve to the same entry and are each counted.
h3. Trigger
One {{DeleteKeys}} request ({{OzoneBucket.deleteKeys}}, or S3 DeleteObjects)
that names the same existing zero byte key twice.
h3. Impact
Bucket {{usedNamespace}} is permanently one too low per extra occurrence and
can go negative. It survives the background purge and an OM restart, and
{{checkBucketQuotaInNamespace}} then accepts more keys than
{{quotaInNamespace}}. The call can be repeated to exceed the namespace quota
without bound. For non empty keys and FSO directories the release goes through
the pending delete counters, so used plus pending (what the quota check and the
client report) stays correct and the bytes quota bypass did not reproduce.
{{usedBytes}} / {{usedNamespace}} still stay one occurrence too low and the
pending counters one occurrence too high, because the purge subtracts once.
h3. Reproduction
PASS, deterministic, client API on a mini cluster: create a zero byte key, call
{{deleteKeys}} with its name listed twice, read the bucket's {{usedNamespace}}.
h3. Suggested fix
Skip entries already resolved in the loop in
{{OMKeysDeleteRequest.validateAndUpdateCache}} (the FSO subclass shares it).
Skipping by name is not enough on FSO, where different spellings resolve to one
entry. Buckets already affected need a quota repair.
h3. Patch
[^CR-8-delete-keys-release-quota-once-for-repeated-key.patch], against
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
The resolve loop keeps a set of entries already accepted and skips a later
occurrence of the same entry. The identity is the DB key for OBS and LEGACY and
{{OmKeyInfo.getPath()}} (parent object ID and file name) for FSO. A repeated
key is treated as deleted by its first occurrence, so the response stays {{OK}}
as today and only the accounting changes. Reporting the repeat as a partial
delete instead would be visible to S3 DeleteObjects and {{deleteKeys}} callers,
so the patch does not do that.
Covered by {{testDuplicateKeysReleaseQuotaOnce}} in the existing
{{TestOMKeysDeleteRequest}} and {{TestOMKeysDeleteRequestWithFSO}}, plus
{{testDuplicateFileReleasesQuotaOnce}} for the different spellings case.
Without the change they fail with "expected: 10L but was: 20L", "expected: 3L
but was: 6L" and "expected: 1L but was: 3L". The key delete request and
response unit suites pass (70 tests), checkstyle is clean. On master at
c99d291e863 the product code and base test hunks apply; the
{{TestOMKeysDeleteRequestWithFSO}} hunk needs a mechanical rebase after
HDDS-16161.
Found by TLA+ model checking and code review of the OM FSO namespace mutation
paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
