[
https://issues.apache.org/jira/browse/HDDS-16793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16793:
------------------------------
Attachment: CR-2-count-only-committed-deleted-keys-in-quota-repair.patch
> Quota repair adds deletedTable entries without the committed flag to the
> pending deletion usage of the bucket, and the purge never releases them
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16793
> URL: https://issues.apache.org/jira/browse/HDDS-16793
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-2-count-only-committed-deleted-keys-in-quota-repair.patch,
> TestBugCR2QuotaRepairCountsUncommittedDeletedKeys.java
>
>
> h3. Mechanism
> {{QuotaRepairTask.countDeletedKey}} (HDDS-15204) adds the replicated size of
> every key info in the {{deletedTable}} entries of a bucket to
> {{snapshotUsedBytes}}, and one unit per key info to
> {{snapshotUsedNamespace}}. The requests that maintain these counters follow a
> narrower rule (HDDS-13756). Only a key that leaves the key table after commit
> is added to them, and it is marked with
> {{OmKeyInfo.withCommittedKeyDeletedFlag(true)}}: by
> {{AbstractOMKeyDeleteResponse}} for DeleteKey and DeleteKeys, by
> {{OMKeyCommitRequest}} for the key it overwrites, and by
> {{OMDirectoriesPurgeResponseWithFSO}} for sub files. When the entry is
> purged, {{KeyDeletingService}} and {{OMKeyPurgeRequest}} release only key
> infos that have the flag.
> Every other key info in {{deletedTable}} has no flag and was never added to
> the counters: the key that {{S3MultipartUploadCompleteRequest}} overwrites,
> the parts of an aborted multipart upload, an overwritten part and the unused
> parts of a completed upload, expired open keys ({{OMOpenKeysDeleteRequest}}),
> the uncommitted blocks of a key commit, and entries written before
> HDDS-13756. A repair that runs while such an entry is waiting adds an amount
> to the counters that no purge removes.
> By history, not run on older code: the recount of {{deletedTable}} was added
> by HDDS-15204 and is in no release tag up to 2.2.1, so only unreleased master
> is affected.
> h3. Trigger
> # A bucket has a {{deletedTable}} entry without the committed flag. Run with
> a key that is overwritten by a completed multipart upload. The other sources
> listed above are from code reading, not run.
> # Quota repair runs for the bucket before {{KeyDeletingService}} purges the
> entry. An admin starts it with {{ozone repair om quota start}}
> ({{OzoneManager.startQuotaRepair}}). The service interval is 60s by default.
> The entry of the overwritten key stays for as long as a snapshot references
> that key (run separately on the same harness, not part of the attached test:
> still present after 30 service runs with a snapshot taken before the
> overwrite).
> # The entry is purged.
> h3. Impact
> * With {{deletedTable}} empty again, {{snapshotUsedBytes}} and
> {{snapshotUsedNamespace}} of the bucket keep the size and one unit for each
> key info that had no flag when the repair ran. Both are part of the usage
> that the quota checks compare. Run: a bucket with a namespace quota of 2 that
> holds one key rejects a second key with {{QUOTA_EXCEEDED}}. The space quota
> check reads {{snapshotUsedBytes}} the same way (argued, not run).
> * No data is affected. The counters stay wrong until quota repair runs again
> at a time when no such entry is waiting (argued, not run).
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugCR2QuotaRepairCountsUncommittedDeletedKeys.java]
> ({{hadoop-ozone/ozone-manager}}) drives one real OM with its
> {{KeyDeletingService}} through {{OzoneManagerProtocol}}, with the testing SCM
> block client of the OM tests and no datanodes. Repair is started with {{new
> QuotaRepairTask(om).repair(buckets)}}, the body of
> {{OzoneManager.startQuotaRepair}}. The service interval is 100 ms and the
> service is held with its own suspend and resume hook so that the repair runs
> while the entry is waiting.
> {{multipartOverwriteThenRepairLeavesPendingCountersStuck}} shows
> {{snapshotUsedBytes=805306368}} and {{snapshotUsedNamespace=1}} after the
> repair and unchanged after the purge, with an empty {{deletedTable}}, and the
> rejected second key. {{keyOverwriteThenRepairReturnsToZero}} is the control
> with a plain key overwrite, where the counters return to zero and the second
> key is accepted. A passing first test means the defect is present.
> h3. Patch
> [^CR-2-count-only-committed-deleted-keys-in-quota-repair.patch], against
> a6332fb59879b80615b9233e953d6813d03ae7a8. It also applies to master at
> 3a22acfaa21.
> {{countDeletedKey}} counts a key info only if {{isDeletedKeyCommitted()}} is
> set, which is the rule the purge uses to release it. The size formula is
> unchanged (replicated size per key info, HDDS-16176 covers that formula). The
> function is shared by the scan of the active {{deletedTable}} and the scans
> of the snapshot {{deletedTable}}s, so both follow the rule. The patch does
> not change which entries carry the flag.
> Covered by {{testQuotaRepairCountsOnlyCommittedDeletedKeys}} in the existing
> {{TestQuotaRepairTask}}: an entry with the flag, an entry without it and an
> entry that holds one key info of each kind (as a key commit writes when it
> overwrites a key and drops uncommitted blocks) must count as two keys.
> Without the change it fails with "expected: 2000L but was: 5000L". The
> existing fixtures ({{putDeletedKey}} in the test and
> {{OMRequestTestUtils.deleteKey}}) now write the flag, as a real delete of a
> committed key does. With the change {{TestQuotaRepairTask}},
> {{TestOMQuotaRepairRequest}}, {{TestOMKeyPurgeRequestAndResponse}},
> {{TestOMDirectoriesPurgeRequestAndResponse}} and {{TestKeyDeletingService}}
> (55 tests) and {{TestOzoneRepairShell}} on a mini cluster (3 tests) pass,
> checkstyle is clean. With the patch the first test of the reproduction fails
> at the counters after the repair (0 instead of 805306368), the control still
> passes.
> Found by code review of the OM key deletion service and key purge paths, as
> part of the TLA+ verification effort under HDDS-15926, on commit
> a6332fb59879b80615b9233e953d6813d03ae7a8. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
