[
https://issues.apache.org/jira/browse/HDDS-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17243:
------------------------------
Attachment:
CR-13-quota-repair-scans-only-own-bucket-rows-of-snapshot-db.patch
> Quota repair counts pending deletes of a bucket again for every snapshot of
> another bucket that holds a copy of them
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17243
> URL: https://issues.apache.org/jira/browse/HDDS-17243
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-13-quota-repair-scans-only-own-bucket-rows-of-snapshot-db.patch,
> TestBugCR13SnapshotDbCrossBucketCount.java
>
>
> h3. Mechanism
> {{QuotaRepairTask.recalculateSnapshotDbPendingDeleteQuota}} recomputes
> {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} from the active DB
> checkpoint and from every snapshot DB on the repaired buckets' snapshot
> chains. For a snapshot DB it scans the whole {{deletedTable}} and
> {{deletedDirTable}} ({{recalculateDeletedKeyUsages}},
> {{recalculateDeletedDirNamespace}}) and credits each row to the bucket it
> belongs to.
> A snapshot DB is a checkpoint of the whole OM DB.
> {{OmSnapshotManager.createOmSnapshotCheckpoint}} removes only the snapshot's
> own bucket prefix from the active {{deletedTable}}, {{deletedDirTable}} and
> {{snapshotRenamedTable}}, so the pending deletes of every other bucket stay
> in the active DB and are also copied into the snapshot DB. The repair counts
> them once from the active DB checkpoint and once more from every snapshot DB
> of another bucket that holds a copy. The code that reads or changes a
> snapshot's deleted tables ({{SnapshotDeletingService}} through
> {{KeyManagerImpl.getDeletedKeyEntries}} and {{getDeletedDirEntries}},
> {{OMSnapshotMoveTableKeysRequest}}) always restricts itself to the snapshot's
> bucket prefix, so the copies are never removed from the snapshot DB, and the
> repair keeps counting them even after the real entries have been purged from
> the active DB.
> h3. Trigger
> * A snapshot of bucket A taken while bucket B (any other bucket in the
> cluster) had pending deletes in the active {{deletedTable}} or
> {{deletedDirTable}}.
> * A quota repair that includes both buckets ({{ozone repair om quota start}},
> admin only; a repair without {{--buckets}} includes every bucket). The repair
> scans only the snapshots of the buckets it repairs, so a repair of bucket B
> alone is not affected.
> h3. Impact
> * {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} of bucket B end too
> high by one copy of its rows per snapshot DB of another bucket that holds
> them (reproduced: 6000 bytes and 2 entries where 3000 and 1 are correct, with
> one snapshot of another bucket; 3000 bytes where 1000 is correct, with two).
> The value is written to RocksDB, and every later repair computes the same
> excess again while those snapshots exist.
> * By the code, not run: the quota checks use used plus snapshotUsed, so
> bucket B can reject writes with {{QUOTA_EXCEEDED}} before it reaches its
> quota. On a real cluster the excess can be smaller than one full copy per
> snapshot, because {{SstFilteringService}} drops the SST files of a snapshot
> DB that hold no key of the snapshot's own bucket, and with them some of the
> copies.
> h3. Reproduction
> [^TestBugCR13SnapshotDbCrossBucketCount.java] goes into
> {{hadoop-ozone/ozone-manager}} (target package and Maven command in the class
> comment) and passes on unmodified source while the bug is present. It deletes
> one key of bucket B, takes a snapshot of bucket A with the real
> {{OMSnapshotCreateRequest}}, then runs {{QuotaRepairTask}} and applies the
> request it submits: bucket B ends at 6000 bytes and 2 entries. A control
> without the snapshot ends at 3000 and 1.
> h3. Patch
> [^CR-13-quota-repair-scans-only-own-bucket-rows-of-snapshot-db.patch] applies
> on top of the patch of HDDS-17241 (against
> 20d95ed37fe35013b5320713665db03c06ee36fa plus that patch); the two also apply
> to master at 64d4a791502. It does not conflict with the patch for deleted
> snapshots that are not purged yet (filed separately): either applies on top
> of the patch of HDDS-17241 alone or together with the other, in either order.
> The snapshot DB scan now iterates only the snapshot's own bucket prefix,
> {{/volume/bucket/}} in {{deletedTable}} and {{/volumeId/bucketId/}} in
> {{deletedDirTable}}: the same rows that snapshot create removes from the
> active DB and that {{SnapshotDeletingService}} moves. Each pending entry is
> then counted once, from the active DB or from the one snapshot DB that holds
> it, also for a bucket with several snapshots. The scan of the active DB
> checkpoint is unchanged, and a snapshot DB scan now reads only its own
> bucket's rows instead of every deleted row in the cluster. This also takes
> {{SstFilteringService}} out of the picture: it deletes SST files of a
> snapshot DB that hold only other buckets' rows without updating the
> snapshot's {{lastTransactionInfo}}, so before this change a filter run during
> the repair could change what was counted without the check of HDDS-17241
> noticing.
> Covered by {{testQuotaRepairCountsOnlyOwnBucketRowsOfSnapshotDb}} in the
> existing {{TestQuotaRepairTask}}: two FSO buckets with deleted keys and
> directories, and two snapshots of one of them. The first bucket's name and ID
> are string prefixes of the other bucket's, so the test also shows that the
> scan stops at the path separator. On top of the HDDS-17241 patch it fails
> without the change ("expected: <1000> but was: <3000>"); with it,
> {{TestQuotaRepairTask}} and {{TestOMQuotaRepairRequest}} (22 tests) pass,
> {{TestOzoneRepairShell}} on a mini cluster passes with both snapshot scan
> patches applied, and checkstyle is clean. With the patch the reproduction's
> snapshot test fails because bucket B is repaired to 3000 and 1, and its
> control passes.
> Found during review of a fix for the OM quota repair, as part of the TLA+
> verification effort under HDDS-15926, on commit
> 20d95ed37fe35013b5320713665db03c06ee36fa. Related to HDDS-17241, whose fix
> changes the same loop. Checked against HDDS issues and apache/ozone pull
> requests for duplicates before filing. The attached patch is a proposal for
> review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
