[
https://issues.apache.org/jira/browse/HDDS-17244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17244:
------------------------------
Attachment: TestBugCR14DeletedSnapshotSkipped.java
> Quota repair skips deleted snapshots that are not purged yet, so the pending
> deletes they still hold drop out of the bucket's snapshotUsedBytes
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17244
> URL: https://issues.apache.org/jira/browse/HDDS-17244
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-14-quota-repair-scans-deleted-snapshots-not-yet-purged.patch,
> TestBugCR14DeletedSnapshotSkipped.java
>
>
> h3. Mechanism
> {{QuotaRepairTask.recalculateSnapshotDbPendingDeleteQuota}} recomputes
> {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} from the active DB
> checkpoint and from the snapshot DBs on the repaired buckets' snapshot
> chains, but it skips every snapshot whose status is not {{SNAPSHOT_ACTIVE}}
> ({{loadActiveSnapshot}} on this commit; the patch of HDDS-17241, which lists
> the snapshots from the checkpoint's snapshot table instead of the live chain,
> keeps the same filter), so deleted snapshots are not scanned.
> Deleting a snapshot ({{OMSnapshotDeleteRequest}}) only sets its status to
> {{SNAPSHOT_DELETED}}. Its deleted keys, deleted directories and renamed
> entries stay in its DB until {{SnapshotDeletingService}} moves them to the
> next active snapshot or to the active DB ({{OMSnapshotMoveTableKeysRequest}},
> which also removes them from the deleted snapshot's DB) and then purges the
> snapshot. None of these requests changes the bucket's counters: an entry is
> charged to {{snapshotUsed*}} from the key delete until {{OMKeyPurgeRequest}}
> or {{OMDirectoriesPurgeRequestWithFSO}} purges it. So while a deleted
> snapshot still holds entries, they are pending and charged, and nothing else
> in the repair counts them. The deleted snapshot's DB is still available
> ({{OmSnapshotManager.getSnapshot(UUID)}} does not require an active snapshot,
> and {{SnapshotDeletingService}} also opens deleted snapshots).
> h3. Trigger
> * A snapshot of the bucket that was deleted and still holds deleted keys or
> directories of the bucket. That lasts until {{SnapshotDeletingService}} has
> moved them out, at least one run of the service, longer for large snapshots
> (the moves are batched) and while the next snapshot in the chain is also
> deleted (the service then skips the snapshot).
> * A quota repair of that bucket ({{ozone repair om quota start}}, admin only)
> in that window.
> h3. Impact
> * {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} of the bucket end too
> low by what the deleted snapshot still holds (reproduced: 0 and 0 where 3000
> and 1 are correct). The value is written to RocksDB and stays until a repair
> runs outside the window.
> * By the code, not run: when those entries are purged later, the purge
> subtracts them again, so the counters stay too low (negative if nothing else
> is pending), and the quota checks, which use used plus snapshotUsed, accept
> writes beyond the bucket's quota by that amount.
> h3. Reproduction
> [^TestBugCR14DeletedSnapshotSkipped.java] goes into
> {{hadoop-ozone/ozone-manager}} (target package and Maven command in the class
> comment) and passes on unmodified source while the bug is present. It deletes
> a key, takes a snapshot with the real {{OMSnapshotCreateRequest}} (which
> moves the deleted key's row from the active DB into the snapshot DB), deletes
> the snapshot with the real {{OMSnapshotDeleteRequest}}, then runs
> {{QuotaRepairTask}} and applies the request it submits: the bucket ends at 0
> and 0 although the row is still in the deleted snapshot's DB. A control with
> the snapshot left active ends at 3000 and 1.
> h3. Patch
> This patch should land together with or after the one of HDDS-17243. Without
> it, the newly scanned deleted snapshots are read in full, every bucket's
> copies included: that adds the overcount of HDDS-17243 for the deleted
> snapshots too, and a long scan of a snapshot that {{SnapshotDeletingService}}
> is about to purge can hold up that purge, which closes the snapshot DB and
> takes its lock while the repair still has it open, and with it the OM's
> double buffer flush until the scan ends (argued from the code, not run). With
> the prefix scan of HDDS-17243 the scan of an emptied deleted snapshot is near
> instant.
> [^CR-14-quota-repair-scans-deleted-snapshots-not-yet-purged.patch] applies on
> top of the patch of HDDS-17241 (against
> 20d95ed37fe35013b5320713665db03c06ee36fa plus that patch); the two also apply
> to master at 64d4a791502. It does not conflict with the patch for pending
> deletes of other buckets counted from snapshot DBs (filed separately): either
> applies on top of the patch of HDDS-17241 alone or together with the other,
> in either order.
> The status check is removed, so the repair scans every snapshot of the bucket
> that is in the checkpoint's snapshot info table, active or deleted. A purged
> snapshot is gone from the table and is not scanned, as before; one purged
> after the checkpoint fails the run through the check HDDS-17241 adds, like
> any other snapshot change during the scan. This does not count an entry
> twice. A move takes the entries out of the deleted snapshot's DB when it
> writes them to the destination, so a move completed before the repair is
> counted once, at the destination. The two writes are separate RocksDB
> commits, but the move updates {{lastTransactionInfo}} of both snapshots in
> the snapshot info cache when it is applied, so a repair that overlaps a move
> fails through the check HDDS-17241 adds.
> Covered by {{testQuotaRepairCountsDeletedSnapshotsNotYetPurged}} in the
> existing {{TestQuotaRepairTask}}: three snapshots, the first two deleted. The
> first still holds its key (its next snapshot is deleted too), and the second
> has already moved its key to the third, active one with the real
> {{OMSnapshotMoveTableKeysRequest}}, so the test checks both the missing count
> and that a moved entry is counted once. On top of the HDDS-17241 patch it
> fails without the change ("expected: <3000> but was: <2000>"); with it,
> {{TestQuotaRepairTask}} and {{TestOMQuotaRepairRequest}} (22 tests) pass,
> {{TestOzoneRepairShell}} on a mini cluster passes with both snapshot scan
> patches applied, and checkstyle is clean. With the patch the reproduction's
> deleted snapshot test fails because the bucket is repaired to 3000 and 1, and
> its control passes.
> Found during review of a fix for the OM quota repair, as part of the TLA+
> verification effort under HDDS-15926, on commit
> 20d95ed37fe35013b5320713665db03c06ee36fa. Related to HDDS-17241, whose fix
> changes the same loop. Checked against HDDS issues and apache/ozone pull
> requests for duplicates before filing. The attached patch is a proposal for
> review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
