Siyao Meng created HDDS-17241:
---------------------------------
Summary: Quota repair can leave bucket snapshotUsedBytes negative
when a snapshot purge commits after the repair's checkpoint and before the
snapshot scan
Key: HDDS-17241
URL: https://issues.apache.org/jira/browse/HDDS-17241
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-11-fail-quota-repair-when-snapshot-changes-after-checkpoint.patch,
TestBugCR11QuotaRepairSnapshotPurgeSkew.java
h3. Mechanism
{{QuotaRepairTask.repairTask}} takes a checkpoint of the active OM DB; the
bucket baselines and all active table counts come from it.
{{recalculateSnapshotDbPendingDeleteQuota}} instead opens each snapshot DB live
through {{OmSnapshotManager.getSnapshot}} and counts its current
{{deletedTable}} and {{deletedDirTable}}. The repair sends the computed value
minus the baseline for {{snapshotUsedBytes}} and {{snapshotUsedNamespace}}, and
{{OMQuotaRepairRequest.updateBucketInfo}} adds it to the current bucket.
A request that removes entries from a snapshot DB and lowers the bucket's
{{snapshotUsed*}} in the same transaction ({{OMKeyPurgeRequest}} or
{{OMDirectoriesPurgeRequestWithFSO}} with a snapshot table key, as sent by the
deep clean of the deleting services) and commits after the checkpoint but
before that snapshot is scanned is then subtracted twice: the scan no longer
sees the entries, the baseline still has them, and the live bucket has already
been lowered. A purge applied after both deleted tables of that snapshot have
been read is harmless. The same undercount happens when a snapshot that is
active in the checkpoint is deleted, has its entries moved to the active DB and
is purged before the scan reaches it: the scan walks the live snapshot chain,
so it skips that snapshot, and the moved entries are in neither the
checkpoint's active tables nor any scanned snapshot. Nothing serializes the
repair with the deleting services, and the counters are not clamped, so
{{OMKeyRequest.checkBucketQuotaInBytes}} and {{checkBucketQuotaInNamespace}}
(which use used plus snapshotUsed) see the bucket that much emptier.
h3. Trigger
* A bucket with a snapshot that still holds deleted keys or directories that
are not yet deep cleaned.
* A quota repair ({{ozone repair om quota start}}, admin only).
* A deep clean purge of that snapshot, or its move and purge after a snapshot
delete, committing between the checkpoint and the scan of that snapshot. The
window covers opening the checkpoint DB, reading the bucket table and scanning
the snapshots that come earlier in the scan.
h3. Impact
* {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} of the bucket end below
their true value by what the purge removed, negative if nothing else is pending
(reproduced: -3000 and -1 where 0 and 0 are correct). The value is written to
RocksDB and stays until a repair runs without the race.
* By the code, not run: quota checks undercount the bucket by the same amount,
so writes beyond the quota are accepted.
h3. Reproduction
[^TestBugCR11QuotaRepairSnapshotPurgeSkew.java] goes into
{{hadoop-ozone/ozone-manager}} (target package and Maven command in the class
comment) and passes on unmodified source while the bug is present. It deletes a
key into a snapshot and runs {{QuotaRepairTask}} with the request it submits
captured and applied. The purge is a real {{OMKeyPurgeRequest}} with the
snapshot table key; the only control is that it commits on the repair's first
call to {{getOmSnapshotManager()}}, which comes from the snapshot scan after
the checkpoint. A control with the purge committed before the repair ends at 0
and 0.
h3. Patch
[^CR-11-fail-quota-repair-when-snapshot-changes-after-checkpoint.patch],
against 20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
41d95dd3286.
The repair now lists the snapshots of each repaired bucket from the
checkpoint's {{snapshotInfoTable}} instead of the live snapshot chain, so a
snapshot removed from the chain after the checkpoint is still visited. Every
request that changes a snapshot DB or removes a snapshot (key and directory
purge, snapshot moves, snapshot delete and purge) already updates the
snapshot's {{lastTransactionInfo}} in the snapshot info cache, or removes the
entry, when it is applied. After scanning a snapshot DB the repair compares the
live value with the one from the checkpoint and fails the run ("Snapshot ...
changed after the quota repair checkpoint was taken, run the quota repair
again") instead of submitting deltas that mix two points in time; a snapshot
that can no longer be opened fails the run as well. A change applied after that
comparison cannot have been seen by the scan, and a change flushed before the
checkpoint is in both, so neither fails the run. The run fails whenever a
snapshot of a repaired bucket changed between the checkpoint and the end of its
scan, which includes some changes that would not have skewed the result;
rerunning is safe. Deep cleaning is on by default, so on a busy cluster a long
repair can fail several times in a row; {{--buckets}} narrows the window to the
snapshots of those buckets. Clamping at zero would hide the symptom but still
be wrong when other deletes are pending. A possible follow up is to pause the
deleting services for the scan instead, by holding {{BOOTSTRAP_LOCK}} from
before the checkpoint until the snapshot scan ends, together with a double
buffer flush, as {{OMDBCheckpointServlet}} does; that always converges but
stalls deletion and follower bootstrap for the length of the scan.
Covered in the existing {{TestQuotaRepairTask}}:
{{testQuotaRepairFailsWhenSnapshotPurgedAfterCheckpoint}} fails without the
change ("expected: 0L but was: -3000L"),
{{testQuotaRepairFailsWhenSnapshotLeavesChainAfterCheckpoint}} fails without it
("expected: 3000L but was: 0L"), and
{{testQuotaRepairSucceedsWhenSnapshotPurgedBeforeCheckpoint}} checks that a
purge flushed before the repair does not fail the run. With the change, the
quota repair, key and directory purge and snapshot move suites (46 tests) pass,
including the existing {{testQuotaRepairSnapshotDbDeletedTableQuota}} for the
case without a race, as does {{TestOzoneRepairShell}} on a mini cluster, and
checkstyle is clean. With the patch the race test of the reproduction fails
because the repair run fails, and its control passes.
Found by code review of the OM volume, bucket and snapshot lifecycle paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
