Siyao Meng created HDDS-16765:
---------------------------------
Summary: A rejected overwrite commit leaves OVERWRITTEN_HSYNC_KEY
on another writer's cached open entry
Key: HDDS-16765
URL: https://issues.apache.org/jira/browse/HDDS-16765
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-3-flag-overwritten-hsync-key-after-commit-validation.patch,
TestBugMC3FailedOverwriteCommit.java
h3. Mechanism
{{OMKeyCommitRequestWithFSO.validateAndUpdateCache}} flags the hsync writer's
open entry with {{OVERWRITTEN_HSYNC_KEY}} and puts it in the open file table
cache ({{:198-213}}) before {{validateAtomicRewrite}} and the bucket quota
checks. When one of those rejects the commit, nothing undoes the cache put. The
error response writes nothing to RocksDB, so the flag lives only until this
OM's double buffer flushes and evicts that epoch. {{OMKeyCommitRequest}}
({{:277-291}}) has the same order.
Each OM flushes and evicts on its own schedule, so the cache content at apply
time differs between OMs.
h3. Trigger
Bucket namespace quota 2, hsync enabled.
# A creates and hsyncs {{f}}.
# B creates {{f}} with overwrite.
# C commits {{g}}.
# B's close is rejected with {{QUOTA_EXCEEDED}}.
# A closes while one OM has not yet flushed and evicted B's rejected commit.
h3. Impact
The lagging OM rejects A's commit with {{KEY_NOT_FOUND}} while the other OMs
apply it. {{fileTable}} and {{openFileTable}} differ permanently between OMs,
and {{getFileStatus}} returns different lengths per OM. If the lagging OM is
the leader, A's close fails although the overwrite never happened. The same
happens on a single OM: A's next hsync or close fails with {{KEY_NOT_FOUND}} if
it is applied before the rejected commit's epoch is evicted.
h3. Reproduction
PASS, 3 of 3 runs on a 3 OM mini cluster with one OM's flush held by
{{OzoneManagerDoubleBuffer#pause}}: [^TestBugMC3FailedOverwriteCommit.java]
({{hadoop-ozone/integration-test}}). 40 rounds without the hook did not hit the
window, so the natural window is narrow on a live OM. An OM that applies the
two entries back to back, such as a follower catching up or an OM replaying its
log after a restart, has both in the same unflushed window (reasoning from
code, not run). A rejection by {{validateAtomicRewrite}} has the same effect as
the quota rejection.
h3. Suggested fix
Move the cache put of the flagged entry below {{validateAtomicRewrite}} and the
quota checks, next to the other cache updates at the end of
{{validateAndUpdateCache}}, in both the FSO and the non FSO commit request.
h3. Patch
[^MC-3-flag-overwritten-hsync-key-after-commit-validation.patch], against
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
c99d291e863.
In both {{OMKeyCommitRequestWithFSO}} and {{OMKeyCommitRequest}} the flagged
copy is still built where it was, but the {{addCacheEntry}} call moves into the
existing "Add to cache of open key table and key table" section, after all
checks that can reject the commit. Nothing between the old and the new position
reads that open key entry, so the success path is unchanged.
Covered by {{testRejectedOverwriteCommitKeepsHsyncedKeyWritable}} in the
existing {{TestOMKeyCommitRequest}}, inherited by
{{TestOMKeyCommitRequestWithFSO}}: writer A hsyncs, writer B's overwrite commit
is rejected with {{QUOTA_EXCEEDED}}, then A's next hsync must return {{OK}}.
Without the change both variants fail with "expected: <OK> but was:
<KEY_NOT_FOUND>". The key commit unit suites (48 tests) and {{TestHSync}} (42
tests) pass, checkstyle is clean. OMs that already diverged are not repaired.
Found by TLA+ model checking and code review of the OM FSO namespace mutation
paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
