Siyao Meng created HDDS-16766:
---------------------------------
Summary: FSO directory delete emptiness check races with flush and
cache eviction, so OMs apply the same delete differently
Key: HDDS-16766
URL: https://issues.apache.org/jira/browse/HDDS-16766
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-4-recheck-db-in-fso-emptiness-check.patch,
TestBugMC4EmptinessCheckRace.java
h3. Mechanism
{{OMFileRequest.checkSubDirectoryExists}} and {{checkSubFileExists}} create a
RocksDB iterator, which is a point in time view, and then for each row treat it
as deleted only if a tombstone is still in the table cache. The double buffer
flush thread ({{OzoneManagerDoubleBuffer.flushBatch}}) and the cache eviction
({{PartialTableCache.evictCache}}, scheduled from the flush thread) do not take
the bucket lock that {{validateAndUpdateCache}} holds.
So a child's delete can be committed and its tombstone evicted after the
iterator was created. The iterator still returns the row, the cache lookup
returns null, and the row is counted as a live child.
h3. Trigger
A directory has one child. A client deletes the child and then the parent, non
recursively. On one OM the flush of the child delete and the eviction of its
tombstone land between iterator creation and the cache lookup in the apply of
the parent delete. On the other OMs they land before or after.
h3. Impact
The OM that hit the window answers {{DIRECTORY_NOT_EMPTY}} and keeps the
directory, the others answer OK and delete it. {{directoryTable}} and the
bucket's {{usedNamespace}} diverge between OMs until the affected OM's DB is
replaced. If the leader hit it, the client gets {{DIRECTORY_NOT_EMPTY}} for an
empty directory. If a follower hit it, the directory reappears after a failover
to that follower. On a single OM the consequence is only the spurious error.
h3. Reproduction
PASS with controlled timing. [^TestBugMC4EmptinessCheckRace.java]
({{hadoop-ozone/integration-test}}), 3 OM HA mini cluster, client API only, one
follower's flush controlled with {{OzoneManagerDoubleBuffer.pause/unpause}}:
hit in 1 of 150 rounds on unmodified source. With an 800 ms sleep inserted
after iterator creation it hits in 9 of 9 rounds. 150 rounds without controlled
timing did not trigger it, so the natural window is microseconds.
h3. Suggested fix
Replace the cache marker check with {{Table.isExist}} on the row's key, which
looks in the cache first and then reads the DB again. Flush commits to RocksDB
before eviction and the bucket write lock prevents the child being recreated
meanwhile, so "no cache entry and row absent now" means deleted. HDDS-14600
added the cache marker check for the same user visible symptom, and HDDS-8526
already rechecks the DB this way for bucket and volume emptiness.
h3. Patch
[^MC-4-recheck-db-in-fso-emptiness-check.patch], against
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
9c6bee807c8.
In {{checkSubDirectoryExists}} and {{checkSubFileExists}} the cache marker
check becomes {{if (!table.isExist(dbKey)) continue;}}. The cost is one point
read per row the iterator returns without a cache entry, and the loop returns
at the first row that still exists, so a non empty directory pays one read.
Covered by the existing
{{TestOMKeyDeleteRequestWithFSO#testDeleteParentAfterChildDeleted}} (the
HDDS-14600 test), now parameterized: the new variant flushes and evicts the
child delete right after the emptiness check opens its iterator, using table
spies, with no sleep and no product hook. Without the change it fails with
"expected: <OK> but was: <DIRECTORY_NOT_EMPTY>". The delete, purge and rename
unit suites (86 tests) and {{TestO3FSWithFSO}}, {{TestOFSWithFSO}},
{{TestDirectoryDeletingServiceWithFSO}} (146 tests) pass, checkstyle is clean.
The patch prevents new occurrences only.
Found by TLA+ model checking and code review of the OM FSO namespace mutation
paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
