Siyao Meng created HDDS-16689:
---------------------------------
Summary: First snapshot load polls for a double buffer flush
inside the ConcurrentHashMap bin lock, stalling all OM writes
Key: HDDS-16689
URL: https://issues.apache.org/jira/browse/HDDS-16689
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
h3. Mechanism
{{SnapshotCache.get}} calls {{cacheLoader.load}} inside {{dbMap.compute}}
({{SnapshotCache.java:225-229}}), so the loader body runs while holding that
key's {{ConcurrentHashMap}} bin lock.
Since HDDS-13633 the loader polls up to
{{ozone.snapshot.checkpoint.dir.creation.poll.timeout}} (20s default) waiting
for a double buffer flush ({{OmSnapshotManager.java:366-383}}). Purge time
{{invalidate()}} ({{SnapshotCache.java:155-170}}) needs the same bin, and it
runs on the {{OMDoubleBufferFlushThread}}.
h3. Impact
A first time {{get(X)}} concurrent with a purge of {{X}} stalls the flush
thread, and with it the double buffer and every OM write, for the poll
duration. Externally observable availability loss, bounded and self resolving
at the poll timeout rather than a permanent deadlock.
h3. Reproduction
{{repro/test_bugCR-MC5_binlock_flush_poll_stall.sh}}. Evidence in
{{repro/logs/ev-cr.log}}.
h3. No patch attached
Fixing this means deciding how the load is serialised. The poll cannot simply
move ahead of the cache lookup: the comment at OmSnapshotManager.java:364-366
says it sits inside the loader on purpose so it only costs a disk check on a
cache miss, and hoisting it would add that cost to every cache hit. Keeping the
miss-only property while getting the poll out of the ConcurrentHashMap bin lock
means either allowing two concurrent first time loads of one snapshot and
discarding the loser, or adding a per key loading latch. Both change load-once
semantics for an expensive resource. Filing the analysis so the approach can be
agreed first.
Found by TLA+ model checking and code review of the OM snapshot cache under
HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated with
Specula (Claude Opus 5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
