Siyao Meng created HDDS-16685:
---------------------------------
Summary: OM deadlock: nested snapshot get() takes SNAPSHOT_DB_LOCK
stripes out of order against acquireAll
Key: HDDS-16685
URL: https://issues.apache.org/jira/browse/HDDS-16685
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
h3. Mechanism
{{OzoneManagerLock.acquireAll}} ({{:158-164}}) takes every stripe of a resource
in ascending index order, blocking, with no backoff and no all or nothing
semantics. {{SnapshotCache.lock()}} reaches it through
{{acquireResourceWriteLock}} ({{:382}}).
{{SnapshotCache.get(UUID)}} instead takes exactly one stripe,
{{acquireReadLock(SNAPSHOT_DB_LOCK, key.toString())}} at {{:217}}, chosen by
the UUID hash, in whatever order the caller opens snapshots. A snapdiff holding
the from snapshot on a high stripe and then requesting the to snapshot on a low
stripe inverts against a concurrent {{acquireAll}} that has already taken the
low stripe.
{{MultiSnapshotLocks}} exists and uses the ordered bulk path, so the ordered
primitive was written on purpose and the snapdiff callers bypass it. The DAG
tracker added by HDDS-13960 orders resources, not stripes within a resource, so
it does not cover this case.
h3. Impact
Permanent deadlock between the snapdiff thread and the checkpoint or follower
bootstrap thread. Every later snapshot operation that needs one of the already
taken stripes blocks behind it. Untimed on both sides, restart only, and
invisible to JVM deadlock detection because read holds have no owner.
h3. Reproduction
{{repro/test_bugMC-5_stripe_inversion_deadlock.sh}} with
{{repro/java/SpeculaMc5StripeInversionTest.java}}. Evidence in
{{repro/logs/ev-mc5.log}}: the Level 0 sequential case completes as expected,
then the two thread case leaves both threads unfinished after 60s, with
{{findDeadlockedThreads()=null}} and thread dumps showing the snapdiff thread
parked in {{SnapshotCache.get}} at {{:217}} and the checkpoint thread parked in
{{acquireAll}} under {{SnapshotCache.lock}}.
h3. No patch attached
Fixing this means deciding how nested snapshot acquisition is ordered.
MultiSnapshotLocks already wraps the ordered bulk path, so routing the snapdiff
callers through it is the natural fix, but that changes how those callers hold
snapshots across a diff and the alternative is to order the stripes inside
SnapshotCache.get itself. A regression test for the deadlock would be slow and
timing dependent in CI. Filing the analysis so the approach can be agreed first.
Found by TLA+ model checking and code review of the OM snapshot cache under
HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated with
Specula (Claude Opus 5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
