Siyao Meng created HDDS-16690:
---------------------------------
Summary: Unbalanced MultiSnapshotLocks.releaseLock() disables DAG
lock order enforcement for the calling thread
Key: HDDS-16690
URL: https://issues.apache.org/jira/browse/HDDS-16690
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
HDDS-16690-CR-R6-unbalanced-release-disables-dag-check.patch
h3. Mechanism
{{MultiSnapshotLocks.releaseLock()}} ({{MultiSnapshotLocks.java:77-86}})
releases unconditionally, and {{DAGResourceLockTracker.unlockResource}}
decrements the per thread counter with no floor ({{:154-157}}), so one
unbalanced release drives the counter to -1. {{canLockResource}} tests {{> 0}}
({{:160-168}}), so with the counter at 0 while the lock is genuinely held, the
DAG check that refuses lock order violating acquisitions stops firing for that
thread.
The unbalanced release is reachable in production:
{{ReclaimableFilter.close()}} calls {{cleanup()}}, which calls
{{snapshotIdLocks.releaseLock()}} whether or not a lock was ever acquired
({{ReclaimableFilter.java:231-237}}), and
{{initializePreviousSnapshotsFromChain}} calls {{close()}} on entry ({{:149}}).
The counters live in {{ThreadLocal}}s on long lived pooled service threads and
are never reset. {{SnapshotDeletingService}} guards its release with
{{isLockAcquired}}; {{ReclaimableFilter}} does not.
h3. Impact
The lock order safety check is silently disabled for the life of that pooled
thread. That check exists to prevent exactly the deadlocks reported separately
for MC-4 and MC-5 in this umbrella, so the consequence is a lost defence rather
than an immediate failure. The resulting deadlock was not demonstrated through
this path, which is the main uncertainty in the rating.
h3. Reproduction
{{repro/test_bugCR-R6_dag_counter_underflow.sh}}. Evidence in
{{repro/logs/ev-cr.log}}.
h3. Patch
{{HDDS-XXXXX-CR-R6-unbalanced-release-disables-dag-check.patch}} is attached.
It applies to 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535 and carries the fix plus
a regression test that was verified to fail without the fix and pass with it.
{code:none}
mvn test -pl hadoop-ozone/ozone-manager -Dtest=TestOzoneManagerLock
{code}
Found by TLA+ model checking and code review of the OM snapshot cache under
HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated with
Specula (Claude Opus 5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
