[
https://issues.apache.org/jira/browse/HDDS-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16690:
------------------------------
Attachment: HDDS-16690-CR-R6-unbalanced-release-disables-dag-check.patch
> Unbalanced MultiSnapshotLocks.releaseLock() disables DAG lock order
> enforcement for the calling thread
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16690
> URL: https://issues.apache.org/jira/browse/HDDS-16690
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> HDDS-16690-CR-R6-unbalanced-release-disables-dag-check.patch
>
>
> h3. Mechanism
> {{MultiSnapshotLocks.releaseLock()}} ({{MultiSnapshotLocks.java:77-86}})
> releases unconditionally, and {{DAGResourceLockTracker.unlockResource}}
> decrements the per thread counter with no floor ({{:154-157}}), so one
> unbalanced release drives the counter to -1. {{canLockResource}} tests {{>
> 0}} ({{:160-168}}), so with the counter at 0 while the lock is genuinely
> held, the DAG check that refuses lock order violating acquisitions stops
> firing for that thread.
> The unbalanced release is reachable in production:
> {{ReclaimableFilter.close()}} calls {{cleanup()}}, which calls
> {{snapshotIdLocks.releaseLock()}} whether or not a lock was ever acquired
> ({{ReclaimableFilter.java:231-237}}), and
> {{initializePreviousSnapshotsFromChain}} calls {{close()}} on entry
> ({{:149}}). The counters live in {{ThreadLocal}}s on long lived pooled
> service threads and are never reset. {{SnapshotDeletingService}} guards its
> release with {{isLockAcquired}}; {{ReclaimableFilter}} does not.
> h3. Impact
> The lock order safety check is silently disabled for the life of that pooled
> thread. That check exists to prevent exactly the deadlocks reported
> separately for MC-4 and MC-5 in this umbrella, so the consequence is a lost
> defence rather than an immediate failure. The resulting deadlock was not
> demonstrated through this path, which is the main uncertainty in the rating.
> h3. Reproduction
> {{repro/test_bugCR-R6_dag_counter_underflow.sh}}. Evidence in
> {{repro/logs/ev-cr.log}}.
> h3. Patch
> {{HDDS-XXXXX-CR-R6-unbalanced-release-disables-dag-check.patch}} is attached.
> It applies to 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535 and carries the fix
> plus a regression test that was verified to fail without the fix and pass
> with it.
> {code:none}
> mvn test -pl hadoop-ozone/ozone-manager -Dtest=TestOzoneManagerLock
> {code}
> Found by TLA+ model checking and code review of the OM snapshot cache under
> HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated
> with Specula (Claude Opus 5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
