[
https://issues.apache.org/jira/browse/HDDS-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16686:
------------------------------
Attachment: HDDS-16686-MC-2-eviction-queue-erasure.patch
> Snapshot cache eviction erases a concurrent enqueue, leaking an open snapshot
> DB and breaking later checkpoints
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16686
> URL: https://issues.apache.org/jira/browse/HDDS-16686
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Attachments: HDDS-16686-MC-2-eviction-queue-erasure.patch
>
>
> h3. Mechanism
> {{SnapshotCache.cleanup(UUID, boolean)}} ({{SnapshotCache.java:379-417}})
> reads {{v.getTotalRefCount()}} at {{:400}} inside the {{dbMap.compute}}
> lambda. On the keep branch it falls through to
> {{pendingEvictionQueue.remove(k)}} at {{:413}}, which executes on every
> branch.
> A releasing thread that drops the last reference between those two points
> enqueues the key outside that bin lock, and the evictor then erases the
> enqueue. The entry stays in {{dbMap}} with refcount 0 and is no longer
> queued, so nothing will ever evict it.
> h3. Impact
> The snapshot DB stays open until OM restart, and {{SnapshotCache.lock()}}'s
> drain check at {{:334-336}} begins failing on every later attempt, which
> breaks admin checkpoint download and follower bootstrap. There is no
> automatic recovery.
> h3. Reproduction
> {{repro/test_bugMC-2_eviction_queue_erasure.sh}} with
> {{repro/java/SpeculaMc2EvictionQueueTest.java}}. Evidence in
> {{repro/logs/ev-mc2.log}} and {{ev-mc2b.log}}.
> Note on probability: the race did not trigger in 4000 unassisted iterations,
> because the window between the refcount read at {{:400}} and the queue
> removal at {{:413}} is a single debug log statement. It was reproduced by
> widening that window with {{repro/level3-snapshotcache-keep-window.patch}},
> which confirms the ordering and demonstrates the consequence: final state
> {{inDbMap=true refCount=0 inQueue=false}}, and the next
> {{snapshotCache.lock()}} throws {{IllegalStateException: Failed to acquire
> lock as cleanup did not drain the cache.}} The natural probability on an
> unpatched build is low; the fix is a one line move of the queue removal onto
> the evict branch.
> h3. Patch
> {{HDDS-XXXXX-MC-2-eviction-queue-erasure.patch}} is attached. It applies to
> 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535 and carries the fix plus a
> regression test that was verified to fail without the fix and pass with it.
> {code:none}
> mvn test -pl hadoop-ozone/ozone-manager -Dtest=TestSnapshotCache
> {code}
> Found by TLA+ model checking and code review of the OM snapshot cache under
> HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated
> with Specula (Claude Opus 5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
