Siyao Meng created HDDS-16687:
---------------------------------
Summary: SnapshotCache.invalidate closes a snapshot RocksDB that a
reader still holds
Key: HDDS-16687
URL: https://issues.apache.org/jira/browse/HDDS-16687
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
h3. Mechanism
{{SnapshotCache.invalidate(UUID)}} ({{SnapshotCache.java:155-170}}) runs
{{dbMap.compute(key, ...)}} and calls {{v.get().close()}} unconditionally.
There is no {{getTotalRefCount()}} check and no {{SNAPSHOT_DB_LOCK}}
acquisition on the path, so a purge closes the snapshot RocksDB under a live
reader.
{{OmSnapshotManager.java:495-496}} states the intent, "so that no operation is
running on the snapshot", that this path does not achieve. The stripe write
lock taken inside {{deleteSnapshotCheckpointDirectories}} delays the directory
deletion but not the close.
h3. Impact
Every subsequent read on the reader's still valid handle fails at
{{RocksDatabase.acquire()}} with "Rocks Database is closed". Affected consumers
are any snapshot reader holding a handle from {{SnapshotCache.get}}: the
deletion services through {{ReclaimableFilter.java:163-165}}, snapdiff through
{{OmSnapshotManager.getActiveSnapshot}}, and FS reads through
{{OzoneManagerRequestHandler}}. The handle can never recover and no downstream
mechanism reopens it, so the reader's operation fails permanently, though the
damage is bounded to that operation rather than the whole OM.
h3. Reproduction
{{repro/test_bugMC-1_purge_closes_referenced_db.sh}} with
{{repro/java/SpeculaMc1PurgeCloseTest.java}}. Triggered through the public API
alone: {{cache.get(key)}}, then {{cache.invalidate(key)}}, then a read through
the handle. The snapshot store is a real {{RDBStore}} built with
{{DBStoreBuilder}}, so the post close read takes the production refusal path
rather than a mock. Evidence in {{repro/logs/ev-mc1.log}}.
h3. No patch attached
Fixing this means deciding what invalidate() does when a reader still holds the
snapshot. Leaving the entry in dbMap changes the documented contract and the
existing assertion in TestSnapshotCache that size() drops to zero after
invalidate, and it makes close() through invalidateAll() leave open DBs behind
at shutdown. Closing only once the last reference goes away needs deferred
close state that does not exist today. Filing the analysis so the semantic can
be agreed first.
Found by TLA+ model checking and code review of the OM snapshot cache under
HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated with
Specula (Claude Opus 5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
