Andrey Yarovoy created HDDS-16289:
-------------------------------------
Summary: Narrow the FSO create/commit apply-path bucket write-lock
to the cache-mutation tail so it does not gate readers of a hot bucket
Key: HDDS-16289
URL: https://issues.apache.org/jira/browse/HDDS-16289
Project: Apache Ozone
Issue Type: Improvement
Reporter: Andrey Yarovoy
*Problem*
On the FSO write-apply path, {{validateAndUpdateCache}} acquires the
{{BUCKET_LOCK}} *write* lock and holds it across a block of work that is mostly
{_}reads of already-committed state{_}, not cache mutation. Because a hot
bucket hashes to a single lock stripe and the OM applies transactions on a
*single* Ratis apply thread, the duration of this write-lock hold is exactly
the window during which every read RPC on that bucket ({{{}getBucketInfo{}}},
{{{}getBucketOwner{}}}, {{{}readKeyInfo{}}}) is blocked — Java's
{{ReentrantReadWriteLock}} blocks incoming readers whenever a writer is queued
at the head, in both fair and non-fair (default) mode.
OM {{/stacks}} analysis showed this as recurring flares in handler threads sit
in {{OzoneManagerLock.acquireLock}} (read side) behind the single apply
thread's {{{}acquireWriteLock{}}}, with only a handful of readers actually
inside the critical section. The critical section is not slow because of a
contended mutation — it is slow because read-heavy work runs under the write
lock.
*Where the time goes*
({{{}OMFileCreateRequestWithFSO.validateAndUpdateCache{}}}, same shape in
{{{}OMKeyCommitRequestWithFSO{}}})
Held under the write lock today:
* {{verifyDirectoryKeysInPath(...)}} — the FSO path walk: one RocksDB point
lookup per path segment ({{{}directoryTable.get{}}}) plus a {{fileTable.get}}
for the leaf. This is the dominant cost and it only *reads* committed/cache
state.
* {{getOmKeyInfoFromFileTable(...)}} — another RocksDB read.
* {{{}checkDirectoryResult{}}}, {{{}checkAllParentsExist{}}},
{{{}getAllMissingParentDirInfo{}}}, quota checks — validation over the values
just read.
* *Only the tail* genuinely needs write exclusivity:
{{{}addOpenFileTableCacheEntry{}}}, {{{}addDirectoryTableCacheEntries{}}}, and
the in-place bucket quota update ({{{}incrUsedNamespace{}}}).
The code already hints at this — the comment above
{{addOpenFileTableCacheEntry}} notes the open-key cache add "can be done
outside of lock."
*Proposed change (investigation + implementation)*
Reorder so the exclusive section covers only the cache mutations that require
it:
# Resolve the path and read existing entries / build {{missingParentInfos}}
and the prepared {{OmKeyInfo}} _before_ taking the write lock (or under the
shared read lock).
# Acquire the bucket write lock and, under it, *re-validate the invariants
that could have changed* (existence/overwrite, quota) against the current
cache, then apply the cache entries and quota update.
# Release.
Because writes are applied by a single thread, this does not change write
throughput — the prior finding that write-lock _scope_ reduction is low-value
for write p99 (single apply thread bounds write commit rate) still holds. The
value here is on a {*}different axis: read availability/latency on hot
buckets{*}, which the current wide critical section directly harms.
*Testing*
* Concurrency test: drive many read RPCs at one bucket while a create/commit
is applied to that bucket; assert readers block only for the (short) mutation
tail, not the path-resolution phase.
* Correctness: existing {{TestOMFileCreateRequestWithFSO}} /
{{TestOMKeyCommitRequestWithFSO}} plus new cases for the
re-validation-under-lock path (concurrent overwrite, quota-exceeded discovered
at re-check).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
