[
https://issues.apache.org/jira/browse/HDDS-16596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-16596:
-------------------------------------
Assignee: Andrey Yarovoy
> Isolate synchronous Ratis-commit and SCM RPC waits from the OM client handler
> pool so a transient bucket-lock convoy cannot escalate to full-pool exhaustion
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16596
> URL: https://issues.apache.org/jira/browse/HDDS-16596
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h2. Problem
> The OM services client RPCs on a single shared IPC handler pool
> ({{{}ozone.om.handler.count.key{}}}). Write requests block their handler
> thread synchronously for the duration of the Ratis commit —
> {{OzoneManagerRatisServer.submitRequestToRatis}} parks the handler in
> {{CompletableFuture.get()}} until the write is replicated — and outbound SCM
> calls ({{{}allocateBlock{}}}, container/pipeline location) likewise park
> their handler for the whole round trip inside {{RetryInvocationHandler}} /
> {{{}Client.getRpcResponse{}}}.
> Because these waits sit on the same pool that serves reads, the pool has no
> reserved headroom. Under sustained write load a large fraction of handlers
> are permanently parked on commit, so the pool runs at near-zero idle at rest.
> When a transient contention event then arrives — for example a bucket
> read/write-lock convoy that would otherwise drain in microseconds — the
> arriving requests consume the few remaining handlers and the localized stall
> escalates into a full-pool outage affecting unrelated buckets and read RPCs.
> h2. Evidence
> On the heavy loaded cluster with mixed IO patterns per-thread analysis of OM
> stack dumps from a saturated OM (200-thread handler pool) versus a healthy OM
> under comparable client load:
> The two profiles carry the same peak lock-convoy magnitude (145 vs 133
> threads), so the convoy itself is not the differentiator. The differentiator
> is headroom: the saturated OM held ~93% of its handlers in commit waits with
> zero idle, so a convoy had nothing to absorb it; the healthy OM held ~93% of
> handlers idle and absorbed convoys of similar size without a pool-wide stall.
> h2. Proposed direction
> Decouple the blocking waits from the client-facing handler pool so a
> transient convoy cannot consume the whole pool:
> * Complete the Ratis commit asynchronously (release the handler once the
> request is submitted, respond on commit) rather than blocking the handler in
> {{{}get(){}}}, *or* run commit waits on a bounded executor separate from the
> handler pool.
> * Isolate outbound SCM RPCs onto a bounded pool with backpressure so a slow
> SCM cannot exhaust client handlers.
> * As a floor, ensure read RPCs retain reserved capacity independent of
> in-flight write commits ({{{}ozone.om.read.threadpool{}}}, default 10, is
> small relative to the handler pool).
> Each has correctness/backpressure implications (in-flight accounting,
> retry/failover, ordering) and should be scoped as design subtasks.
> h2. Testing
> * A concurrency benchmark that saturates the write path and confirms
> read-RPC latency and idle-handler count stay bounded while commits are in
> flight.
> * A fault-injection test with a deliberately slow downstream (Ratis or SCM)
> asserting the handler pool retains headroom rather than reaching full
> occupancy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
