[
https://issues.apache.org/jira/browse/HDDS-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-17561:
----------------------------------
Labels: pull-request-available (was: )
> Deliver blocking XceiverClientGrpc replies on the waiting thread
> ----------------------------------------------------------------
>
> Key: HDDS-17561
> URL: https://issues.apache.org/jira/browse/HDDS-17561
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Datanode
> Affects Versions: 2.2.1
> Reporter: Sergey Soldatov
> Assignee: Sergey Soldatov
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> A blocking {{XceiverClientGrpc#sendCommand}} (ReadChunk, GetBlock and the
> other read-path commands) is built on top of the async path. For every
> request it opens a new bi-directional gRPC stream, sends one request,
> half-closes the stream, and waits on a {{CompletableFuture}}. The reply then
> travels:
> # gRPC transport (Netty event loop) ->
> # gRPC's default executor, where the {{StreamObserver}} parses the response
> and completes the future ->
> # the caller thread, woken up from {{get()}}.
> So each blocking request pays two thread hand-offs plus the setup and
> teardown of an HTTP/2 stream on both the client and the datanode, even though
> the caller is just sitting idle waiting for the reply.
> h3. Proposal
> * *Reply on the waiting thread.* New {{CallerRunsReplyFuture}}: a
> {{CompletableFuture}} that is also the call's {{Executor}}. While a thread
> waits in {{get()}} / {{join()}}, the gRPC callbacks of the call are queued
> and run by that thread, so the response is parsed and the future completed on
> the caller (the same technique gRPC blocking stubs use). Before the caller
> starts waiting and after it returns, callbacks run on a shared daemon
> executor, never on the transport thread.
> * *Pooled streams for untraced blocking calls.* A blocking call without a
> trace context (the production default) leases an idle stream of the channel,
> or opens one, and sends exactly one request on it at a time. The datanode
> serves the requests of a stream in order, so the next reply always belongs to
> the current request and no request IDs are needed. After the reply the stream
> goes back to the pool. The number of streams is bounded by the client
> semaphore (max outstanding requests).
> * *Client-side deadline.* A pooled stream has no gRPC deadline, so the
> request deadline is enforced by the waiter (and by a timer when nobody
> waits). On timeout the request fails with {{DEADLINE_EXCEEDED}}, as before,
> and the stream is cancelled and dropped.
> * *Stream lifecycle.*
> ** Idle streams are half-closed after {{scm.container.client.idle.threshold}}
> by a shared reaper thread, and immediately when the channel leaves {{READY}}
> (e.g. the datanode sends GOAWAY on shutdown), so open streams do not hold up
> the datanode's graceful shutdown.
> ** Half-closed streams that the datanode never completes are cancelled.
> ** A stream is dropped, never reused, when it fails, is completed by the
> datanode, receives an unsolicited reply, or when its request is abandoned
> (timeout, interrupt, cancel).
> * *Unchanged paths.* Traced blocking calls still open one stream per call
> (trace headers are per stream), but also complete on the waiting thread. The
> async path ({{sendCommandAsync(ContainerCommandRequestProto)}}) is unchanged.
> * *Smaller fixes along the way.* If issuing a fan-out {{sendCommand}} to one
> of the datanodes fails, the requests already issued to the others are
> cancelled instead of being left in flight. {{sortDatanodes}} returns early
> for single-node pipelines.
> * {{README.gRPC.md}} is updated to describe the new behavior.
> h3. Compatibility
> Client-only change. No protobuf, RPC or datanode changes; the datanode
> already handles several sequential requests on one stream.
> h3. Testing
> New {{TestXceiverClientGrpcCallerRuns}} (33 tests against an in-process gRPC
> server) covers:
> * replies completing on the caller for {{get()}} and {{join()}}, and off the
> submitter when nobody waits;
> * stream reuse for sequential calls, distinct streams for concurrent calls,
> no more streams than semaphore permits;
> * deadline expiry, timed waits shorter than the request deadline, interrupted
> waiters, cancel from another thread, late waiters;
> * stream errors, datanode-completed streams, unsolicited replies, cancel
> racing with the reply;
> * idle-threshold reaping, half-closed streams never completed by the
> datanode, server shutdown not being held up;
> * traced calls using a stream per call, dead channels failing fast without
> leaking permits, and fan-out cancellation.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
