[
https://issues.apache.org/jira/browse/HDDS-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116515#comment-18116515
]
Wei-Chiu Chuang commented on HDDS-16418:
----------------------------------------
Thanks [~hanifou] I'm more inclined to (B) + (A):
Other gRPC server/client builder imposes timeout and interval default, it would
make sense to have a default consistent with them
For example, XceiverServerGrpc, while not configurable, has:
.maxConnectionIdle(15, TimeUnit.MINUTES)
.keepAliveTime(5, TimeUnit.MINUTES)
.keepAliveTimeout(30, TimeUnit.SECONDS)
Side note, out of scope: a few gRPC server/client don't configure
timeout/intervals:
GrpcOzoneManagerServer, XceiverServerGrpc, ReplicationServer
> Make Inter-SCM gRPC timeouts and keepalive operator-tunable
> -----------------------------------------------------------
>
> Key: HDDS-16418
> URL: https://issues.apache.org/jira/browse/HDDS-16418
> Project: Apache Ozone
> Issue Type: Improvement
> Components: SCM HA
> Reporter: Wei-Chiu Chuang
> Assignee: Mohammadhani Fouladgar
> Priority: Major
> Labels: configuration, grpc, scm-ha
>
> h3. Problem
> SCM HA followers (and Recon) download the leader SCM RocksDB checkpoint over
> Inter-SCM gRPC during install snapshot. Transfers can run for a long time and
> depend on network stability, {{scm.db}} size, and disk throughput.
> Today operators have almost no way to tune gRPC timeout and connection
> behavior for this path:
> * *Client* ({{InterSCMGrpcClient}}): only
> {{ozone.scm.ha.grpc.deadline.interval}} exists for the stub RPC deadline. No
> documented client keepalive / idle settings on the channel. (See also
> HDDS-16417 for incorrect application of the deadline value.)
> * *Server* ({{InterSCMGrpcProtocolService}}): only port
> ({{ozone.scm.grpc.port}}) and {{maxInboundMessageSize}} are set. Keepalive,
> max connection idle, and related Netty server options use gRPC-java defaults
> and are not exposed in {{ozone-default.xml}}.
> By contrast, datanode container gRPC in {{XceiverServerGrpc}} sets explicit
> {{maxConnectionIdle}}, {{keepAliveTime}}, and {{keepAliveTimeout}} (currently
> hardcoded, not config either, but documented in code comments).
> Operators troubleshooting snapshot download stalls, proxy idle timeouts, or
> load-balancer disconnects cannot align Inter-SCM gRPC behavior with their
> environment without code changes.
> h3. Proposal
> Introduce *operator-tunable* Inter-SCM gRPC timeout and keepalive settings
> under a consistent {{ozone.scm.ha.grpc.*}} prefix (names to be finalized in
> review), for example:
> * Client RPC deadline (retain / clarify
> {{ozone.scm.ha.grpc.deadline.interval}})
> * Server: {{maxConnectionIdle}}, {{keepAliveTime}}, {{keepAliveTimeout}} (and
> any other knobs needed for long-lived streaming RPCs)
> * Client channel: matching keepalive / idle settings where applicable so
> client and server policies are coherent
> Defaults should remain safe for large checkpoint downloads (generous
> idle/deadline vs datanode chunk RPCs). Document each property in
> {{ozone-default.xml}} with units and interaction with TLS.
> h3. Affected code
> *
> {{hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcClient.java}}
> *
> {{hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcProtocolService.java}}
> *
> {{hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java}}
> * {{hadoop-ozone/recon/.../StorageContainerServiceProviderImpl.java}} (Recon
> uses {{InterSCMGrpcClient}})
> h3. Acceptance criteria
> * New config keys wired on both Inter-SCM gRPC client and server builders.
> * Defaults preserve current effective behavior (after fixing HDDS-16417
> deadline units).
> * Unit or integration test coverage that config values are read and applied
> (mock or small local server/client test acceptable).
> * Release note if defaults or operator-visible behavior changes.
> h3. Related
> * HDDS-16417 — deadline interval TimeUnit bug on the existing client property.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
