John Zhuge created SPARK-59724:
----------------------------------
Summary: [Spark Connect] Coalesce per-response ReleaseExecute RPCs
to cut per-statement round-trip overhead
Key: SPARK-59724
URL: https://issues.apache.org/jira/browse/SPARK-59724
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 4.1.0
Reporter: John Zhuge
A reattachable statement sends ExecutePlan plus one *ReleaseExecute*
synchronously per response batch. Many release RPCs for a small statement like
"USE default". The latency is acerbated when one or more hops are inserted
between Spark Connect client and server, e.g. a reserve proxy.
Releases are important to release sources on the server, however
* after every response, ExecuteResponseObserver keeps only
the last observerRetryBufferSize (default 10m)
* Release timing does not seem to affect correctness.
Periodic maintenance checks also help
{noformat}
spark.connect.execute.manager.detachedTimeout — default 5m
spark.connect.execute.manager.maintenanceInterval — default 30s
{noformat}
Thus I'd like to propose coalescing multiple releases, disabled by default,
{noformat}
spark.connect.execute.reattachable.release.batch = 1 (default)
{noformat}
SPARK-59032 is about release leak; this is the round-trip overhead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
