Hyukjin Kwon created SPARK-60997:
------------------------------------
Summary: Apply the per-RPC deadlines of PySpark Connect in the
Rust client, and reattach an execution past its deadline
Key: SPARK-60997
URL: https://issues.apache.org/jira/browse/SPARK-60997
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect 4.3.0 applies per-RPC deadlines, {{RpcDeadlines}} of
{{pyspark.sql.connect.client}}: 10 minutes for each stream of a reattachable
{{ExecutePlan}} and each {{ReattachExecute}}, an hour for {{AnalyzePlan}} and
{{AddArtifacts}}, and 10 minutes for {{Config}}, {{Interrupt}},
{{ReleaseSession}}, {{ArtifactStatus}}, {{CloneSession}}, {{GetStatus}} and
{{FetchErrorDetails}}. When the deadline of an execution's stream fires, the
operation continues on the server and the client opens a new
{{ReattachExecute}} stream to resume its results
({{ExecutePlanResponseReattachableIterator}}); a {{DEADLINE_EXCEEDED}} of
another call raises {{SparkConnectGrpcException}}. So a call on a connection
that broke without an error does not wait forever.
The Rust client sets no deadline on its calls, so such a call waits until the
connection fails, and a {{DEADLINE_EXCEEDED}} of an execution's stream, as from
a proxy, fails the execution where PySpark reattaches. Apply the deadlines of
PySpark 4.3.0 and reattach an execution whose stream exceeds its deadline.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
