Kaicheng Zhou created SPARK-59125:
-------------------------------------
Summary: Spark Connect sends cleanup RPC after the session is
released
Key: SPARK-59125
URL: https://issues.apache.org/jira/browse/SPARK-59125
Project: Spark
Issue Type: Bug
Components: Connect, PySpark
Affects Versions: 4.2.0, 4.3.0, 4.4.0
Reporter: Kaicheng Zhou
SparkConnectClient._on_exit() unconditionally invokes _cleanup_ml_cache() for
clients that remain open. If release_session() was called explicitly, the
server-side session has already been released, but the client is not marked as
closed. Consequently, the exit handler sends an unnecessary ML cache cleanup
RPC to the released session. The client should record a successful session
release and make _on_exit() a no-op afterward. This avoids redundant RPCs and
potential delays during interpreter shutdown.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
