[
https://issues.apache.org/jira/browse/SPARK-61184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61184:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Send the server-side session ID that the client observed, and check that of
> the responses, so that a session ends when its server restarts, as PySpark
> does
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61184
> URL: https://issues.apache.org/jira/browse/SPARK-61184
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect keeps the server-side session ID of the responses and sends
> it as client_observed_server_side_session_id on its requests (ExecutePlan,
> AnalyzePlan, Config, AddArtifacts, ArtifactStatus, Interrupt,
> ReattachExecute, ReleaseExecute, FetchErrorDetails). After a restart of the
> server, the server raises INVALID_HANDLE.SESSION_CHANGED ("The existing Spark
> server driver instance has restarted. Please reconnect."), and the client
> closes its session, so the next calls raise NO_ACTIVE_SESSION. The client
> also checks the server-side session ID of each response
> (_verify_response_integrity) and raises a PySparkAssertionError when it
> changes.
> The Rust client neither sends nor checks it. After a restart of the server, a
> session silently goes on in a new session of the server: its configurations
> are back to their defaults and its temporary views are gone. For example,
> after spark.conf.set("spark.sql.shuffle.partitions", "7") and a restart of
> the server, spark.conf.get("spark.sql.shuffle.partitions") gives "200", where
> PySpark raises NO_ACTIVE_SESSION.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
