Hyukjin Kwon created SPARK-61129:
------------------------------------
Summary: Make SparkSession and DataFrame picklable, of the active
session, in the Python client as PySpark does
Key: SPARK-61129
URL: https://issues.apache.org/jira/browse/SPARK-61129
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect's SparkSession and DataFrame can be pickled with cloudpickle.
- A session pickles as a function of its ID, which gives back the active
session of that ID (SparkSession._getActiveSessionIfMatches). Where no session
is active, such as on a Python worker of the server, it raises
PySparkRuntimeError NO_ACTIVE_SESSION. Where another session is active, it
raises PySparkAssertionError.
- A DataFrame pickles of its plan and its session.
So a UDF or UDTF that captures the session or a DataFrame is sent to the
server. If it is called there, it raises PythonException NO_ACTIVE_SESSION, as
Apache Spark's test_udtf_access_spark_session_connect checks. In the same
process, a pickled DataFrame unpickles as a DataFrame of the same plan.
The Python client of spark-connect-rust cannot pickle either: "TypeError:
cannot pickle 'pyspark.sql.session.SparkSession' object", and the same for its
DataFrame. Such a UDF or UDTF fails on the client, and
test_udtf_access_spark_session_connect fails. The client also lacks
SparkSession._getActiveSessionIfMatches.
The client's SparkSession and DataFrame should pickle as PySpark's do.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
