Hyukjin Kwon created SPARK-61125:
------------------------------------
Summary: Pickle the handler of a UDTF when it is used, and raise
UDTF_SERIALIZATION_ERROR, in the Python client as PySpark does
Key: SPARK-61125
URL: https://issues.apache.org/jira/browse/SPARK-61125
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect pickles the handler class of a UDTF when the UDTF is used, as
it builds the {{PythonUDTF}} of the plan, and raises a {{PySparkPicklingError}}
of {{UDTF_SERIALIZATION_ERROR}} ("Please check the stack trace and make sure
the function is serializable.") for a {{pickle.PicklingError}}. Apache Spark's
{{test_udtf_pickle_error}} checks it, with a handler that refers to a file
opened for writing.
The Python client pickles the handler when {{udtf}} decorates the class, and
raises the {{pickle.PicklingError}} as it is. So the error is raised where the
UDTF is defined, rather than where it is used, and of another type.
The fix is to pickle the handler when the UDTF is used, and to raise
{{UDTF_SERIALIZATION_ERROR}}, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
