Hyukjin Kwon created SPARK-61142:
------------------------------------
Summary: Make the functions of pyspark.sql.functions those of
pyspark.sql.functions.builtin, which pickle by reference, in the Python client
as PySpark does
Key: SPARK-61142
URL: https://issues.apache.org/jira/browse/SPARK-61142
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark 4.3.0, the functions of pyspark.sql.functions are defined in
pyspark.sql.functions.builtin (all but pandas_udf and arrow_udf, of
pyspark.sql.pandas.functions), so pickle and cloudpickle pickle them by
reference, by that name.
In the Python client, 518 of them, such as abs and array_sort, are of the
module functions_generated, which is not importable. cloudpickle pickles them
by value, and fails on the Rust functions that they call:
_pickle.PicklingError: Can't pickle <built-in function pyfunc_invoke_function>:
import of module 'functions' failed
So a UDF, a UDTF or a foreachBatch function that captures one of them cannot be
sent. Apache Spark's test_parity_pandas_transform_with_state_state_variable.py
fails on this client (STREAMING_CONNECT_SERIALIZATION_ERROR of foreachBatch
functions that capture array_sort and col).
The functions should be of pyspark.sql.functions.builtin, as PySpark's are.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
