[
https://issues.apache.org/jira/browse/SPARK-61142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61142:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Make the functions of pyspark.sql.functions those of
> pyspark.sql.functions.builtin, which pickle by reference, in the Python
> client as PySpark does
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61142
> URL: https://issues.apache.org/jira/browse/SPARK-61142
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark 4.3.0, the functions of pyspark.sql.functions are defined in
> pyspark.sql.functions.builtin (all but pandas_udf and arrow_udf, of
> pyspark.sql.pandas.functions), so pickle and cloudpickle pickle them by
> reference, by that name.
> In the Python client, 518 of them, such as abs and array_sort, are of the
> module functions_generated, which is not importable. cloudpickle pickles them
> by value, and fails on the Rust functions that they call:
> _pickle.PicklingError: Can't pickle <built-in function
> pyfunc_invoke_function>: import of module 'functions' failed
> So a UDF, a UDTF or a foreachBatch function that captures one of them cannot
> be sent. Apache Spark's
> test_parity_pandas_transform_with_state_state_variable.py fails on this
> client (STREAMING_CONNECT_SERIALIZATION_ERROR of foreachBatch functions that
> capture array_sort and col).
> The functions should be of pyspark.sql.functions.builtin, as PySpark's are.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
