Hyukjin Kwon created SPARK-61164:
------------------------------------
Summary: Make AnalyzeArgument, PartitioningColumn, OrderingColumn
and SelectedColumn the frozen dataclasses of PySpark in the Python client
Key: SPARK-61164
URL: https://issues.apache.org/jira/browse/SPARK-61164
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's AnalyzeArgument, PartitioningColumn, OrderingColumn and
SelectedColumn, of the analyze method of a polymorphic UDTF, are frozen
dataclasses of pyspark.sql.udtf. A handler that captures them, such as one
whose analyze returns an AnalyzeResult defined outside of it, pickles with
them, by reference, and the Python worker of the server, which runs analyze,
unpickles them as PySpark's.
In the Python client, they are classes of the Rust extension, which cannot be
pickled:
PySparkPicklingError: [UDTF_SERIALIZATION_ERROR] Cannot serialize the UDTF
'TestUDTF' ... TypeError: cannot pickle 'pyspark.sql.udtf.PartitioningColumn'
object
They also lack the behavior of dataclasses (fields, equality, hashing, repr,
FrozenInstanceError). Apache Spark's
test_udtf_with_scalar_analyze_returning_wrong_result and
test_udtf_with_table_analyze_returning_wrong_result of test_parity_udtf.py fail
on this client, in two variants each.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
