[
https://issues.apache.org/jira/browse/SPARK-61164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61164:
------------------------------------
Assignee: Hyukjin Kwon
> Make AnalyzeArgument, PartitioningColumn, OrderingColumn and SelectedColumn
> the frozen dataclasses of PySpark in the Python client
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61164
> URL: https://issues.apache.org/jira/browse/SPARK-61164
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark's AnalyzeArgument, PartitioningColumn, OrderingColumn and
> SelectedColumn, of the analyze method of a polymorphic UDTF, are frozen
> dataclasses of pyspark.sql.udtf. A handler that captures them, such as one
> whose analyze returns an AnalyzeResult defined outside of it, pickles with
> them, by reference, and the Python worker of the server, which runs analyze,
> unpickles them as PySpark's.
> In the Python client, they are classes of the Rust extension, which cannot be
> pickled:
> PySparkPicklingError: [UDTF_SERIALIZATION_ERROR] Cannot serialize the UDTF
> 'TestUDTF' ... TypeError: cannot pickle 'pyspark.sql.udtf.PartitioningColumn'
> object
> They also lack the behavior of dataclasses (fields, equality, hashing, repr,
> FrozenInstanceError). Apache Spark's
> test_udtf_with_scalar_analyze_returning_wrong_result and
> test_udtf_with_table_analyze_returning_wrong_result of test_parity_udtf.py
> fail on this client, in two variants each.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
