[
https://issues.apache.org/jira/browse/SPARK-61108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61108:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Name the nested fields of collected rows by the schema, and raise for
> calendar intervals in collect, in the Python client as PySpark does
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61108
> URL: https://issues.apache.org/jira/browse/SPARK-61108
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{DataFrame.collect}} (and so {{first}}, {{head}},
> {{take}} and {{tail}}) checks the Arrow schema of the result with
> {{from_arrow_schema}}. It then builds the rows by the schema of the result:
> * The fields of nested structs have the names of the schema. Duplicate names
> are kept: {{SELECT named_struct('a', 1, 'a', 2) AS v}} collects
> {{Row(v=Row(a=1, a=2))}}, while the server's Arrow data names them {{a_0}},
> {{a_1}}. {{toLocalIterator}} names them the same way.
> * A calendar interval raises {{UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION}}
> ("month_day_nano_interval is not supported in conversion to Arrow").
> The Python client instead named nested fields by the Arrow data:
> {{Row(v=Row(a_0=1, a_1=2))}}, where {{v.a}} raised
> {{ATTRIBUTE_NOT_SUPPORTED}}. It also collected a calendar interval as a
> string such as "14 months 25 days 18367500000000 ns".
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
