[
https://issues.apache.org/jira/browse/SPARK-61107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61107.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 307
[https://github.com/apache/spark-connect-rust/pull/307]
> Cast the table of toArrow to the Arrow schema of the DataFrame in the Python
> client as PySpark does
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-61107
> URL: https://issues.apache.org/jira/browse/SPARK-61107
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{DataFrame.toArrow}} first converts the schema of the
> DataFrame to Arrow, with {{to_arrow_schema(df.schema,
> error_on_duplicated_field_names_in_struct=True, timezone="UTC")}}. It then
> runs the query and casts the table to that schema. So:
> * The table has the Arrow types of the Spark types. Strings and binaries are
> {{string}} and {{binary}} even with
> {{spark.sql.execution.arrow.useLargeVarTypes}}. Timestamps are of UTC.
> * Nested struct fields have the names of the schema, not the deduplicated
> names of the server's Arrow data. Duplicate names raise
> {{DUPLICATED_FIELD_NAME_IN_ARROW_STRUCT}}.
> * Types that PySpark does not convert to Arrow raise
> {{UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION}} before the query runs. These
> are year-month intervals, calendar intervals, char and varchar.
> * An array or map of NullType that is not nullable raises PyArrow's "A null
> type field may not be non-nullable".
> The Python client instead returned the Arrow data of the server as it was.
> That gave {{large_string}} columns, the names {{a_0}}, {{a_1}} for duplicate
> fields, and {{month_interval}} columns that PyArrow cannot read ({{KeyError:
> 21}}).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
