Hyukjin Kwon created SPARK-61107:
------------------------------------
Summary: Cast the table of toArrow to the Arrow schema of the
DataFrame in the Python client as PySpark does
Key: SPARK-61107
URL: https://issues.apache.org/jira/browse/SPARK-61107
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{DataFrame.toArrow}} first converts the schema of the
DataFrame to Arrow, with {{to_arrow_schema(df.schema,
error_on_duplicated_field_names_in_struct=True, timezone="UTC")}}. It then runs
the query and casts the table to that schema. So:
* The table has the Arrow types of the Spark types. Strings and binaries are
{{string}} and {{binary}} even with
{{spark.sql.execution.arrow.useLargeVarTypes}}. Timestamps are of UTC.
* Nested struct fields have the names of the schema, not the deduplicated names
of the server's Arrow data. Duplicate names raise
{{DUPLICATED_FIELD_NAME_IN_ARROW_STRUCT}}.
* Types that PySpark does not convert to Arrow raise
{{UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION}} before the query runs. These are
year-month intervals, calendar intervals, char and varchar.
* An array or map of NullType that is not nullable raises PyArrow's "A null
type field may not be non-nullable".
The Python client instead returned the Arrow data of the server as it was. That
gave {{large_string}} columns, the names {{a_0}}, {{a_1}} for duplicate fields,
and {{month_interval}} columns that PyArrow cannot read ({{KeyError: 21}}).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
