Hyukjin Kwon created SPARK-61106:
------------------------------------
Summary: Convert the values of local rows of a schema, and raise
their errors, as PyArrow does in createDataFrame in the Python client
Key: SPARK-61106
URL: https://issues.apache.org/jira/browse/SPARK-61106
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{spark.createDataFrame}} of local rows with a schema
passes the values of most atomic types to PyArrow as they are. PyArrow then
converts them, or raises its own errors:
* An int, date, day-time interval or time field takes any value that has
{{__index__}}, or else {{__int__}}: a {{Decimal}}, a float or a NumPy scalar. A
bool raises {{ArrowTypeError}} ("Expected integer, got bool").
** {{Decimal('1.5')}} is 1 in an int field, 1970-01-02 in a date field and 1
microsecond in a day-time interval field.
** A NaN in a date, day-time interval or time field raises "cannot convert
float NaN to integer".
* A value that PyArrow cannot convert raises {{pyarrow.lib.ArrowInvalid}}, or
{{pyarrow.lib.ArrowTypeError}}, of PyArrow's message:
** "Could not convert 'x' with type str: tried to convert to int32"
** "tried to convert to float32" for a float field
** "Value 1099511627776 too large to fit in C integer type"
** "PyLong is too large to fit int64" for an int beyond int64 in a float or
double field
** "Integer too large for date32"
** "object of type <class 'str'> cannot be converted to int"
The Python client converts these values in Rust, emulating PyArrow. It instead:
* raised plain {{ValueError}}s and {{TypeError}}s
* named float32 "float"
* raised for a {{Decimal}} in an int, date or day-time interval field
* raised {{OverflowError}} for a large int in a date field
* took a bool, or a NaN, in a time field
Of 374 cases of a value of each kind in a field of each atomic type, 119
differed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
