Hyukjin Kwon created SPARK-61079:
------------------------------------
Summary: Convert pandas input with an explicit schema as PySpark's
Arrow conversion does in createDataFrame of the Rust client
Key: SPARK-61079
URL: https://issues.apache.org/jira/browse/SPARK-61079
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
createDataFrame of a pandas DataFrame with an explicit schema in the Rust
client converts each cell itself and differs from PySpark 4.3.0's Arrow
conversion: NaN or None in a numeric column into an int field fails, an int64
257 into a tinyint field wraps to 1 silently, a null in a non-nullable field
raises another error, naive timestamps are read in the client's time zone
instead of spark.sql.session.timeZone, nanoseconds are truncated silently, lone
surrogates and datetimes in date fields fail differently, nested NaN is kept
instead of becoming null, and an int beyond int64 in an inferred object column
becomes a double instead of raising OverflowError.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
