Fangchen Li created SPARK-59467:
-----------------------------------
Summary: Timezone-naive ArrowDtype timestamps are interpreted as
UTC instead of the session timezone in pandas conversion
Key: SPARK-59467
URL: https://issues.apache.org/jira/browse/SPARK-59467
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 4.4.0
Reporter: Fangchen Li
_check_series_convert_timestamps_internal in pyspark.sql.pandas.types only
recognizes numpy datetime64 and pd.DatetimeTZDtype columns. A timezone-naive
column backed by pd.ArrowDtype (for example, timestamp[us][pyarrow]) falls
through unchanged and is interpreted as UTC rather than in the session
timezone, so in a non-UTC session the value is shifted by the session's UTC
offset:
spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
ts = [datetime.datetime(2020, 1, 1, 12, 0)]
numpy_col = pd.Series(ts, dtype="datetime64[ns]")
arrow_col = pa.array(ts,
pa.timestamp("us")).to_pandas(types_mapper=pd.ArrowDtype)
spark.createDataFrame(pd.DataFrame(\{"t": numpy_col}), "t timestamp").collect()
# [Row(t=datetime.datetime(2020, 1, 1, 12, 0))] -- correct
spark.createDataFrame(pd.DataFrame(\{"t": arrow_col}), "t timestamp").collect()
# [Row(t=datetime.datetime(2020, 1, 1, 4, 0))] -- off by 8 hours
This affects createDataFrame with a TimestampType schema and pandas UDFs that
return ArrowDtype timestamp Series, and it contradicts the documented behavior
of spark.sql.session.timeZone for pandas timestamps.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
