[
https://issues.apache.org/jira/browse/SPARK-60081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60081:
-----------------------------------
Labels: pull-request-available (was: )
> Python UDFs get the wrong value for TIMESTAMPs in the repeated DST hour
> -----------------------------------------------------------------------
>
> Key: SPARK-60081
> URL: https://issues.apache.org/jira/browse/SPARK-60081
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 4.2.0, 4.1.1, 5.0.0
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: pull-request-available
>
> When clocks go back for DST, one local hour happens twice. For a TIMESTAMP in
> the second occurrence of that hour, a Python UDF gets a value that is one
> hour too early. Even an identity UDF {{lambda x: x}} returns a different
> value.
>
> {code:java}
> import os, time
> os.environ["TZ"] = "America/Los_Angeles"
> time.tzset()
> from pyspark.sql import SparkSession, functions as F
> from pyspark.sql.types import LongType, TimestampType
> spark = SparkSession.builder.config("spark.sql.session.timeZone",
> "America/Los_Angeles").getOrCreate()
> # 2021-11-07 09:30 UTC = 01:30 PST, the second 01:30 of the night the clocks
> go back.
> # Its epoch second is 1636277400.
> df = spark.sql("SELECT timestamp'2021-11-07 09:30:00Z' AS t")# 1) The value a
> Python UDF receives
> udf_input = F.udf(lambda x: int(x.timestamp()), LongType())
> df.select(udf_input("t")).first()[0]
> # expected: 1636277400
> # Spark 4.2.0: 1636273800 (one hour early)# 2) An identity UDF should return
> its input unchanged
> identity = F.udf(lambda x: x, TimestampType())
> df.select(F.unix_timestamp(identity("t"))).first()[0]
> # expected: 1636277400
> # Spark 4.2.0: 1636273800 (one hour early)
> {code}
>
> The results above use 4.2.0's default, an Arrow-optimized UDF. Both paths are
> wrong on 4.1.1, 4.2.0 and master:
> || ||What the UDF receives||What an identity UDF returns||
> |Arrow UDF ({{{}useArrow=True{}}}, default in 4.2)|1 hour early|1 hour early|
> |Pickled UDF ({{{}useArrow=False{}}}, default in 4.1)|correct|1 hour early|
>
> The first 01:30 (PDT, {{{}08:30Z{}}}) and all other times are correct.
> *Cause*
> Python marks the second occurrence with {{{}datetime.fold = 1{}}}, and
> PySpark drops {{fold}} in two places:
> # Arrow input conversion ({{{}pyspark/sql/conversion.py{}}}):
> {{value.astimezone().replace(tzinfo=None)}} always gives {{{}fold=0{}}}. A
> fix: {{{}datetime.datetime.fromtimestamp(value.timestamp()){}}}.
> # {{TimestampType.toInternal}} ({{{}pyspark/sql/types.py{}}}, used for UDF
> results on the pickled path and by {{{}createDataFrame{}}}):
> {{time.mktime(dt.timetuple())}} ignores {{{}fold{}}}. A fix:
> {{{}dt.timestamp(){}}}.
> Since pyspark-4.2 makes Arrow UDFs the default, every Python UDF now reads
> these timestamps wrongly by default, while the pickled path reads them
> correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
