Yen-Hsiang, Wang created SPARK-60143:
----------------------------------------
Summary: Make make_timestamp tests in test_functions independent
of the local timezone
Key: SPARK-60143
URL: https://issues.apache.org/jira/browse/SPARK-60143
Project: Spark
Issue Type: Test
Components: PySpark
Affects Versions: 4.4.0
Reporter: Yen-Hsiang, Wang
Three tests in {{pyspark.sql.tests.test_functions}} fail deterministically when
the test process runs in a non-UTC local timezone:
* {{FunctionsTests.test_try_make_timestamp_ltz}}
* {{FunctionsTests.test_try_make_timestamp}}
* {{FunctionsTests.test_make_timestamp}}
They pass on CI only because the PySpark jobs run in Docker containers whose
default timezone is UTC. Neither the developer docs nor
{{python/run-tests}} require or set {{TZ=UTC}} for local runs, so anyone
running these tests on a laptop outside UTC hits the failures.
h3. Root cause 1: timezone abbreviation passed to Spark
({{test_try_make_timestamp_ltz}})
The test derives the "local timezone" with:
{code:python}
# use local timezone here to avoid flakiness
data = [(2024, 5, 22, 10, 30, 0,
datetime.datetime.now().astimezone().tzinfo.__str__())]
{code}
{{tzinfo.__str__()}} returns an abbreviation such as {{CST}} or {{PDT}}, not
an IANA zone ID. Abbreviations are ambiguous, and Java resolves
them differently from the OS:
* {{TZ=Asia/Taipei}}: Python yields {{'CST'}} (China Standard Time, UTC+8),
but Java interprets {{CST}} as US Central Time (UTC-5 in May).
10:30 becomes 23:30 local, a 13-hour shift.
* {{TZ=America/Los_Angeles}}: Python yields {{'PDT'}}, which Spark rejects
outright with {{[INVALID_TIMEZONE] The timezone: PDT is invalid}}.
So the comment's intent ("avoid flakiness") is not met: the test is
guaranteed to fail in these timezones.
h3. Root cause 2: expected values hardcoded in UTC ({{test_make_timestamp}},
{{test_try_make_timestamp}})
The input is {{2024-05-22 10:30 CET}} and the expected value is hardcoded:
{code:python}
result_with_tz = datetime.datetime(2024, 5, 22, 8, 30)
result_frac_with_tz = datetime.datetime(2024, 5, 22, 8, 30, 45, 123000)
{code}
10:30 CET (UTC+2 in May) is 08:30 *UTC*, but {{collect()}} returns a naive
datetime in the *local* timezone. The expectation therefore only
holds when the local timezone is UTC (e.g. 16:30 in Asia/Taipei, 01:30 in
America/Los_Angeles).
h3. How to reproduce
From a Spark checkout built with {{build/sbt -Phive package}}:
{noformat}
TZ=Asia/Taipei python/run-tests --testnames "pyspark.sql.tests.test_functions
FunctionsTests.test_try_make_timestamp_ltz"
TZ=Asia/Taipei python/run-tests --testnames "pyspark.sql.tests.test_functions
FunctionsTests.test_make_timestamp"
TZ=America/Los_Angeles python/run-tests --testnames
"pyspark.sql.tests.test_functions FunctionsTests.test_try_make_timestamp_ltz"
{noformat}
All fail; the same commands with {{TZ=UTC}} pass.
Minimal reproduction of root cause 1:
{code:python}
# with TZ=Asia/Taipei
spark.sql("SELECT make_timestamp(2024, 5, 22, 10, 30, 0, 'CST') AS
ts").first().ts
# datetime.datetime(2024, 5, 22, 23, 30) <- expected 10:30
{code}
h3. Proposed fix (test-only)
* In {{test_try_make_timestamp_ltz}}, pass an unambiguous zone ID (e.g.
{{"UTC"}}) instead of {{tzinfo.__str__()}}.
* Compute expected values by converting the known UTC instant to local time
instead of hardcoding them, e.g.:
{code:python}
datetime.datetime(2024, 5, 22, 8, 30,
tzinfo=datetime.timezone.utc).astimezone().replace(tzinfo=None)
{code}
This keeps the semantics under test unchanged and makes the tests pass under
any local timezone. Verified locally under UTC, Asia/Taipei and
America/Los_Angeles.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
