[
https://issues.apache.org/jira/browse/SPARK-60143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60143:
-----------------------------------
Labels: pull-request-available (was: )
> Make make_timestamp tests in test_functions independent of the local timezone
>
> ---------------------------------------------------------------------------------
>
> Key: SPARK-60143
> URL: https://issues.apache.org/jira/browse/SPARK-60143
> Project: Spark
> Issue Type: Test
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: Yen-Hsiang, Wang
> Priority: Minor
> Labels: pull-request-available
>
> Three tests in {{pyspark.sql.tests.test_functions}} fail deterministically
> when the test process runs in a non-UTC local timezone:
> * {{FunctionsTests.test_try_make_timestamp_ltz}}
> * {{FunctionsTests.test_try_make_timestamp}}
> * {{FunctionsTests.test_make_timestamp}}
> They pass on CI only because the PySpark jobs run in Docker containers whose
> default timezone is UTC. Neither the developer docs nor
> {{python/run-tests}} require or set {{TZ=UTC}} for local runs, so anyone
> running these tests on a laptop outside UTC hits the failures.
> h3. Root cause 1: timezone abbreviation passed to Spark
> ({{{}test_try_make_timestamp_ltz{}}})
> The test derives the "local timezone" with:
> {code:python}
> # use local timezone here to avoid flakiness
> data = [(2024, 5, 22, 10, 30, 0,
> datetime.datetime.now().astimezone().tzinfo.__str__())]
> {code}
> {{tzinfo._{_}str{_}_()}} returns an abbreviation such as {{CST}} or
> {{{}PDT{}}}, not an IANA zone ID. Abbreviations are ambiguous, and Java
> resolves
> them differently from the OS:
> * {{{}TZ=Asia/Taipei{}}}: Python yields {{'CST'}} (China Standard Time,
> UTC+8), but Java interprets {{CST}} as US Central Time (UTC-5 in May).
> 10:30 becomes 23:30 local, a 13-hour shift.
> * {{{}TZ=America/Los_Angeles{}}}: Python yields {{{}'PDT'{}}}, which Spark
> rejects outright with {{{}[INVALID_TIMEZONE] The timezone: PDT is invalid{}}}.
> So the comment's intent ("avoid flakiness") is not met: the test is
> guaranteed to fail in these timezones.
> h3. Root cause 2: expected values hardcoded in UTC
> ({{{}test_make_timestamp{}}}, {{{}test_try_make_timestamp{}}})
> The input is {{2024-05-22 10:30 CET}} and the expected value is hardcoded:
> {code:python}
> result_with_tz = datetime.datetime(2024, 5, 22, 8, 30)
> result_frac_with_tz = datetime.datetime(2024, 5, 22, 8, 30, 45, 123000)
> {code}
> 10:30 CET (UTC+2 in May) is 08:30 {*}UTC{*}, but {{collect()}} returns a
> naive datetime in the *local* timezone. The expectation therefore only
> holds when the local timezone is UTC (e.g. 16:30 in Asia/Taipei, 01:30 in
> America/Los_Angeles).
> h3. How to reproduce
> From a Spark checkout built with {{{}build/sbt -Phive package{}}}:
> {noformat}
> TZ=Asia/Taipei python/run-tests --testnames
> "pyspark.sql.tests.test_functions FunctionsTests.test_try_make_timestamp_ltz"
> TZ=Asia/Taipei python/run-tests --testnames
> "pyspark.sql.tests.test_functions FunctionsTests.test_make_timestamp"
> TZ=America/Los_Angeles python/run-tests --testnames
> "pyspark.sql.tests.test_functions FunctionsTests.test_try_make_timestamp_ltz"
> {noformat}
> All fail; the same commands with {{TZ=UTC}} pass.
> Minimal reproduction of root cause 1:
> {code:python}
> # with TZ=Asia/Taipei
> spark.sql("SELECT make_timestamp(2024, 5, 22, 10, 30, 0, 'CST') AS
> ts").first().ts
> # datetime.datetime(2024, 5, 22, 23, 30) <- expected 10:30
> {code}
> h3. Proposed fix (test-only)
> * In {{{}test_try_make_timestamp_ltz{}}}, pass an unambiguous zone ID (e.g.
> {{{}"UTC"{}}}) instead of {{{}tzinfo._{_}str{_}_(){}}}.
> * Compute expected values by converting the known UTC instant to local time
> instead of hardcoding them, e.g.:
> {code:python}
> datetime.datetime(2024, 5, 22, 8, 30,
> tzinfo=datetime.timezone.utc).astimezone().replace(tzinfo=None)
> {code}
> This keeps the semantics under test unchanged and makes the tests pass under
> any local timezone. Verified locally under UTC, Asia/Taipei and
> America/Los_Angeles.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
