[
https://issues.apache.org/jira/browse/SPARK-60143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yen-Hsiang, Wang updated SPARK-60143:
-------------------------------------
Description:
Three tests in {{pyspark.sql.tests.test_functions}} fail deterministically when
the test process runs in a non-UTC local timezone:
* {{FunctionsTests.test_try_make_timestamp_ltz}}
* {{FunctionsTests.test_try_make_timestamp}}
* {{FunctionsTests.test_make_timestamp}}
They pass on CI only because the PySpark jobs run in Docker containers whose
default timezone is UTC. Neither the developer docs nor
{{python/run-tests}} require or set {{TZ=UTC}} for local runs, so anyone
running these tests on a laptop outside UTC hits the failures.
h3. Root cause 1: timezone abbreviation passed to Spark
({{{}test_try_make_timestamp_ltz{}}})
The test derives the "local timezone" with:
{code:python}
# use local timezone here to avoid flakiness
data = [(2024, 5, 22, 10, 30, 0,
datetime.datetime.now().astimezone().tzinfo.__str__())]
{code}
{{tzinfo._{_}str{_}_()}} returns an abbreviation such as {{CST}} or
{{{}PDT{}}}, not an IANA zone ID. Abbreviations are ambiguous, and Java
resolves
them differently from the OS:
* {{{}TZ=Asia/Taipei{}}}: Python yields {{'CST'}} (China Standard Time,
UTC+8), but Java interprets {{CST}} as US Central Time (UTC-5 in May).
10:30 becomes 23:30 local, a 13-hour shift.
* {{{}TZ=America/Los_Angeles{}}}: Python yields {{{}'PDT'{}}}, which Spark
rejects outright with {{{}[INVALID_TIMEZONE] The timezone: PDT is invalid{}}}.
So the comment's intent ("avoid flakiness") is not met: the test is guaranteed
to fail in these timezones.
h3. Root cause 2: expected values hardcoded in UTC
({{{}test_make_timestamp{}}}, {{{}test_try_make_timestamp{}}})
The input is {{2024-05-22 10:30 CET}} and the expected value is hardcoded:
{code:python}
result_with_tz = datetime.datetime(2024, 5, 22, 8, 30)
result_frac_with_tz = datetime.datetime(2024, 5, 22, 8, 30, 45, 123000)
{code}
10:30 CET (UTC+2 in May) is 08:30 {*}UTC{*}, but {{collect()}} returns a naive
datetime in the *local* timezone. The expectation therefore only
holds when the local timezone is UTC (e.g. 16:30 in Asia/Taipei, 01:30 in
America/Los_Angeles).
h3. How to reproduce
>From a Spark checkout built with {{{}build/sbt -Phive package{}}}:
{noformat}
TZ=Asia/Taipei python/run-tests --testnames "pyspark.sql.tests.test_functions
FunctionsTests.test_try_make_timestamp_ltz"
TZ=Asia/Taipei python/run-tests --testnames "pyspark.sql.tests.test_functions
FunctionsTests.test_make_timestamp"
TZ=America/Los_Angeles python/run-tests --testnames
"pyspark.sql.tests.test_functions FunctionsTests.test_try_make_timestamp_ltz"
{noformat}
All fail; the same commands with {{TZ=UTC}} pass.
Minimal reproduction of root cause 1:
{code:python}
# with TZ=Asia/Taipei
spark.sql("SELECT make_timestamp(2024, 5, 22, 10, 30, 0, 'CST') AS
ts").first().ts
# datetime.datetime(2024, 5, 22, 23, 30) <- expected 10:30
{code}
h3. Proposed fix (test-only)
* In {{{}test_try_make_timestamp_ltz{}}}, pass an unambiguous zone ID (e.g.
{{{}"UTC"{}}}) instead of {{{}tzinfo._{_}str{_}_(){}}}.
* Compute expected values by converting the known UTC instant to local time
instead of hardcoding them, e.g.:
{code:python}
datetime.datetime(2024, 5, 22, 8, 30,
tzinfo=datetime.timezone.utc).astimezone().replace(tzinfo=None)
{code}
This keeps the semantics under test unchanged and makes the tests pass under
any local timezone. Verified locally under UTC, Asia/Taipei and
America/Los_Angeles.
was:
Three tests in {{pyspark.sql.tests.test_functions}} fail deterministically when
the test process runs in a non-UTC local timezone:
* {{FunctionsTests.test_try_make_timestamp_ltz}}
* {{FunctionsTests.test_try_make_timestamp}}
* {{FunctionsTests.test_make_timestamp}}
They pass on CI only because the PySpark jobs run in Docker containers whose
default timezone is UTC. Neither the developer docs nor
{{python/run-tests}} require or set {{TZ=UTC}} for local runs, so anyone
running these tests on a laptop outside UTC hits the failures.
h3. Root cause 1: timezone abbreviation passed to Spark
({{test_try_make_timestamp_ltz}})
The test derives the "local timezone" with:
{code:python}
# use local timezone here to avoid flakiness
data = [(2024, 5, 22, 10, 30, 0,
datetime.datetime.now().astimezone().tzinfo.__str__())]
{code}
{{tzinfo.__str__()}} returns an abbreviation such as {{CST}} or {{PDT}}, not
an IANA zone ID. Abbreviations are ambiguous, and Java resolves
them differently from the OS:
* {{TZ=Asia/Taipei}}: Python yields {{'CST'}} (China Standard Time, UTC+8),
but Java interprets {{CST}} as US Central Time (UTC-5 in May).
10:30 becomes 23:30 local, a 13-hour shift.
* {{TZ=America/Los_Angeles}}: Python yields {{'PDT'}}, which Spark rejects
outright with {{[INVALID_TIMEZONE] The timezone: PDT is invalid}}.
So the comment's intent ("avoid flakiness") is not met: the test is
guaranteed to fail in these timezones.
h3. Root cause 2: expected values hardcoded in UTC ({{test_make_timestamp}},
{{test_try_make_timestamp}})
The input is {{2024-05-22 10:30 CET}} and the expected value is hardcoded:
{code:python}
result_with_tz = datetime.datetime(2024, 5, 22, 8, 30)
result_frac_with_tz = datetime.datetime(2024, 5, 22, 8, 30, 45, 123000)
{code}
10:30 CET (UTC+2 in May) is 08:30 *UTC*, but {{collect()}} returns a naive
datetime in the *local* timezone. The expectation therefore only
holds when the local timezone is UTC (e.g. 16:30 in Asia/Taipei, 01:30 in
America/Los_Angeles).
h3. How to reproduce
From a Spark checkout built with {{build/sbt -Phive package}}:
{noformat}
TZ=Asia/Taipei python/run-tests --testnames "pyspark.sql.tests.test_functions
FunctionsTests.test_try_make_timestamp_ltz"
TZ=Asia/Taipei python/run-tests --testnames "pyspark.sql.tests.test_functions
FunctionsTests.test_make_timestamp"
TZ=America/Los_Angeles python/run-tests --testnames
"pyspark.sql.tests.test_functions FunctionsTests.test_try_make_timestamp_ltz"
{noformat}
All fail; the same commands with {{TZ=UTC}} pass.
Minimal reproduction of root cause 1:
{code:python}
# with TZ=Asia/Taipei
spark.sql("SELECT make_timestamp(2024, 5, 22, 10, 30, 0, 'CST') AS
ts").first().ts
# datetime.datetime(2024, 5, 22, 23, 30) <- expected 10:30
{code}
h3. Proposed fix (test-only)
* In {{test_try_make_timestamp_ltz}}, pass an unambiguous zone ID (e.g.
{{"UTC"}}) instead of {{tzinfo.__str__()}}.
* Compute expected values by converting the known UTC instant to local time
instead of hardcoding them, e.g.:
{code:python}
datetime.datetime(2024, 5, 22, 8, 30,
tzinfo=datetime.timezone.utc).astimezone().replace(tzinfo=None)
{code}
This keeps the semantics under test unchanged and makes the tests pass under
any local timezone. Verified locally under UTC, Asia/Taipei and
America/Los_Angeles.
> Make make_timestamp tests in test_functions independent of the local timezone
>
> ---------------------------------------------------------------------------------
>
> Key: SPARK-60143
> URL: https://issues.apache.org/jira/browse/SPARK-60143
> Project: Spark
> Issue Type: Test
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: Yen-Hsiang, Wang
> Priority: Minor
>
> Three tests in {{pyspark.sql.tests.test_functions}} fail deterministically
> when the test process runs in a non-UTC local timezone:
> * {{FunctionsTests.test_try_make_timestamp_ltz}}
> * {{FunctionsTests.test_try_make_timestamp}}
> * {{FunctionsTests.test_make_timestamp}}
> They pass on CI only because the PySpark jobs run in Docker containers whose
> default timezone is UTC. Neither the developer docs nor
> {{python/run-tests}} require or set {{TZ=UTC}} for local runs, so anyone
> running these tests on a laptop outside UTC hits the failures.
> h3. Root cause 1: timezone abbreviation passed to Spark
> ({{{}test_try_make_timestamp_ltz{}}})
> The test derives the "local timezone" with:
> {code:python}
> # use local timezone here to avoid flakiness
> data = [(2024, 5, 22, 10, 30, 0,
> datetime.datetime.now().astimezone().tzinfo.__str__())]
> {code}
> {{tzinfo._{_}str{_}_()}} returns an abbreviation such as {{CST}} or
> {{{}PDT{}}}, not an IANA zone ID. Abbreviations are ambiguous, and Java
> resolves
> them differently from the OS:
> * {{{}TZ=Asia/Taipei{}}}: Python yields {{'CST'}} (China Standard Time,
> UTC+8), but Java interprets {{CST}} as US Central Time (UTC-5 in May).
> 10:30 becomes 23:30 local, a 13-hour shift.
> * {{{}TZ=America/Los_Angeles{}}}: Python yields {{{}'PDT'{}}}, which Spark
> rejects outright with {{{}[INVALID_TIMEZONE] The timezone: PDT is invalid{}}}.
> So the comment's intent ("avoid flakiness") is not met: the test is
> guaranteed to fail in these timezones.
> h3. Root cause 2: expected values hardcoded in UTC
> ({{{}test_make_timestamp{}}}, {{{}test_try_make_timestamp{}}})
> The input is {{2024-05-22 10:30 CET}} and the expected value is hardcoded:
> {code:python}
> result_with_tz = datetime.datetime(2024, 5, 22, 8, 30)
> result_frac_with_tz = datetime.datetime(2024, 5, 22, 8, 30, 45, 123000)
> {code}
> 10:30 CET (UTC+2 in May) is 08:30 {*}UTC{*}, but {{collect()}} returns a
> naive datetime in the *local* timezone. The expectation therefore only
> holds when the local timezone is UTC (e.g. 16:30 in Asia/Taipei, 01:30 in
> America/Los_Angeles).
> h3. How to reproduce
> From a Spark checkout built with {{{}build/sbt -Phive package{}}}:
> {noformat}
> TZ=Asia/Taipei python/run-tests --testnames
> "pyspark.sql.tests.test_functions FunctionsTests.test_try_make_timestamp_ltz"
> TZ=Asia/Taipei python/run-tests --testnames
> "pyspark.sql.tests.test_functions FunctionsTests.test_make_timestamp"
> TZ=America/Los_Angeles python/run-tests --testnames
> "pyspark.sql.tests.test_functions FunctionsTests.test_try_make_timestamp_ltz"
> {noformat}
> All fail; the same commands with {{TZ=UTC}} pass.
> Minimal reproduction of root cause 1:
> {code:python}
> # with TZ=Asia/Taipei
> spark.sql("SELECT make_timestamp(2024, 5, 22, 10, 30, 0, 'CST') AS
> ts").first().ts
> # datetime.datetime(2024, 5, 22, 23, 30) <- expected 10:30
> {code}
> h3. Proposed fix (test-only)
> * In {{{}test_try_make_timestamp_ltz{}}}, pass an unambiguous zone ID (e.g.
> {{{}"UTC"{}}}) instead of {{{}tzinfo._{_}str{_}_(){}}}.
> * Compute expected values by converting the known UTC instant to local time
> instead of hardcoding them, e.g.:
> {code:python}
> datetime.datetime(2024, 5, 22, 8, 30,
> tzinfo=datetime.timezone.utc).astimezone().replace(tzinfo=None)
> {code}
> This keeps the semantics under test unchanged and makes the tests pass under
> any local timezone. Verified locally under UTC, Asia/Taipei and
> America/Los_Angeles.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
