[
https://issues.apache.org/jira/browse/SPARK-57749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-57749.
------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 56863
[https://github.com/apache/spark/pull/56863]
> Fix midnight-sensitive inline-table TIME -> TIMESTAMP_NTZ golden test in
> cast.sql
> ---------------------------------------------------------------------------------
>
> Key: SPARK-57749
> URL: https://issues.apache.org/jira/browse/SPARK-57749
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> h2. Summary
> Make the inline-table {{TIME -> TIMESTAMP_NTZ}} coverage added in SPARK-57618
> deterministic. The golden case
> {code:sql}
> SELECT CAST(x AS DATE) = current_date() FROM VALUES (CAST(TIME'12:34:56' AS
> TIMESTAMP_NTZ)) t(x);
> {code}
> asserts {{true}} but is sensitive to a midnight boundary and can flip to
> {{false}}.
> h2. Background
> A foldable inline-table row that carries no {{CURRENT_LIKE}} pattern is
> early-evaluated at analysis time by {{ResolveInlineTables}}
> ({{EvaluateUnresolvedInlineTable.earlyEvalIfPossible}} ->
> {{EvalInlineTables.eval}}), which runs the interpreted cast's
> {{currentDate(zoneId)}} = {{LocalDate.now(zoneId)}}. The sibling
> {{current_date()}} keeps its {{CURRENT_LIKE}} bit, survives to the optimizer,
> and is stabilized separately by {{ComputeCurrentTime}} from a different clock
> read.
> If analysis finishes just before midnight and optimization starts just after,
> {{x}}'s date is yesterday while {{current_date()}} is today, so the row
> returns {{false}} and the golden asserting {{true}} fails. This is the
> divergence the {{CURRENT_LIKE}} deferral exists to prevent; the
> {{CAST}}-widened pruning in {{ComputeCurrentTime}} does not help here because
> the inline-table cast is consumed by the analyzer before
> {{ComputeCurrentTime}} runs.
> h2. Fix
> * Drop the {{= current_date()}} comparison for the inline-table case (it
> cannot be stabilized on the inline-table path) and keep deterministic
> inline-table coverage via the date-independent round-trip, e.g. {{SELECT
> CAST(x AS TIME(6)) FROM VALUES (CAST(TIME'12:34:56' AS TIMESTAMP_NTZ)) t(x)}}
> -> the date cancels on the way back to {{TIME}}.
> * Correct the accompanying comment: the inline-table cast is early-evaluated
> at analysis time and is *not* stabilized by {{ComputeCurrentTime}}, so its
> date anchor is {{LocalDate.now()}} at analysis, not necessarily
> {{current_date()}}.
> * Regenerate {{cast.sql}} / {{nonansi/cast.sql}} golden files.
> The non-inline-table {{CAST(CAST(TIME ... AS TIMESTAMP_NTZ) AS DATE) =
> current_date()}} case stays: it goes through {{ComputeCurrentTime}} and
> shares the same date literal, so it is deterministic.
> h2. Notes
> This mirrors the Structured Streaming out-of-scope note in SPARK-57618: the
> inline-table early-eval path for {{TIME -> TIMESTAMP_NTZ}} uses an
> analysis-time current date rather than the optimizer-stabilized one.
> Test-only change; no behavior change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
