Max Gekk created SPARK-57749:
--------------------------------
Summary: Fix midnight-sensitive inline-table TIME -> TIMESTAMP_NTZ
golden test in cast.sql
Key: SPARK-57749
URL: https://issues.apache.org/jira/browse/SPARK-57749
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
h2. Summary
Make the inline-table {{TIME -> TIMESTAMP_NTZ}} coverage added in SPARK-57618
deterministic. The golden case
{code:sql}
SELECT CAST(x AS DATE) = current_date() FROM VALUES (CAST(TIME'12:34:56' AS
TIMESTAMP_NTZ)) t(x);
{code}
asserts {{true}} but is sensitive to a midnight boundary and can flip to
{{false}}.
h2. Background
A foldable inline-table row that carries no {{CURRENT_LIKE}} pattern is
early-evaluated at analysis time by {{ResolveInlineTables}}
({{EvaluateUnresolvedInlineTable.earlyEvalIfPossible}} ->
{{EvalInlineTables.eval}}), which runs the interpreted cast's
{{currentDate(zoneId)}} = {{LocalDate.now(zoneId)}}. The sibling
{{current_date()}} keeps its {{CURRENT_LIKE}} bit, survives to the optimizer,
and is stabilized separately by {{ComputeCurrentTime}} from a different clock
read.
If analysis finishes just before midnight and optimization starts just after,
{{x}}'s date is yesterday while {{current_date()}} is today, so the row returns
{{false}} and the golden asserting {{true}} fails. This is the divergence the
{{CURRENT_LIKE}} deferral exists to prevent; the {{CAST}}-widened pruning in
{{ComputeCurrentTime}} does not help here because the inline-table cast is
consumed by the analyzer before {{ComputeCurrentTime}} runs.
h2. Fix
* Drop the {{= current_date()}} comparison for the inline-table case (it cannot
be stabilized on the inline-table path) and keep deterministic inline-table
coverage via the date-independent round-trip, e.g. {{SELECT CAST(x AS TIME(6))
FROM VALUES (CAST(TIME'12:34:56' AS TIMESTAMP_NTZ)) t(x)}} -> the date cancels
on the way back to {{TIME}}.
* Correct the accompanying comment: the inline-table cast is early-evaluated at
analysis time and is *not* stabilized by {{ComputeCurrentTime}}, so its date
anchor is {{LocalDate.now()}} at analysis, not necessarily {{current_date()}}.
* Regenerate {{cast.sql}} / {{nonansi/cast.sql}} golden files.
The non-inline-table {{CAST(CAST(TIME ... AS TIMESTAMP_NTZ) AS DATE) =
current_date()}} case stays: it goes through {{ComputeCurrentTime}} and shares
the same date literal, so it is deterministic.
h2. Notes
This mirrors the Structured Streaming out-of-scope note in SPARK-57618: the
inline-table early-eval path for {{TIME -> TIMESTAMP_NTZ}} uses an
analysis-time current date rather than the optimizer-stabilized one. Test-only
change; no behavior change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
