Max Gekk created SPARK-57323:
--------------------------------
Summary: Support casting between DATE and nanosecond-precision
timestamp types TIMESTAMP_NTZ(p) and TIMESTAMP_LTZ(p)
Key: SPARK-57323
URL: https://issues.apache.org/jira/browse/SPARK-57323
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
The nanosecond-precision timestamp types {{TIMESTAMP_NTZ(p)}} and
{{TIMESTAMP_LTZ(p)}} (precision {{p}} in [7, 9], represented by
{{TimestampNTZNanosType}} / {{TimestampLTZNanosType}}) currently have no CAST
support to or from {{DATE}}. The Catalyst {{Cast}} expression only implements
{{DATE}} <-> {{TIMESTAMP}} and {{DATE}} <-> {{TIMESTAMP_NTZ}} for the
microsecond GA types, plus the micros <-> nanos casts added in SPARK-57293.
This task adds explicit CAST support in both directions:
* {{DATE}} -> {{TIMESTAMP_NTZ(p)}}
* {{DATE}} -> {{TIMESTAMP_LTZ(p)}}
* {{TIMESTAMP_NTZ(p)}} -> {{DATE}}
* {{TIMESTAMP_LTZ(p)}} -> {{DATE}}
h2. Semantics
Parity with the existing microsecond {{DATE}} <-> {{TIMESTAMP}} casts:
* {{DATE}} -> {{TIMESTAMP_LTZ(p)}}: midnight of the date in the session time
zone, sub-microsecond part = 0.
* {{DATE}} -> {{TIMESTAMP_NTZ(p)}}: midnight UTC, sub-microsecond part = 0.
* {{TIMESTAMP_LTZ(p)}} -> {{DATE}}: calendar date in the session time zone; the
time-of-day and sub-microsecond digits are dropped.
* {{TIMESTAMP_NTZ(p)}} -> {{DATE}}: calendar date on the UTC wall-clock grid;
the time-of-day and sub-microsecond digits are dropped.
h2. Scope
* Explicit CAST only. {{canCast}} / {{canAnsiCast}} allow all four conversions.
* {{canANSIStoreAssign}} blocks {{DATE}} <-> nanos in both directions (no
silent store assignment); an explicit CAST is required. This avoids silently
dropping sub-microsecond digits and time-of-day, consistent with the nanos ->
micros narrowing rule from SPARK-57293.
* No implicit up-cast / type-coercion changes ({{UpCastRule}} and
{{findWiderDateTimeType}} are unchanged).
Implementation is confined to {{Cast.scala}} (eligibility rules,
{{needsTimeZone}}, {{forceNullable}}, interpreted and codegen cast methods)
with tests in {{CastSuiteBase}}, covering both ANSI-off and ANSI-on modes,
time-zone sensitivity for the LTZ directions, and null handling.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
