[
https://issues.apache.org/jira/browse/SPARK-57833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-57833:
-----------------------------
Shepherd: Max Gekk
> Support nanosecond-precision timestamps in the timestampadd and timestampdiff
> functions
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-57833
> URL: https://issues.apache.org/jira/browse/SPARK-57833
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
> precision).
> h2. Problem
> {{TimestampAdd}} (datetimeExpressions.scala ~L4278-4288) types its timestamp
> arg as {{AnyTimestampType}} (excludes nanosecond) and evaluates
> {{DateTimeUtils.timestampAdd(unit, Long, ...)}}; {{TimestampDiff}}
> (~L4367-4380) types both args as {{TimestampType}} and operates on {{Long}}.
> Nanosecond args raise {{UNEXPECTED_INPUT_TYPE}}. This is distinct from
> SPARK-57501, which added the {{+}} / {{-}} day-time-interval OPERATOR
> ({{TimestampAddInterval}}), not these named functions.
> h2. Goal
> {{timestampadd(unit, qty, ts_nanos)}} returns a nanosecond timestamp (same
> type/precision) with {{nanosWithinMicro}} preserved for unit >= MICROSECOND
> (and correctly rescaled for a NANOSECOND unit if added);
> {{timestampdiff(unit, start, end)}} accepts nanosecond endpoints and, for
> sub-microsecond units, computes the difference using the nanosecond remainder.
> h2. Scope
> Extend both {{inputTypes}} to accept {{AnyTimestampNanoType}}; add
> {{TimestampNanosVal}} eval/codegen; add {{DateTimeUtils}} helpers for
> nanosecond add/diff; decide and document NANOSECOND-unit handling.
> h2. Acceptance criteria
> * add/diff over NTZ/LTZ nanosecond timestamps preserve/consume the nanosecond
> remainder; microsecond behavior unchanged.
> h2. Testing
> {{DateExpressionsSuite}}; nanos golden files.
> h2. Dependencies
> None hard - coordinate with the overflow/range-hardening sub-task
> (overflow-safe add/diff helpers).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
