Max Gekk created SPARK-57833:
--------------------------------
Summary: Support nanosecond-precision timestamps in the
timestampadd and timestampdiff functions
Key: SPARK-57833
URL: https://issues.apache.org/jira/browse/SPARK-57833
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
precision).
h2. Problem
{{TimestampAdd}} (datetimeExpressions.scala ~L4278-4288) types its timestamp
arg as {{AnyTimestampType}} (excludes nanosecond) and evaluates
{{DateTimeUtils.timestampAdd(unit, Long, ...)}}; {{TimestampDiff}}
(~L4367-4380) types both args as {{TimestampType}} and operates on {{Long}}.
Nanosecond args raise {{UNEXPECTED_INPUT_TYPE}}. This is distinct from
SPARK-57501, which added the {{+}} / {{-}} day-time-interval OPERATOR
({{TimestampAddInterval}}), not these named functions.
h2. Goal
{{timestampadd(unit, qty, ts_nanos)}} returns a nanosecond timestamp (same
type/precision) with {{nanosWithinMicro}} preserved for unit >= MICROSECOND
(and correctly rescaled for a NANOSECOND unit if added); {{timestampdiff(unit,
start, end)}} accepts nanosecond endpoints and, for sub-microsecond units,
computes the difference using the nanosecond remainder.
h2. Scope
Extend both {{inputTypes}} to accept {{AnyTimestampNanoType}}; add
{{TimestampNanosVal}} eval/codegen; add {{DateTimeUtils}} helpers for
nanosecond add/diff; decide and document NANOSECOND-unit handling.
h2. Acceptance criteria
* add/diff over NTZ/LTZ nanosecond timestamps preserve/consume the nanosecond
remainder; microsecond behavior unchanged.
h2. Testing
{{DateExpressionsSuite}}; nanos golden files.
h2. Dependencies
None hard - coordinate with the overflow/range-hardening sub-task
(overflow-safe add/diff helpers).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
