[
https://issues.apache.org/jira/browse/SPARK-57832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-57832:
-----------------------------
Shepherd: Max Gekk
> Support timestamp - timestamp subtraction for nanosecond-precision timestamps
> (microsecond-precision interval result)
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-57832
> URL: https://issues.apache.org/jira/browse/SPARK-57832
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
> precision).
> h2. Problem
> {{SubtractTimestamps}} (datetimeExpressions.scala ~L4040-4090) types both
> operands as {{AnyTimestampType}} (microsecond only) and evaluates on {{Long}}
> micros, returning {{DayTimeIntervalType()}} (or legacy
> {{CalendarIntervalType}}). Subtracting two nanosecond timestamps ({{ts_nanos
> - ts_nanos}}) currently fails analysis. The SPIP had deferred this because
> the day-time interval result type has only microsecond precision; per
> decision we now support it and accept the microsecond-precision result.
> h2. Goal
> Allow {{TIMESTAMP_NTZ(p) - TIMESTAMP_NTZ(p)}} and {{TIMESTAMP_LTZ(p) -
> TIMESTAMP_LTZ(p)}} (and microsecond/nanosecond mixed operands via existing
> widening, SPARK-57454), returning a {{DayTimeIntervalType}} computed from the
> two {{TimestampNanosVal}} values. The result is microsecond-precision: the
> sub-microsecond remainder difference is dropped, with the rounding/truncation
> rule documented (default: truncate toward the microsecond grid, i.e. compute
> from epoch micros).
> h2. Scope
> Extend {{SubtractTimestamps.inputTypes}} to accept {{AnyTimestampNanoType}};
> add a {{TimestampNanosVal}} eval/codegen branch computing {{leftMicros -
> rightMicros}} (documented handling of the {{nanosWithinMicro}} remainder);
> keep the result type {{DayTimeIntervalType()}} / legacy
> {{CalendarIntervalType}} unchanged; ensure mixed-precision operands resolve
> through type coercion. Wire the nanosecond cases into the subtraction path in
> {{BinaryArithmeticWithDatetimeResolver}} if needed.
> h2. Non-goals
> No new sub-microsecond interval type; the day-time interval remains
> microsecond-precision (this is the accepted precision loss).
> h2. Acceptance criteria
> * {{ts_nanos - ts_nanos}} returns a {{DayTimeIntervalType}} value equal to
> the microsecond-grid difference; NTZ/LTZ zone semantics match the microsecond
> behavior; mixed microsecond/nanosecond operands work; interpreted + codegen
> tested.
> h2. Testing
> {{DateExpressionsSuite}}; {{timestamp-ntz-nanos.sql}} /
> {{timestamp-ltz-nanos.sql}} golden files.
> h2. Dependencies
> None hard - coordinate with the overflow/range-hardening sub-task; mixed
> microsecond/nanosecond operands via resolved SPARK-57454.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
