Max Gekk created SPARK-57827:
--------------------------------
Summary: Support nanosecond precision in make_timestamp,
make_timestamp_ntz and make_timestamp_ltz
Key: SPARK-57827
URL: https://issues.apache.org/jira/browse/SPARK-57827
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
precision).
h2. Problem
The public builders produce {{MakeTimestamp}} / {{MakeTimestampNTZ}} /
{{MakeTimestampLTZ}} / {{MakeTimestampFromDateTime}} with microsecond output;
the {{sec}} argument is {{DecimalType(16, 6)}} (datetimeExpressions.scala
~L3403-3512 / ~L3017-3036 / ~L3722-3734). Nanosecond-capable expressions
{{MakeTimestampNTZNanos}} / {{MakeTimestampLTZNanos}} exist but are internal
(used only by {{ComputeCurrentTime}}) and unregistered.
h2. Goal
Allow constructing nanosecond-precision timestamps from parts, either by
widening {{sec}} to accept up to 9 fractional digits or by exposing the
nanosecond make-timestamp expressions, and preserve TIME(9) sub-microsecond
digits in {{make_timestamp(date, time)}}.
h2. Scope
Widen {{sec}} precision / add nanosecond codepaths in {{MakeTimestamp*}};
register the nanosecond variants or extend the existing builders; preserve the
NTZ/LTZ family.
h2. Acceptance criteria
* {{make_timestamp(..., 5.123456789)}} and {{make_timestamp(date, time(9))}}
produce a nanosecond timestamp with the correct remainder.
h2. Testing
{{DateExpressionsSuite}}; nanos golden files.
h2. Dependencies
None - independent. The internal nanosecond make-timestamp expressions it
exposes are reused by the current_timestamp(p)/localtimestamp(p) sub-task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
