Max Gekk created SPARK-57836:
--------------------------------
Summary: Produce and accept nanosecond precision in to_timestamp,
to_timestamp_ntz, to_timestamp_ltz and try_to_timestamp
Key: SPARK-57836
URL: https://issues.apache.org/jira/browse/SPARK-57836
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
precision).
h2. Problem
{{ParseToTimestamp}} (datetimeExpressions.scala ~L2641-2674) and the builders
(~L1506-1591) pin the target {{dataType}} to {{TimestampNTZType}} /
{{TimestampType}} / {{SQLConf.get.timestampType}} (all microsecond), and
{{inputTypes}} omit nanosecond. So there is no way to PARSE a string into a
nanosecond timestamp, and a nanosecond input is narrowed to microseconds.
h2. Goal
Allow {{to_timestamp}}, {{to_timestamp_ntz}}, {{to_timestamp_ltz}} and
{{try_to_timestamp}} to produce nanosecond-precision output (at least via
{{TIMESTAMP_NTZ(p)}} / {{TIMESTAMP_LTZ(p)}} target semantics), parse up to 9
fractional digits, and accept nanosecond inputs without narrowing.
h2. Scope
Thread a target precision/type through the builders; use the nanosecond-aware
{{TimestampFormatter}} for parsing; accept {{AnyTimestampNanoType}} inputs in
{{ToTimestamp}} / {{GetTimestamp}} {{inputTypes}}.
h2. Acceptance criteria
* Parsing a 7-9 digit fractional string yields a nanosecond timestamp; the
{{try_}} variant returns null instead of throwing on overflow/invalid input.
h2. Testing
{{DateExpressionsSuite}}; nanos golden files.
h2. Dependencies
None hard - coordinate with the overflow/range-hardening sub-task (parse
overflow error taxonomy).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
