[
https://issues.apache.org/jira/browse/SPARK-57836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-57836:
-----------------------------
Shepherd: Max Gekk
> Produce and accept nanosecond precision in to_timestamp, to_timestamp_ntz,
> to_timestamp_ltz and try_to_timestamp
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-57836
> URL: https://issues.apache.org/jira/browse/SPARK-57836
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
> precision).
> h2. Problem
> {{ParseToTimestamp}} (datetimeExpressions.scala ~L2641-2674) and the builders
> (~L1506-1591) pin the target {{dataType}} to {{TimestampNTZType}} /
> {{TimestampType}} / {{SQLConf.get.timestampType}} (all microsecond), and
> {{inputTypes}} omit nanosecond. So there is no way to PARSE a string into a
> nanosecond timestamp, and a nanosecond input is narrowed to microseconds.
> h2. Goal
> Allow {{to_timestamp}}, {{to_timestamp_ntz}}, {{to_timestamp_ltz}} and
> {{try_to_timestamp}} to produce nanosecond-precision output (at least via
> {{TIMESTAMP_NTZ(p)}} / {{TIMESTAMP_LTZ(p)}} target semantics), parse up to 9
> fractional digits, and accept nanosecond inputs without narrowing.
> h2. Scope
> Thread a target precision/type through the builders; use the nanosecond-aware
> {{TimestampFormatter}} for parsing; accept {{AnyTimestampNanoType}} inputs in
> {{ToTimestamp}} / {{GetTimestamp}} {{inputTypes}}.
> h2. Acceptance criteria
> * Parsing a 7-9 digit fractional string yields a nanosecond timestamp; the
> {{try_}} variant returns null instead of throwing on overflow/invalid input.
> h2. Testing
> {{DateExpressionsSuite}}; nanos golden files.
> h2. Dependencies
> None hard - coordinate with the overflow/range-hardening sub-task (parse
> overflow error taxonomy).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
