[
https://issues.apache.org/jira/browse/SPARK-57811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-57811:
-----------------------------
Shepherd: Max Gekk
> Support string to nanosecond-precision timestamp coercion in comparisons and
> predicates
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-57811
> URL: https://issues.apache.org/jira/browse/SPARK-57811
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
> precision).
> h2. Problem
> {{findCommonTypeForBinaryComparison}} (analysis/TypeCoercion.scala ~L144-147)
> special-cases only {{TimestampType + StringType}}, and
> {{StringPromotionTypeCoercion}} (~L58-61) matches only the microsecond
> {{TimestampTypeExpression}} (DataTypeExpression.scala ~L45). A string operand
> compared to a nanosecond timestamp column is therefore not coerced like the
> microsecond case: non-ANSI may promote the timestamp to string, ANSI may
> fail. Column-vs-column nanos/micro is already correct (SPARK-57454); this is
> the string-literal path only.
> h2. Goal
> Coerce a string operand compared to a nanosecond timestamp column to the
> nanosecond type (parity with microsecond), in both the binary-comparison
> common-type path and the equality string-promotion path, for ANSI and
> non-ANSI.
> h2. Scope
> Extend {{findCommonTypeForBinaryComparison}} and
> {{StringPromotionTypeCoercion}} (and the {{DataTypeExpression}} matchers) to
> include {{AnyTimestampNanoType}}.
> h2. Acceptance criteria
> * {{ts_nanos = '2020-01-02 03:04:05.123456789'}} and range predicates resolve
> by casting the string to the nanosecond type; results correct in both ANSI
> and non-ANSI modes.
> h2. Testing
> {{TypeCoercionSuite}} / {{AnsiTypeCoercionSuite}}; nanos golden files.
> h2. Dependencies
> None - independent (widening resolved in SPARK-57454). Feeds the
> cross-precision test-coverage sub-task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
