[
https://issues.apache.org/jira/browse/SPARK-57812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093000#comment-18093000
]
Rakesh Raushan commented on SPARK-57812:
----------------------------------------
I am working on it
> Support catalog column-statistics serialization for nanosecond-precision
> timestamps
> -----------------------------------------------------------------------------------
>
> Key: SPARK-57812
> URL: https://issues.apache.org/jira/browse/SPARK-57812
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
> precision).
> h2. Problem
> {{CatalogColumnStat.toExternalString}} and {{fromExternalString}}
> (catalyst/catalog/interface.scala ~L1040-1081) handle only {{TimestampType}}
> / {{TimestampNTZType}}; nanosecond min/max hit
> {{columnStatisticsSerializationNotSupportedError}} /
> {{columnStatisticsDeserializationNotSupportedError}}. In-query stat
> aggregation already works ({{CommandUtils.statExprs}} uses {{case _:
> DatetimeType}}); the failure is on persisting/reading catalog stats.
> h2. Goal
> Serialize and deserialize nanosecond timestamp min/max column statistics
> to/from the catalog at full precision.
> h2. Scope
> Add nanosecond arms to {{toExternalString}} / {{fromExternalString}} using
> the nanos-aware formatter/parser (a round-trippable text form including the
> fractional digits).
> h2. Acceptance criteria
> * {{ANALYZE TABLE ... FOR COLUMNS <nanos_col>}} persists and reloads min/max
> without error; values round-trip.
> h2. Testing
> {{StatisticsCollectionSuite}} / {{StatisticsSuite}}.
> h2. Dependencies
> None - independent. Enables the CBO FilterEstimation sub-task (which needs
> persisted nanosecond min/max).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
