Max Gekk created SPARK-57835:
--------------------------------
Summary: Define read behavior for persisted nanosecond-typed
tables when the preview flag is off
Key: SPARK-57835
URL: https://issues.apache.org/jira/browse/SPARK-57835
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
precision).
h2. Problem
{{DataType.fromJson}} for nanosecond precision calls
{{checkTimestampNanosTypesEnabled()}} (DataType.scala ~L255-276), so a table
created with {{spark.sql.timestampNanosTypes.enabled=true}} cannot be loaded
with the flag off (DataTypeSuite ~L1596-1610). This cross-flag / cross-version
policy is undefined for real catalog tables/views.
h2. Goal
Define and implement a clear policy for reading a persisted nanosecond-typed
table/view when the preview flag is off (e.g. a specific actionable error),
covered by metastore / DESCRIBE / SHOW CREATE / view round-trip tests.
h2. Scope
Decide the policy; make the catalog restore path raise a clear, consistent
error (or permit read); add e2e tests including {{DESCRIBE}}, {{SHOW CREATE
TABLE}}, and view persistence with nanosecond columns.
h2. Acceptance criteria
* Loading a nanosecond table with the flag off yields the defined behavior with
a clear message; DDL command outputs render nanosecond columns correctly.
h2. Testing
{{HiveExternalCatalogSuite}}, command/DDL suites.
h2. Dependencies
Relates to the isHiveCompatibleDataType sub-task (both touch catalog restore);
can land independently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
