[
https://issues.apache.org/jira/browse/SPARK-59841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-59841:
-----------------------------------
Priority: Blocker (was: Major)
> Align table and changelog state-option handling and metadata reuse
> ------------------------------------------------------------------
>
> Key: SPARK-59841
> URL: https://issues.apache.org/jira/browse/SPARK-59841
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0, 5.0.0
> Reporter: Gengliang Wang
> Priority: Blocker
> Fix For: 4.3.0
>
>
> {{TableCatalog.tableStateOptionKeys()}} was introduced in Spark 4.3.0 by [PR
> #57799|https://github.com/apache/spark/pull/57799] ([commit
> 4b4a2f7b69d5|https://github.com/apache/spark/commit/4b4a2f7b69d555a2b28a130092f7d75411cb4c49]).
> That change made ordinary table loading and metadata reuse honor declared
> table-state options, while {{loadChangelog}} retained its all-options
> contract.
> TableCatalog currently has different option-handling contracts for ordinary
> table reads and changelog reads:
> * {{tableStateOptionKeys()}} declares the connector options that select table
> state. Spark passes only those options to {{loadTable(Identifier,
> TableContext, stateOptions)}} and uses them for table metadata reuse.
> Complete per-read options remain available for scan planning.
> * {{loadChangelog(Identifier, ChangelogContext, options)}} receives all query
> options, and changelog resolution bypasses the corresponding state-aware
> table-loading/cache path. There is no separate changelog state-option
> declaration.
> State options do reach CDC connectors, but Spark does not apply the same
> option projection and metadata-reuse rules to the two paths. This complicates
> deduplicating loads and refreshes, including the ongoing work on concurrent
> table refresh.
> For example, a current-table read and a changes-from-version-1 read with
> matching table-state options should be able to share the underlying captured
> table state. Their scan options and changelog range still describe distinct
> reads. Different state options must remain isolated, and different changelog
> contexts must not be treated as equivalent reads.
> The goal is to define consistent state-option, caching, and refresh semantics
> for ordinary and changelog reads, while retaining complete scan options and
> preserving the requested CDC context through refresh.
> The API approach is undecided. This ticket does not prescribe changing the
> Changelog hierarchy, adding a baseTable accessor, or any particular wrapper
> or refresh-request abstraction. The implementation should be evaluated for
> compatibility with existing connector APIs for the Spark 4.3.0 release.
> References:
> * [TableCatalog option
> contracts|https://github.com/apache/spark/blob/29acd89ec5f4806c1f6a335d8d3136baf111cd08/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java#L104]
> * [Current changelog
> resolution|https://github.com/apache/spark/blob/29acd89ec5f4806c1f6a335d8d3136baf111cd08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala#L382]
> * [Exploratory prototype for discussion; API not
> decided|https://github.com/gengliangwang/spark/pull/20]
> Target release: Spark 4.3.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
