Serge Rielau created SPARK-59751:
------------------------------------
Summary: Preserve CHAR/VARCHAR scan policy across cache mutation,
refresh, and rename
Key: SPARK-59751
URL: https://issues.apache.org/jira/browse/SPARK-59751
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Serge Rielau
First-class CHAR/VARCHAR under SPARK-58794 binds an analyzed scan mode
({{{}PreserveNative{}}} / {{{}SparkStandard{}}}) onto relation identity so
incompatible scans cannot reuse each other's cached results. SPARK-58814 / PR
58317 covers format round-trips and ORC truncation, plus the cache-lifecycle
repairs that were required to keep that identity from serving stale rows on the
paths already in that PR.
This follow-up is the remaining cache-policy contract. Today {{None}} still
means "unbound / not first-class". A cache created with first-class flags off
can be rebound when a later session with first-class modes recaches or renames
the table. V1 and Hive rename still snapshot one cache entry, so coexisting
PreserveNative and SparkStandard variants can lose a mode or storage level.
Hive refresh coexistence and a simultaneous three-policy mutation matrix are
also not complete.
Scope:
* Represent the pre-first-class policy as an explicit bound state (or
equivalent), distinct from unbound.
* Preserve that policy, plus PreserveNative and SparkStandard, through
recache, refreshTable, and rename for V1, V2, and Hive.
* Keep ordinary cache substitution mode-sensitive; ignore scan mode only for
mutation/rename discovery.
* Restore only direct table caches (including analyzer CHAR/VARCHAR padding
Projects); do not promote dependent-query or time-travel caches.
* Reproduce with coexisting variants in one SharedState, including a
mutation/rename from a session using a different policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
