[
https://issues.apache.org/jira/browse/SPARK-59751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118497#comment-18118497
]
Serge Rielau commented on SPARK-59751:
--------------------------------------
Needed work from PR 58317 review of {{b8d27c0e}}
The latest comments are non-blocking, but they describe the actual contract
this JIRA should implement. Do not patch them onto SPARK-58814.
Design (do this first)
Cache reuse, mutation identity, direct-cache origin, and restoration are still
separate helpers. Make one {{CacheManager}} lifecycle API:
# A descriptor with resolved direct-cache origin, analyzed plan, bound scan
mode, and storage level.
# Route V1, V2, and Hive mutation, refresh, and rename through that API.
Ordinary cache substitution stays mode-sensitive; mutation ignores only scan
mode.
# Restore each variant in isolated session configuration that shares
{{SharedState}} / {{{}CacheManager{}}}. Do not wrap restoration in caller
{{{}withSQLConf{}}}.
# Keep format/reader-mode work on SPARK-58814.
Must-fix from the latest review
* Isolate rename restoration from caller {{{}SQLConf{}}}. V2
{{RenameTableExec}} and V1/Hive {{AlterTableRenameCommand}} both use
{{{}withSQLConf{}}}; concurrent work can observe restoration-only policy.
* Match direct V2 caches by resolved table identity, not {{CACHE TABLE}}
display name. Unqualified or namespace-qualified caches can be dropped on
rename.
* Bind {{Legacy}} as an explicit policy and preserve it with PreserveNative
and SparkStandard through recache, refresh, and rename.
* Exercise actually bound CHAR/VARCHAR columns in the V1 insert and save
regressions. Current fixtures are {{{}INT{}}}/{{{}STRING{}}} and {{{}LONG{}}},
so both config branches stay unbound and a regression to {{recacheByPlan}}
still passes.
Nits to include while doing the redesign
* Check the cheap direct-cache predicate before walking the plan.
* Bind scan mode inside the existing padding traversal when padding runs; keep
a standalone walk only when padding is skipped.
* Correct {{lookupCacheDescriptorsByV2Relation}} Scaladoc: default
{{directNamedCacheOnly = false}} also returns dependent plans.
> Preserve CHAR/VARCHAR scan policy across cache mutation, refresh, and rename
> ----------------------------------------------------------------------------
>
> Key: SPARK-59751
> URL: https://issues.apache.org/jira/browse/SPARK-59751
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 5.0.0
> Reporter: Serge Rielau
> Priority: Major
>
> First-class CHAR/VARCHAR under SPARK-58794 binds an analyzed scan mode
> ({{{}PreserveNative{}}} / {{{}SparkStandard{}}}) onto relation identity so
> incompatible scans cannot reuse each other's cached results. SPARK-58814 / PR
> 58317 covers format round-trips and ORC truncation, plus the cache-lifecycle
> repairs that were required to keep that identity from serving stale rows on
> the paths already in that PR.
> This follow-up is the remaining cache-policy contract. Today {{None}} still
> means "unbound / not first-class". A cache created with first-class flags off
> can be rebound when a later session with first-class modes recaches or
> renames the table. V1 and Hive rename still snapshot one cache entry, so
> coexisting PreserveNative and SparkStandard variants can lose a mode or
> storage level. Hive refresh coexistence and a simultaneous three-policy
> mutation matrix are also not complete.
> Scope:
> * Represent the pre-first-class policy as an explicit bound state (or
> equivalent), distinct from unbound.
> * Preserve that policy, plus PreserveNative and SparkStandard, through
> recache, refreshTable, and rename for V1, V2, and Hive.
> * Keep ordinary cache substitution mode-sensitive; ignore scan mode only for
> mutation/rename discovery.
> * Restore only direct table caches (including analyzer CHAR/VARCHAR padding
> Projects); do not promote dependent-query or time-travel caches.
> * Reproduce with coexisting variants in one SharedState, including a
> mutation/rename from a session using a different policy.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
