[
https://issues.apache.org/jira/browse/SPARK-56173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-56173:
-----------------------------
Summary: flag flip, and delete FallBackFileSourceV2 (was: V2 file write
cache invalidation, flag flip, and delete FallBackFileSourceV2)
> flag flip, and delete FallBackFileSourceV2
> ------------------------------------------
>
> Key: SPARK-56173
> URL: https://issues.apache.org/jira/browse/SPARK-56173
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Yang Jie
> Priority: Major
>
> Enable V2 file write by default.
> # Fix cache invalidation: switch from `recacheByPlan` to `recacheByPath` for
> `FileTable` in `DataSourceV2Strategy.refreshCache`, matching V1 behavior. Add
> `fileIndex.refresh()` after write.
> # Skip `checkPartitioningMatchesV2Table` for `FileTable` (partitioning is
> inferred from disk, empty for new paths).
> # Delete `FallBackFileSourceV2` and remove from analyzer rule chains. Remove
> `V2_FILE_WRITE_ENABLED` config.
> # Regression fixes: `FileTable.columns()` restores NOT NULL constraints from
> catalog schema; `FileFormatDataWriter.writeAll` wraps errors with
> `TASK_WRITE_FAILED`; `V2SessionCatalog.loadTable` sets `catalogTable` on
> `FileTable`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
