Yang Jie created SPARK-56173:
--------------------------------
Summary: V2 file write cache invalidation, flag flip, and delete
FallBackFileSourceV2
Key: SPARK-56173
URL: https://issues.apache.org/jira/browse/SPARK-56173
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.2.0
Reporter: Yang Jie
Enable V2 file write by default.
# Fix cache invalidation: switch from `recacheByPlan` to `recacheByPath` for
`FileTable` in `DataSourceV2Strategy.refreshCache`, matching V1 behavior. Add
`fileIndex.refresh()` after write.
# Skip `checkPartitioningMatchesV2Table` for `FileTable` (partitioning is
inferred from disk, empty for new paths).
# Delete `FallBackFileSourceV2` and remove from analyzer rule chains. Remove
`V2_FILE_WRITE_ENABLED` config.
# Regression fixes: `FileTable.columns()` restores NOT NULL constraints from
catalog schema; `FileFormatDataWriter.writeAll` wraps errors with
`TASK_WRITE_FAILED`; `V2SessionCatalog.loadTable` sets `catalogTable` on
`FileTable`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
