[
https://issues.apache.org/jira/browse/SPARK-57205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-57205:
-----------------------------
Description:
SPARK-40259 (#57360) made DSv2 scan merging a Spark-side mechanism: PlanMerger
drives V2ScanRelationPushDown.rebuildScan to rebuild the merged scan, and a
source only declares that what its scans read is determined by the filters
pushed and the columns pruned, through the SCAN_MERGING table capability. No
built-in source declares it yet.
This declares it on FileTable, so the built-in file formats (Parquet, ORC, CSV,
JSON, text, Avro) take part in scan merging on their DSv2 read path. Today two
scans of the same file table that differ only in their projected columns are
not canonically identical, because a file source folds its data filters into
the FileScan object and FileScan.equals compares them. On the V1 path the same
filters sit above an identical LogicalRelation, so V1 merges shapes that V2
does not.
Out of scope, both already listed as follow-ups on SPARK-40259: merging two
scans whose strict (fully enforced) filters differ, which is what keeps two
differing partition filters apart on the V2 path; and widening nested columns
to the union of the fields each side reads, which is what keeps Parquet and ORC
nested columns apart while nested schema pruning is on.
Summary: Declare SCAN_MERGING on the built-in file sources (was: Add
SupportsScanMerging to merge equivalent V2 file scans)
> Declare SCAN_MERGING on the built-in file sources
> -------------------------------------------------
>
> Key: SPARK-57205
> URL: https://issues.apache.org/jira/browse/SPARK-57205
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0, 5.0.0
> Reporter: Yang Jie
> Priority: Major
> Labels: pull-request-available
>
> SPARK-40259 (#57360) made DSv2 scan merging a Spark-side mechanism:
> PlanMerger drives V2ScanRelationPushDown.rebuildScan to rebuild the merged
> scan, and a source only declares that what its scans read is determined by
> the filters pushed and the columns pruned, through the SCAN_MERGING table
> capability. No built-in source declares it yet.
> This declares it on FileTable, so the built-in file formats (Parquet, ORC,
> CSV, JSON, text, Avro) take part in scan merging on their DSv2 read path.
> Today two scans of the same file table that differ only in their projected
> columns are not canonically identical, because a file source folds its data
> filters into the FileScan object and FileScan.equals compares them. On the V1
> path the same filters sit above an identical LogicalRelation, so V1 merges
> shapes that V2 does not.
> Out of scope, both already listed as follow-ups on SPARK-40259: merging two
> scans whose strict (fully enforced) filters differ, which is what keeps two
> differing partition filters apart on the V2 path; and widening nested columns
> to the union of the fields each side reads, which is what keeps Parquet and
> ORC nested columns apart while nested schema pruning is on.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
