Peter Toth created SPARK-59620:
----------------------------------
Summary: Late-materialization storage-filter pushdown via splicing
Key: SPARK-59620
URL: https://issues.apache.org/jira/browse/SPARK-59620
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
A runtime bloom filter from join runtime filtering is applied today as a Filter
above the scan. The scan still reads every value page of every row group, even
where the filter drops almost every row. On a selective join over a wide table
that read is the dominant cost.
This asks for such a filter to be pushed into the scan instead, so the
vectorized Parquet reader can read the filter's key column first, decide which
rows survive, and skip the column pages that no surviving row touches. A row
group where nothing survives then costs one key-column read and no value-column
IO at all.
The scan reports what it saved through new SQL metrics, so a user can see when
the filter is paying off and when it is not.
Behind a new SQL config, off by default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
