[
https://issues.apache.org/jira/browse/SPARK-59620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59620:
-----------------------------------
Labels: pull-request-available (was: )
> Late-materialization storage-filter pushdown via splicing
> ---------------------------------------------------------
>
> Key: SPARK-59620
> URL: https://issues.apache.org/jira/browse/SPARK-59620
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Peter Toth
> Priority: Major
> Labels: pull-request-available
>
> A runtime bloom filter from join runtime filtering is applied today as a
> Filter above the scan. The scan still reads every value page of every row
> group, even where the filter drops almost every row. On a selective join over
> a wide table that read is the dominant cost.
> This asks for such a filter to be pushed into the scan instead, so the
> vectorized Parquet reader can read the filter's key column first, decide
> which rows survive, and skip the column pages that no surviving row touches.
> A row group where nothing survives then costs one key-column read and no
> value-column IO at all.
> The scan reports what it saved through new SQL metrics, so a user can see
> when the filter is paying off and when it is not.
> Behind a new SQL config, off by default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
