[
https://issues.apache.org/jira/browse/SPARK-58272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58272:
-----------------------------------
Labels: pull-request-available (was: )
> Enable runtime Bloom filters for safely materialized cached join inputs
> -----------------------------------------------------------------------
>
> Key: SPARK-58272
> URL: https://issues.apache.org/jira/browse/SPARK-58272
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Chao Sun
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> Spark injects a runtime Bloom filter only when the join creation side exposes
> a likely selective predicate above a scan. Persisting that selected input
> replaces its logical subtree with InMemoryRelation, hiding the predicate even
> after all cached partitions are materialized. Consequently a large shuffle
> join may process every application-side row despite the cache already
> containing the exact candidate join keys, size, and row count.
> {code:sql}
> CACHE TABLE selected_keys AS
> SELECT request_id FROM requests WHERE tenant_id = 42;
> SELECT events.*
> FROM events JOIN selected_keys
> ON events.request_id = selected_keys.request_id;
> {code}
> A visible requests filter can trigger a runtime Bloom filter; the equivalent
> materialized selected_keys input currently does not.
> h3. Proposed improvement
> Teach runtime-filter injection to recognize a fully materialized cached
> creation side with accurate, generation-consistent row-count and size
> statistics. Permit the optimization only when replaying the cached input is
> safe and there is evidence of pruning benefit, while retaining existing
> join-type, application-side scan-size, deterministic-key, and feature-enable
> safeguards.
> Safety must fail closed for incomplete cache materialization, partially
> consumed partitions, stale updates from an earlier cache generation,
> non-repeatable or opaque execution, best-effort file reads, and storage
> levels that cannot provide reliable replay. A dedicated size threshold can
> distinguish a materialized cache from a creation side that would need its
> original query recomputed.
> Prior discussion: https://github.com/apache/spark/pull/39377 and SPARK-41867.
> This proposal uses exact materialized cache statistics and explicit
> replay/correctness safeguards rather than generally looking through arbitrary
> cached lineage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
