Chao Sun created SPARK-58272:
--------------------------------
Summary: Enable runtime Bloom filters for safely materialized
cached join inputs
Key: SPARK-58272
URL: https://issues.apache.org/jira/browse/SPARK-58272
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Chao Sun
h3. Problem
Spark injects a runtime Bloom filter only when the join creation side exposes a
likely selective predicate above a scan. Persisting that selected input
replaces its logical subtree with InMemoryRelation, hiding the predicate even
after all cached partitions are materialized. Consequently a large shuffle join
may process every application-side row despite the cache already containing the
exact candidate join keys, size, and row count.
{code:sql}
CACHE TABLE selected_keys AS
SELECT request_id FROM requests WHERE tenant_id = 42;
SELECT events.*
FROM events JOIN selected_keys
ON events.request_id = selected_keys.request_id;
{code}
A visible requests filter can trigger a runtime Bloom filter; the equivalent
materialized selected_keys input currently does not.
h3. Proposed improvement
Teach runtime-filter injection to recognize a fully materialized cached
creation side with accurate, generation-consistent row-count and size
statistics. Permit the optimization only when replaying the cached input is
safe and there is evidence of pruning benefit, while retaining existing
join-type, application-side scan-size, deterministic-key, and feature-enable
safeguards.
Safety must fail closed for incomplete cache materialization, partially
consumed partitions, stale updates from an earlier cache generation,
non-repeatable or opaque execution, best-effort file reads, and storage levels
that cannot provide reliable replay. A dedicated size threshold can distinguish
a materialized cache from a creation side that would need its original query
recomputed.
Prior discussion: https://github.com/apache/spark/pull/39377 and SPARK-41867.
This proposal uses exact materialized cache statistics and explicit
replay/correctness safeguards rather than generally looking through arbitrary
cached lineage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
