andygrove opened a new pull request, #2793: URL: https://github.com/apache/datafusion-ballista/pull/2793
# Which issue does this PR close? No single issue. Split out of #2792. Related: #2662 (q21 at SF1000), #2520 (performance review tracking). # Rationale for this change DataFusion decorrelates `IN`/`EXISTS`/`NOT EXISTS` subqueries into semi and anti joins and places them above the whole `FROM` join tree, so the subquery filters the joined result. TPC-H q18 joins 600M rows before its subquery keeps 6,400 orders. Spark handles this with `PushDownLeftSemiAntiJoin`. # What changes are included in this PR? A logical rule, `SemiJoinPushdown`, controlled by `ballista.optimizer.semi_join_pushdown` (default `true`). It moves semi and anti joins below passthrough projections, inner joins (onto the side that holds their keys) and the preserved side of outer joins. It stops where the move is sound but slower: - **A `Filter`.** The semi join should see the filter's output. TPC-H q4's `EXISTS` probes 5.7M filtered orders instead of all 150M; pushing below the filter took it from 3.3 s to 9.6 s at SF100. - **A join whose other input is filtered.** That join can remove most rows itself. q21's `EXISTS` and `NOT EXISTS` moved onto all 379M `l1` rows and doubled its run time at SF100 (32 s to 65 s). Unit tests cover each case, including the two it must not push through. With every change in #2792 together, on one machine at TPC-H SF100 (2 executors x 8 cores, 16 partitions), q18 goes from 42.2 s to 16.0 s. This rule is the part that filters `orders` down to the qualifying orders before joining `lineitem`; the shuffle spill fix in a separate PR contributes too. All 22 queries pass `--verify` against DataFusion at SF10 on that branch. # Are there any user-facing changes? One new config key, `ballista.optimizer.semi_join_pushdown`, enabled by default. Plans for queries with subqueries change. `docs/source/user-guide/configs.md` is regenerated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
