Dandandan opened a new pull request, #24442: URL: https://github.com/apache/datafusion/pull/24442
## Which issue does this PR close? - N/A ## Rationale for this change `JoinSelection` picks the hash join build side from plan-time estimates. When those estimates are wrong the join builds its hash table from the larger input. Across TPC-H SF1, 6 of 62 hash joins do this — q9 builds a 1.5M-row table to probe with 319k rows. A partitioned hash join can just measure instead: both inputs are hash-partitioned on the join keys, so partition `i` only ever meets partition `i` and the choice is local to the operator. ## What changes are included in this PR? Behind `datafusion.optimizer.adaptive_join_build_side` (default off), a partitioned hash join reads a bounded prefix of both inputs and builds from whichever ends first. If neither ends within `adaptive_join_build_side_peek_rows`, the planned side is kept. The decision is per partition, so it also adapts to skew. Restricted to inner joins in `Partitioned` mode with no join filter, and only when the join is not producing a dynamic filter — those merge build-side key bounds across all partitions, which a per-partition swap would make unsound. ## Are these changes tested? Yes — new unit tests cover the swap (output identical to the planned orientation, and the small side builds) and the fallback when neither side ends within the budget. All 22 TPC-H queries return identical results with the flag on. ## Results This reduces hash table memory but **does not make anything faster**: | | build memory off | on | |---|---|---| | q8 | 27.8 MB | 9.5 MB (−66%) | | q9 | 106.6 MB | 71.9 MB (−33%) | Runtime at SF1 is unchanged within noise, and q9 was slower in 3 of 4 runs. Swapping trades fewer build rows for proportionally more probe lookups, and the two costs appear close enough to cancel. Please treat this as a memory optimization only. ## Are there any user-facing changes? Two new config options, both inert unless enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
