andygrove opened a new pull request, #2685:
URL: https://github.com/apache/datafusion-ballista/pull/2685
# Which issue does this PR close?
Part of #2652. This is split out of #2657 to make that PR smaller to review.
# Rationale for this change
A pipelined job is planned with the static planner, so AQE never picks its
join strategies from measured sizes. Every hash table in a pipelined job is
resident at the same time, a hash join build cannot spill, and a failed
pipelined task fails the whole job. Planning a large join as a partitioned hash
join is therefore much riskier than in blocking mode, and the planner needs to
fall back to a join that can spill when the build side may not fit.
# What changes are included in this PR?
- `state/pipelined_join_selection.rs` adds:
- `PipelinedJoinSelection`, a physical optimizer rule. It turns every
`Partitioned` hash join whose estimated build size per partition exceeds
`ballista.optimizer.hash_join_max_build_partition_bytes` back into a
`SortMergeJoinExec`, with the sorts it needs. A build of unknown size is
treated as too large. This is stricter than AQE, which keeps the hash join when
the size is unknown. `CollectLeft` joins and small builds are left alone, and a
limit of `0` disables the check.
- `create_pipelined_physical_plan`, which plans a logical plan with
`prefer_hash_join = true`, as AQE does, and then applies the rule.
- The size estimate reuses AQE's `estimate_output_byte_size`, which is now
`pub(crate)`. It uses `total_byte_size` when known and estimates from the row
count otherwise.
Nothing calls this yet, so the module carries `#[allow(dead_code)]`. #2657
calls `create_pipelined_physical_plan` from `TaskManager::submit_job` for
pipelined jobs and removes the attribute.
This PR adds tests:
- Unit tests for the rule cover an oversized, small and unknown-size build,
estimation from the row count, keeping a projection over the join, the zero
limit and `CollectLeft` joins.
- Planner tests check that small joins are planned as hash joins whatever
`prefer_hash_join` says, and joins over the build limit as sort-merge joins.
# Are there any user-facing changes?
No. The rule is not used by any query yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
