sunchao opened a new pull request, #25491: URL: https://github.com/apache/datafusion/pull/25491
## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/6013. This supplies the DataFusion API needed for executor-level broadcast hash-table reuse; the embedding executor's cache is a separate change. ## Rationale for this change Embedding executors can create an independent hash-join plan for each task while repeatedly probing the same broadcast relation. `CollectLeft` shares a build within one plan instance, so separate task plans still decode and build independently. An explicit reusable build lets the embedding executor prepare once and attach it to fresh joins. ## What changes are included in this PR? Add `PreparedHashJoinBuild`, `HashJoinExec::prepare_build`, and `HashJoinExecBuilder::with_prepared_build` for `CollectLeft` inner joins with matching direct-column keys and fixed-width or UTF-8 build columns. The object owns immutable rows, hash buckets, row-index chains, bounds and a reservation in a caller-supplied durable memory pool. Each consuming join retains its own probe bookkeeping, residual predicate and dynamic-filter state. The caller owns cache identity, single-flight coordination, cancellation and eviction. Preparation admits retained buffers, compact-concatenation overlap, hash-index storage and temporary hash/validity buffers. Output gathers own their supported build-column buffers. Views, dictionaries, nested build columns and null-aware joins are outside this initial API. Attached plans use the prepared row count, recompute properties without inheriting an unused build child's ordering, preserve leases through compatible resets/projections, reject incompatible child/key rewrites, and reject serialization that would lose the prepared rows. Existing joins retain their current execution path. ## What is the testing strategy for this PR? The corresponding port on public `branch-55` at `7d3835c71` passed all 427 `joins::hash_join` unit tests, including 20 prepared-build regressions. These cover reuse across independent consumers, multiple probe partitions, residual predicates, dynamic-filter isolation, nulls and duplicate keys, output ownership, memory admission and release, errors/cancellation, plan properties, resets, and rewrite validation. ```text cargo test -p datafusion-physical-plan --lib joins::hash_join --no-default-features ``` The protobuf serialization regression also passed (1 test): ```text cargo test -p datafusion-proto --test proto_integration prepared_hash_join_cannot_be_serialized_without_its_build ``` It verifies that serializing a prepared join fails instead of serializing its empty schema placeholder. Formatting and `git diff --check` pass. This PR targets `main`. Local execution of the `main` test build is currently blocked because its Arrow 60.0.0 dependency is not available from the configured dependency registry. The public DataFusion 55.1.0 validation above uses Arrow 59.2.0 and does not replace testing this exact `main` revision in CI. The draft should remain unmerged until that validation completes. No end-to-end performance result is claimed. ## Are there any user-facing changes? New opt-in Rust APIs for embedding executors. Normal SQL planning and join selection do not enable reuse automatically. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
