abokhalill opened a new pull request, #25494:
URL: https://github.com/apache/datafusion/pull/25494
## Which issue does this PR close?
- N/A
## Rationale for this change
`pre_selection_scatter` scatters the short-circuited RHS result back to the
full batch length. For each run of selected rows it slices the RHS array and
iterates it, appending one `Option<bool>` at a time:
```rust
right_result
.slice(right_array_pos, len)
.iter()
.for_each(|v| result_array_builder.append_option(v));
```
Both the RHS and the builder are bit-packed, so this sets one bit per row
where `BooleanBufferBuilder::append_packed_range` copies a word at a time. The
per-run `slice` also clones and drops an `Arc`, which shows up as the hottest
instruction in the function when the mask selects many short runs.
Borrowing the RHS buffers once outside the loop removes the slice entirely,
and the copy becomes a range copy for both the values and the validity bitmap.
Benchmarks: TPC-H SF30, parquet, 24 threads
q6, 388.7 ms → 356.5 ms, -8.3%
q12, 471.2 ms → 451.7 ms, -4.1%
q19, 657.0 ms → 644.0 ms, -2.0%
q3, 723.6 ms → 727.5 ms, +0.6%
q7, 1350.9 ms → 1359.6 ms, +0.7%
q10, 910.0 ms → 911.9 ms, +0.2%
The first three win every round of six; the rest are within noise.
The function is only reached for short-circuit AND/OR, so the gain depends
on how much of the filter reaches the RHS. Varying selectivity with `l_quantity
< K` on `lineitem`:
4% of rows selected, -1.3%
8%, -3.6%
16%, -7.1%
32%, -5.7%
48% and above, within noise
It peaks where the mask has many short runs, and decays at both ends: a
nearly empty mask has little to copy, and a nearly full one is a single long
run.
## What changes are included in this PR?
* Borrow the RHS values and nulls buffers once instead of slicing per run
* Copy each run with `append_packed_range` rather than per-row
`append_option`
* Only build a validity bitmap when the RHS actually has nulls
## What is the testing strategy for this PR?
Existing tests pass, including `test_pre_selection_scatter`.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
