Rich-T-kid opened a new pull request, #27025: URL: https://github.com/apache/datafusion/pull/27025
## Which issue does this PR close? Part of #26848. This is the non-null path; #26966 handles nullable keys. ## Why For a dictionary with non-null keys and values, scattering hashes overwrites every output row. `with_hashes` was still zeroing its reusable 8,192-row buffer first, and the scatter loop loaded and stored each row before starting the next key-indexed load. ## What changed `with_hashes` now keeps the initialized buffer contents when the first column is a non-null dictionary, since that column replaces every slot. The non-null scatter loop gathers eight dictionary hashes before updating the output, which gives the CPU independent loads to overlap. Nullable keys or values continue through the existing path. Short hash buffers also keep the existing truncation behavior. ## Results Existing `with_hashes` benchmark, 8,192 rows, Apple M4 Max. These are paired geometric means from two runs per side in `main / this PR / this PR / main` order, using separate target directories. Base was `198ab43f8`. | case | main | this PR | change | |---|---:|---:|---:| | one dictionary column, no nulls | 1.89 us | 1.39 us | -26.6% | | three dictionary columns, no nulls | 5.71 us | 5.02 us | -12.1% | I also ran the existing 3% null-key cases as a regression check. They stayed on the old scatter path and varied within the run-to-run noise. ## Test coverage The new test checks reuse after a non-zero buffer, null keys, null dictionary values, and a hash buffer shorter than the input array. Locally checked: - `cargo fmt --all -- --check` - `cargo test -p datafusion-common` - `cargo test -p datafusion-common --features force_hash_collisions` - `cargo clippy -p datafusion-common --all-targets -- -D warnings` - `cargo clippy -p datafusion-common --all-targets --features force_hash_collisions -- -D warnings` - the repository pre-push checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
