shehab-ali opened a new pull request, #23980:
URL: https://github.com/apache/datafusion/pull/23980
## Which issue does this PR close?
- Part of #23237.
## Rationale for this change
Issue #23237 identifies hash join candidate equality filtering as a potential
performance bottleneck. Before changing that implementation, we need stable
benchmarks that reproduce the relevant workloads and let us measure the
effect
of a proposed optimization independently.
The existing hash join benchmark suite covers high-fanout joins, but it does
not isolate these two cases:
1. A single hot hash bucket in which all build-side rows use the same long
string key.
2. A skewed composite-key join in which candidate pairs must be checked
across
multiple key columns.
This PR intentionally adds only the benchmarks. The implementation change for
`equal_rows_arr` will be submitted separately so that the benchmark workloads
can be reviewed and established independently of the optimization.
## What changes are included in this PR?
This PR adds two hash join benchmarks:
- **Q24: single-hot-bucket long string-key join**
- Creates 3,000 build-side rows with the same long string key.
- Selects matching probe-side rows that use that same key.
- Causes every selected probe row to fan out across the entire build-side
bucket.
- Uses `count(*)` so the benchmark emphasizes hash match and candidate
equality processing without materializing and retaining the full joined
output.
- **Q25: skewed composite-key join**
- Uses an integer key and a long string key.
- Distributes the build-side integer key over 256 values while all selected
probe rows use one value.
- Produces a high-fanout set of candidate matches for one integer-key
value.
- Requires the second string-key column to be checked as part of composite
join-key equality.
- Uses `count(*)` to focus the measurement on the join match path.
Both benchmarks:
- Assert that the input data is available.
- Assert that the physical plan contains `HashJoinExec`.
- Use the no-statistics configuration and force Partitioned hash join mode.
- Are registered in the `hj` benchmark runner with descriptive build/probe
metadata.
- Can be used to compare the existing implementation against the follow-up
optimization for #23237.
## Are these changes tested?
No hash join implementation is changed by this PR.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
