Rachelint opened a new pull request, #26098: URL: https://github.com/apache/datafusion/pull/26098
## Which issue does this PR close? None. Benchmark-only experiment based on #26097 (`85b64a74cec1eab1e5a590234cd242fa0ece843d`). ## Rationale for this change Compare continuous Partial flushing with ordinary Final aggregation against main, and expose independent switches for follow-up experiments. ## What changes are included in this PR? - Revert the fixed-width-only eligibility change (`88d36d4d`) so string keys and string state are eligible again; preserve the original nested-state exclusions. - Preserve #26097's 2 MiB allocated-byte Partial threshold, in addition to the group-count threshold. - Add three experimental execution options, all defaulting to true: `hash_aggregate_bucket_compaction`, `hash_aggregate_detect_repeated_groups`, and `hash_aggregate_final_buckets`. - Disabling recurrence detection bypasses emitted-key hashing/sampling and keeps threshold flushing active. Disabling Final buckets leaves Partial flushing enabled. - No StringView allocation reuse, payload reserve, or column-wise scatter changes from #25959 are included. Requested comparison: all three switches false, `hash_aggregate_bucket_threshold=262144`, against main at `7f0ab426eb84a967a93c42ee0bd6e928e49adeb6`. Existing skip-partial aggregation remains enabled. Main uses its ordinary behavior. This compares the full experiment with main, not just the incremental switch diff. ## What is the testing strategy for this PR? Added SQLLogicTest coverage for integer, long StringView, and mixed keys with all three switches disabled, plus Final buckets enabled with compaction disabled. Updated configuration documentation and information_schema fixtures. - `cargo fmt --all` and format check: passed. - `cargo clippy --all-targets --all-features -- -D warnings`: passed. - Configuration docs regenerated; `./ci/scripts/doc_prettier_check.sh --write --allow-dirty`: passed. - Required workspace extended tests: passed (12,382 Rust tests and 527 SQLLogicTest files). - ECS runtime source hashes match the three modified Rust source files. ECS comparison: Rust 1.99.0, `release-nonlto`, 12 partitions on CPUs 0-11, batch size 8192, 99,997,497 partitioned input rows. Two reverse-order rounds; four iterations per cell, first excluded (six warm samples per variant). Unmodified benchmark runner, entire-query wall-time medians: | Query | main (ms) | Candidate (ms) | Change | |---|---:|---:|---:| | Q10 | 106.80 | 110.82 | +3.76% | | Q11 | 123.25 | 127.66 | +3.58% | | Q12 | 404.42 | 390.75 | -3.38% | | Q31 | 460.88 | 456.69 | -0.91% | | Q32 | 1776.68 | 1790.72 | +0.79% | | Q33 | 2286.17 | 2016.63 | -11.79% | | Q34 | 2277.13 | 2048.87 | -10.02% | | Q35 | 419.98 | 354.63 | -15.56% | Q33/Q34/Q35 gains repeat in both orders. Q10/Q11 regress slightly; Q31/Q32 remain near flat. The benchmark checks success and output row counts; SQLLogicTest provides value-level correctness coverage. These results compare the full experiment against the pinned current main; they do not isolate unrelated changes between the parent PR base and main. ## Are there any user-facing changes? Three experimental configuration switches. The parent feature is disabled by default (`hash_aggregate_bucket_threshold=0`). This PR is an experimental benchmark vehicle, not intended for merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
