Rachelint opened a new pull request, #26100: URL: https://github.com/apache/datafusion/pull/26100
## Which issue does this PR close? None. Experimental performance comparison; not intended for merge as-is. ## Rationale for this change Measure continuous Partial hash aggregation flushing directly on main, independently of Final bucket aggregation. Compare a distinct-group threshold with an allocated-byte threshold across the complete partitioned ClickBench suite. Base commit: `7b1e55d4c2291acf6b9e7eb20dcdc62f25a8fe98`. ## What changes are included in this PR? - Add independent `partial_aggregation_flush_rows` and `partial_aggregation_flush_bytes` execution options. Both default to zero (disabled); reaching either enabled threshold emits the current Partial state and continues aggregating. - Rows means distinct group rows held in the table, not processed input rows. Thresholds are checked after each batch and may overshoot. - Reuse main's existing state materialization and output path; preserve memory reservation accounting and include threshold-flushed groups in the skip-partial probe. - Apply to unordered Partial aggregation without a soft group limit or nested aggregate state. String keys are supported. - No recurrence detection, Final buckets, compaction, StringView allocation reuse, or payload reservation changes are included. ## What is the testing strategy for this PR? - `cargo fmt --all` and format check. - `cargo check -p datafusion-physical-plan`. - `cargo clippy --all-targets --all-features -- -D warnings`. - Eleven hash-stream tests, including skip-partial accounting with no flush, byte-only, group-only, and both thresholds enabled. - `aggregate_partial_flush.slt` compares disabled, byte-only, and group-only modes for integer, long StringView, mixed, nullable, and nested-state aggregates; `information_schema.slt` verifies configuration exposure. - Configuration docs regenerated; `./ci/scripts/doc_prettier_check.sh --write --allow-dirty`. ECS benchmark methodology: unmodified dfbench runner, all q0-q42 in `clickbench_partitioned`, 99,997,497 input rows, 12 partitions pinned to CPUs 0-11, batch size 8192, Rust 1.99.0, `release-nonlto`. Baseline is the exact main commit above. Candidate modes: 262,144 groups, 32,768 groups, and 2 MiB, with only one threshold enabled at a time. Two rounds in reverse query/variant order, four iterations per cell with the first discarded: six warm samples per query/mode. Output row counts must match main for every query. Full-suite ECS results (negative changes mean faster). The sum column adds the 43 per-query medians; geometric mean equally weights all queries. Counts use a descriptive 3% cutoff, not a significance test. | Mode | Sum of query medians (s) | vs main | Geomean vs main | >3% faster / >3% slower | |---|---:|---:|---:|---:| | main | 28.479 | +0.00% | +0.00% | 0 / 0 | | rows262144 | 27.447 | -3.63% | -2.06% | 11 / 1 | | rows32768 | 27.073 | -4.94% | -3.28% | 14 / 2 | | bytes2m | 27.010 | -5.16% | -3.40% | 16 / 2 | 2 MiB and 32,768 groups are very close on the full-suite score (about 0.23% apart). This run does not establish that byte thresholds are inherently better. <details><summary>All 43 query medians (ms)</summary> | Query | main | 262144 groups | 32768 groups | 2 MiB | |---|---:|---:|---:|---:| | Q0 | 1.90 | 1.94 | 1.91 | 1.88 | | Q1 | 19.37 | 19.99 | 19.52 | 19.16 | | Q2 | 53.25 | 52.47 | 52.16 | 52.42 | | Q3 | 53.08 | 53.45 | 51.57 | 52.29 | | Q4 | 381.27 | 357.72 | 341.59 | 345.78 | | Q5 | 414.22 | 412.02 | 410.75 | 403.82 | | Q6 | 1.99 | 1.95 | 1.95 | 1.97 | | Q7 | 21.36 | 21.34 | 20.72 | 21.68 | | Q8 | 505.27 | 477.47 | 454.32 | 462.02 | | Q9 | 642.79 | 646.11 | 654.34 | 654.83 | | Q10 | 107.85 | 106.19 | 110.82 | 110.31 | | Q11 | 121.28 | 121.36 | 125.17 | 125.69 | | Q12 | 402.41 | 401.38 | 399.95 | 388.92 | | Q13 | 604.00 | 588.03 | 592.88 | 589.18 | | Q14 | 433.66 | 418.88 | 426.53 | 408.36 | | Q15 | 436.76 | 407.64 | 397.66 | 383.36 | | Q16 | 1007.32 | 928.01 | 876.73 | 879.92 | | Q17 | 1002.08 | 916.85 | 881.90 | 872.12 | | Q18 | 2224.60 | 1843.61 | 1762.73 | 1734.82 | | Q19 | 45.94 | 46.25 | 45.26 | 45.53 | | Q20 | 682.74 | 682.90 | 684.60 | 686.67 | | Q21 | 683.12 | 680.12 | 681.23 | 677.68 | | Q22 | 1389.44 | 1386.13 | 1385.11 | 1402.52 | | Q23 | 4807.42 | 4875.22 | 4877.88 | 4853.55 | | Q24 | 56.97 | 56.60 | 56.04 | 56.48 | | Q25 | 145.47 | 145.64 | 147.44 | 146.39 | | Q26 | 56.71 | 57.88 | 59.59 | 58.67 | | Q27 | 689.99 | 687.04 | 688.80 | 685.57 | | Q28 | 3537.37 | 3581.99 | 3566.17 | 3522.65 | | Q29 | 55.04 | 54.50 | 54.65 | 54.79 | | Q30 | 444.73 | 442.55 | 416.92 | 411.42 | | Q31 | 462.51 | 458.87 | 445.08 | 448.31 | | Q32 | 1771.62 | 1683.82 | 1685.73 | 1676.94 | | Q33 | 2195.63 | 2029.70 | 1976.48 | 2000.65 | | Q34 | 2199.20 | 2022.59 | 1979.17 | 2021.79 | | Q35 | 422.13 | 380.81 | 349.12 | 361.43 | | Q36 | 80.81 | 80.14 | 79.41 | 78.33 | | Q37 | 45.32 | 45.20 | 45.36 | 45.45 | | Q38 | 50.31 | 50.70 | 50.33 | 50.31 | | Q39 | 164.86 | 163.37 | 157.39 | 158.80 | | Q40 | 19.17 | 19.23 | 19.16 | 19.38 | | Q41 | 19.05 | 18.89 | 19.14 | 18.78 | | Q42 | 19.45 | 19.97 | 19.60 | 19.71 | </details> Same-binary control: all 43 queries, the same two reverse-order rounds and six warm samples per query/mode, using candidate `634a59fff` for both modes. | Metric | Both thresholds disabled | 2 MiB only | Change | |---|---:|---:|---:| | Sum of query medians | 28.325 s | 27.013 s | -4.63% | | Geometric mean of query ratios | 1.0 | 0.9714 | -2.86% | Per-round full-suite changes: -4.20% and -5.16%. Fourteen queries improve by more than 3%; two regress by more than 3% (a descriptive cutoff, not statistical significance). The same-binary control is important: Q32 is essentially unchanged (-0.14%), so its approximately 5% difference between main and candidate builds should not be attributed to flushing. Q33/Q34/Q35 retain -9.02%/-9.08%/-16.50% improvements with the same binary. This confirms real flush gains while keeping code-generation/environment differences out of that attribution. ## Are there any user-facing changes? Two experimental execution options, both disabled by default. Adding public configuration fields is an API change. Default Final aggregation, repartitioning, and coalescing behavior remains unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
