HappenLee opened a new pull request, #69425:
URL: https://github.com/apache/doris/pull/69425
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Bucketed aggregation always allocates 256 hash tables per sink, even when a
query has only one sink and one source. It also selects a bucket from
mixed-hash bits 24–31. Once a phmap table reaches capacity `2^18 - 1`, bit 24
also participates in its probe index, so fixing that bit for a bucket confines
initial probes to half the index space.
This change adds `bucketed_agg_num_buckets`: `0` selects one bucket when
both sink and source have one instance, and 256 otherwise; explicit values must
be powers of two from 1 through 256. The planner captures the value and sends
it in an optional Thrift field. A missing field preserves 256 buckets for older
FEs.
Bucket routing uses the high byte of the mixed hash, independently of
phmap's practical probe-index and fingerprint bits. The single-bucket path
reuses `lazy_emplace_batch`, a single sink skips source-side merging, and
inline COUNT avoids allocating an unused places buffer. Initialization, memory
estimates, source traversal, completion and NULL-key merging all honor the
active count. Profiles expose `NumBuckets`, `NumSinkInstances` and
`PartitionTime`.
### Release note
Add the `bucketed_agg_num_buckets` session variable to select an automatic
or explicit bucket count for bucketed aggregation. The default automatic mode
uses one bucket for serial execution and 256 otherwise. Invalid counts are
rejected. SQL aggregation semantics are unchanged.
### Check List (For Author)
- Test:
- [x] Unit Test: `run-fe-ut.sh --run
'org.apache.doris.qe.SessionVariableBucketedAggTest,org.apache.doris.nereids.glue.translator.BucketedAggregateTranslatorTest'`
— **35 passed**, zero failures/errors.
- [x] Manual checks: clang-format 16, FE Checkstyle, BE build hygiene
and `git diff --check` passed.
- [x] Manual result audit: all **120 supplied regression result
sections** match an independent recomputation of the 32,768 input rows; the
supplied `.out` file is unchanged.
- [ ] BE Unit Test: configuration completed with ASAN and `-j 48`, but
the build stopped because local third-party artifacts are incomplete
(`libsimdutf.a`; `libpaimon_c.a` is also absent). The tests did not execute.
- [ ] Regression test: the added matrix has not been executed locally
against a Doris cluster.
- clang-tidy was attempted using the generated compilation database, but
missing PCH/compiler resources prevented analysis. It is **not** reported as
passing.
- Performance benchmarks have not been run.
- Behavior changed:
- [x] Yes: automatic bucket selection and an explicit session override;
query results are unchanged.
- Does this need documentation?
- [x] Yes: the session-variable reference should describe
`bucketed_agg_num_buckets`; its inline description is included here, and a
website documentation PR is still pending.
<details>
<summary>Review checkpoints</summary>
Execution flow: session variable → captured plan value → optional Thrift
field → once-only shared initialization → independent sink tables → completion
publication → per-bucket merge/output → shared-state cleanup.
| Checkpoint | Conclusion |
| --- | --- |
| Goal | Automatic and explicit counts cover serial and parallel execution;
FE tests pass, BE/runtime execution remains pending. |
| Scope | Changes are limited to bucketed aggregation, its session/Thrift
plumbing and tests. |
| Concurrency | Existing `call_once`, completion release/acquire operations
and per-bucket CAS ownership are preserved; no new lock order. |
| Lifecycle | Tables and arenas retain query-scoped ownership; the
single-sink shortcut transfers no ownership. No new global initialization. |
| Configuration | Validated, forwarded, captured per plan and effective
without restart. |
| Compatibility | Optional field; absent means 256. Older BEs ignore the
field. No persisted or exchanged bucket layout changes. |
| Thrift IDs | Checked current master and branches 3.1/4.0/4.1/4.2: only
master defines this struct, and field 6 is unused. |
| Parallel paths | Sink/source loops and all existing
numeric/string/composite, nullable and inline-count variants use the active
count. |
| Conditions | Power-of-two/range validation protects mask routing and the
fixed maximum state array; automatic selection checks both task counts. |
| Test coverage | BE routing/configuration/concurrent initialization/invalid
inputs; FE forwarding/validation/plan capture; SQL NULLs, multiple batches,
distinct groups, empty input, LIMIT, invalid SET and count/parallelism
combinations. |
| Expected results | All 120 supplied sections independently checked; this
does not replace runtime regression execution. |
| Observability | Effective bucket count, sink count and partition time are
visible in the profile. |
| Transactions/persistence | No EditLog, transaction, visible-version,
delete-bitmap or storage-format changes. |
| Writes/failure cleanup | No new data-write path; existing cancellation,
LIMIT and shared-state cleanup remain in use. |
| FE/BE propagation | The sole bucketed-node construction captures the
session value; serialization and BE validation cover its execution path. |
| Performance/memory | Avoids serial scatter/table overhead and unused COUNT
storage; routing no longer fixes practical phmap probe bits. Benchmark
validation remains pending. |
| Additional review | Preserved newer master COUNT(slot) eligibility and
regression protections; checked const-column materialization and NULL-key
merging. |
</details>
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
