morningman opened a new issue, #66715: URL: https://github.com/apache/doris/issues/66715
### Description Umbrella issue for the BE build-time optimization series: make a cold BE build several times faster and shrink the incremental rebuild radius of the hottest headers, with zero runtime behavior change. The batch was developed and measured on one branch and published as the combined reference PR #66510, which stays open as the whole-batch reference carrying the end-to-end measurements. It is being landed as the focused PR series below; the split PRs supersede it. **Headline numbers** - #66400 (part 1, merged): cold full BE build **~-22%** by cutting hot-header include edges. - The rest of the batch, measured as one pairwise A/B at `./build.sh --compile-bench -j 6` on the same idle machine (details in #66510): **21m45s → 11m10s (-48.7%)** end-to-end, sum of per-TU wall **-51.3%**, compile units 8380 → 7281. - Incremental rebuild radius: today touching `storage/olap_common.h` rebuilds **all 318** first-party TUs (through the PCH) plus the 219MB PCH itself; after the final PR in the series it rebuilds 182, and the PCH depends on 9 doris headers instead of 31. ### PR series **Landed** - [x] #66400 — 1/x: cut hot include edges under `exec_env.h` / `runtime_state.h` / `thread_context.h` (~-22% cold build) - [x] #66615 — fix the BE unit-test & benchmark build on macOS arm64; also fixes `-fno-access-control` never actually reaching `doris_be_test` on any platform - [x] #66616 — build-timing and header-closure sweep tooling (`closure-sweep.sh --no-pch`, `rebuild_radius.py`, `tu-bench.sh`): the measurement gate the rest of the series points at - [x] #66617 — remove unreachable mixed-width decimal registrations and hand-written constant-constant paths in binary arithmetic (sentinel TU -39% compile time, registration surface pinned by a contract UT) - [x] #66672 — 3/x: three more waves of hot include-edge cuts (`column.h`, RLE decoders, pipeline `dependency.h`), same seed-then-cut shape as #66400 **In review** - [ ] #66712 — introduce `ENABLE_UNITY_BUILD` (default ON) and pilot unity builds on three glue targets (-11.8% end-to-end at -j14; information_schema segment 13×, http 10.4×) **Planned — numbers measured on the combined branch, to be re-validated in each PR** - [ ] Unity builds for Exec and Exprs, the two heaviest targets (-12.9% and -11.2% end-to-end on the combined branch) - [ ] Unity builds for the remaining targets (Cloud / Storage / Runtime / Load / IO / Util / …, -20.7% on top; the conversion also surfaced 4 latent bugs that get fixed on the way) - [ ] `extern template` declarations for the existing explicit instantiations (~-2% cold build, `doris_be` -47MB, cross-TU duplicate weak symbols -68%) - [ ] Decouple `storage/olap_common.h` from the bottom of the include graph and slim the PCH — the incremental-rebuild-radius PR (318 → 182 dependents; PCH trigger line 31 → 9 doris headers) - [ ] Small standalone: `DORIS_DEV_DEBUG_INFO=line-tables` developer knob (-10.8% per TU for local debug builds; default output bit-identical) ### How each PR is written Every PR body carries its own measured numbers with the measurement conditions (-j level, PCH on/off, paired runs), its verification evidence (no-PCH closure sweep over all TUs, incremental rebuilds, BE UT, regression/microbenchmarks where semantics are touched), and a disclosure section for negative results and known blind spots — e.g. all local measurements are macOS/clang20, so Linux gcc/clang coverage relies on the PR CI lines. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
