This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7b00434015c [opt](build) Add ENABLE_UNITY_BUILD and pilot unity builds
on three glue targets (#66712)
7b00434015c is described below
commit 7b00434015ca962f2d5c1964b402062c82018c45
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri Aug 14 13:54:17 2026 +0800
[opt](build) Add ENABLE_UNITY_BUILD and pilot unity builds on three glue
targets (#66712)
> Part of the BE build-time optimization series tracked in #66715.
>
> Split out of **https://github.com/apache/doris/pull/66510**, which
carries the whole
> BE build-time batch. After the header-closure surgery in **#66400**
and **#66672**,
> this PR opens the second line of that batch: CMake unity builds. It
adds the
> infrastructure switch and converts the three lowest-risk glue segments
as a pilot;
> the heavier targets (Exec, Exprs, and the rest) follow in separate PRs
once this
> one has proven the mechanism cross-platform.
### What problem does this PR solve?
Related PR: #66510, #66672
Problem Summary:
Most of the BE's cold-build time is not spent compiling our code — it is
spent
**re-parsing the same shared header closure once per small `.cpp`**. For
the glue
directories this ratio is extreme: the 51 InformationSchema scanners
cost ~124 CPU s
(with PCH) of which almost everything is the closure re-parse; the ~58
http handlers
sum to ~5.6 min of slot time for a few thousand lines of handler logic.
CMake's built-in `UNITY_BUILD` (CMake ≥ 3.16; we require 3.19.2)
concatenates groups
of `.cpp` files into jumbo TUs, so a closure is parsed **once per
batch** instead of
once per file. This PR:
1. **Adds the switch** — `option(ENABLE_UNITY_BUILD ON)` in
`be/CMakeLists.txt`,
plumbed through `build.sh` and `run-be-ut.sh` exactly like `ENABLE_PCH`,
overridable from the environment / `custom_env.sh`. Every per-target
`UNITY_BUILD` property is set unconditionally from it -- including an
explicit
OFF, so the switch also wins over CMake's own `CMAKE_UNITY_BUILD` in a
reused
cache. Turn it OFF for per-file diagnostics, per-file tooling
(clang-tidy,
coverage), or the finest-grained incremental rebuilds.
2. **Pilots unity on three glue segments** (and only there — everything
else is
explicitly opted out per file, not by omission):
- **InformationSchema**: all 51 scanner TUs → 1 unity TU
(`UNITY_BUILD_BATCH_SIZE 0`).
- **Service, scoped to `http/`**: ~58 handler TUs → 1 unity TU. The
non-http
service sources (service entry points, arrow_flight) are heterogeneous
heavy TUs
that gain nothing from merging and stay individual.
- **Storage, scoped to `index/`**: ~111 TUs sharing one CLucene-heavy
closure →
4 unity TUs of ≤ 32 sources (batch size bounds jumbo-TU size and
memory).
3. **Makes the merged TUs legal C++** with two hygiene commits that
stand on their
own even without unity:
- 27 http action files each carried a private copy of the same
file-scope
constants (`HEADER_JSON` ×16, `TABLET_ID` ×7, `SCHEMA_HASH` ×4, …). They
move to
one shared header as C++17 inline variables, values unchanged.
- `prefix_query` declared `get_prefix_terms(IndexReader*)` relying on a
header-level `CL_NS_USE(index)` using-directive. Inside
`doris::segment_v2` that
unqualified name silently flips to `doris::segment_v2::IndexReader` as
soon as
any sibling source brings that type into scope — a landmine with or
without
unity. The declaration now spells out `lucene::index::IndexReader`.
4. **Fixes a latent bug unity exposed**: `schema_scanner_helper.h`
opened with
`#ifndef _SCHEMA_SCANNER_HELPER_H_` but never defined the macro (and one
include
sat outside the guard), so the guard never worked. Harmless while every
TU
included it exactly once; breaks immediately under unity. Now `#pragma
once`.
5. **Keeps the compile-bench tooling honest under unity** — with the
default ON,
`build.sh --compile-bench` produces a build directory whose sources are
jumbo TUs,
which two of the already-merged tools silently mis-read: `cut_impact.py`
dropped
every unity dependency block (and with it all 211 pilot sources) from
its TU set,
and `report.py` filed each batch's time under the build directory
instead of the
module it came from. `cut_impact.py` now refuses a unity build directory
with the
rebuild instruction — its analysis needs per-source closures, and
splitting a
batch's union back over its members would over-estimate instead of
under-report —
`report.py` attributes a unity TU to its target's directory, and
`rebuild_radius.py` says when its per-header counts are batch-granular.
`build-support/tests/test-compile-bench-unity.sh` pins all three
behaviours.
Files whose file-scope macros must not leak into unity siblings stay
individual via
`SKIP_UNITY_BUILD_INCLUSION`: `http_parser.cpp` (CR/LF),
`be_thread_stack_action.cpp`
(`UNW_LOCAL_ONLY`), and four index files (`CL_MAX_PATH` and friends,
`IS_CHINESE_CHAR`, `APPLY_FOR_PRIMITITYPE`). A future file whose
file-scope symbols
clash inside a unity TU can opt out the same way.
### Measured results
All numbers from the development branch this series is split from (which
also
carried the #66672 include cuts), macOS arm64 + clang 20, `-j14`,
`ENABLE_PCH=ON`, cold builds, back-to-back A/B:
| metric | before | after |
|---|---|---|
| build phase wall | 11m38s | **10m16s (-82.4 s / -11.8%)** |
| Σ per-TU CPU | 147.9 min | **134.8 min (-8.9%)** |
| TU count | 8384 | 8180 |
| failures | 0 | 0 |
Per-segment slot time (sum of compile time the segment occupies across
slots):
| segment | before | after | ratio |
|---|---|---|---|
| information_schema | 221.7 s | 17.0 s | **13×** |
| service/http | 273.6 s | 26.3 s | **10.4×** |
| storage/index | 264.2 s | 120.1 s | **2.2×** (remainder includes the
four macro SKIPs) |
Why unity rather than more PCH: a serial cold micro-benchmark of the
InformationSchema target alone measured **9.9× with PCH** (124 s → 12.5
s) and
**14.6× without PCH** (206 s → 14.1 s) — and unity *without* PCH still
beats
individual TUs *with* PCH by 8.8×. Unity removes the repeated parse
instead of
amortizing it, and the two compose.
Side effects on artifacts: `libInformationSchema.a` 334 MB → 29 MB,
`libStorage.a`
1492 MB → 1318 MB (linkonce_odr instantiations dedup inside each unity
TU). The
largest unity TU peaks at 2.1 GB RSS — *below* the largest existing
individual TU in
the tree (3.9 GB), so `-jN` memory envelopes are unchanged.
### Risk and verification
- **Unity changes TU grouping only; no code changes ride along** beyond
the two
hygiene commits described above (constant dedup with identical values,
one
qualified name, one include guard).
- **Archive symbol parity** was checked per target on the development
branch:
InformationSchema keeps all 2303 external defined symbols (plus 5 weak
`unique_ptr<SchemaXxxScanner>` instantiations that dedup), Service keeps
all 4054,
Storage keeps all external symbols with 8 weak linkonce_odr template
instantiations deduping away — which is the point of unity, not a loss.
- **This exact branch, rebased onto current master, full BE build from
scratch**
(macOS arm64, clang 20, `ENABLE_PCH=ON`, `ENABLE_UNITY_BUILD=ON`):
**8349/8349 ninja edges, zero failures, `doris_be` links.** The six
expected unity
TUs (InformationSchema ×1, Service http ×1, storage/index ×4) all
compile.
This includes `schema_tso_status_scanner.cpp`, added upstream after the
pilot was
measured — it lands inside the InformationSchema unity TU via the
existing
`GLOB_RECURSE` with zero CMakeLists edits, which is the intended
maintenance story.
- **The OFF path is verified on the same tree**: reconfiguring with
`ENABLE_UNITY_BUILD=OFF` removes every `unity_*.cxx` entry from
`compile_commands.json` and flips **exactly 219 ninja edges** — the
three targets'
per-file objects plus their archives and the final link, nothing else.
All of them
compile per-file with zero failures and `doris_be` links again. The
blast radius
of the switch is precisely the three pilot targets.
- **`OFF` also wins over a native-unity cache**: configuring with
`-DENABLE_UNITY_BUILD=OFF -DCMAKE_UNITY_BUILD=ON` produced 28 unity TUs
across the
three pilot targets before the gate fix and **0** after, while the
ordinary
`-DENABLE_UNITY_BUILD=ON` configure still produces exactly the 6
advertised
batches (InformationSchema ×1, Service ×1, Storage ×4).
- **These three segments have been building as unity TUs on the
development branch
since 2026-08-08**, through repeated full-tree builds and the BE UT
builds that
verified #66672 (the UT binaries link against these same target
libraries).
### Proactive disclosure
- **Cross-platform is the blind spot, and the default is deliberately ON
so this
PR's own CI closes it.** Every local build and measurement above is
macOS arm64 +
clang 20. Nothing here is platform-specific by construction, but with
the default
ON, the Linux compile lanes and every regression pipeline in this PR's
CI run
against unity builds — that is the validation. Please give the Linux gcc
lane in
particular a look. If some environment trips over unity after merge, the
escape hatches are, in order: per-user `ENABLE_UNITY_BUILD=OFF` (env or
`custom_env.sh`), per-file `SKIP_UNITY_BUILD_INCLUSION`, or a one-line
default
flip — the infrastructure stays either way.
- **The incremental-rebuild trade-off is real**: touching one `.cpp`
inside a unity
batch recompiles the whole batch. For the glue chosen here a batch
compiles in
~16–30 s, comparable to single mid-weight TUs elsewhere in the tree; and
`ENABLE_UNITY_BUILD=OFF` restores per-file granularity for workflows
that need it.
This is also why the pilot targets are glue directories and not the
hot-edit paths.
- **contrib (openblas/clucene) was evaluated and deliberately left
alone**: the
f2c-generated LAPACK sources and the snowball stemmers define clashing
file-scope
statics (`static c__1` and friends) — structurally un-unifiable without
rewriting
generated code — and contrib compiles once and rarely changes.
- **The rest of Storage and Service is opted out per file, on purpose.**
Merging
heavy heterogeneous TUs earns nothing (the closure parse is not the
dominant cost
there) and risks monster TUs. Follow-up PRs extend unity to Exec, Exprs
and the
remaining targets with the same SKIP discipline; on the development
branch the
full rollout takes the same tree from 10m16s to **6m18s**.
---
be/CMakeLists.txt | 15 +++
be/src/information_schema/CMakeLists.txt | 8 ++
be/src/information_schema/schema_scanner_helper.h | 4 +-
be/src/service/CMakeLists.txt | 14 +++
.../http/action/action_constants.h} | 43 +++-----
.../service/http/action/batch_download_action.cpp | 2 +-
.../http/action/check_encryption_action.cpp | 2 +-
.../http/action/check_tablet_segment_action.cpp | 3 +-
be/src/service/http/action/checksum_action.cpp | 3 +-
be/src/service/http/action/compaction_action.cpp | 7 +-
.../http/action/compaction_score_action.cpp | 2 +-
be/src/service/http/action/config_action.cpp | 2 +-
.../service/http/action/delete_bitmap_action.cpp | 7 +-
be/src/service/http/action/download_action.cpp | 2 +-
be/src/service/http/action/file_cache_action.cpp | 4 +-
be/src/service/http/action/health_action.cpp | 3 +-
be/src/service/http/action/http_stream.cpp | 3 +-
be/src/service/http/action/jeprofile_actions.cpp | 2 +-
be/src/service/http/action/load_channel_action.cpp | 3 +-
be/src/service/http/action/load_stream_action.cpp | 3 +-
be/src/service/http/action/meta_action.cpp | 3 +-
be/src/service/http/action/pad_rowset_action.cpp | 2 +-
be/src/service/http/action/peer_cache_action.cpp | 3 +-
.../service/http/action/reload_tablet_action.cpp | 5 +-
.../service/http/action/restore_tablet_action.cpp | 4 +-
.../http/action/show_nested_index_file_action.cpp | 3 +-
be/src/service/http/action/snapshot_action.cpp | 4 +-
be/src/service/http/action/stream_load.cpp | 2 +-
.../http/action/tablet_migration_action.cpp | 2 +-
.../http/action/tablets_distribution_action.cpp | 3 +-
be/src/service/http/action/tablets_info_action.cpp | 3 +-
be/src/service/http/action/version_action.cpp | 3 +-
be/src/storage/CMakeLists.txt | 17 +++
.../storage/index/inverted/query/prefix_query.cpp | 4 +-
be/src/storage/index/inverted/query/prefix_query.h | 5 +-
build-support/compile-bench/cut_impact.py | 23 +++-
build-support/compile-bench/rebuild_radius.py | 9 +-
build-support/compile-bench/report.py | 23 +++-
build-support/tests/test-compile-bench-unity.sh | 116 +++++++++++++++++++++
build.sh | 2 +
run-be-ut.sh | 2 +
41 files changed, 277 insertions(+), 93 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 4bbd22dd774..0741d0a17dd 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -80,6 +80,20 @@ add_definitions(-DGLOG_CUSTOM_PREFIX_SUPPORT)
option(GLIBC_COMPATIBILITY "Enable compatibility with older glibc libraries."
ON)
option(USE_LIBCPP "Use libc++" OFF)
option(USE_JEMALLOC "Use jemalloc" ON)
+# Merge groups of .cpp files into jumbo translation units for much faster full
+# builds. Turn OFF for precise per-file diagnostics, per-file tooling
+# (clang-tidy/coverage), or the finest-grained incremental rebuilds.
+option(ENABLE_UNITY_BUILD "Enable CMake unity builds for BE targets" ON)
+# Normalize to a strict ON/OFF. Every pilot target sets its UNITY_BUILD
property
+# unconditionally from this value: OFF must be an explicit OFF, otherwise a
+# cache that also carries CMake's own CMAKE_UNITY_BUILD=ON leaves the property
+# ON (it initializes UNITY_BUILD at add_library time) and the escape hatch does
+# nothing.
+if (ENABLE_UNITY_BUILD)
+ set(DORIS_UNITY_BUILD ON)
+else()
+ set(DORIS_UNITY_BUILD OFF)
+endif()
if (OS_MACOSX)
set(GLIBC_COMPATIBILITY OFF)
set(USE_LIBCPP ON)
@@ -103,6 +117,7 @@ message(STATUS "USE_LIBCPP is ${USE_LIBCPP}")
message(STATUS "USE_JEMALLOC is ${USE_JEMALLOC}")
message(STATUS "USE_UNWIND is ${USE_UNWIND}")
message(STATUS "ENABLE_PCH is ${ENABLE_PCH}")
+message(STATUS "ENABLE_UNITY_BUILD is ${ENABLE_UNITY_BUILD}")
message(STATUS "USE_AVX2 is ${USE_AVX2}")
# set CMAKE_BUILD_TYPE
diff --git a/be/src/information_schema/CMakeLists.txt
b/be/src/information_schema/CMakeLists.txt
index f06b42180dc..a77f44d7963 100644
--- a/be/src/information_schema/CMakeLists.txt
+++ b/be/src/information_schema/CMakeLists.txt
@@ -25,3 +25,11 @@ file(GLOB_RECURSE SRC_FILES CONFIGURE_DEPENDS *.cpp)
add_library(InformationSchema STATIC ${SRC_FILES})
pch_reuse(InformationSchema)
+
+# Unity build: these schema scanners are homogeneous glue code whose per-TU
+# cost is dominated by re-parsing the same header closure once per file.
+# Batch size 0 merges all sources into a single unity TU (~10x faster than
+# compiling them individually). A new file whose file-scope symbols clash
+# inside the unity TU can opt out via SKIP_UNITY_BUILD_INCLUSION.
+set_target_properties(InformationSchema PROPERTIES UNITY_BUILD
${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 0)
diff --git a/be/src/information_schema/schema_scanner_helper.h
b/be/src/information_schema/schema_scanner_helper.h
index fbe23adae5f..decee9d804e 100644
--- a/be/src/information_schema/schema_scanner_helper.h
+++ b/be/src/information_schema/schema_scanner_helper.h
@@ -15,8 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-#include <cstdint>
-#ifndef _SCHEMA_SCANNER_HELPER_H_
+#pragma once
#include <stdint.h>
@@ -45,4 +44,3 @@ public:
};
} // namespace doris
-#endif
diff --git a/be/src/service/CMakeLists.txt b/be/src/service/CMakeLists.txt
index 938a3711c43..28985a2e031 100644
--- a/be/src/service/CMakeLists.txt
+++ b/be/src/service/CMakeLists.txt
@@ -36,6 +36,20 @@ add_library(Service STATIC ${SRC_FILES})
pch_reuse(Service)
+# Unity build scoped to the http glue: ~60 small handlers re-parsing the same
+# header closure once per file. Non-http sources (service entry points,
+# arrow_flight) are heterogeneous heavy TUs that gain nothing from merging.
+# http_parser.cpp and be_thread_stack_action.cpp stay individual because their
+# file-scope macros (CR/LF, UNW_LOCAL_ONLY) must not leak into unity siblings.
+set(SERVICE_UNITY_SKIP ${SRC_FILES})
+list(FILTER SERVICE_UNITY_SKIP EXCLUDE REGEX ".*/service/http/.*")
+list(APPEND SERVICE_UNITY_SKIP
+ ${CMAKE_CURRENT_SOURCE_DIR}/http/http_parser.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/http/action/be_thread_stack_action.cpp)
+set_source_files_properties(${SERVICE_UNITY_SKIP} PROPERTIES
SKIP_UNITY_BUILD_INCLUSION ON)
+set_target_properties(Service PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 0)
+
if (${MAKE_TEST} STREQUAL "OFF" AND ${BUILD_BENCHMARK} STREQUAL "OFF")
add_executable(doris_be
doris_main.cpp
diff --git a/be/src/storage/index/inverted/query/prefix_query.h
b/be/src/service/http/action/action_constants.h
similarity index 52%
copy from be/src/storage/index/inverted/query/prefix_query.h
copy to be/src/service/http/action/action_constants.h
index 1f146cc235d..e3cc8161573 100644
--- a/be/src/storage/index/inverted/query/prefix_query.h
+++ b/be/src/service/http/action/action_constants.h
@@ -17,29 +17,20 @@
#pragma once
-#include "storage/index/inverted/query/query.h"
-
-CL_NS_USE(index)
-
-namespace doris::segment_v2 {
-
-class PrefixQuery : public Query {
-public:
- PrefixQuery(SearcherPtr searcher, IndexQueryContextPtr context);
- ~PrefixQuery() override = default;
-
- void add(const InvertedIndexQueryInfo& query_info) override;
- void search(roaring::Roaring& roaring) override;
-
- void get_prefix_terms(IndexReader* reader, const std::wstring& field_name,
- const std::string& prefix, std::vector<std::string>&
prefix_terms,
- int32_t max_expansions = 50);
-
-private:
- SearcherPtr _searcher;
- IndexQueryContextPtr _context;
-
- UnionTermIterPtr _lead1;
-};
-
-} // namespace doris::segment_v2
\ No newline at end of file
+#include <cstddef>
+#include <string>
+
+namespace doris {
+
+// Constants shared by the http action handlers. Each of these used to be
+// copy-pasted as a file-scope constant in many action .cpp files; single
+// definitions let those sources live together in one unity TU.
+inline const std::string HEADER_JSON = "application/json";
+inline const std::string TABLET_ID = "tablet_id";
+inline const std::string SCHEMA_HASH = "schema_hash";
+inline const std::string OP = "op";
+inline const std::string PATH = "path";
+inline const std::string TOKEN_PARAMETER = "token";
+inline constexpr size_t MEBIBYTE = 1024 * 1024;
+
+} // namespace doris
diff --git a/be/src/service/http/action/batch_download_action.cpp
b/be/src/service/http/action/batch_download_action.cpp
index 4ccd7ec9671..ecbc84b2eea 100644
--- a/be/src/service/http/action/batch_download_action.cpp
+++ b/be/src/service/http/action/batch_download_action.cpp
@@ -29,6 +29,7 @@
#include "common/status.h"
#include "io/fs/local_file_system.h"
#include "runtime/exec_env.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_method.h"
#include "service/http/http_request.h"
@@ -40,7 +41,6 @@ namespace {
const std::string CHECK_PARAMETER = "check";
const std::string LIST_PARAMETER = "list";
const std::string DIR_PARAMETER = "dir";
-const std::string TOKEN_PARAMETER = "token";
} // namespace
BatchDownloadAction::BatchDownloadAction(
diff --git a/be/src/service/http/action/check_encryption_action.cpp
b/be/src/service/http/action/check_encryption_action.cpp
index 9db83df00ce..e8aa9691bc5 100644
--- a/be/src/service/http/action/check_encryption_action.cpp
+++ b/be/src/service/http/action/check_encryption_action.cpp
@@ -36,6 +36,7 @@
#include "io/fs/file_system.h"
#include "io/fs/path.h"
#include "runtime/exec_env.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_status.h"
@@ -44,7 +45,6 @@
namespace doris {
-const std::string TABLET_ID = "tablet_id";
const std::string GET_FOOTER = "get_footer";
CheckEncryptionAction::CheckEncryptionAction(ExecEnv* exec_env,
TPrivilegeHier::type hier,
diff --git a/be/src/service/http/action/check_tablet_segment_action.cpp
b/be/src/service/http/action/check_tablet_segment_action.cpp
index e86a7adc074..96ecebcad66 100644
--- a/be/src/service/http/action/check_tablet_segment_action.cpp
+++ b/be/src/service/http/action/check_tablet_segment_action.cpp
@@ -25,6 +25,7 @@
#include <string>
#include "service/backend_options.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -35,8 +36,6 @@
namespace doris {
-const static std::string HEADER_JSON = "application/json";
-
CheckTabletSegmentAction::CheckTabletSegmentAction(ExecEnv* exec_env,
StorageEngine& engine,
TPrivilegeHier::type hier,
TPrivilegeType::type type)
diff --git a/be/src/service/http/action/checksum_action.cpp
b/be/src/service/http/action/checksum_action.cpp
index bd8e36c9c9c..1aa24385ad7 100644
--- a/be/src/service/http/action/checksum_action.cpp
+++ b/be/src/service/http/action/checksum_action.cpp
@@ -24,6 +24,7 @@
#include "boost/lexical_cast.hpp"
#include "common/logging.h"
#include "common/status.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_request.h"
#include "service/http/http_status.h"
@@ -32,11 +33,9 @@
namespace doris {
-const std::string TABLET_ID = "tablet_id";
// do not use name "VERSION",
// or will be conflict with "VERSION" in thrift/config.h
const std::string TABLET_VERSION = "version";
-const std::string SCHEMA_HASH = "schema_hash";
ChecksumAction::ChecksumAction(ExecEnv* exec_env, StorageEngine& engine,
TPrivilegeHier::type hier,
TPrivilegeType::type type)
diff --git a/be/src/service/http/action/compaction_action.cpp
b/be/src/service/http/action/compaction_action.cpp
index 68b77808390..88c9bb1d169 100644
--- a/be/src/service/http/action/compaction_action.cpp
+++ b/be/src/service/http/action/compaction_action.cpp
@@ -33,6 +33,7 @@
#include "common/metrics/doris_metrics.h"
#include "common/status.h"
#include "service/backend_options.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -50,12 +51,6 @@
namespace doris {
using namespace ErrorCode;
-namespace {
-
-constexpr std::string_view HEADER_JSON = "application/json";
-
-} // namespace
-
CompactionAction::CompactionAction(CompactionActionType ctype, ExecEnv*
exec_env,
StorageEngine& engine, TPrivilegeHier::type
hier,
TPrivilegeType::type ptype)
diff --git a/be/src/service/http/action/compaction_score_action.cpp
b/be/src/service/http/action/compaction_score_action.cpp
index 7db935e1381..04febc7e1a4 100644
--- a/be/src/service/http/action/compaction_score_action.cpp
+++ b/be/src/service/http/action/compaction_score_action.cpp
@@ -42,6 +42,7 @@
#include "cloud/cloud_tablet_mgr.h"
#include "cloud/config.h"
#include "common/status.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_handler_with_auth.h"
#include "service/http/http_headers.h"
@@ -58,7 +59,6 @@ const std::string SYNC_META = "sync_meta";
const std::string COMPACTION_SCORE = "compaction_score";
constexpr size_t DEFAULT_TOP_N = std::numeric_limits<size_t>::max();
constexpr bool DEFAULT_SYNC_META = false;
-constexpr std::string_view TABLET_ID = "tablet_id";
template <typename T>
concept CompactionScoreAccessble = requires(T t) {
diff --git a/be/src/service/http/action/config_action.cpp
b/be/src/service/http/action/config_action.cpp
index 7fbf6666aa9..11f5aa00709 100644
--- a/be/src/service/http/action/config_action.cpp
+++ b/be/src/service/http/action/config_action.cpp
@@ -35,6 +35,7 @@
#include "common/config.h"
#include "common/logging.h"
#include "common/status.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -42,7 +43,6 @@
namespace doris {
-const static std::string HEADER_JSON = "application/json";
const static std::string PERSIST_PARAM = "persist";
const std::string CONF_ITEM = "conf_item";
diff --git a/be/src/service/http/action/delete_bitmap_action.cpp
b/be/src/service/http/action/delete_bitmap_action.cpp
index 58113471832..bd8b0603741 100644
--- a/be/src/service/http/action/delete_bitmap_action.cpp
+++ b/be/src/service/http/action/delete_bitmap_action.cpp
@@ -41,6 +41,7 @@
#include "common/logging.h"
#include "common/metrics/doris_metrics.h"
#include "common/status.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -52,12 +53,6 @@
namespace doris {
using namespace ErrorCode;
-namespace {
-
-constexpr std::string_view HEADER_JSON = "application/json";
-
-} // namespace
-
DeleteBitmapAction::DeleteBitmapAction(DeleteBitmapActionType ctype, ExecEnv*
exec_env,
BaseStorageEngine& engine,
TPrivilegeHier::type hier,
TPrivilegeType::type ptype)
diff --git a/be/src/service/http/action/download_action.cpp
b/be/src/service/http/action/download_action.cpp
index 683947901a5..74061d09bb0 100644
--- a/be/src/service/http/action/download_action.cpp
+++ b/be/src/service/http/action/download_action.cpp
@@ -26,6 +26,7 @@
#include "common/status.h"
#include "io/fs/local_file_system.h"
#include "runtime/exec_env.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_request.h"
#include "service/http/utils.h"
@@ -33,7 +34,6 @@
namespace doris {
namespace {
const std::string FILE_PARAMETER = "file";
-const std::string TOKEN_PARAMETER = "token";
const std::string CHANNEL_PARAMETER = "channel";
const std::string CHANNEL_INGEST_BINLOG_TYPE = "ingest_binlog";
const std::string ACQUIRE_MD5_PARAMETER = "acquire_md5";
diff --git a/be/src/service/http/action/file_cache_action.cpp
b/be/src/service/http/action/file_cache_action.cpp
index d291bc8df1d..2403f970139 100644
--- a/be/src/service/http/action/file_cache_action.cpp
+++ b/be/src/service/http/action/file_cache_action.cpp
@@ -32,6 +32,7 @@
#include "io/cache/block_file_cache_factory.h"
#include "io/cache/file_cache_common.h"
#include "io/cache/fs_file_cache_storage.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -42,10 +43,7 @@
namespace doris {
-constexpr static std::string_view HEADER_JSON = "application/json";
-constexpr static std::string_view OP = "op";
constexpr static std::string_view SYNC = "sync";
-constexpr static std::string_view PATH = "path";
constexpr static std::string_view CLEAR = "clear";
constexpr static std::string_view RESET = "reset";
constexpr static std::string_view HASH = "hash";
diff --git a/be/src/service/http/action/health_action.cpp
b/be/src/service/http/action/health_action.cpp
index 93ee2f46d63..27618940f4b 100644
--- a/be/src/service/http/action/health_action.cpp
+++ b/be/src/service/http/action/health_action.cpp
@@ -21,6 +21,7 @@
#include <string>
#include "runtime/exec_env.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -28,8 +29,6 @@
namespace doris {
-const static std::string HEADER_JSON = "application/json";
-
void HealthAction::handle(HttpRequest* req) {
std::string status;
std::string msg;
diff --git a/be/src/service/http/action/http_stream.cpp
b/be/src/service/http/action/http_stream.cpp
index ce33b3d8ef6..c374b21de7d 100644
--- a/be/src/service/http/action/http_stream.cpp
+++ b/be/src/service/http/action/http_stream.cpp
@@ -49,6 +49,7 @@
#include "runtime/cluster_info.h"
#include "runtime/exec_env.h"
#include "runtime/fragment_mgr.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_common.h"
#include "service/http/http_headers.h"
@@ -68,8 +69,6 @@ using namespace ErrorCode;
namespace {
-constexpr size_t MEBIBYTE = 1024 * 1024;
-
bool is_compressed_file_scan(const TPipelineFragmentParams& params) {
if (!params.__isset.file_scan_params) {
return false;
diff --git a/be/src/service/http/action/jeprofile_actions.cpp
b/be/src/service/http/action/jeprofile_actions.cpp
index e8c36b00cf1..d7702c8fb63 100644
--- a/be/src/service/http/action/jeprofile_actions.cpp
+++ b/be/src/service/http/action/jeprofile_actions.cpp
@@ -24,6 +24,7 @@
#include "agent/utils.h"
#include "runtime/memory/heap_profiler.h"
+#include "service/http/action/action_constants.h"
#include "service/http/ev_http_server.h"
#include "service/http/http_channel.h"
#include "service/http/http_handler.h"
@@ -33,7 +34,6 @@
namespace doris {
-const static std::string HEADER_JSON = "application/json";
const static std::string START_HEAP_PROFILE_NOTICE =
"`curl http://be_host:be_webport/jeheap/active/true` to start heap
profiler, note that "
"`JEMALLOC_CONF` in `be/conf/be.conf` must contain `prof:true`, will
only track and sample "
diff --git a/be/src/service/http/action/load_channel_action.cpp
b/be/src/service/http/action/load_channel_action.cpp
index 2a2e8c3714d..44f0e0aaf83 100644
--- a/be/src/service/http/action/load_channel_action.cpp
+++ b/be/src/service/http/action/load_channel_action.cpp
@@ -28,6 +28,7 @@
#include "load/channel/load_channel_mgr.h"
#include "runtime/exec_env.h"
#include "service/backend_options.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -38,8 +39,6 @@
namespace doris {
-const static std::string HEADER_JSON = "application/json";
-
void LoadChannelAction::handle(HttpRequest* req) {
req->add_output_header(HttpHeaders::CONTENT_TYPE, HEADER_JSON.c_str());
HttpChannel::send_reply(req, HttpStatus::OK,
_get_load_channels().ToString());
diff --git a/be/src/service/http/action/load_stream_action.cpp
b/be/src/service/http/action/load_stream_action.cpp
index 364deb7413f..f3300a82c22 100644
--- a/be/src/service/http/action/load_stream_action.cpp
+++ b/be/src/service/http/action/load_stream_action.cpp
@@ -28,6 +28,7 @@
#include "load/channel/load_stream_mgr.h"
#include "runtime/exec_env.h"
#include "service/backend_options.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -38,8 +39,6 @@
namespace doris {
-const static std::string HEADER_JSON = "application/json";
-
void LoadStreamAction::handle(HttpRequest* req) {
req->add_output_header(HttpHeaders::CONTENT_TYPE, HEADER_JSON.c_str());
HttpChannel::send_reply(req, HttpStatus::OK,
_get_load_streams().ToString());
diff --git a/be/src/service/http/action/meta_action.cpp
b/be/src/service/http/action/meta_action.cpp
index 4dec64d0532..b41dce36336 100644
--- a/be/src/service/http/action/meta_action.cpp
+++ b/be/src/service/http/action/meta_action.cpp
@@ -29,6 +29,7 @@
#include "cloud/config.h"
#include "common/logging.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -42,8 +43,6 @@
namespace doris {
-const static std::string HEADER_JSON = "application/json";
-const static std::string OP = "op";
const static std::string DATA_SIZE = "data_size";
const static std::string HEADER = "header";
diff --git a/be/src/service/http/action/pad_rowset_action.cpp
b/be/src/service/http/action/pad_rowset_action.cpp
index 017371af586..84ffd046de6 100644
--- a/be/src/service/http/action/pad_rowset_action.cpp
+++ b/be/src/service/http/action/pad_rowset_action.cpp
@@ -27,6 +27,7 @@
#include <string>
#include <vector>
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_request.h"
#include "service/http/http_status.h"
@@ -42,7 +43,6 @@
namespace doris {
namespace {
-const std::string TABLET_ID = "tablet_id";
const std::string START_VERSION = "start_version";
const std::string END_VERSION = "end_version";
diff --git a/be/src/service/http/action/peer_cache_action.cpp
b/be/src/service/http/action/peer_cache_action.cpp
index 29271c9aa55..12992802eb0 100644
--- a/be/src/service/http/action/peer_cache_action.cpp
+++ b/be/src/service/http/action/peer_cache_action.cpp
@@ -25,6 +25,7 @@
#include "cloud/cloud_storage_engine.h"
#include "cloud/cloud_warm_up_manager.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -32,8 +33,6 @@
namespace doris {
-static constexpr std::string_view HEADER_JSON = "application/json";
-
// Serialize a single TabletPeerCandidates into a rapidjson object
static void tablet_peer_to_json(int64_t tablet_id, const TabletPeerCandidates&
tpc,
rapidjson::Writer<rapidjson::StringBuffer>&
writer) {
diff --git a/be/src/service/http/action/reload_tablet_action.cpp
b/be/src/service/http/action/reload_tablet_action.cpp
index 9932e84efcf..cf15e50aa34 100644
--- a/be/src/service/http/action/reload_tablet_action.cpp
+++ b/be/src/service/http/action/reload_tablet_action.cpp
@@ -27,6 +27,7 @@
#include "common/logging.h"
#include "common/status.h"
#include "runtime/exec_env.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_request.h"
#include "service/http/http_status.h"
@@ -34,10 +35,6 @@
namespace doris {
-const std::string PATH = "path";
-const std::string TABLET_ID = "tablet_id";
-const std::string SCHEMA_HASH = "schema_hash";
-
ReloadTabletAction::ReloadTabletAction(ExecEnv* exec_env, StorageEngine&
engine,
TPrivilegeHier::type hier,
TPrivilegeType::type type)
: HttpHandlerWithAuth(exec_env, hier, type), _engine(engine) {}
diff --git a/be/src/service/http/action/restore_tablet_action.cpp
b/be/src/service/http/action/restore_tablet_action.cpp
index beeba3449ac..6036d254a63 100644
--- a/be/src/service/http/action/restore_tablet_action.cpp
+++ b/be/src/service/http/action/restore_tablet_action.cpp
@@ -35,6 +35,7 @@
#include "io/fs/local_file_system.h"
#include "io/fs/path.h"
#include "runtime/exec_env.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_request.h"
#include "service/http/http_status.h"
@@ -49,9 +50,6 @@ using std::filesystem::path;
namespace doris {
-const std::string TABLET_ID = "tablet_id";
-const std::string SCHEMA_HASH = "schema_hash";
-
RestoreTabletAction::RestoreTabletAction(ExecEnv* exec_env, StorageEngine&
engine,
TPrivilegeHier::type hier,
TPrivilegeType::type type)
: HttpHandlerWithAuth(exec_env, hier, type), _engine(engine) {}
diff --git a/be/src/service/http/action/show_nested_index_file_action.cpp
b/be/src/service/http/action/show_nested_index_file_action.cpp
index 84450da5bc0..1e7f737107c 100644
--- a/be/src/service/http/action/show_nested_index_file_action.cpp
+++ b/be/src/service/http/action/show_nested_index_file_action.cpp
@@ -23,6 +23,7 @@
#include <string>
#include "common/status.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -34,8 +35,6 @@
namespace doris {
using namespace ErrorCode;
-const static std::string HEADER_JSON = "application/json";
-
ShowNestedIndexFileAction::ShowNestedIndexFileAction(ExecEnv* exec_env,
TPrivilegeHier::type hier,
TPrivilegeType::type
ptype)
: HttpHandlerWithAuth(exec_env, hier, ptype) {}
diff --git a/be/src/service/http/action/snapshot_action.cpp
b/be/src/service/http/action/snapshot_action.cpp
index abd714928ba..82536c83aa1 100644
--- a/be/src/service/http/action/snapshot_action.cpp
+++ b/be/src/service/http/action/snapshot_action.cpp
@@ -26,6 +26,7 @@
#include "common/logging.h"
#include "common/status.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_request.h"
#include "service/http/http_status.h"
@@ -34,9 +35,6 @@
namespace doris {
-const std::string TABLET_ID = "tablet_id";
-const std::string SCHEMA_HASH = "schema_hash";
-
SnapshotAction::SnapshotAction(ExecEnv* exec_env, StorageEngine& engine,
TPrivilegeHier::type hier,
TPrivilegeType::type type)
: HttpHandlerWithAuth(exec_env, hier, type), _engine(engine) {}
diff --git a/be/src/service/http/action/stream_load.cpp
b/be/src/service/http/action/stream_load.cpp
index 81d1fe9ccef..8f47edaba8a 100644
--- a/be/src/service/http/action/stream_load.cpp
+++ b/be/src/service/http/action/stream_load.cpp
@@ -57,6 +57,7 @@
#include "load/stream_load/stream_load_recorder.h"
#include "runtime/cluster_info.h"
#include "runtime/exec_env.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_common.h"
#include "service/http/http_headers.h"
@@ -84,7 +85,6 @@ bvar::LatencyRecorder
g_stream_load_commit_and_publish_latency_ms("stream_load",
"commit_and_publish_ms");
static constexpr size_t MIN_CHUNK_SIZE = 64 * 1024;
-static constexpr size_t MEBIBYTE = 1024 * 1024;
static const std::string CHUNK = "chunked";
static const std::string OFF_MODE = "off_mode";
static const std::string SYNC_MODE = "sync_mode";
diff --git a/be/src/service/http/action/tablet_migration_action.cpp
b/be/src/service/http/action/tablet_migration_action.cpp
index 0883511cf8b..779b543bdaa 100644
--- a/be/src/service/http/action/tablet_migration_action.cpp
+++ b/be/src/service/http/action/tablet_migration_action.cpp
@@ -24,6 +24,7 @@
#include "common/config.h"
#include "common/status.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -34,7 +35,6 @@
#include "storage/task/engine_storage_migration_task.h"
namespace doris {
-const static std::string HEADER_JSON = "application/json";
void TabletMigrationAction::_init_migration_action() {
int32_t max_thread_num = config::max_tablet_migration_threads;
diff --git a/be/src/service/http/action/tablets_distribution_action.cpp
b/be/src/service/http/action/tablets_distribution_action.cpp
index 7d1b78baae6..faa0efdaa40 100644
--- a/be/src/service/http/action/tablets_distribution_action.cpp
+++ b/be/src/service/http/action/tablets_distribution_action.cpp
@@ -29,6 +29,7 @@
#include "absl/strings/substitute.h"
#include "common/status.h"
#include "service/backend_options.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -40,8 +41,6 @@
namespace doris {
-const static std::string HEADER_JSON = "application/json";
-
TabletsDistributionAction::TabletsDistributionAction(ExecEnv* exec_env,
StorageEngine& engine,
TPrivilegeHier::type hier,
TPrivilegeType::type type)
diff --git a/be/src/service/http/action/tablets_info_action.cpp
b/be/src/service/http/action/tablets_info_action.cpp
index 90c3614b40b..c3f082c6499 100644
--- a/be/src/service/http/action/tablets_info_action.cpp
+++ b/be/src/service/http/action/tablets_info_action.cpp
@@ -29,6 +29,7 @@
#include "cloud/config.h"
#include "runtime/exec_env.h"
#include "service/backend_options.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -39,8 +40,6 @@
namespace doris {
-const static std::string HEADER_JSON = "application/json";
-
TabletsInfoAction::TabletsInfoAction(ExecEnv* exec_env, TPrivilegeHier::type
hier,
TPrivilegeType::type type)
: HttpHandlerWithAuth(exec_env, hier, type) {}
diff --git a/be/src/service/http/action/version_action.cpp
b/be/src/service/http/action/version_action.cpp
index 76f6f064af0..c638e61790c 100644
--- a/be/src/service/http/action/version_action.cpp
+++ b/be/src/service/http/action/version_action.cpp
@@ -20,6 +20,7 @@
#include <string>
#include "common/version_internal.h"
+#include "service/http/action/action_constants.h"
#include "service/http/http_channel.h"
#include "service/http/http_headers.h"
#include "service/http/http_request.h"
@@ -28,8 +29,6 @@
namespace doris {
-const static std::string HEADER_JSON = "application/json";
-
VersionAction::VersionAction(ExecEnv* exec_env, TPrivilegeHier::type hier,
TPrivilegeType::type type)
: HttpHandlerWithAuth(exec_env, hier, type) {}
diff --git a/be/src/storage/CMakeLists.txt b/be/src/storage/CMakeLists.txt
index e7a82b486db..888b16ebe32 100644
--- a/be/src/storage/CMakeLists.txt
+++ b/be/src/storage/CMakeLists.txt
@@ -45,3 +45,20 @@ if (OS_MACOSX)
endif()
pch_reuse(Storage)
+
+# Unity build scoped to storage/index/: 115 homogeneous reader/writer/iterator
+# TUs sharing one (CLucene-heavy) header closure. The rest of Storage keeps
+# compiling individually. Batch 32 bounds jumbo-TU size and memory. Four index
+# files stay individual because their file-scope macros (CL_MAX_PATH and
+# friends, IS_CHINESE_CHAR, APPLY_FOR_PRIMITITYPE) must not leak into unity
+# siblings.
+set(STORAGE_UNITY_SKIP ${SRC_FILES})
+list(FILTER STORAGE_UNITY_SKIP EXCLUDE REGEX ".*/storage/index/.*")
+list(APPEND STORAGE_UNITY_SKIP
+
${CMAKE_CURRENT_SOURCE_DIR}/index/inverted/inverted_index_compound_reader.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/index/inverted/inverted_index_fs_directory.cpp
+
${CMAKE_CURRENT_SOURCE_DIR}/index/inverted/tokenizer/basic/basic_tokenizer.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/index/zone_map/zone_map_index.cpp)
+set_source_files_properties(${STORAGE_UNITY_SKIP} PROPERTIES
SKIP_UNITY_BUILD_INCLUSION ON)
+set_target_properties(Storage PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 32)
diff --git a/be/src/storage/index/inverted/query/prefix_query.cpp
b/be/src/storage/index/inverted/query/prefix_query.cpp
index fa1a92592da..4153a5696b4 100644
--- a/be/src/storage/index/inverted/query/prefix_query.cpp
+++ b/be/src/storage/index/inverted/query/prefix_query.cpp
@@ -49,8 +49,8 @@ void PrefixQuery::search(roaring::Roaring& roaring) {
}
}
-void PrefixQuery::get_prefix_terms(IndexReader* reader, const std::wstring&
field_name,
- const std::string& prefix,
+void PrefixQuery::get_prefix_terms(lucene::index::IndexReader* reader,
+ const std::wstring& field_name, const
std::string& prefix,
std::vector<std::string>& prefix_terms,
int32_t max_expansions) {
std::wstring ws_prefix = StringUtil::string_to_wstring(prefix);
diff --git a/be/src/storage/index/inverted/query/prefix_query.h
b/be/src/storage/index/inverted/query/prefix_query.h
index 1f146cc235d..e21f884498f 100644
--- a/be/src/storage/index/inverted/query/prefix_query.h
+++ b/be/src/storage/index/inverted/query/prefix_query.h
@@ -31,7 +31,10 @@ public:
void add(const InvertedIndexQueryInfo& query_info) override;
void search(roaring::Roaring& roaring) override;
- void get_prefix_terms(IndexReader* reader, const std::wstring& field_name,
+ // Explicitly qualified: unqualified IndexReader inside doris::segment_v2
+ // resolves to doris::segment_v2::IndexReader whenever that type is in
+ // scope (e.g. in a unity TU), not to the CL_NS_USE(index) one intended.
+ void get_prefix_terms(lucene::index::IndexReader* reader, const
std::wstring& field_name,
const std::string& prefix, std::vector<std::string>&
prefix_terms,
int32_t max_expansions = 50);
diff --git a/build-support/compile-bench/cut_impact.py
b/build-support/compile-bench/cut_impact.py
index 5507c01076a..e9c2d388fcc 100644
--- a/build-support/compile-bench/cut_impact.py
+++ b/build-support/compile-bench/cut_impact.py
@@ -67,6 +67,9 @@ REPO_ROOT = os.path.dirname(os.path.dirname(SCRIPT_DIR))
SOURCE_EXTS = (".cpp", ".cc", ".c", ".cxx")
HEADER_EXTS = (".h", ".hpp", ".hh", ".inc", ".ipp")
+# CMake writes unity sources as <target-dir>/Unity/unity_<n>_cxx.cxx (or _c.c).
+UNITY_SOURCE_RE = re.compile(r"/Unity/unity_\d+_c(?:xx)?\.c(?:xx)?$")
+
INCLUDE_RE = re.compile(r'^\s*#\s*include\s+(["<])([^">]+)[">]', re.M)
COMMENT_RE = re.compile(r"//[^\n]*|/\*.*?\*/", re.S)
TOKEN_RE = re.compile(r"[A-Za-z_]\w{2,}")
@@ -170,7 +173,15 @@ class IncludeGraph:
def load_tus(build_dir, graph):
- """source abs path -> set of project files in its real (ninja) closure."""
+ """source abs path -> set of project files in its real (ninja) closure.
+
+ Refuses a unity build dir instead of analyzing it. A unity TU concatenates
+ many sources into one jumbo .cxx, so ninja reports a single flat closure
for
+ the whole batch: the per-source closures this tool needs are simply not in
+ the database. Splitting the union back over the members would be worse than
+ useless -- every member would appear to reach every header any sibling
pulls
+ in, inflating both the affected-TU counts and the seeding advice.
+ """
proc = subprocess.Popen(
["ninja", "-C", build_dir, "-t", "deps"],
stdout=subprocess.PIPE,
@@ -178,6 +189,7 @@ def load_tus(build_dir, graph):
text=True,
)
tus = {}
+ unity = 0 # unity TUs seen: their member sources are not individually
visible
src_prefix = os.path.join(REPO_ROOT, "be", "src") + os.sep
gen_prefix = os.path.join(REPO_ROOT, "gensrc", "build") + os.sep
cur = None # dep set of the current block's TU; None while skipping a
block
@@ -195,6 +207,8 @@ def load_tus(build_dir, graph):
cur = tus.setdefault(key, set())
cur.add(key)
else:
+ if UNITY_SOURCE_RE.search(p):
+ unity += 1
cur = None # foreign TU (contrib etc.): skip whole block
elif cur is not None and p in graph.files:
cur.add(sys.intern(p))
@@ -202,6 +216,13 @@ def load_tus(build_dir, graph):
expect_source = line.rstrip().endswith("(VALID)")
cur = None
proc.wait()
+ if unity:
+ sys.exit(
+ "error: {} unity translation units in {}\n"
+ " Unity TUs hide their member sources from `ninja -t deps`, so
this\n"
+ " analysis would silently run on a partial TU set. Reconfigure
that\n"
+ " build directory with unity builds off and rebuild:\n"
+ " ENABLE_UNITY_BUILD=OFF ./build.sh
--compile-bench".format(unity, build_dir))
return tus
diff --git a/build-support/compile-bench/rebuild_radius.py
b/build-support/compile-bench/rebuild_radius.py
index ceaa6a6e5a5..400836ebecd 100644
--- a/build-support/compile-bench/rebuild_radius.py
+++ b/build-support/compile-bench/rebuild_radius.py
@@ -72,6 +72,7 @@ def main():
counts = {h: 0 for h in headers}
pch_deps = set()
objects = 0
+ unity_objects = 0
proc = subprocess.Popen(["ninja", "-C", args.build_dir, "-t", "deps"],
stdout=subprocess.PIPE, text=True,
errors="replace",
@@ -79,7 +80,7 @@ def main():
target, target_is_pch, hits = None, False, set()
def flush():
- nonlocal objects
+ nonlocal objects, unity_objects
if target is None:
return
if target_is_pch:
@@ -89,6 +90,8 @@ def main():
counts[h] += 1
if target.endswith((".o", ".obj")):
objects += 1
+ if "/Unity/unity_" in target:
+ unity_objects += 1
for line in proc.stdout:
if line[:1] not in (" ", "\t", "\n"):
@@ -105,6 +108,10 @@ def main():
proc.wait()
print(f"{objects:,} object targets in the deps database\n")
+ if unity_objects:
+ print(f"note: {unity_objects:,} of them are unity TUs, so the sources
they batch "
+ "count once per\n batch instead of once per file. For
per-file counts "
+ "rebuild with ENABLE_UNITY_BUILD=OFF.\n")
print(f"{'header':58s} {'dependents':>10s} via PCH (= rebuilds
everything)")
for h in headers:
flag = "YES" if h in pch_deps else "no"
diff --git a/build-support/compile-bench/report.py
b/build-support/compile-bench/report.py
index d70ed286f00..e8f95e414a5 100755
--- a/build-support/compile-bench/report.py
+++ b/build-support/compile-bench/report.py
@@ -38,6 +38,7 @@ Outputs inside <run_dir>: report.txt (human) and summary.json
(machine).
import argparse
import json
import os
+import re
import sys
from collections import defaultdict
@@ -47,6 +48,10 @@ TOP_TEMPLATES = 20
TOP_TAIL = 10
WIDTH = 78
+# be/<build dir>/<mirrored source
dir>/CMakeFiles/<target>.dir/Unity/unity_<n>_cxx.cxx
+UNITY_SOURCE_RE = re.compile(
+
r"^be/build[^/]*/(.+?)/CMakeFiles/[^/]+\.dir/Unity/unity_\d+_c(?:xx)?\.c(?:xx)?$")
+
def section(title):
text = "-- " + title + " "
@@ -166,9 +171,25 @@ def shorten_header(path, meta):
return path
+def unity_source_dir(rel_src):
+ """Logical source directory of a generated unity TU, or None.
+
+ CMake writes unity sources into the build tree, mirroring the source
layout:
+
be/build_Release_compile_bench/src/storage/CMakeFiles/Storage.dir/Unity/unity_0_cxx.cxx
+ which batches sources from be/src/storage. Without this the whole batch
rolls
+ up under the build directory instead of the module it came from.
+
+ Attribution is only as fine as the target's own directory: a batch that
+ merges just be/src/service/http sources still lands on be/src/service, and
a
+ unity TU is one timing for all of its members by construction.
+ """
+ match = UNITY_SOURCE_RE.match(rel_src)
+ return "be/" + match.group(1) if match else None
+
+
def group_keys(rel_src):
"""Return (level1, level2) directory grouping keys for a
doris_home-relative source."""
- rel_dir = os.path.dirname(rel_src)
+ rel_dir = unity_source_dir(rel_src) or os.path.dirname(rel_src)
if rel_dir.startswith("be/src"):
prefix, rest = "be/src", rel_dir[len("be/src"):].strip("/")
else:
diff --git a/build-support/tests/test-compile-bench-unity.sh
b/build-support/tests/test-compile-bench-unity.sh
new file mode 100755
index 00000000000..2145a9b9908
--- /dev/null
+++ b/build-support/tests/test-compile-bench-unity.sh
@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# The compile-bench tools read a build directory that may or may not have been
+# configured with ENABLE_UNITY_BUILD=ON, and a unity TU looks nothing like the
+# per-file shape they were written against. Both behaviours below are invisible
+# on a real build (one is a refusal, the other a grouping key), so they need a
+# fixture to stay honest.
+
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+python3 - "${ROOT}/compile-bench" <<'PY'
+import importlib.util
+import os
+import stat
+import sys
+import tempfile
+import types
+
+bench_dir = sys.argv[1]
+
+
+def load(name):
+ spec = importlib.util.spec_from_file_location(name,
os.path.join(bench_dir, name + ".py"))
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+cut_impact = load("cut_impact")
+report = load("report")
+repo = cut_impact.REPO_ROOT
+failures = []
+
+
+def check(label, cond):
+ print((" ok " if cond else " FAIL ") + label)
+ if not cond:
+ failures.append(label)
+
+
+def load_tus_with(deps_output):
+ """Run cut_impact.load_tus against a canned `ninja -t deps` database."""
+ tmp = tempfile.mkdtemp()
+ fake_ninja = os.path.join(tmp, "ninja")
+ with open(fake_ninja, "w") as fh:
+ fh.write("#!/bin/sh\ncat <<'EOF'\n" + deps_output + "\nEOF\n")
+ os.chmod(fake_ninja, os.stat(fake_ninja).st_mode | stat.S_IEXEC)
+ old_path = os.environ["PATH"]
+ os.environ["PATH"] = tmp + os.pathsep + old_path
+ try:
+ return cut_impact.load_tus(os.path.join(tmp, "build"),
types.SimpleNamespace(files=set()))
+ finally:
+ os.environ["PATH"] = old_path
+
+
+print("cut_impact.load_tus")
+
+# GREEN: the per-file shape the analysis is written against.
+standalone = (
+ "src/service/CMakeFiles/Service.dir/http/action/health_action.cpp.o: "
+ "#deps 2, deps mtime 1 (VALID)\n"
+ " {repo}/be/src/service/http/action/health_action.cpp\n"
+ " {repo}/be/src/service/http/action/health_action.h"
+).format(repo=repo)
+tus = load_tus_with(standalone)
+check("keeps a standalone TU", list(tus) == [repo +
"/be/src/service/http/action/health_action.cpp"])
+
+# RED: a unity TU. Its member sources are invisible here, so the analysis must
+# refuse rather than quietly run on whatever is left.
+unity = (
+
"src/information_schema/CMakeFiles/InformationSchema.dir/Unity/unity_0_cxx.cxx.o:
"
+ "#deps 2, deps mtime 1 (VALID)\n"
+ "
src/information_schema/CMakeFiles/InformationSchema.dir/Unity/unity_0_cxx.cxx\n"
+ " {repo}/be/src/information_schema/schema_scanner.cpp"
+).format(repo=repo)
+try:
+ load_tus_with(unity)
+ check("refuses a unity deps database", False)
+except SystemExit as exc:
+ message = str(exc)
+ check("refuses a unity deps database", "unity translation units" in
message)
+ check("names the way out", "ENABLE_UNITY_BUILD=OFF" in message)
+
+print("report.group_keys")
+unity_src =
("be/build_Release_compile_bench/src/storage/CMakeFiles/Storage.dir"
+ "/Unity/unity_0_cxx.cxx")
+check("unity TU groups under the module it batches",
+ report.group_keys(unity_src) == ("be/src/storage", "be/src/storage"))
+check("ordinary source is unchanged",
+ report.group_keys("be/src/service/http/action/health_action.cpp")
+ == ("be/src/service", "be/src/service/http"))
+check("build-tree non-unity path is unchanged",
+ report.group_keys("be/build_Release/src/agent/foo.cpp")[0] == "be")
+
+if failures:
+ sys.exit("{} check(s) failed".format(len(failures)))
+print("all checks passed")
+PY
diff --git a/build.sh b/build.sh
index e25b9aa0959..790da28c7f3 100755
--- a/build.sh
+++ b/build.sh
@@ -767,6 +767,7 @@ echo "Get params:
DENABLE_CLANG_COVERAGE -- ${DENABLE_CLANG_COVERAGE}
DISPLAY_BUILD_TIME -- ${DISPLAY_BUILD_TIME}
ENABLE_PCH -- ${ENABLE_PCH}
+ ENABLE_UNITY_BUILD -- ${ENABLE_UNITY_BUILD:-ON}
EXTRA_FE_MODULES -- ${EXTRA_FE_MODULES}
EXTRA_BE_MODULES -- ${EXTRA_BE_MODULES}
EXTRA_CLOUD_MODULES -- ${EXTRA_CLOUD_MODULES}
@@ -943,6 +944,7 @@ if [[ "${BUILD_BE}" -eq 1 ]]; then
-DSTRIP_DEBUG_INFO="${STRIP_DEBUG_INFO}" \
-DDISPLAY_BUILD_TIME="${DISPLAY_BUILD_TIME}" \
-DENABLE_PCH="${ENABLE_PCH}" \
+ -DENABLE_UNITY_BUILD="${ENABLE_UNITY_BUILD:-ON}" \
-DUSE_JEMALLOC="${USE_JEMALLOC}" \
-DUSE_AVX2="${USE_AVX2}" \
-DARM_MARCH="${ARM_MARCH}" \
diff --git a/run-be-ut.sh b/run-be-ut.sh
index 3ff9cd573ee..7eeece8329e 100755
--- a/run-be-ut.sh
+++ b/run-be-ut.sh
@@ -223,6 +223,7 @@ echo "Get params:
PARALLEL -- ${PARALLEL}
CLEAN -- ${CLEAN}
ENABLE_PCH -- ${ENABLE_PCH}
+ ENABLE_UNITY_BUILD -- ${ENABLE_UNITY_BUILD:-ON}
EXTRA_BE_MODULES -- ${EXTRA_BE_MODULES}
"
echo "Build Backend UT"
@@ -335,6 +336,7 @@ cd "${CMAKE_BUILD_DIR}"
${CMAKE_USE_CCACHE_CXX:+${CMAKE_USE_CCACHE_CXX}} \
${CMAKE_USE_CCACHE_C:+${CMAKE_USE_CCACHE_C}} \
-DENABLE_PCH="${ENABLE_PCH}" \
+ -DENABLE_UNITY_BUILD="${ENABLE_UNITY_BUILD:-ON}" \
-DDORIS_JAVA_HOME="${JAVA_HOME}" \
-DBUILD_AZURE="${BUILD_AZURE}" \
"${BE_EXTRA_CMAKE_ARGS[@]}" \
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
