This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d340d568362 [opt](build) Enable unity builds for the Exec and Exprs
targets (#66776)
d340d568362 is described below
commit d340d568362887cb21ac79c7a6f20a58ff16513f
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri Aug 14 21:41:27 2026 +0800
[opt](build) Enable unity builds for the Exec and Exprs targets (#66776)
> Part of the BE build-time optimization series tracked in #66715.
>
> Split out of **https://github.com/apache/doris/pull/66510**, which
carries the whole
> BE build-time batch. #66712 introduced the `ENABLE_UNITY_BUILD` switch
and piloted
> unity builds on three low-risk glue targets. This PR extends unity to
**Exec and
> Exprs** — the two heaviest targets in the BE build and the largest
single source of
> the unity line's win. The remaining targets follow in one more PR.
### What problem does this PR solve?
Related PR: #66510, #66712
Problem Summary:
Same mechanism as #66712: most of the cold-build cost of glue-heavy
targets is
**re-parsing the shared header closure once per small `.cpp`**, and
CMake's
`UNITY_BUILD` makes a batch pay that parse once. What is new here is the
scale —
Exec (174 TUs, `libExec.a` 2450 MB) and Exprs (287 TUs, `libExprs.a`
2721 MB) are
the two heaviest targets in the tree, and their glue shares the heaviest
closures
(`operator.h`/`dependency.h` for Exec; the vexpr/factory closure for
Exprs).
The four commits:
1. **Deduplicate exec file-scope names that clash under unity** (no
behavior
change): `file_scanner.cpp`/`file_scanner_v2.cpp` both defined the
Iceberg
delete content codes and `is_iceberg_position_deletes_sys_table()` in
anonymous
namespaces — the shared trio moves to `iceberg_scan_semantics.h`
(`file_scanner_v2_test.cpp` carried a third copy, kept file-local by
#66615
because this header move had not landed yet; it now uses the header
too).
`vtablet_writer.cpp`/`vtablet_writer_v2.cpp` both defined a file-scope
`CLOSE_WAIT_EVENT_FALLBACK_MS` — scoped into `IndexChannel` and
`VTabletWriterV2`; v2's file-scope `on_partitions_created()` trampoline
renamed
`on_partitions_created_v2` (the two functions cast to different writer
types).
`exchange_sink_operator.cpp`'s namespace-scope `timer_name` renamed
`wait_for_dependency_timer_name` (shadowed unity siblings' locals under
`-Wshadow -Werror`).
2. **Unity for the whole Exec target**: 167 of 174 TUs join 14 unity
batches of
≤12 sources, gated on `ENABLE_UNITY_BUILD` like the pilot targets. Opted
out:
five files whose file-scope macros must not leak into siblings, plus the
two
heaviest template-instantiation TUs (`operator.cpp`,
`hashjoin_build_sink.cpp`)
which would dominate any batch they join; `scan_operator.cpp` is both.
3. **Three latent defects the Exprs conversion surfaced** (stand on
their own):
`dictionary_factory.h` had **no include guard at all** — any TU reaching
it
through two include paths fails with a class redefinition, and under
unity the
clang error recovery poisoned unrelated batch members with spurious
`-Warray-bounds` diagnostics. Now `#pragma once`. And
`function_dict_get_many.cpp` had copy-pasted the `DictGetState` struct
from
`function_dict_get.cpp` at namespace scope — renamed `DictGetManyState`
so the
two TUs can share a batch. And `function_variant_element_v2.cpp` kept
`OwnedPathSegment` in an anonymous namespace while using it as a field
of the
externally-visible `ResolvedVariantElementV2Path::Impl` — gcc's
`-Wsubobject-linkage` (`-Werror`) rejects exactly that once the file is
`#include`d into a unity batch instead of being the main file of its TU
(clang has no such warning); the struct moves to namespace scope.
4. **Unity for the Exprs glue**: 246 of 287 TUs join 31 unity batches of
≤8
sources, same switch. Opted out: the flex/bison/gperf generated tables,
seven
macro-leaking files, the 30 heavy template-instantiation TUs (>15 s wall
or
>2.2 GB RSS in the compile bench: the min_max/collect/topn/percentile
aggregate
family, `in.cpp`, `multiply.cpp`, `function_array_aggregation.cpp`, …)
whose
per-file codegen would only stack into jumbo poles — and three files
that
tests compile a second time by `#include`ing the `.cpp`
(`function_variant_element.cpp`, `uuid.cpp`,
`function_jsonb_transform.cpp`),
see the verification section.
### Measured results
All numbers from the development branch this series is split from, macOS
arm64 +
clang 20, `-j14`, `ENABLE_PCH=ON`, cold builds, back-to-back A/B. The
baseline is
the #66712 state of that branch (10m16s), so the two waves compose with
the pilot:
| wave | build phase wall | target slot time | archive size |
|---|---|---|---|
| Exec unity | 10m16s → **8m57s (-79.2 s / -12.9%)** | 1533 s → 579 s
(2.6×) | `libExec.a` 2450 MB → 641 MB |
| Exprs unity | 8m57s → **7m57s (-60.0 s / -11.2%)** | 2147 s → 1333 s |
`libExprs.a` 2721 MB → 1392 MB |
Jumbo-TU envelope: the largest Exec unity TU compiles in 32 s / 2.6 GB
RSS
standalone, the largest Exprs one in 25 s / 1.95 GB — both below the
largest
existing individual TU in the tree (3.9 GB), so `-jN` memory envelopes
are
unchanged.
### Risk and verification
- **Unity changes TU grouping only.** The code commits riding along are
hygiene:
constants deduplicated with identical values, one constant scoped into
its class,
two renames, one `#pragma once`. No logic change.
- **Archive symbol parity** (checked on the development branch): Exec
keeps all
external defined symbols — three weak linkonce_odr template
instantiations dedup
away, which is the point of unity, not a loss. Exprs likewise (one weak
instantiation dedups; the `DictGetManyState` rename carries its
`shared_ptr`
machinery under the new name).
- **This exact branch, rebased onto current master, full BE build from
scratch**
(macOS arm64, clang 20, `ENABLE_PCH=ON`, `ENABLE_UNITY_BUILD=ON`):
**7981/7981 ninja edges, zero failures, `doris_be` links (325 MB).**
Exec
produces exactly 14 unity TUs and Exprs exactly 31, as advertised. This
includes
`pipeline/rec_cte_shared_state.cpp`, added upstream after the waves were
measured — it lands inside an Exec unity batch via the existing
`GLOB_RECURSE`
with zero CMakeLists edits, which is the intended maintenance story.
- **The first CI round of this PR did its job and caught two issues;
both are
fixed in the current revision.**
1. *BE UT lane, duplicate symbols at link*: three test files compile a
src
`.cpp` a second time by `#include`ing it
(`function_variant_element_test`,
`function_uuid_test`, `function_json_object_flatten_test`). Pre-unity
this
linked only by archive-member selectivity: the test object defines the
symbols first and the library member is never pulled. A unity batch,
however, is pulled in for its *siblings* and brings a second strong
definition. The three `#include`d files
(`function_variant_element.cpp`, `uuid.cpp`,
`function_jsonb_transform.cpp`)
are now `SKIP_UNITY_BUILD_INCLUSION` — individual archive members
restore
exactly the shadowing semantics master links with today.
2. *Performance lane (the one gcc lane), `-Werror=subobject-linkage`*:
`function_variant_element_v2.cpp` held `OwnedPathSegment` in an
anonymous
namespace as a field type of the externally-visible `...Path::Impl`. gcc
only raises `-Wsubobject-linkage` when the definition sits in an
`#include`d file — which is what unity turns a `.cpp` into; clang has no
such warning, so every local build was green. The struct moves to
namespace scope (name unique to the TU); fixed at the source rather than
SKIPped.
- **The OFF path, checked on the same tree**: reconfiguring with
`ENABLE_UNITY_BUILD=OFF` drops **all 51** `unity_*.cxx` entries from
`compile_commands.json` (Exec 14, Exprs 31, the #66712 pilots 6) and the
TU
count goes 8464 → 9042 — the batches return to exactly their 629 member
files.
Reconfiguring back ON restores exactly the same 51 batches. The switch
semantics themselves (including winning over a stale `CMAKE_UNITY_BUILD`
cache) were established in #66712.
- **These two targets have been building as unity TUs on the development
branch
since 2026-08-08**, through repeated full-tree builds and the BE UT
builds that
verified #66672 (the UT binaries link against these same target
libraries).
- The `file_scanner_v2_test.cpp` hunk was compile-verified standalone
against this
branch (`-fsyntax-only` with the test TU's full include closure).
### Proactive disclosure
- **Cross-platform is the blind spot, closed by this PR's own CI** —
every local
build and measurement above is macOS arm64 + clang 20. With
`ENABLE_UNITY_BUILD`
defaulting ON since #66712, the Linux compile lanes and every regression
pipeline
in this PR's CI run against unity Exec/Exprs — that is the validation,
and the
first round proved it works: the BE UT and gcc lanes each caught one
real
unity interaction (detailed above), fixed in this revision. Escape
hatches, in order:
per-user `ENABLE_UNITY_BUILD=OFF`, per-file
`SKIP_UNITY_BUILD_INCLUSION`, or a
one-line default flip.
- **The incremental-rebuild trade-off is real**: touching one `.cpp`
inside a
batch recompiles the whole batch (≤12 sources for Exec, ≤8 for Exprs; a
batch
compiles in ~25–32 s). This is why the heaviest, most-edited TUs
(`operator.cpp`, `hashjoin_build_sink.cpp`, the aggregate families,
`in.cpp`,
`multiply.cpp`, …) are deliberately SKIPped and keep per-file
granularity, and
`ENABLE_UNITY_BUILD=OFF` restores it everywhere.
- **The SKIP lists are coverage policy, not leftovers**: 7 Exec + 38
Exprs files
stay individual on purpose — generated parsers (flex/bison/gperf), files
whose
file-scope macros would leak into siblings, and the heavy codegen TUs
where
merging saves no closure parse worth the jumbo-TU cost. A future file
whose
file-scope symbols clash inside a unity TU opts out the same one-line
way.
- Unity covers the *glue* of these targets, not the codegen-heavy
families — the
30 heavy Exprs SKIPs mean the headline per-target ratios (2.6× Exec slot
time)
are earned on the batched part; the SKIPped monsters keep their cost and
their
per-file granularity.
---
be/src/exec/CMakeLists.txt | 18 +++++++
be/src/exec/operator/exchange_sink_operator.cpp | 15 +++---
be/src/exec/scan/file_scanner.cpp | 15 +-----
be/src/exec/scan/file_scanner_v2.cpp | 12 -----
be/src/exec/sink/writer/vtablet_writer.cpp | 2 -
be/src/exec/sink/writer/vtablet_writer.h | 2 +
be/src/exec/sink/writer/vtablet_writer_v2.cpp | 6 +--
be/src/exec/sink/writer/vtablet_writer_v2.h | 2 +
be/src/exprs/CMakeLists.txt | 63 ++++++++++++++++++++++
be/src/exprs/function/dictionary_factory.h | 2 +
be/src/exprs/function/function_dict_get_many.cpp | 6 +--
.../exprs/function/function_variant_element_v2.cpp | 7 ++-
be/src/format/table/iceberg_scan_semantics.h | 13 +++++
be/test/exec/scan/file_scanner_v2_test.cpp | 4 +-
14 files changed, 118 insertions(+), 49 deletions(-)
diff --git a/be/src/exec/CMakeLists.txt b/be/src/exec/CMakeLists.txt
index 10f59e25dbb..7ec05378911 100644
--- a/be/src/exec/CMakeLists.txt
+++ b/be/src/exec/CMakeLists.txt
@@ -38,6 +38,24 @@ add_library(Exec STATIC
pch_reuse(Exec)
+# Unity build for the whole Exec target: the operator/sink/scan glue TUs share
+# one heavy (operator.h/dependency.h) header closure, so batching them cuts the
+# repeated frontend work. Batch 12 bounds jumbo-TU size and memory. Opted out:
+# five files whose file-scope macros must not leak into unity siblings, plus
+# the two heaviest template-instantiation TUs which would dominate any batch
+# they join (scan_operator.cpp is both).
+set(EXEC_UNITY_SKIP
+ ${CMAKE_CURRENT_SOURCE_DIR}/common/arrow_column_to_doris_column.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/operator/hashjoin_build_sink.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/operator/join_build_sink_operator.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/operator/operator.cpp
+
${CMAKE_CURRENT_SOURCE_DIR}/operator/partitioned_aggregation_sink_operator.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/operator/scan_operator.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/sink/writer/vfile_result_writer.cpp)
+set_source_files_properties(${EXEC_UNITY_SKIP} PROPERTIES
SKIP_UNITY_BUILD_INCLUSION ON)
+set_target_properties(Exec PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 12)
+
if (${BUILD_TASK_EXECUTOR_SIMULATOR} STREQUAL "ON")
file(GLOB_RECURSE SIMULATOR_FILES CONFIGURE_DEPENDS
"${CMAKE_CURRENT_SOURCE_DIR}/scan/task_executor/tools/simulator/*.cpp"
diff --git a/be/src/exec/operator/exchange_sink_operator.cpp
b/be/src/exec/operator/exchange_sink_operator.cpp
index 2011a44d2ca..3fe4201ad83 100644
--- a/be/src/exec/operator/exchange_sink_operator.cpp
+++ b/be/src/exec/operator/exchange_sink_operator.cpp
@@ -49,7 +49,7 @@ bool ExchangeSinkLocalState::transfer_large_data_by_brpc()
const {
return _parent->cast<ExchangeSinkOperatorX>()._transfer_large_data_by_brpc;
}
-static const std::string timer_name = "WaitForDependencyTime";
+static const std::string wait_for_dependency_timer_name =
"WaitForDependencyTime";
Status ExchangeSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo&
info) {
RETURN_IF_ERROR(Base::init(state, info));
@@ -81,17 +81,18 @@ Status ExchangeSinkLocalState::init(RuntimeState* state,
LocalSinkStateInfo& inf
"");
_merge_block_timer = ADD_TIMER(custom_profile(), "MergeBlockTime");
_local_bytes_send_counter = ADD_COUNTER(custom_profile(),
"LocalBytesSent", TUnit::BYTES);
- _wait_for_dependency_timer = ADD_TIMER_WITH_LEVEL(common_profile(),
timer_name, 1);
- _wait_queue_timer =
- ADD_CHILD_TIMER_WITH_LEVEL(common_profile(),
"WaitForRpcBufferQueue", timer_name, 1);
+ _wait_for_dependency_timer =
+ ADD_TIMER_WITH_LEVEL(common_profile(),
wait_for_dependency_timer_name, 1);
+ _wait_queue_timer = ADD_CHILD_TIMER_WITH_LEVEL(common_profile(),
"WaitForRpcBufferQueue",
+
wait_for_dependency_timer_name, 1);
_create_channels();
// Make sure brpc stub is ready before execution.
for (auto& channel : channels) {
RETURN_IF_ERROR(channel->init(state));
}
- _wait_broadcast_buffer_timer =
- ADD_CHILD_TIMER(common_profile(), "WaitForBroadcastBuffer",
timer_name);
+ _wait_broadcast_buffer_timer = ADD_CHILD_TIMER(common_profile(),
"WaitForBroadcastBuffer",
+
wait_for_dependency_timer_name);
auto& p = _parent->cast<ExchangeSinkOperatorX>();
_part_type = p._part_type;
@@ -262,7 +263,7 @@ Status ExchangeSinkLocalState::open(RuntimeState* state) {
_wait_channel_timer.push_back(common_profile()->add_nonzero_counter(
fmt::format("WaitForLocalExchangeBuffer{}",
_local_channels_dependency.size()),
- TUnit ::TIME_NS, timer_name, 1));
+ TUnit ::TIME_NS, wait_for_dependency_timer_name,
1));
}
}
}
diff --git a/be/src/exec/scan/file_scanner.cpp
b/be/src/exec/scan/file_scanner.cpp
index 6bdf487f204..b262cc6c900 100644
--- a/be/src/exec/scan/file_scanner.cpp
+++ b/be/src/exec/scan/file_scanner.cpp
@@ -71,6 +71,7 @@
#include "format/table/hudi_reader.h"
#include "format/table/iceberg_position_delete_sys_table_reader.h"
#include "format/table/iceberg_reader.h"
+#include "format/table/iceberg_scan_semantics.h"
#include "format/table/iceberg_sys_table_jni_reader.h"
#include "format/table/jdbc_jni_reader.h"
#include "format/table/max_compute_jni_reader.h"
@@ -100,20 +101,6 @@ class ShardedKVCache;
namespace doris {
using namespace ErrorCode;
-namespace {
-constexpr int kIcebergPositionDeleteContent = 1;
-constexpr int kIcebergDeletionVectorContent = 3;
-
-bool is_iceberg_position_deletes_sys_table(const TFileRangeDesc& range) {
- return range.__isset.table_format_params &&
- range.table_format_params.table_format_type == "iceberg" &&
- range.table_format_params.__isset.iceberg_params &&
- range.table_format_params.iceberg_params.__isset.content &&
- (range.table_format_params.iceberg_params.content ==
kIcebergPositionDeleteContent ||
- range.table_format_params.iceberg_params.content ==
kIcebergDeletionVectorContent);
-}
-} // namespace
-
const std::string FileScanner::FileReadBytesProfile = "FileReadBytes";
const std::string FileScanner::FileReadTimeProfile = "FileReadTime";
diff --git a/be/src/exec/scan/file_scanner_v2.cpp
b/be/src/exec/scan/file_scanner_v2.cpp
index 420bbd62a42..23f55c421fc 100644
--- a/be/src/exec/scan/file_scanner_v2.cpp
+++ b/be/src/exec/scan/file_scanner_v2.cpp
@@ -76,9 +76,6 @@
namespace doris {
namespace {
-constexpr int kIcebergPositionDeleteContent = 1;
-constexpr int kIcebergDeletionVectorContent = 3;
-
std::string table_format_name(const TFileRangeDesc& range) {
return range.__isset.table_format_params ?
range.table_format_params.table_format_type
: "NotSet";
@@ -136,15 +133,6 @@ bool is_supported_jni_table_format(const TFileRangeDesc&
range) {
table_format == "max_compute" || table_format == "trino_connector";
}
-bool is_iceberg_position_deletes_sys_table(const TFileRangeDesc& range) {
- return range.__isset.table_format_params &&
- range.table_format_params.table_format_type == "iceberg" &&
- range.table_format_params.__isset.iceberg_params &&
- range.table_format_params.iceberg_params.__isset.content &&
- (range.table_format_params.iceberg_params.content ==
kIcebergPositionDeleteContent ||
- range.table_format_params.iceberg_params.content ==
kIcebergDeletionVectorContent);
-}
-
bool is_csv_format(TFileFormatType::type format_type) {
switch (format_type) {
case TFileFormatType::FORMAT_CSV_PLAIN:
diff --git a/be/src/exec/sink/writer/vtablet_writer.cpp
b/be/src/exec/sink/writer/vtablet_writer.cpp
index dcdfe55136f..e1a3a29914d 100644
--- a/be/src/exec/sink/writer/vtablet_writer.cpp
+++ b/be/src/exec/sink/writer/vtablet_writer.cpp
@@ -127,8 +127,6 @@ static const std::vector<int32_t>&
adaptive_local_bucket_seqs(const VOlapTablePa
return partition.local_bucket_seqs;
}
-static constexpr int64_t CLOSE_WAIT_EVENT_FALLBACK_MS = 1000;
-
Status IndexChannel::init(RuntimeState* state, const
std::vector<TTabletWithPartition>& tablets,
bool incremental) {
SCOPED_CONSUME_MEM_TRACKER(_index_channel_tracker.get());
diff --git a/be/src/exec/sink/writer/vtablet_writer.h
b/be/src/exec/sink/writer/vtablet_writer.h
index 6b9b8c3d787..1605faba306 100644
--- a/be/src/exec/sink/writer/vtablet_writer.h
+++ b/be/src/exec/sink/writer/vtablet_writer.h
@@ -583,6 +583,8 @@ private:
friend class VTabletWriter;
friend class VRowDistribution;
+ static constexpr int64_t CLOSE_WAIT_EVENT_FALLBACK_MS = 1000;
+
int _max_failed_replicas(int64_t tablet_id);
int _load_required_replicas_num(int64_t tablet_id);
diff --git a/be/src/exec/sink/writer/vtablet_writer_v2.cpp
b/be/src/exec/sink/writer/vtablet_writer_v2.cpp
index 1b790f966ae..8a5fe58500a 100644
--- a/be/src/exec/sink/writer/vtablet_writer_v2.cpp
+++ b/be/src/exec/sink/writer/vtablet_writer_v2.cpp
@@ -60,8 +60,6 @@ namespace doris {
extern bvar::Adder<int64_t> g_sink_load_back_pressure_version_time_ms;
-static constexpr int64_t CLOSE_WAIT_EVENT_FALLBACK_MS = 1000;
-
VTabletWriterV2::VTabletWriterV2(const TDataSink& t_sink, const
VExprContextSPtrs& output_exprs,
std::shared_ptr<Dependency> dep,
std::shared_ptr<Dependency> fin_dep)
@@ -85,7 +83,7 @@ Status
VTabletWriterV2::on_partitions_created(TCreatePartitionResult* result) {
return Status::OK();
}
-static Status on_partitions_created(void* writer, TCreatePartitionResult*
result) {
+static Status on_partitions_created_v2(void* writer, TCreatePartitionResult*
result) {
return
static_cast<VTabletWriterV2*>(writer)->on_partitions_created(result);
}
@@ -139,7 +137,7 @@ Status VTabletWriterV2::_init_row_distribution() {
.vec_output_expr_ctxs = &_vec_output_expr_ctxs,
.schema = _schema,
.caller = (void*)this,
- .create_partition_callback =
&::doris::on_partitions_created});
+ .create_partition_callback =
&::doris::on_partitions_created_v2});
return _row_distribution.open(_output_row_desc);
}
diff --git a/be/src/exec/sink/writer/vtablet_writer_v2.h
b/be/src/exec/sink/writer/vtablet_writer_v2.h
index 19ae8854bf8..a3d86a613bb 100644
--- a/be/src/exec/sink/writer/vtablet_writer_v2.h
+++ b/be/src/exec/sink/writer/vtablet_writer_v2.h
@@ -118,6 +118,8 @@ private:
std::shared_ptr<LoadStreamMap> load_stream_map);
private:
+ static constexpr int64_t CLOSE_WAIT_EVENT_FALLBACK_MS = 1000;
+
Status _init_row_distribution();
Status _init(RuntimeState* state, RuntimeProfile* profile);
diff --git a/be/src/exprs/CMakeLists.txt b/be/src/exprs/CMakeLists.txt
index 62cc03b4c5c..b7bf6916a90 100644
--- a/be/src/exprs/CMakeLists.txt
+++ b/be/src/exprs/CMakeLists.txt
@@ -38,6 +38,69 @@ target_link_libraries(Exprs
pch_reuse(Exprs)
+# Unity build for the Exprs glue: ~250 registration/wrapper TUs share the
+# vexpr/factory header closure. Batch 8 bounds jumbo-TU size and memory.
+# Opted out and kept as individual TUs:
+# - the flex/bison outputs and tldLookup.generated.cpp (generated tables and
+# macros would clash),
+# - files whose file-scope macros must not leak into unity siblings
+# (fmod_fast, function_bitmap_variadic,
function_date_or_datetime_computation,
+# function_datetime_floor_ceil, function_string_misc, json_functions,
+# math_functions),
+# - the heavy template-instantiation TUs (>15s wall or >2.2GB RSS in the
+# compile bench: the min_max/collect/topn/percentile aggregate family,
+# in.cpp, multiply.cpp, round.cpp, function_array_aggregation.cpp, ...);
+# their codegen is per-file and would only stack into jumbo poles,
+# - files that tests compile a second time by #including the .cpp
+# (function_variant_element, uuid, function_jsonb_transform): the test
+# object must shadow a never-pulled archive member, but a unity batch is
+# pulled in for its siblings and the linker sees a duplicate definition.
+set(EXPRS_UNITY_SKIP
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_array_agg.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_avg.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_bit.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_collect_limit.cpp
+
${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_collect_no_limit.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_distinct.cpp
+
${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_group_array_intersect.cpp
+
${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_group_array_union.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_histogram.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_map_combinator.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_min_max_any.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_min_max_by.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_min_max_max.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_min_max_min.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_percentile.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_reader.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_sum.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_topn_array.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_topn_weighted.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/aggregate/aggregate_function_uniq.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/ai/ai_functions.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/array/function_array_aggregation.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/cast/function_cast_decimal.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/fmod_fast.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/function_bitmap_variadic.cpp
+
${CMAKE_CURRENT_SOURCE_DIR}/function/function_date_or_datetime_computation.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/function_datetime_floor_ceil.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/function_jsonb_transform.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/function_string_misc.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/function_variant_element.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/in.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/minus.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/multiply.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/plus.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/round.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/url/tldLookup.generated.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/function/uuid.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/json_functions.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/math_functions.cpp
+ ${GENSRC_DIR}/geo/wkt_lex.l.cpp
+ ${GENSRC_DIR}/geo/wkt_yacc.y.cpp)
+set_source_files_properties(${EXPRS_UNITY_SKIP} PROPERTIES
SKIP_UNITY_BUILD_INCLUSION ON)
+set_target_properties(Exprs PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 8)
+
# Flex/Bison generation rules for geo WKT parsing
add_custom_command(
OUTPUT ${GENSRC_DIR}/geo/wkt_lex.l.cpp ${GENSRC_DIR}/geo/wkt_lex.l.h
diff --git a/be/src/exprs/function/dictionary_factory.h
b/be/src/exprs/function/dictionary_factory.h
index bab27987bf3..5d0630d5974 100644
--- a/be/src/exprs/function/dictionary_factory.h
+++ b/be/src/exprs/function/dictionary_factory.h
@@ -15,6 +15,8 @@
// specific language governing permissions and limitations
// under the License.
+#pragma once
+
#include <gen_cpp/BackendService_types.h>
#include <mutex>
diff --git a/be/src/exprs/function/function_dict_get_many.cpp
b/be/src/exprs/function/function_dict_get_many.cpp
index 93e1f9fae5d..9d06fb5b6db 100644
--- a/be/src/exprs/function/function_dict_get_many.cpp
+++ b/be/src/exprs/function/function_dict_get_many.cpp
@@ -35,7 +35,7 @@
namespace doris {
-struct DictGetState {
+struct DictGetManyState {
std::shared_ptr<const IDictionary> dict;
///TODO:
// 1. we do not need to check dict every time(shoud only check in open)
@@ -65,7 +65,7 @@ public:
if (scope == FunctionContext::THREAD_LOCAL) {
return Status::OK();
}
- std::shared_ptr<DictGetState> state = std::make_shared<DictGetState>();
+ std::shared_ptr<DictGetManyState> state =
std::make_shared<DictGetManyState>();
context->set_function_state(scope, state);
DCHECK(context->get_num_args() == 3);
auto dict_fn = context->dict_function();
@@ -87,7 +87,7 @@ public:
Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
uint32_t result, size_t input_rows_count) const
override {
- auto* dict_state = reinterpret_cast<DictGetState*>(
+ auto* dict_state = reinterpret_cast<DictGetManyState*>(
context->get_function_state(FunctionContext::FRAGMENT_LOCAL));
if (!dict_state) {
return Status::RuntimeError("funciton context for function '{}'
must have dict_state;",
diff --git a/be/src/exprs/function/function_variant_element_v2.cpp
b/be/src/exprs/function/function_variant_element_v2.cpp
index 644a033f4af..9bfd5dea113 100644
--- a/be/src/exprs/function/function_variant_element_v2.cpp
+++ b/be/src/exprs/function/function_variant_element_v2.cpp
@@ -32,16 +32,15 @@
namespace doris {
-namespace {
-
+// Not in the anonymous namespace: as a field of the externally-visible Impl,
an
+// internal-linkage type trips gcc's -Wsubobject-linkage once this file is
+// #included into a unity batch instead of being the main file of its TU.
struct OwnedPathSegment {
VariantElementV2PathSegment::Kind kind;
PaddedPODArray<char> key;
int64_t index = 0;
};
-} // namespace
-
struct ResolvedVariantElementV2Path::Impl {
DorisVector<OwnedPathSegment> segments;
};
diff --git a/be/src/format/table/iceberg_scan_semantics.h
b/be/src/format/table/iceberg_scan_semantics.h
index c708a3d6222..274da536649 100644
--- a/be/src/format/table/iceberg_scan_semantics.h
+++ b/be/src/format/table/iceberg_scan_semantics.h
@@ -36,4 +36,17 @@ inline bool supports_iceberg_scan_semantics_v2(const
TFileScanRangeParams* param
params->iceberg_scan_semantics_version >=
ICEBERG_SCAN_SEMANTICS_VERSION_2;
}
+// Iceberg manifest-entry content codes (spec: 1 = position deletes, 3 =
deletion vectors).
+inline constexpr int kIcebergPositionDeleteContent = 1;
+inline constexpr int kIcebergDeletionVectorContent = 3;
+
+inline bool is_iceberg_position_deletes_sys_table(const TFileRangeDesc& range)
{
+ return range.__isset.table_format_params &&
+ range.table_format_params.table_format_type == "iceberg" &&
+ range.table_format_params.__isset.iceberg_params &&
+ range.table_format_params.iceberg_params.__isset.content &&
+ (range.table_format_params.iceberg_params.content ==
kIcebergPositionDeleteContent ||
+ range.table_format_params.iceberg_params.content ==
kIcebergDeletionVectorContent);
+}
+
} // namespace doris
diff --git a/be/test/exec/scan/file_scanner_v2_test.cpp
b/be/test/exec/scan/file_scanner_v2_test.cpp
index 353c08043ad..328b0e9c33e 100644
--- a/be/test/exec/scan/file_scanner_v2_test.cpp
+++ b/be/test/exec/scan/file_scanner_v2_test.cpp
@@ -46,15 +46,13 @@
#include "exprs/vdirect_in_predicate.h"
#include "exprs/vliteral.h"
#include "exprs/vslot_ref.h"
+#include "format/table/iceberg_scan_semantics.h"
#include "format_v2/expr/cast.h"
#include "testutil/mock/mock_runtime_state.h"
namespace doris {
namespace {
-constexpr int kIcebergPositionDeleteContent = 1;
-constexpr int kIcebergDeletionVectorContent = 3;
-
TFileRangeDesc range_with_format(std::string table_format,
TFileFormatType::type format_type) {
TFileRangeDesc range;
range.__set_format_type(format_type);
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
