This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b41856ab7e9 [fix](build) Move the macOS toolchain to LLVM 22 so ASAN
works on macOS 26.4+ (#68595)
b41856ab7e9 is described below
commit b41856ab7e9915e3b9ffc76caa96a27baf62aa54
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Sep 30 21:00:34 2026 +0800
[fix](build) Move the macOS toolchain to LLVM 22 so ASAN works on macOS
26.4+ (#68595)
### What problem does this PR solve?
Issue Number: N/A
Related PR: apache/doris-website#4180 (the macOS build guide change that
goes with this PR)
Problem Summary:
**Context.**
On macOS the only entry point for the BE unit tests is `run-be-ut.sh`,
and it defaults to `BUILD_TYPE_UT=ASAN`. Since macOS 26.4 that default
cannot start a single process: every binary linked against llvm.org's
`libclang_rt.asan_osx_dynamic` deadlocks during runtime initialisation,
before `main()`. `env.sh` selects that same toolchain on macOS
(`llvm@20`), so a developer who follows the documented setup gets a
build that hangs rather than one that fails.
How the macOS toolchain gets chosen matters for the rest of this
description:
- **Selection.** `env.sh` writes `custom_env_mac.sh`, which puts `$(brew
--prefix)/opt/<formula>/bin` on `PATH` for every formula in its
`CELLARS` list. Unless `DORIS_CLANG_HOME` is already set, `env.sh` then
takes the first `clang` on `PATH` and derives `CC`/`CXX` from it.
- **Installation.** `CELLARS` only selects; it installs nothing.
Developers install Homebrew formulae by following the macOS build guide,
and each CI workflow installs from its own `brew install` list.
- **Third-party libraries.** `thirdparty/installed` comes from one of
two places:
- the published `doris-thirdparty-prebuilt-darwin-*.tar.xz`, which
`apache/doris-thirdparty`'s `build-target.yml` builds by running
`thirdparty/build-thirdparty.sh` from apache/doris master;
- a local run of that same script. `build.sh` also starts one by itself
when `thirdparty/installed` lacks a library it checks for.
**1. The problem, and what it cost**
*The deadlock.* `sample` on the stalled process shows the whole chain:
```
__malloc_init (libsystem_malloc) <- early libSystem
init calls malloc_default_zone()
`- wrap_malloc_default_zone (asan runtime)
`- AsanInitFromRtl -> AsanInitInternal -> InitializeShadowMemory
`- MemoryRangeIsAvailable -> MemoryMappingLayout::Next ->
get_dyld_hdr()
`- dyld_shared_cache_iterate_text_swift <- macOS 26.4
reimplemented this in Swift
`- _Block_copy -> malloc
`- __sanitizer_mz_malloc (asan's own malloc)
`- AsanInitFromRtl() <- re-enters init
`- StaticSpinMutex::LockSlow <- spins on the
lock it already holds
```
The cause is an OS-side change, not a Doris one. It also does not mean
ASAN is broken on macOS 26: Apple's own clang sanitizer runtime runs
fine on the same host, and only llvm.org's compiler-rt is affected.
Upstream fixed it by weak-importing `_dyld_get_dyld_header` and using
it, when present, instead of walking the shared cache
(llvm/llvm-project#182943, main `2e7d07a`; backport #188913,
release/22.x `7b6514c`). **The fix shipped only in 22.1.8.** 20.1.8 is
the last 20.x release and 21.1.8 the last 21.x, so no version of
`llvm@20` can ever be made to work.
*Why it looks like a hang and not an error.*
- `be/CMakeLists.txt` reaches `storage/index/ann` unconditionally, and
that directory's `cmake-protect` target calls `add_subdirectory()` on
`contrib/openblas`.
- OpenBLAS runs an instrumented `getarch` probe from its **configure**
step (`contrib/openblas/cmake/prebuild.cmake:1513`,
`execute_process(COMMAND .../getarch 0 ...)`), and `execute_process` has
no timeout.
- So every ASAN build stops at `-- Running getarch` and never moves
again. It sits at about 90% CPU and prints no diagnostic.
`run-be-ut.sh` on macOS was therefore unusable at its default setting,
and the failure gave the developer nothing to act on.
*The second obstacle, once the toolchain is bumped.*
- clang 22 added `-Wc2y-extensions` and folds it into `-Wpedantic`.
- `__COUNTER__` only reached the C standard in C2y (WG14 N3457).
`be/src/runtime/runtime_profile.h:73-85` uses it to give two
`SCOPED_TIMER` / `SCOPED_RAW_TIMER` expansions on the same line distinct
names.
- As a result, a clang 22 build of any TU that includes that header
fails under `-Werror`. Three representative unity TUs were enough to hit
it, so the failure is not confined to one module.
*The third obstacle: clang 22 cannot build the third-party libraries.*
The first revision of this PR moved every macOS toolchain reference to
`llvm@22`, including the third-party build. CI failed both macOS
third-party jobs (7m49s, 6m23s) on the first library that exercises a
new clang 22 error:
```
unixODBC 2.3.7 src/SQLBrowseConnectW.c:424:82:
error: incompatible pointer types passing 'SQLSMALLINT *' (aka 'short *')
to parameter of type 'int *' [-Wincompatible-pointer-types]
```
clang 16, clang 20 and Apple clang 21 report this as a warning; clang 22
makes it an error. unixODBC is only the *first* failure: the build
aborts there, so every library after it is untested against clang 22.
The macOS third-party libraries therefore have to stay on `llvm@20`
while the BE moves to `llvm@22`.
Leaving the third-party build alone does not achieve that, because it
sources `env.sh`:
- With `CELLARS` naming `llvm@22`, a machine without `llvm@22` gets a
non-existent directory on `PATH`.
- `command -v clang` then falls through to `/usr/bin/clang` (Apple
clang), which is worse than either explicit choice.
- The job that publishes the prebuilt archive in
`apache/doris-thirdparty` is exactly such a machine: it installs only
`llvm@20` and sets no `DORIS_CLANG_HOME`.
*Two latent defects clang 22 then surfaced.* The bump is not a pure
version change. clang 22's stricter diagnostics stop the build on two
pre-existing bugs, and both are worth fixing on their own merits.
| # | Site | Diagnostic | What is actually wrong |
|---|---|---|---|
| 1 | `be/src/load/group_commit/wal/wal_dirs_info.cpp:98` |
`-Wunused-result` | `LOG(INFO) << "… err: {}", e.what();` — the `,` is
the comma operator, not an argument separator, so the statement is
`(LOG(INFO) << "…{}") , (e.what())`. The `{}` is never substituted, and
`e.what()` is evaluated and then discarded: **the error message has
never been logged**, only the literal `{}`. A repo-wide scan for the
same shape finds exactly this one site (`cloud/src/common/logging.h`
matches only inside macro definitions and is not a bug). |
| 2 | `common/cpp/sync_point.cpp:208,210` | `-Wthread-safety-analysis` |
The function holds `std::unique_lock lock(mutex_)`, then releases and
re-takes the mutex with a raw `mutex_.unlock()` / `mutex_.lock()` pair
around the callback, which bypasses the lock's ownership tracking. If
the callback throws, the re-lock is skipped while `~unique_lock` still
believes it owns the mutex, so the destructor unlocks a mutex that is
not held (UB) and `num_callbacks_running_` is never decremented.
`lock.unlock()` / `lock.lock()` still runs the callback unlocked, as
intended, and keeps the ownership state correct on every path. |
**2. What this PR does, and why it helps**
- **`env.sh`**: the macOS `CELLARS` list moves from `llvm@20` to
`llvm@22`, with a comment recording that 22.1.8 is the minimum and why.
This is the single line that decides which clang a macOS developer
builds the BE with.
- **`thirdparty/build-thirdparty.sh`**: on Darwin, right after sourcing
`env.sh` and next to the existing environment sanitization, the script
unconditionally points `DORIS_CLANG_HOME`, `CC` and `CXX` at `$(brew
--prefix llvm@20)` and puts its `bin/` first on `PATH`. If `llvm@20` is
not installed, it stops with a `brew install llvm@20` hint rather than
falling back to another compiler.
- The override has to come after `env.sh` and has to be unconditional.
`env.sh` sources `custom_env.sh`, which may export `DORIS_CLANG_HOME`
for the BE, and `build.sh` has already exported the BE's `llvm@22`
values before it starts this script.
- Every macOS third-party build runs this script: `build.sh`'s automatic
rebuild, a manual run, the pull request check in `build-thirdparty.yml`,
and the `apache/doris-thirdparty` job that publishes
`doris-thirdparty-prebuilt-darwin-*.tar.xz`. This one place therefore
keeps all of them, and the published archive, on `llvm@20`.
- `.github/workflows/build-thirdparty.yml` is unchanged: its macOS jobs
already install `llvm@20`.
- **`be/CMakeLists.txt`**: add `-Wno-c2y-extensions` for clang 19 and
newer, in its own `add_compile_options` call **after** `-Wpedantic`.
- The position matters: a later `-Wpedantic` turns the group back on.
That is also why passing the flag through `EXTRA_CXX_FLAGS` does not
work: that variable lands near the front of the command line.
- The version gate matters because the `c2y-extensions` group only
exists from clang 19 on. clang rejects an unknown `-Wno-` option like
any other unknown warning option (`-Wunknown-warning-option`, fatal
under `-Werror`), and this file accepts clang 16 and newer.
- For clang 19 and newer the command line is exactly what it was.
- **`.github/workflows/be-ut-mac.yml`**: install `llvm@22` for the BE
and keep `llvm@20` next to it.
- The job builds the BE on top of the downloaded darwin-arm64 prebuilt.
When that archive lacks a library `build.sh` checks for, `build.sh`
rebuilds the third-party libraries from source, and that rebuild now
needs `llvm@20`. The archive falls behind like this whenever master
changes the set of checked libraries before the next archive is
published.
- Both formulae are keg-only, so installing both does not change which
one the BE uses.
- **`be/src/load/group_commit/wal/wal_dirs_info.cpp` and
`common/cpp/sync_point.cpp`**: the two fixes from the table above. Each
is one line, the smallest change that removes the defect rather than
suppressing the warning. No `-Wno-unused-result` /
`-Wno-thread-safety-analysis` is added, because those diagnostics point
at real bugs and should keep firing.
- **apache/doris-website#4180**: the macOS build guide installs
`llvm@22` and `llvm@20` instead of only `llvm@20`. Without it, a
developer who follows the guide after this PR has no `llvm@22`, and the
BE silently builds with Apple clang.
What this PR deliberately does not do is add `llvm@20` to `env.sh`'s
`CELLARS`:
- `CELLARS` installs nothing.
- `build-thirdparty.sh` does not use `PATH` to find `llvm@20`.
- The `CELLARS` loop prepends each entry to `PATH`. An `llvm@20` entry
after `llvm@22` would put the BE back on `llvm@20` and bring the
deadlock back.
**3. The classes, and how they call each other**
```
env.sh CELLARS := llvm@22 (selects a
compiler; installs nothing)
|- generates custom_env_mac.sh: $(brew --prefix)/opt/<cellar>/bin
prepended to PATH
|- DORIS_CLANG_HOME := dirname($(command -v clang))/.. -> CC / CXX /
ASAN_SYMBOLIZER_PATH
'- is sourced by:
|- build.sh, run-be-ut.sh -> BE: llvm@22
'- thirdparty/build-thirdparty.sh:55
'- :100-110 on Darwin: DORIS_CLANG_HOME / CC / CXX := $(brew
--prefix llvm@20)
llvm@20 missing -> exit 1, "brew install llvm@20"
-> third-party
libraries: llvm@20
run by: build.sh (library missing from
thirdparty/installed), manual runs,
build-thirdparty.yml (PR check), doris-thirdparty
build-target.yml (publisher)
brew install lists, i.e. who installs which LLVM:
macOS build guide (apache/doris-website#4180) llvm@22 llvm@20
.github/workflows/be-ut-mac.yml llvm@22 llvm@20
.github/workflows/build-thirdparty.yml llvm@20
apache/doris-thirdparty build-target.yml llvm@20
be/CMakeLists.txt if (COMPILER_CLANG)
|- add_compile_options(-Wpedantic ...) <- enables the c2y
group
'- if (CMAKE_CXX_COMPILER_VERSION >= 19)
add_compile_options(-Wno-c2y-extensions) <- must stay after
the line above
|
be/src/runtime/runtime_profile.h:73-85
'- MACRO_CONCAT(SCOPED_TIMER, __COUNTER__) <- the only
__COUNTER__ use in be/src, be/test
|
be/src/storage/index/ann/cmake-protect/CMakeLists.txt:48
'- add_subdirectory(contrib/openblas)
'- cmake/prebuild.cmake:1513 execute_process(getarch) <- where
the hang surfaced
```
### Release note
None
### Check List (For Author)
- Test: Manual test on macOS 26.5.1 (arm64, dyld-1378), plus CI.
- `clang -fsanitize=address` hello world exits 0 with llvm@22 and still
deadlocks with llvm@20 on the same host.
- `sh run-be-ut.sh` with no environment overrides now selects
`Clang-22.1.8` and clears the `-- Running getarch` point that previously
hung forever: zero FAILED targets, and `doris_be_test` links.
- `sh run-be-ut.sh --run --filter=FormatRoundTest.*` starts the ASAN
binary and passes 8 tests.
- `-Wno-c2y-extensions` gate, tested on a `__COUNTER__` TU under
`-Wpedantic -Werror`: clang 16.0.6 rejects the flag (`unknown warning
option`), which is why it is gated; clang 20.1.8 accepts it; clang
22.1.8 fails without it and passes with it.
- `build-thirdparty.sh` compiler override, checked under `bash -x` with
both `custom_env.sh` and `build.sh` exporting the `llvm@22` values: the
effective `CC`/`CXX` and the first clang on `PATH` are llvm@20. With
`llvm@20` missing, the script exits 1 with the brew hint before building
anything.
- `be-ut-mac.yml`: the "Build BE" step script passes `bash -n` on bash
3.2 and 5.3, and its `cellars` array holds both `llvm@22` and `llvm@20`.
The workflow runs only on pushes to master and on schedule, so this PR
cannot run it.
- `build-support/check-build-hygiene.sh` passes.
- CI: `Build Third Party Libraries (macOS)`, `(macOS-arm64)` and
`(Linux)` pass at `a955357a30f` (run 36542107899), which already
contains the `build-thirdparty.sh` override. The first revision, which
moved these builds to llvm@22, failed both macOS jobs on unixODBC.
- Regression test / unit test: N/A (toolchain, flag and workflow
change).
- Behavior changed: Yes, for macOS builds only.
- The BE builds with llvm@22 instead of llvm@20.
- The third-party libraries, including the published prebuilt, stay on
llvm@20. Building them from source now requires llvm@20 installed next
to llvm@22; without it, the build stops with a `brew install llvm@20`
hint.
- Existing macOS developers need to run `brew install llvm@22`. Without
it, `env.sh` falls back to Apple clang in `/usr/bin` for newly
configured build directories.
- Does this need documentation: Yes, apache/doris-website#4180.
---------
Co-authored-by: Claude Code <[email protected]>
---
.github/workflows/be-ut-mac.yml | 5 +++++
be/CMakeLists.txt | 12 ++++++++++++
be/src/load/group_commit/wal/wal_dirs_info.cpp | 2 +-
common/cpp/sync_point.cpp | 6 +++---
env.sh | 8 +++++++-
thirdparty/build-thirdparty.sh | 22 ++++++++++++++++++++++
6 files changed, 50 insertions(+), 5 deletions(-)
diff --git a/.github/workflows/be-ut-mac.yml b/.github/workflows/be-ut-mac.yml
index 201e8d0a640..686335c26c7 100644
--- a/.github/workflows/be-ut-mac.yml
+++ b/.github/workflows/be-ut-mac.yml
@@ -70,6 +70,11 @@ jobs:
'openjdk@11'
'maven'
'node'
+ # llvm@22 builds the BE (env.sh selects it). llvm@20 builds the
+ # third-party libraries: build.sh rebuilds them from source when
the
+ # prebuilt archive below is missing one it checks for, and
+ # thirdparty/build-thirdparty.sh refuses to run on macOS without
llvm@20.
+ 'llvm@22'
'llvm@20'
'libomp'
)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 02da8f5dc2e..874426d1dc9 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -468,6 +468,18 @@ if (COMPILER_CLANG)
# -Werror failure. The diagnostic carries no
correctness
# signal, so drop it.
-Wno-pass-failed)
+ # clang 22 reports __COUNTER__ under -Wpedantic as a C2y extension
+ # (-Wc2y-extensions). runtime_profile.h expands it to give two
SCOPED_TIMER /
+ # SCOPED_RAW_TIMER uses on the same line distinct names -- a long-standing
+ # clang/gcc/MSVC extension that only reached the standard in C2y, and the
+ # diagnostic carries no correctness signal. The c2y-extensions group only
exists
+ # from clang 19 on, and clang rejects an unknown -Wno- option like any
other
+ # unknown warning option (-Wunknown-warning-option, fatal under -Werror),
so
+ # clang 16-18 must not see the flag. It must also stay after -Wpedantic
above:
+ # a -Wpedantic later on the command line turns the group back on.
+ if (CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL "19")
+ add_compile_options(-Wno-c2y-extensions)
+ endif()
if (USE_LIBCPP)
add_compile_options($<$<COMPILE_LANGUAGE:CXX>:-stdlib=libc++>)
add_definitions(-DUSE_LIBCPP)
diff --git a/be/src/load/group_commit/wal/wal_dirs_info.cpp
b/be/src/load/group_commit/wal/wal_dirs_info.cpp
index 171f17b43e6..0b0880a1348 100644
--- a/be/src/load/group_commit/wal/wal_dirs_info.cpp
+++ b/be/src/load/group_commit/wal/wal_dirs_info.cpp
@@ -95,7 +95,7 @@ Status WalDirInfo::update_wal_dir_used(size_t used) {
try {
RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir,
&wal_dir_size));
} catch (const std::exception& e) {
- LOG(INFO) << "failed to update wal dir used info, err: {}",
e.what();
+ LOG(INFO) << "failed to update wal dir used info, err: " <<
e.what();
return Status::OK();
}
set_used(wal_dir_size);
diff --git a/common/cpp/sync_point.cpp b/common/cpp/sync_point.cpp
index 209d5dcf0d5..ae9fc35a7c1 100644
--- a/common/cpp/sync_point.cpp
+++ b/common/cpp/sync_point.cpp
@@ -204,10 +204,10 @@ void SyncPoint::Data::process(const std::string& point,
std::vector<std::any>&&
auto callback_pair = callbacks_.find(point);
if (callback_pair != callbacks_.end()) {
num_callbacks_running_++;
- auto callback = callback_pair->second;
- mutex_.unlock();
+ auto callback = callback_pair->second;
+ lock.unlock();
callback(std::move(cb_arg));
- mutex_.lock();
+ lock.lock();
num_callbacks_running_--;
}
cleared_points_.insert(point);
diff --git a/env.sh b/env.sh
index 510b52a224b..2726225bf67 100755
--- a/env.sh
+++ b/env.sh
@@ -69,7 +69,13 @@ CELLARS=(
wget
pcre
maven
- llvm@20
+ # 22.1.8 is the first compiler-rt whose ASAN runtime survives macOS 26.4+:
+ # dyld reimplemented dyld_shared_cache_iterate_text in Swift there and its
+ # _Block_copy allocation re-enters ASAN init, so every -fsanitize=address
+ # binary built with llvm@20 deadlocks before main()
(llvm/llvm-project#182943,
+ # fixed in 22.1.8; no 20.x or 21.x backport exists). That is the BE's
compiler:
+ # thirdparty/build-thirdparty.sh keeps the third-party libraries on
llvm@20.
+ llvm@22
m4
)
for cellar in "\${CELLARS[@]}"; do
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index 9ac1e01d02d..eee801bd91f 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -87,6 +87,28 @@ unset CMAKE_TOOLCHAIN_FILE \
VCPKG_DEFAULT_TRIPLET \
CONDA_PREFIX
+# The macOS third-party libraries stay on LLVM 20 while env.sh gives the BE
LLVM 22 (the
+# first compiler-rt whose ASAN runtime survives macOS 26.4+). clang 22 turns
+# -Wincompatible-pointer-types into an error and stops at unixODBC 2.3.7
+# (SQLBrowseConnectW.c passes SQLSMALLINT* where int* is expected), and no
package after it
+# has been built with clang 22. Every macOS third-party build comes through
here - the
+# rebuild build.sh starts on its own, a manual run, the pull request check and
the
+# apache/doris-thirdparty job that publishes
doris-thirdparty-prebuilt-darwin-*.tar.xz - so
+# this is the one place that decides their compiler. Like the unset above, it
has to come
+# after env.sh: custom_env.sh may point DORIS_CLANG_HOME at another LLVM for
the BE. CC/CXX
+# carry the compiler and PATH the rest of the LLVM tools, as when env.sh named
llvm@20.
+if [[ "$(uname -s)" == 'Darwin' ]]; then
+ DORIS_CLANG_HOME="$(brew --prefix llvm@20)"
+ if [[ ! -x "${DORIS_CLANG_HOME}/bin/clang" ]]; then
+ echo "The macOS third-party build needs LLVM 20
(${DORIS_CLANG_HOME}/bin/clang is missing): brew install llvm@20" >&2
+ exit 1
+ fi
+ export DORIS_CLANG_HOME
+ export CC="${DORIS_CLANG_HOME}/bin/clang"
+ export CXX="${DORIS_CLANG_HOME}/bin/clang++"
+ export PATH="${DORIS_CLANG_HOME}/bin:${PATH}"
+fi
+
# Check args
usage() {
echo "
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
