morningman opened a new pull request, #67877:
URL: https://github.com/apache/doris/pull/67877
### What problem does this PR solve?
Related PR: #64093
Problem Summary:
**Every aarch64 BE built from master or branch-4.1 (since 4.1.4-rc01)
crashes before `main()` on a 64 KiB-page kernel** (Kunpeng 920 + Kylin V10 /
openEuler / CentOS aarch64, the common ARM server stack). The core from a 4.1.4
arm64 deployment shows:
```
access_mem (addr=0) src/aarch64/Ginit.c
is_plt_entry src/aarch64/Gstep.c
_ULaarch64_step
_ULaarch64_tdep_trace
unw_backtrace
<bthread/mutex.cpp static initializer: backtrace(dummy_buf, 4)> ← bRPC
"Warm up backtrace before main()"
__libc_csu_init
```
SIGSEGV with `si_addr=0`, AUXV `PAGESZ=65536`, and inside the libunwind
cursor `ip=0`, `validate=0`, `mem_validate_func=msync_validate`,
`last_good_addr[]={0,0,0,0}`.
How it happens:
1. #64093 links GNU libunwind statically into the aarch64 BE (it forces
`USE_UNWIND=ON` on Linux, references `unw_*` from `phdr_cache.cpp` without the
`__x86_64__` guard, and builds jemalloc with `--enable-prof-libunwind`).
libunwind's weak `backtrace` alias then replaces glibc's `backtrace()` for
every caller in the process: bRPC's pre-main warm-up,
`StackTrace::tryCapture()` on aarch64, jemalloc heap profiling. The 4.1.3 arm64
package was immune only because it was built with `USE_UNWIND=OFF` (which the
ARM compilation doc still prescribes and #64093 removed): it has
`backtrace@GLIBC_2.17` and zero `_ULaarch64_*` symbols.
2. libunwind 1.6.2 hard-codes `PAGE_SIZE 4096` in `src/aarch64/Ginit.c`. On
a 64 KiB-page kernel `mincore()`/`msync()` reject its 4 KiB-aligned probes with
`EINVAL`, so `validate_mem()` fails for 15/16 of all addresses (the core's
`msync_validate` + empty `last_good_addr` are exactly this).
3. `unw_step()` therefore gets `unw_is_signal_frame() < 0` for a valid IP,
takes the "IP points to non-mapped memory, use LR" recovery path and loads
`uc.regs[30]` — a slot `_Uaarch64_getcontext_trace` never writes (it saves only
FP/SP/PC). On the reporting host that stack garbage is 0.
4. `dwarf_step(ip=0)` finds nothing, and the fallback calls `is_plt_entry()`
with `c->validate` restored to 0, so `access_mem` executes `ldr x8, [x1]` with
`x1=0`.
The PHDR cache being empty before `main()` is not what breaks this (step 3
has already poisoned `ip` before any FDE lookup, and the same crash reproduces
without the Doris hook); changing the cache initialisation order would not fix
it, and every later `backtrace()` on such a host rolls the same dice.
Fix: **upgrade libunwind 1.6.2 → 1.8.3**, which was never bumped since it
was introduced in 2023 (#21938):
- 1.7.0+: `unw_page_size = sysconf(_SC_PAGESIZE)` instead of the hard-coded
4096.
- 1.8.0+: `src/mi/Gaddress_validator.c` validates through the pipe-write
probe only, so `mincore`/`msync` alignment no longer matters at all;
`unw_step()` sets `c->validate = 1` before the DWARF-failure fallback, so a
garbage IP is rejected instead of dereferenced.
Changes:
- `thirdparty/vars.sh`: libunwind 1.6.2 → 1.8.3 (md5
`13bc7b41462ac6ea157d350eaf6c1503`).
- `thirdparty/patches/libunwind-1.8.3-doris-phdr-cache.patch`: the #64093
hook rebased onto 1.8.3's `as->iterate_phdr_function` call site; semantics
unchanged (`doris_unwind_iterate_phdr` still takes precedence whenever it is
linked in). 1.8 also exposes `unw_set_iterate_phdr_function()`, which could
replace the source patch in a follow-up.
- `thirdparty/build-thirdparty.sh`: `--disable-tests
--disable-documentation`; only `libunwind.a` is consumed.
- `thirdparty/test/libunwind-page-size-test.sh` (+ run at the end of the
Linux thirdparty CI build): reproduces the failure on any Linux host. An
`LD_PRELOAD` shim gives libunwind a 64 KiB-page kernel's behaviour (`EINVAL`
for unaligned `mincore`/`msync`, `sysconf(_SC_PAGESIZE)=65536`), and a program
shaped like the bRPC warm-up calls `backtrace()` from a constructor with a
zeroed stack. Against 1.6.2 it segfaults with the customer's exact stack and
cursor state; against 1.8.3 it passes.
No BE source change: every `unw_*` API the BE, jemalloc and the x86_64
stack-trace code use is unchanged in 1.8.3, `cxx_exceptions` stays off on
x86_64/aarch64, and the weak `backtrace` alias is still provided.
Follow-ups outside this PR: the doris-website ARM compilation page still
says `export USE_UNWIND=OFF` is required, which has been a silent no-op since
#64093.
### Release note
Fix the aarch64 BE crashing before `main()` (SIGSEGV in
`access_mem`/`is_plt_entry` under `unw_backtrace`) on 64 KiB-page kernels such
as Kunpeng 920 with Kylin, openEuler or CentOS, by upgrading the bundled
libunwind from 1.6.2 to 1.8.3.
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
- Symbolised the customer core against the matching 4.1.4 arm64
`doris_be` (gdb): crash stack, `si_addr=0`, `PAGESZ=65536`, cursor
`ip=0`/`validate=0`, `mem_validate_func=msync_validate`, `last_good_addr={0}`,
`uc.regs[30]=0`; `backtrace` == `unw_backtrace` in the 4.1.4 binary vs
`backtrace@GLIBC_2.17` in 4.1.3.
- Reproduced the identical crash on a 4 KiB arm64 Linux container by
running the real 4.1.4 `doris_be --version` under the 64 KiB shim and zeroing
`uc.regs[30]` in gdb.
- Built libunwind 1.6.2 (+Doris patch) and 1.8.3 (+rebased patch)
for aarch64 with the exact recipe from `build_libunwind()`; ran
`thirdparty/test/libunwind-page-size-test.sh` against both: 1.6.2 FAIL
(SIGSEGV, same stack), 1.8.3 PASS with and without the Doris hook, and with
`ip` forced to 0 in gdb.
- Same build + test on x86_64 (emulated ubuntu-22.04): PASS.
- The rebased patch applies with both GNU `patch` and Apple `patch`;
`thirdparty/test/download-thirdparty-*-test.sh`,
`juicefs-default-mirror-test.sh`, `azure-vcpkg-retry-test.sh` pass; shellcheck
reports no new findings.
- Not done here: a Linux `doris_be` rebuilt with 1.8.3 on a real 64
KiB-page host. That is the one check that still needs an aarch64 machine.
- Behavior changed:
- [x] Yes. Bundled libunwind 1.6.2 → 1.8.3; libunwind's own test
programs and man pages are no longer built. Unwinding semantics for the BE
otherwise unchanged.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_016MdauUJh8AtE8SozZ29Dmv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
