LuciferYang commented on issue #67871:
URL: https://github.com/apache/doris/issues/67871#issuecomment-5634896273
Reproduced outside CI, and the mechanism in the triage above is confirmed.
One correction to my earlier comment first.
Correction: my claim that the failing build's runtime libc is glibc 2.28 and
exports `getentropy` was wrong. That Build ID came from build 1040773, which is
a different crash (the Iceberg null dereference). The logs of the builds that
carry this signature contain no libc Build ID at all, because ASAN could not
unwind past frame 1. So the "old libc" branch was never ruled out, and it is in
fact the trigger.
What sets the date: #67752 reverted the compilation image from `almalinux:8`
back to `centos:7`, merged 2026-09-10 10:32 +0800. almalinux:8 carries glibc
2.28, which exports `getentropy@@GLIBC_2.25`; centos:7 carries glibc 2.17,
which does not export it at all. The BE UT core dumps start that day (26 of 76
builds) and reach 51 of 63 the next, which is what a staged image refresh
across agents looks like, and it also explains why the same test passes on some
agents minutes apart.
Reproduction, with no Doris build involved. A 25-line program that calls
`getentropy(buf, 32)` on the main thread and on a spawned thread, compiled with
the repo's own clang 20.1.7 and `-fsanitize=address`, linked four ways, then
run under glibc 2.17 (`centos:7`, x86_64) and glibc 2.28 (the current build
image):
| link | `getentropy` in `.dynsym` | glibc 2.17 | glibc 2.28 |
|---|---|---|---|
| ASAN only | WEAK, 5 bytes, ASAN trampoline | SEGV at pc 0 | ok |
| plus Doris `getrandom.c.o` (BE UT today, forced object) | WEAK, 5 bytes,
ASAN trampoline | SEGV at pc 0 | ok |
| plus `libglibc-compatibility.a` on the link line | WEAK, 5 bytes, ASAN
trampoline | SEGV at pc 0 | ok |
| plus Doris `getentropy.c.o` forced in | GLOBAL, 637 bytes, Doris | ok | ok
|
The crash is the same shape as CI's, including the register pattern, `rdi` a
buffer and `rsi` the requested length, and `#1` being the caller of
`getentropy` rather than an interceptor frame:
```
==1==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc
0x000000000000 ...)
#0 0x000000000000 (<unknown module>)
#1 0x5555556a5ef4 (/ent/t_archive+0x151ef4)
rdi = 0x00007bfffcbde020 rsi = 0x0000000000000020
```
With `ASAN_OPTIONS=verbosity=1`, glibc 2.17 prints `AddressSanitizer: failed
to intercept 'getentropy'` on every one of these binaries, so `real_getentropy`
is null in all four. Only the last one survives, because there the call binds
to Doris' own definition and never reaches the interceptor.
Two things follow for the fix. Forcing `getrandom` alone does not protect
this path, which the second row shows, so the existing explicit object is not
enough. Putting `libglibc-compatibility.a` on the link line is also not enough,
which the third row shows: ASAN's weak definition already satisfies the
reference by the time the archive is scanned, so the member is never extracted.
What works is making Doris' `getentropy` unavoidable, that is moving
`musl/getentropy.c` into `glibc-compatibility-explicit` next to
`musl/getrandom.c`, keeping the existing `HAVE_SYS_RANDOM_H` guard.
I can put up that change with the symbol table check the triage asks for.
The reason to also keep the CentOS 7 runtime in the loop for any regression
test is visible in the table: every variant passes on glibc 2.28, so a test
that only runs on a modern runtime proves nothing here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
