Doris-Breakwater commented on issue #67871:
URL: https://github.com/apache/doris/issues/67871#issuecomment-5634831507
## Initial triage
This is a valid, high-impact BE-UT infrastructure failure. The available
evidence points away from the Remote Doris cancellation logic itself and toward
ASAN/libc symbol interposition during Abseil entropy initialization. The
affected scope is currently **x86 Linux ASAN_UT**; there is not yet evidence
that a non-sanitized BE or a production deployment is affected.
The issue currently has no labels or assignee. It should be routed to the BE
build/glibc-compatibility owners and treated as a high-priority CI reliability
regression because one occurrence aborts the whole test binary.
### What is verified
- The failing worker is created in `FlightRemoteDorisStream::open()`, but it
crashes before the in-process Flight server enters `DoGet`. Arrow/gRPC startup
is therefore the trigger surface, not proof of a Remote Doris cancellation race.
- Doris uses Abseil `20250512.1`. On glibc headers >= 2.25, that Abseil
version's `ReadSeedMaterialFromOSEntropy()` calls `getentropy(buffer,
min(remaining, 256))` before falling back to `/dev/urandom`.
- The failing build used Clang 20.1.8 with `-fsanitize=address` and
`GLIBC_COMPATIBILITY=ON`.
- The crash registers are consistent with the exact Abseil call above: `rdi`
is a buffer and `rsi == 0x100` (256), followed by a jump to PC 0.
- Clang 20's static ASAN runtime supplies weak `getentropy` and `getrandom`
interceptor trampolines. Its `getentropy` interceptor unconditionally calls the
function pointer stored in `__interception::real_getentropy`.
- Doris puts `musl/getrandom.c` in `glibc-compatibility-explicit`, but
`musl/getentropy.c` remains only in the `glibc-compatibility` static archive. A
minimal link using the same Clang 20 toolchain and these Doris sources produced
a strong Doris `getrandom`, but a weak ASAN-trampoline `getentropy`; the
archive member containing Doris' `getentropy` was not extracted. This occurs
because the ASAN archive is linked with `--whole-archive` before user
objects/libraries and its weak definition already satisfies later `getentropy`
references.
- PR #67701 did not change `getrandom.c`, `getentropy.c`, their
explicit/archive membership, or their relative link order. It added forced
undefined references for math symbols and expanded `lance_symbol_versions.c`.
Its timing is worth retaining as a correlation, but the current source diff
does not establish it as the entropy crash's cause.
### Leading root-cause hypothesis
The issue's interposition hypothesis is directionally strong, but the likely
null target is **`real_getentropy` rather than `real_getrandom`**:
1. Abseil calls the ASAN-provided weak `getentropy` trampoline.
2. At ASAN startup, compiler-rt resolves the real function with
`dlsym(RTLD_NEXT, "getentropy")`.
3. If the runtime libc does not export `getentropy` (for example, glibc
older than 2.25), compiler-rt's fallback finds its own trampoline through
`RTLD_DEFAULT`, rejects that self-reference, and leaves `real_getentropy ==
nullptr`.
4. The first Abseil entropy request then calls through that null pointer,
yielding PC 0 with `(buffer, 256)` still in the argument registers.
This explains the observed signature and why Doris' explicit `getrandom`
object does not by itself protect this path. It is still a hypothesis until the
failing binary and its runtime libc are inspected. The reported nondeterminism
may come from heterogeneous agent runtimes or differing final symbol
resolution; it should not be attributed to a code race without that evidence.
### Information needed to confirm it
Please preserve one failing `doris_be_test` binary (or rerun on the same
image) and collect:
```bash
getconf GNU_LIBC_VERSION
readelf -WsW be/ut_build_ASAN/test/doris_be_test | grep -E
'getentropy|getrandom'
nm -an be/ut_build_ASAN/test/doris_be_test | grep -E
'(___interceptor_)?getentropy|(___interceptor_)?getrandom|real_getentropy|real_getrandom'
ASAN_OPTIONS=verbosity=1 be/ut_build_ASAN/test/doris_be_test \
--gtest_filter=RemoteDorisV2ReaderTest.RuntimeCancellationInterruptsBlockedFlightDoGet
```
For a core under gdb, also inspect:
```gdb
p/x __interception::real_getentropy
p/x __interception::real_getrandom
info address getentropy
info address getrandom
```
The decisive result would be a weak `getentropy` resolving to the ASAN
trampoline together with `real_getentropy == 0`, ideally accompanied by
`AddressSanitizer: failed to intercept 'getentropy'`. Please collect the same
symbol/glibc information from one passing agent as a comparison. The issue
already contains sufficient test steps and representative logs; a Doris Profile
is not relevant to this failure.
### Recommended next steps
1. Run the symbol/glibc checks above on one failing and one passing agent
before changing code.
2. If confirmed, make Doris' compatibility `getentropy` definition
unavoidable in the final link (for example, move `musl/getentropy.c` into the
explicit object target or force its archive extraction), while retaining the
existing syscall-backed `getrandom` implementation. Verify the final symbol
table, not only a successful link.
3. Add a link-level regression check requiring the intended strong
`getentropy`/`getrandom` definitions in ASAN_UT, plus repeated execution of the
focused test on the oldest supported glibc runtime and on a current runtime. A
runtime-only test on modern glibc would miss this failure mode.
4. Treat CI observability separately: emit the active gtest name as a
TeamCity service message or otherwise persist it before test execution, because
gtest XML is not completed when the process aborts.
Breakwater-GitHub-Analysis-Slot: slot_d1b38fd4389f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
