leeks8888 commented on issue #70496:
URL: https://github.com/apache/doris/issues/70496#issuecomment-6904163366

   We see the same crash families on **Apache Doris 3.1.4** (BE commit 
`7f5ba43de6`, official `apache-doris-3.1.4-bin-x64-noavx2` binary, brpc 1.4.0), 
so this does not look specific to 4.1.x. Sharing our data and a core-dump 
analysis in case it helps narrow it down.
   
   ### Environment
   | Item | Value |
   |---|---|
   | Doris | 3.1.4 (`7f5ba43de6`), 3 FE + 28 BE |
   | BE host | Rocky Linux 9, kernel `5.14.0-611.5.1.el9_7` |
   | BE runtime | Docker, Ubuntu 22.04 base (glibc 2.35), OpenJDK 17.0.21 |
   | BE size | 16 cores (one node 32), 125 GB RAM, `mem_limit = 80%` |
   
   The crashes started before we moved the image from Ubuntu 20.04 to 22.04 and 
have the same rate after, so the base image / glibc is not the trigger.
   
   ### Crash statistics (BE `be.out`, 2026-05-01 – 2026-11-13, 991 crashes)
   | Family | Count | Top frames |
   |---|---|---|
   | bthread SIGSEGV | 663 (67%) | `bthread::TaskControl::worker_thread`, 
`TaskGroup::task_runner` / `usleep`, `ExecutionQueueBase::_execute_tasks`; 
fault addr `0x0` / `0x38` |
   | brpc Stream SIGABRT | 109 (11%) | `Stream::Create ← StreamAccept ← 
PInternalService::open_load_stream` with `Check failed: 0 == 
Socket::Address(fake_sock_id, &ptr)`; `Stream::OnReceived ← 
ParseStreamingMessage` |
   | JVM (in-process) | 101 (10%) | C1 `GraphBuilder`, 
`SymbolTable::do_lookup`, `Symbol::increment_refcount`, 
`ProtectionDomainEntry::object_no_keepalive` |
   | other | 118 | `je_large_prof_info_get`, 
`VersionGraph::capture_consistent_versions`, `MetadataAdder<TabletMeta>`, 
`InvertedIndexFileInfo`, `bvar::Reducer<…>::SeriesSampler::take_sample` (5) |
   
   - Crashes are spread evenly across all 28 BEs (21–47 each), so not a single 
bad host.
   - Rate follows load: ~7.5–12.6/day on weekdays, ~4/day on weekends, peak 
08:00–17:00. Monthly totals grew from 68 (May) to 291 (Oct).
   - Ruled out: kernel OOM (none), `vm.max_map_count` (2,000,000), fd limit 
(≈5k of 65k), swap, THP.
   - Setting `enable_memtable_on_sink_node=false` for ~2 h did not change the 
rate. That change only skips the `open_load_stream` path.
   - The query id printed in the crash header usually belongs to a SELECT that 
had already finished OK, so there is no single culprit query.
   
   The JVM crashes inside the BE process (symbol table / metadata corruption) 
and the variety of unrelated victims make us think this is **native heap 
corruption (use-after-free or stray write) inside `doris_be`**, with bthread / 
brpc / JVM structures as the victims.
   
   ### Core dump analysis: bthread SIGSEGV @0x38
   We captured a full core (124 GB virtual, 41 GB on disk) of a 
`bthread::TaskControl::worker_thread` SIGSEGV. The unwinder stops at the signal 
handler, so we read the saved `ucontext` registers directly:
   
   ```
   RIP = <doris_be>+0xd03b710   ; symbol-less frame, the same "...710" address 
in every
                                ; worker_thread SIGSEGV we have
   CR2 = 0x38, TRAPNO = 14, ERR = 4, RAX = 0
   ```
   
   The instruction at RIP is a GCC isolated-NULL-path trap 
(`-fisolate-erroneous-paths-dereference`):
   ```
   mov 0x38,%rax
   ud2
   ```
   Only `bthread::TaskGroup::run_main_task` branches to that trap, after 
`wait_task()` returns a tid. This is the inlined `address_meta(tid)` → 
`ResourcePool<TaskMeta>::address_resource()` lookup:
   ```
   rmt+158: call TaskGroup::wait_task
   rmt+190: shr $0x18 ; group = _block_groups[id >> 24]  ; NULL -> trap 
(rmt+205)
   rmt+218: block = group->blocks[(id >> 8) & 0xffff]     ; NULL -> trap 
(rmt+225)
   rmt+234: cmp %rax, 0x8000(%r12) ; offset vs block->nitem
   rmt+242: jbe trap                                      ; nitem <= offset -> 
trap  <== taken
   ```
   State in the core:
   - tid from the run queue: `0x00000ae2_0005f300` (version `0xae2`, slot 
`0x5f300` → block `0x5f3`, item 0)
   - the group and block pointers are valid
   - **`block->nitem` (block+0x8000) is 0**
   - yet **item 0 of that block is a fully constructed `TaskMeta` whose `tid` 
field is `0x00000ae20005f300`**, with a valid fn pointer and arg
   
   `get_resource()` increments `nitem` right after constructing the item, 
before the tid can be published to any queue. The core is taken after all 
threads stopped, so this is not a visibility race. **Something overwrote 
`block->nitem` (8 bytes at block+0x8000) with 0 after the TaskMeta was 
created.** That is a stray write or use-after-free into memory that the 
ResourcePool block now owns.
   
   ### bvar AgentCombiner fixes
   #64040 / #67276 (brpc#2949 / brpc#3291) fix a use-after-free that writes 
zeros into freed TLS agent memory. That pattern would fit the zeroed `nitem` 
above. But this issue reports the same crash families on 4.1.5 (`917b20be838` = 
4.1.5-rc03, which has both patch files). So either those patches don't cover 
the remaining writer, or the release binary was built without rebuilding 
thirdparty. Could a maintainer confirm whether the official 4.1.5 binary 
actually includes the patched brpc?
   
   We can run gdb commands on the core if that helps; we can't share the core 
itself because it contains customer data.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to