morningman opened a new pull request, #68942:
URL: https://github.com/apache/doris/pull/68942
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
**In short.** The NonConcurrent regression pipeline fails a few percent of
its runs with `Lsan fail` while every case passes. LeakSanitizer finds, at the
BE's graceful exit, one stream load's `ResourceContext` that was never
released, but its report only shows allocation stacks, which are the same for
every stream load, so nobody can tell which load it was. This PR makes the BE
print, at graceful exit in memleak-check mode, every `ResourceContext` still
alive with its task id, the references still held, its memory tracker and
workload group, and whether its `QueryContext` is already gone. Nothing changes
unless `enable_graceful_exit_check=true`.
**Background.**
- With `enable_graceful_exit_check=true` (memleak-check mode, used by the
regression pipelines on the ASAN build), `doris_main` stops every service and
runs `ExecEnv::destroy()` instead of `_exit()`, so that LeakSanitizer checks
the heap when the process exits. The NonConcurrent pipeline greps `be.out` for
`detected memory leaks` after `stop_be.sh --grace` and reports `Lsan fail`.
- Every query or load has a `ResourceContext`: CPU/memory/IO contexts, a
task controller, the query memory tracker and the workload group. The
`QueryContext` owns it, and the components that work for the task (load
channel, memtables, scanner context, `RuntimeQueryStatisticsMgr`, threads
attached to the task, ...) hold references of their own while they run.
**The problem, and what it cost.**
- `Lsan fail` hits 1–5% of NonConcurrent runs depending on the month (448 of
the last 14000, back to March 2026), on PRs that have nothing to do with it;
authors rerun or skip. Every report TeamCity still keeps (10 runs from
2026-10-05 to 2026-10-12, on #68707, #68281, #68791, #68803, #68798, #68543,
#68901, #68919, #68879) has the same shape, down to the byte counts, and all of
its leaks are indirect:
| Leaked | Size | Origin |
|---|---|---|
| one `ResourceContext` with its CPU/memory/IO contexts,
`QueryTaskController` and both trackers (`Load#Id=…`, `Memtable#Load#Id=…`) |
6.5 KB, 94 objects | an HTTP stream load: `StreamLoadAction::_process_put` →
`StreamLoadExecutor` → `FragmentMgr::_get_or_create_query_ctx` |
| the `normal` workload group created by
`WorkloadGroupMgr::create_internal_wg()` | 6.1 KB + 4.3 KB | kept alive by
`ResourceContext::_workload_group` |
| the `std::make_shared` block of the load's `QueryContext` | 2.5 KB |
kept allocated by the `weak_ptr` in `QueryTaskController` |
- The `QueryContext` itself was destroyed: in `be.INFO`, each of the 21894
`query_id: … use wg:` lines of the run has its `Query … deconstructed`. So one
reference to the load's `ResourceContext` is never released, from a place
LeakSanitizer does not scan (a bthread stack, whose memory brpc maps itself) or
a count that is never decremented. LeakSanitizer cannot tell which of these two
it is.
- A NonConcurrent run issues about 430 HTTP stream loads (about 380 through
the load channel, 150 of them audit-log loads), and the report cannot say which
one leaked: the stacks are identical, and the one thing that would identify the
load, the tracker label with its id, is inside the leaked memory, which
LeakSanitizer does not print. Without the load there is nothing to follow, so
the flake stays unexplained.
**How this PR fixes it.**
- In memleak-check mode each `ResourceContext` registers itself in a
process-wide set when it is constructed and removes itself when it is
destroyed. The set stores the addresses bitwise inverted, so LeakSanitizer does
not take it for a reference and still reports a leaked context.
- `ResourceContext::describe_live_contexts()` returns one line per context
still alive: address, strong references still held, task id, finished flag,
running time, memory tracker (label, consumption, peak), workload group and,
for a query, whether its `QueryContext` is still alive.
- `doris_main` calls it right after `ExecEnv::destroy()` on the graceful
exit path and writes the result to `be.WARNING` and to `be.out`, where the
LeakSanitizer report follows once `main()` returns. The printed address falls
inside one of the `Objects leaked above` ranges of that report.
**Results.** What the next `Lsan fail` will tell, compared with today:
| | Before | After |
|---|---|---|
| Which query or load leaked | unknown | `task_id=…`; its label, table and
test are then in `be.INFO` |
| How many references are left | unknown | `references=N` |
| Whether the `QueryContext` is gone | only by pairing ~20k log lines |
`query_context=destroyed` |
| Where to look | the LeakSanitizer report in `be.out` | the same `be.out`,
just before that report (the build log's `head -n 100 be.out` shows it) |
| Cost with `enable_graceful_exit_check=false` (default) | — | one boolean
check per `ResourceContext` construction and destruction |
A line looks like this (values made up):
```
1 resource contexts are still alive. LeakSanitizer reports a leaked context
as the memory block that holds it, so match the addresses below against the
ranges it prints under 'Objects leaked above'.
0x7c0000001010: references=1, task_id=1eac0c4ec4ec0000-0000000000000003,
finished=true, running_ms=7569,
mem_tracker=Load#Id=1eac0c4ec4ec0000-0000000000000003 (consumption=0,
peak=17.18 MB), workload_group=normal(1), query_context=destroyed
```
**Classes, and how they call each other.**
- `ResourceContext` (`resource_context.{h,cpp}`): registers in the
constructor and unregisters in the destructor when memleak-check mode is on;
`describe_live_contexts()` formats the live ones.
- `QueryTaskController`: new `is_query_context_alive()`.
- `doris_main`: prints the description after `ExecEnv::destroy()`.
```
doris_main (graceful exit, enable_graceful_exit_check=true)
├─ ExecEnv::destroy()
└─ ResourceContext::describe_live_contexts(100) ──► be.WARNING + be.out
├─ registry of disguised addresses ◄── ResourceContext() /
~ResourceContext()
└─ per live context:
├─ TaskController: task id, finished, running time
├─ MemoryContext → MemTrackerLimiter: label, consumption, peak
├─ WorkloadGroup: name, id
└─ QueryTaskController::is_query_context_alive()
```
No holder of `ResourceContext` changes, and neither do the LeakSanitizer
options or suppressions. Once the leaked load is known, the fix for the leak
itself will be a separate PR.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Added `ResourceContextLeakCheckTest`
(`be/test/runtime/workload_management/resource_context_test.cpp`): a context is
listed with its task id, references and tracker until it is released; contexts
created outside memleak-check mode are not tracked; a query's context shows
`query_context=destroyed` once its `QueryContext` is gone; the output is
capped. Locally: clang-format 16, `build-support/check-build-hygiene.sh`, and
`-fsyntax-only` of every changed file with the BE UT ASAN flags against
regenerated gensrc. The BE UT binary was not built or run locally; the BE UT
pipeline runs the new test.
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
