zclllyybb commented on issue #65605: URL: https://github.com/apache/doris/issues/65605#issuecomment-4969637062
Breakwater-GitHub-Analysis-Slot: slot_a501243ab2e4 This content is generated by AI for reference only. Initial triage for maintainers: The attached heap graph is actionable, but it does not prove a classic unreleased-pointer leak by itself. The largest visible branch is about 3.0 GB under `butil::ResourcePool::add_block` out of a 5.49 GB focused heap. The visible caller chain is: `AsyncResultWriter::process_block -> VTabletWriterV2::open -> _open_streams/_open_streams_to_backend -> LoadStreamStubs::open -> LoadStreamStub::open -> brpc::Channel::CallMethod -> bthread_timer_add -> TimerThread::Bucket::schedule -> butil::ResourcePool::add_block`. For Doris 3.1.4, this maps to the load stream open path. `VTabletWriterV2::open()` opens load streams to destination BEs, and `LoadStreamStub::open()` creates a brpc stream and calls `open_load_stream` with `open_load_stream_timeout_ms` (default 60000 ms). brpc schedules an RPC timeout task via `bthread_timer_add`. In brpc 1.4.0, canceled timer tasks are recycled by the timer thread later, and `butil::ResourcePool` retains allocated blocks as a high-water cache rather than returning them to the OS immediately. So the current evidence is most consistent with a large brpc timer-task/resource-pool high-water mark driven by load stream opens. This can still be a real production problem because it is charged as `UntrackedMemory` and can keep RSS high, but the screenshot alone is not enough to distinguish: 1. high load-stream open rate or many concurrent loads/backends/streams with a 60s timeout causing a large timer-task high-water mark; 2. repeated failed/slow `open_load_stream` RPCs causing many timeout tasks to remain pending until the timer thread catches up; 3. an actual lifecycle bug where failed/canceled load stream opens are not being completed/closed normally. I checked the 3.1.4 load-stream code path and did not see an obvious permanent `LoadStreamMapPool` retention in the normal close/cancel paths: successful close calls `release()` and the last sink erases the load stream map, and cancel also cancels streams and releases the map. That makes the brpc timer path the first place to verify rather than assuming scanner block memory is leaking. Missing information needed to confirm the root cause: - Raw heap profiles from at least two timestamps, preferably shortly after restart and when `UntrackedMemory` is high, not only the rendered image. - BE memory tracker or memory overview output at the same timestamps, plus BE RSS. - BE logs covering the growth window, especially lines containing `open load stream`, `failed to open stream`, `Failed to connect to backend`, `releasing streams`, `closing olap table sink`, and `canceled olap table sink`, with the related `load_id`s. - Load workload shape: load type, load QPS, concurrent load count, number of BEs, tablet/replica scale, `load_stream_per_node`, `open_load_stream_timeout_ms`, and whether failures/retries happen during the period. - brpc/bthread vars if available, especially `bthread_timer_*`, collected while memory is growing and after load traffic stops. - Whether memory plateaus after stopping all load traffic for longer than `open_load_stream_timeout_ms`, or continues to grow with no load activity. Recommended next step: Try to reproduce with continuous load traffic while collecting repeated heap profiles and the matching load logs. If the `ResourcePool::add_block` memory grows with load-stream open rate and then plateaus after traffic stops, this is likely brpc timer/resource-pool high-water retention; reducing load concurrency, `load_stream_per_node`, or carefully lowering `open_load_stream_timeout_ms` can be used as a diagnostic mitigation. If it keeps increasing after load traffic stops and after the timeout window has passed, the suspicious area is the load-stream open/cancel/error lifecycle around `LoadStreamStub::open()` and brpc `Controller::EndRPC` timer cleanup. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
