Hi all, I want to refactor Doris BE memory tracking tool MemTracker. MemTracker: tracks memory consumption via manual calls to Consume()/Release(), it contains an optional limit and can be arranged into a tree structure.
motivation: Currently, only a small amount of memory in BE is tracked, which makes it impossible to locate and limit the memory usage of process and queries, affecting the stability of Doris. Details: 1. The online task fails due to the OOM of the BE process; 2. Cannot effectively limit the memory usage of a single query; 3. Users often report that BE consumes a lot of memory and cannot effectively locate the memory hotspot location; 4. Hope for a clearer MemTracker hierarchy to improve readability; 5. Hope to support the detection of memory leaks; Implementation ideas: Based on tcmalloc mew/delete hook and TLS, all memory new/delete/malloc/free/etc. of the BE process can be automatically counted. Similar to ClickHouse overloading JeMalloc. Since Memory Tracking Tool is an important feature, I'd like to propose it as a new Doris improvement Proposal and present a detailed design for this feature. Part of the work has actually been done, in the following pr: https://github.com/apache/incubator-doris/pull/6979 https://github.com/apache/incubator-doris/pull/8322 https://github.com/apache/incubator-doris/pull/8476 This belongs to issue: https://github.com/apache/incubator-doris/issues/7553 Furthermore, my wiki account is zouxiny...@gmail.com and I request the editing rights in Doris wiki. Best, Xinyi Zou
