Hi Xinyi: Nice Job! I have created DSIP-002[1] for this proposal
[1] https://cwiki.apache.org/confluence/display/DORIS/DSIP-002%3A+Refactor+memory+tracker+on+BE -- 此致！Best Regards 陈明雨 Mingyu Chen Email: chenmin...@apache.org At 2022-03-19 21:02:16, "XinYi Zou" <zouxiny...@gmail.com> wrote: >Hi all, >I want to refactor Doris BE memory tracking tool MemTracker. > >MemTracker: tracks memory consumption via manual calls to >Consume()/Release(), it contains an optional limit and can be arranged into >a tree structure. > >motivation: >Currently, only a small amount of memory in BE is tracked, which makes it >impossible to locate and limit the memory usage of process and queries, >affecting the stability of Doris. Details: >1. The online task fails due to the OOM of the BE process; >2. Cannot effectively limit the memory usage of a single query; >3. Users often report that BE consumes a lot of memory and cannot >effectively locate the memory hotspot location; >4. Hope for a clearer MemTracker hierarchy to improve readability; >5. Hope to support the detection of memory leaks; > >Implementation ideas: >Based on tcmalloc mew/delete hook and TLS, all memory >new/delete/malloc/free/etc. of the BE process can be automatically counted. >Similar to ClickHouse overloading JeMalloc. > >Since Memory Tracking Tool is an important feature, I'd like to propose it >as a new Doris improvement Proposal and present a detailed design for this >feature. > >Part of the work has actually been done, in the following pr: >https://github.com/apache/incubator-doris/pull/6979 >https://github.com/apache/incubator-doris/pull/8322 >https://github.com/apache/incubator-doris/pull/8476 >This belongs to issue: https://github.com/apache/incubator-doris/issues/7553 > >Furthermore, my wiki account is zouxiny...@gmail.com and I request the >editing rights in Doris wiki. > >Best, >Xinyi Zou
