This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 1cb1f00d12e [opt](memory) Opt memory heap profile (#2334)
1cb1f00d12e is described below
commit 1cb1f00d12e67b343adb7648ca9147cc756282de
Author: Xinyi Zou <[email protected]>
AuthorDate: Wed May 7 16:17:33 2025 +0800
[opt](memory) Opt memory heap profile (#2334)
## Versions
- [x] dev
- [x] 3.0
- [x] 2.1
- [x] 2.0
## Languages
- [ ] Chinese
- [ ] English
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
---
community/developer-guide/debug-tool.md | 343 ++++++++-------------
.../heap-profile-memory-analysis.md | 12 +-
.../current/developer-guide/debug-tool.md | 252 +++++++--------
.../heap-profile-memory-analysis.md | 12 +-
.../heap-profile-memory-analysis.md | 12 +-
.../heap-profile-memory-analysis.md | 12 +-
.../heap-profile-memory-analysis.md | 12 +-
.../heap-profile-memory-analysis.md | 12 +-
8 files changed, 316 insertions(+), 351 deletions(-)
diff --git a/community/developer-guide/debug-tool.md
b/community/developer-guide/debug-tool.md
index d4fbd355fc8..1d629171afd 100644
--- a/community/developer-guide/debug-tool.md
+++ b/community/developer-guide/debug-tool.md
@@ -88,148 +88,25 @@ Fe is a java process. Here are just a few simple and
commonly used java debuggin
## BE debugging
-### Preparing
-
-[pprof](https://github.com/google/pprof): from gperftools, it is used to
transform the content generated by gperftools into a format that is easy for
people to read, such as PDF, SVG, text, etc.
-
-[graphviz](http://www.graphviz.org/): in the absence of this library, pprof
can only be converted to text format, but this way is not easy to view. After
the library is installed, pprof can be converted to SVG, PDF and other formats,
and the call relationship is clearer.
-
-[perf](https://perf.wiki.kernel.org/index.php/main_page): Linux kernel comes
with performance analysis tool. [here](http://www.brendangregg.com/perf.html)
there are some examples of perf usage.
-
-[flamegraph](https://github.com/brendangregg/flamegraph): a visualization tool
used to show the output of perf in the form of flame graph.
-
### Memory
Debugging memory is generally divided into two aspects. One is whether the
total amount of memory use is reasonable. On the one hand, the excessive amount
of memory use may be due to memory leak in the system, on the other hand, it
may be due to improper use of program memory. The second is whether there is a
problem of memory overrun and illegal access, such as program access to memory
with an illegal address, use of uninitialized memory, etc. For the debugging of
memory, we usually use [...]
-Doris 1.2.1 and previous versions use TCMalloc. Doris 1.2.2 starts to use
Jemalloc by default. Select the memory debugging method according to the Doris
version used. If you need to switch TCMalloc, you can compile `USE_JEMALLOC=OFF
sh build.sh --be`.
-
-When we find that the memory usage is too large, we can first check the BE log
to see if there is a large memory application.
-
-###### TCMalloc
-
-When using TCMalloc, when a large memory application is encountered, the
application stack will be printed to the be.out file, and the general
expression is as follows:
-
-```
-tcmalloc: large alloc 1396277248 bytes == 0x3f3488000 @ 0x2af6f63 0x2c4095b
0x134d278 0x134bdcb 0x133d105 0x133d1d0 0x19930ed
-```
-
-This indicates that Doris be is trying to apply memory of '1396277248 bytes'
on this stack. We can use the 'addr2line' command to restore the stack to a
letter that we can understand. The specific example is shown below.
+#### Jemalloc HEAP PROFILE
-```
-$ addr2line -e lib/doris_be 0x2af6f63 0x2c4095b 0x134d278 0x134bdcb 0x133d105
0x133d1d0 0x19930ed
+> Doris 1.2.2 version starts to use Jemalloc as the memory allocator by
default.
-/home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1335
-/home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1357
-/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.cpp:267
-/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.hpp:86
-/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:239
-/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:213
-thread.cpp:?
-```
+For the principle analysis of Heap Profile, refer to [Heap Profiling Principle
Analysis](https://cn.pingcap.com/blog/an-explanation-of-the-heap-profiling-principle/).
It should be noted that Heap Profile records virtual memory
-##### JEMALLOC
+Supports real-time and periodic Heap Dump, and then uses `jeprof` to parse the
generated Heap Profile.
-Most of Doris's large memory applications use Allocator, such as HashTable and
data serialization. This part of the memory application is expected and will be
effectively managed. Other large memory applications are not expected and will
be applied The stack is printed to the be.INFO file, which is usually used for
debugging, and the general expression is as follows:
-```
-MemHook alloc large memory: 8.2GB, stacktrace:
-Alloc Stacktrace:
- @ 0x55a6a5cf6b4d doris::ThreadMemTrackerMgr::consume()
- @ 0x55a6a5cf99bf malloc
- @ 0x55a6ae0caf98 operator new()
- @ 0x55a6a57cb013 doris::segment_v2::PageIO::read_and_decompress_page()
- @ 0x55a6a57719c0 doris::segment_v2::ColumnReader::read_page()
- ……
-```
-
-#### HEAP PROFILE
-
-##### TCMalloc
-
-Sometimes the application of memory is not caused by the application of large
memory, but by the continuous accumulation of small memory. Then there is no
way to locate the specific application information by viewing the log, so you
need to get the information through other ways.
-
-At this time, we can take advantage of TCMalloc's
[heapprofile](https://gperftools.github.io/gperftools/heapprofile.html). If the
heapprofile function is set, we can get the overall memory application usage of
the process. The usage is to set the 'heapprofile' environment variable before
starting Doris be. For example:
-
-```
-export HEAPPROFILE=/tmp/doris_be.hprof
-./bin/start_be.sh --daemon
-```
-
-In this way, when the dump condition of the heapprofile is met, the overall
memory usage will be written to the file in the specified path. Later, we can
use the 'pprof' tool to analyze the output content.
-
-```
-$ pprof --text lib/doris_be /tmp/doris_be.hprof.0012.heap | head -30
-
-Using local file lib/doris_be.
-Using local file /tmp/doris_be.hprof.0012.heap.
-Total: 668.6 MB
- 610.6 91.3% 91.3% 610.6 91.3%
doris::SystemAllocator::allocate_via_malloc (inline)
- 18.1 2.7% 94.0% 18.1 2.7% _objalloc_alloc
- 5.6 0.8% 94.9% 63.4 9.5% doris::RowBatch::RowBatch
- 5.1 0.8% 95.6% 7.1 1.1% butil::ResourcePool::add_block (inline)
- 3.7 0.5% 96.2% 3.7 0.5% butil::iobuf::create_block (inline)
- 3.4 0.5% 96.7% 3.4 0.5% butil::FlatMap::init
- 3.2 0.5% 97.2% 5.2 0.8% butil::ObjectPool::add_block (inline)
- 2.6 0.4% 97.6% 2.6 0.4% __gnu_cxx::new_allocator::allocate
(inline)
- 2.0 0.3% 97.9% 2.0 0.3% butil::ObjectPool::add_block_group
(inline)
- 2.0 0.3% 98.2% 2.0 0.3% butil::ResourcePool::add_block_group
(inline)
- 1.7 0.3% 98.4% 1.7 0.3% doris::SegmentReader::_load_index
-```
-
-Contents of each column of the above documents:
-
-* Column 1: the memory size directly applied by the function, in MB
-* Column 4: the total memory size of the function and all the functions it
calls.
-* The second column and the fifth column are the proportion values of the
first column and the fourth column respectively.
-* The third column is the cumulative value of the second column.
-
-Of course, it can also generate call relation pictures, which is more
convenient for analysis. For example, the following command can generate a call
graph in SVG format.
-
-```
-pprof --svg lib/doris_be /tmp/doris_be.hprof.0012.heap > heap.svg
-```
-
-**NOTE: turning on this option will affect the execution performance of the
program. Please be careful to turn on the online instance.**
-
-###### pprof remote server
-
-Although heapprofile can get all the memory usage information, it has some
limitations. 1. Restart be. 2. You need to enable this command all the time,
which will affect the performance of the whole process.
-
-For Doris be, you can also use the way of opening and closing the heap profile
dynamically to analyze the memory application of the process. Doris supports
the [remote server debugging of
gperftools](https://gperftools.github.io/gperftools/pprof_remote_servers.html).
Then you can use 'pprof' to directly perform dynamic head profile on the remote
running Doris be. For example, we can check the memory usage increment of Doris
through the following command
-
-```
-$ pprof --text --seconds=60 http://be_host:be_webport/pprof/heap
-
-Total: 1296.4 MB
- 484.9 37.4% 37.4% 484.9 37.4% doris::StorageByteBuffer::create
- 272.2 21.0% 58.4% 273.3 21.1% doris::RowBlock::init
- 157.5 12.1% 70.5% 157.5 12.1% doris::RowBatch::RowBatch
- 90.7 7.0% 77.5% 90.7 7.0%
doris::SystemAllocator::allocate_via_malloc
- 66.6 5.1% 82.7% 66.6 5.1% doris::IntegerColumnReader::init
- 47.9 3.7% 86.4% 47.9 3.7% __gnu_cxx::new_allocator::allocate
- 20.8 1.6% 88.0% 35.4 2.7% doris::SegmentReader::_load_index
- 12.7 1.0% 89.0% 12.7 1.0% doris::DecimalColumnReader::init
- 12.7 1.0% 89.9% 12.7 1.0% doris::LargeIntColumnReader::init
- 12.7 1.0% 90.9% 12.7 1.0% doris::StringColumnDirectReader::init
- 12.3 0.9% 91.9% 12.3 0.9% std::__cxx11::basic_string::_M_mutate
- 10.4 0.8% 92.7% 10.4 0.8%
doris::VectorizedRowBatch::VectorizedRowBatch
- 10.0 0.8% 93.4% 10.0 0.8%
doris::PlainTextLineReader::PlainTextLineReader
-```
-
-The output of this command is the same as the output and view mode of heap
profile, which will not be described in detail here. Statistics will be enabled
only during execution of this command, which has a limited impact on process
performance compared with heap profile.
-
-##### JEMALLOC
-
-For the analysis of the principle of Heap Profile, please refer to [Analysis
of the Principle of Heap
Profiling](https://cn.pingcap.com/blog/an-explanation-of-the-heap-profiling-principle/).
It should be noted that Heap Profile records virtual memory
-
-Supports both real-time and periodic Heap Dump, and then uses `jeprof` to
parse the generated Heap Profile.
-
-###### 1. realtime heap dump
+##### 1. Real-time Heap Dump, used to analyze real-time memory
Change `prof:false` in `JEMALLOC_CONF` in `be.conf` to `prof:true`, change
`prof_active:false` to `prof_active:true` and restart Doris BE, then use the
Jemalloc Heap Dump HTTP interface to generate a Heap Profile file on the
corresponding BE machine.
> For Doris 2.1.8 and 3.0.4 and later versions, `prof` in `JEMALLOC_CONF` is
> already `true` by default, no need to modify.
-> For Doris versions before 2.1.8 and 3.0.4, there is no `prof_active` in
`JEMALLOC_CONF`, just change `prof:false` to `prof:true`.
+
+For Doris versions before 2.1.8 and 3.0.4, there is no `prof_active` in
`JEMALLOC_CONF`, just change `prof:false` to `prof:true`.
```shell
curl http://be_host:be_webport/jeheap/dump
@@ -241,9 +118,9 @@ The default sampling interval is 512K, which usually only
records 10% of the mem
If you are doing performance testing, keep `prof:false` to avoid the
performance loss of Heap Dump.
-###### 2. regular heap dump
+##### 2. Regular Heap Dump for long-term memory observation
-First, change `prof:false` of `JEMALLOC_CONF` in `be.conf` to `prof:true`. The
directory where the Heap Profile file is located defaults to
`${DORIS_HOME}/log`, and the file name prefix is `JEMALLOC_PROF_PRFIX` in
`be.conf`, which defaults to `jemalloc_heap_profile_`.
+Change `prof:false` of `JEMALLOC_CONF` in `be.conf` to `prof:true`. The
directory where the Heap Profile file is located is `${DORIS_HOME}/log` by
default. The file name prefix is `JEMALLOC_PROF_PRFIX` in `be.conf`, and the
default is `jemalloc_heap_profile_`.
> Before Doris 2.1.6, `JEMALLOC_PROF_PRFIX` is empty and needs to be changed
> to any value as the profile file name
@@ -264,52 +141,55 @@ Change `prof_leak` and `prof_final` in `JEMALLOC_CONF` in
`be.conf` to `true` an
4. Dump the cumulative value (growth) of memory instead of the real-time value:
Change `prof_accum` in `JEMALLOC_CONF` in `be.conf` to `true` and restart BE.
+
Use `jeprof --alloc_space` to display the cumulative value of heap dump.
##### 3. `jeprof` parses Heap Profile
-Use `be/bin/jeprof` to parse the Heap Profile of the above dump. If the
process memory is too large, the parsing process may take several minutes,
please wait patiently.
+Use `be/bin/jeprof` to parse the Heap Profile of the above dump. If the
process memory is too large, the parsing process may take several minutes.
Please wait patiently.
If there is no `jeprof` binary in the `be/bin` directory of the Doris BE
deployment path, you can package the `jeprof` in the `doris/tools` directory
and upload it to the server.
-> Requires addr2line version 2.35.2 and above, see QA-1 below for details
-> Try to have Heap Dump and `jeprof` analyze Heap Profile on the same server,
that is, analyze Heap Profile directly on the machine running Doris BE, see
QA-2 below for details
+> The addr2line version is required to be 2.35.2 or above, see QA-1 below for
details
+> Try to have Heap Dump and `jeprof` analyze Heap Profile on the same server,
that is, analyze Heap Profile directly on the machine running Doris BE as much
as possible, see QA-2 below for details
1. Analyze a single Heap Profile file
```shell
-jeprof --dot lib/doris_be heap_dump_file_1
+jeprof --dot ${DORIS_HOME}/lib/doris_be ${DORIS_HOME}/log/profile_file
```
-After executing the above command, a dot syntax graph will be output in the
terminal. Paste it to the [online dot drawing
website](http://www.webgraphviz.com/) to generate a memory allocation graph,
and then analyze it.
+After executing the above command, paste the text output by the terminal to
the [online dot drawing website](http://www.webgraphviz.com/) to generate a
memory allocation graph, and then analyze it.
-If the server is convenient for file transfer, you can also use the following
command to directly generate a call relationship graph result.pdf file and
transfer it to the local computer for viewing. You need to install the
dependencies required for drawing.
+If the server is convenient for file transfer, you can also use the following
command to directly generate a call relationship graph. The result.pdf file is
transferred to the local computer for viewing. You need to install the
dependencies required for drawing.
```shell
yum install ghostscript graphviz
-jeprof --pdf lib/doris_be heap_dump_file_1 > result.pdf
+jeprof --pdf ${DORIS_HOME}/lib/doris_be ${DORIS_HOME}/log/profile_file >
result.pdf
```
-2. Analyze the diff of two Heap Profile files
+[graphviz](http://www.graphviz.org/): Without this library, pprof can only be
converted to text format, but this method is not easy to view. After installing
this library, pprof can be converted to svg, pdf and other formats, and the
call relationship is clearer.
+
+2. Analyze the diff of two heap profile files
```shell
-jeprof --dot lib/doris_be --base=heap_dump_file_1 heap_dump_file_2
+jeprof --dot ${DORIS_HOME}/lib/doris_be --base=${DORIS_HOME}/log/profile_file
${DORIS_HOME}/log/profile_file2
```
-Multiple heap files can be generated by running the above command multiple
times over a period of time. You can select an earlier heap file as a baseline
and compare it with a later heap file to analyze their diff. The method for
generating a call relationship graph is the same as above.
+Multiple heap files can be generated by running the above command multiple
times over a period of time. You can select an earlier heap file as a baseline
and compare and analyze their diff with a later heap file. The method for
generating a call graph is the same as above.
##### 4. QA
-1. Many errors occurred after running jeprof: `addr2line: Dwarf Error: found
dwarf version xxx, this reader only handles version xxx`
+1. Many errors appear after running jeprof: `addr2line: Dwarf Error: found
dwarf version xxx, this reader only handles version xxx`.
-After GCC 11, DWARF-v5 is used by default, which requires Binutils 2.35.2 and
above. Doris Ldb_toolchain uses GCC 11. See:
https://gcc.gnu.org/gcc-11/changes.html.
+GCC 11 and later use DWARF-v5 by default, which requires Binutils 2.35.2 and
above. Doris Ldb_toolchain uses GCC 11. See:
https://gcc.gnu.org/gcc-11/changes.html.
Replace addr2line to 2.35.2, refer to:
```
// Download addr2line source code
wget https://ftp.gnu.org/gnu/binutils/binutils-2.35.tar.bz2
-//Install dependencies, if needed
+// Install dependencies, if necessary
yum install make gcc gcc-c++ binutils
// Compile & install addr2line
@@ -319,7 +199,7 @@ cd binutils-2.35
make
make install
-// verify
+// Verify
addr2line -h
// Replace addr2line
@@ -330,22 +210,23 @@ cp addr2line /bin/addr2line
cp addr2line /usr/bin/addr2line
hash -r
```
+Note that addr2line 2.3.9 cannot be used, which may be incompatible and cause
the memory to keep growing.
-Note that you cannot use addr2line 2.3.9, which may be incompatible and cause
memory to keep growing.
+2. Many errors appear after running `jeprof`: `addr2line: DWARF error: invalid
or unhandled FORM value: 0x25`, and the parsed Heap stack is the memory address
of the code, not the function name
-2. After running `jeprof`, many errors appear: `addr2line: DWARF error:
invalid or unhandled FORM value: 0x25`. The parsed Heap stack is the memory
address of the code, not the function name
+Usually, it is because the execution of Heap Dump and the execution of
`jeprof` to parse Heap Profile are not on the same server, which causes
`jeprof` to fail to parse the function name using the symbol table. Try to
complete the operation of Dump Heap and `jeprof` parsing on the same machine,
that is, try to parse the Heap Profile directly on the machine running Doris BE.
-Usually, it is because the execution of Heap Dump and the execution of
`jeprof` to parse Heap Profile are not on the same server, which causes
`jeprof` to fail to parse the function name using the symbol table. Try to
complete the operation of Dump Heap and `jeprof` parsing on the same machine,
that is, parse the Heap Profile directly on the machine running Doris BE as
much as possible.
-
-Or confirm the Linux kernel version of the machine running Doris BE, download
the `be/bin/doris_be` binary file and the Heap Profile file to the machine with
the same kernel version to execute `jeprof`.
+Or confirm the Linux kernel version of the machine running Doris BE, download
the `be/bin/doris_be` binary file and the Heap Profile file to the machine with
the same kernel version and execute `jeprof`.
3. If the Heap stack after directly parsing the Heap Profile on the machine
running Doris BE is still the memory address of the code, not the function name
Use the following script to manually parse the Heap Profile and modify these
variables:
- heap: the file name of the Heap Profile.
-- bin: `be/bin/doris_be` binary file name
-- llvm_symbolizer: Path to the llvm symbol table parser, preferably the
version used to compile the `be/bin/doris_be` binary.
+
+- bin: the file name of the `be/bin/doris_be` binary
+
+- llvm_symbolizer: the path of the llvm symbol table parser, the version
should preferably be the version used to compile the `be/bin/doris_be` binary.
```
#!/bin/bash
@@ -404,98 +285,138 @@ fi
- After the above operation, if the Heap stack is still the memory address of
the code, try `USE_JEMALLOC=OFF ./build.sh --be` to compile Doris BE using
TCMalloc, and then refer to the above section to use TCMalloc Heap Profile to
analyze memory.
-#### LSAN
+#### TCMalloc HEAP PROFILE
-[LSAN](https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer)
is an address checking tool, GCC has been integrated. When we compile the
code, we can enable this function by turning on the corresponding compilation
options. When the program has a determinable memory leak, it prints the leak
stack. Doris be has integrated this tool, only need to compile with the
following command to generate be binary with memory leak detection version.
+> Doris 1.2.1 and earlier versions use TCMalloc. Doris 1.2.2 version uses
Jemalloc by default. To switch to TCMalloc, you can compile like this:
`USE_JEMALLOC=OFF sh build.sh --be`.
+
+When using TCMalloc, when a large memory application is encountered, the
application stack will be printed to the be.out file, and the general
expression is as follows:
```
-BUILD_TYPE=LSAN ./build.sh
+tcmalloc: large alloc 1396277248 bytes == 0x3f3488000 @ 0x2af6f63 0x2c4095b
0x134d278 0x134bdcb 0x133d105 0x133d1d0 0x19930ed
```
-When the system detects a memory leak, it will output the corresponding
information in be. Out. For the following demonstration, we intentionally
insert a memory leak code into the code. We insert the following code into the
`open` function of `StorageEngine`.
+This indicates that Doris be is trying to apply memory of '1396277248 bytes'
on this stack. We can use the 'addr2line' command to restore the stack to a
letter that we can understand. The specific example is shown below.
```
- char* leak_buf = new char[1024];
- strcpy(leak_buf, "hello world");
- LOG(INFO) << leak_buf;
+$ addr2line -e lib/doris_be 0x2af6f63 0x2c4095b 0x134d278 0x134bdcb 0x133d105
0x133d1d0 0x19930ed
+
+/home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1335
+/home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1357
+/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.cpp:267
+/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.hpp:86
+/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:239
+/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:213
+thread.cpp:?
```
-We get the following output in be.out
+Sometimes the application of memory is not caused by the application of large
memory, but by the continuous accumulation of small memory. Then there is no
way to locate the specific application information by viewing the log, so you
need to get the information through other ways.
+
+At this time, we can take advantage of TCMalloc's
[heapprofile](https://gperftools.github.io/gperftools/heapprofile.html). If the
heapprofile function is set, we can get the overall memory application usage of
the process. The usage is to set the 'heapprofile' environment variable before
starting Doris be. For example:
```
-=================================================================
-==24732==ERROR: LeakSanitizer: detected memory leaks
+export HEAPPROFILE=/tmp/doris_be.hprof
+./bin/start_be.sh --daemon
+```
-Direct leak of 1024 byte(s) in 1 object(s) allocated from:
- #0 0xd10586 in operator new[](unsigned long)
../../../../gcc-7.3.0/libsanitizer/lsan/lsan_interceptors.cc:164
- #1 0xe333a2 in doris::StorageEngine::open(doris::EngineOptions const&,
doris::StorageEngine**)
/home/ssd0/zc/palo/doris/core/be/src/olap/storage_engine.cpp:104
- #2 0xd3cc96 in main
/home/ssd0/zc/palo/doris/core/be/src/service/doris_main.cpp:159
- #3 0x7f573b5eebd4 in __libc_start_main
(/opt/compiler/gcc-4.8.2/lib64/libc.so.6+0x21bd4)
+In this way, when the dump condition of the heapprofile is met, the overall
memory usage will be written to the file in the specified path. Later, we can
use the 'pprof' tool to analyze the output content.
-SUMMARY: LeakSanitizer: 1024 byte(s) leaked in 1 allocation(s).
```
+$ pprof --text lib/doris_be /tmp/doris_be.hprof.0012.heap | head -30
-From the above output, we can see that 1024 bytes have been leaked, and the
stack information of memory application has been printed out.
+Using local file lib/doris_be.
+Using local file /tmp/doris_be.hprof.0012.heap.
+Total: 668.6 MB
+ 610.6 91.3% 91.3% 610.6 91.3%
doris::SystemAllocator::allocate_via_malloc (inline)
+ 18.1 2.7% 94.0% 18.1 2.7% _objalloc_alloc
+ 5.6 0.8% 94.9% 63.4 9.5% doris::RowBatch::RowBatch
+ 5.1 0.8% 95.6% 7.1 1.1% butil::ResourcePool::add_block (inline)
+ 3.7 0.5% 96.2% 3.7 0.5% butil::iobuf::create_block (inline)
+ 3.4 0.5% 96.7% 3.4 0.5% butil::FlatMap::init
+ 3.2 0.5% 97.2% 5.2 0.8% butil::ObjectPool::add_block (inline)
+ 2.6 0.4% 97.6% 2.6 0.4% __gnu_cxx::new_allocator::allocate
(inline)
+ 2.0 0.3% 97.9% 2.0 0.3% butil::ObjectPool::add_block_group
(inline)
+ 2.0 0.3% 98.2% 2.0 0.3% butil::ResourcePool::add_block_group
(inline)
+ 1.7 0.3% 98.4% 1.7 0.3% doris::SegmentReader::_load_index
+```
+
+Contents of each column of the above documents:
+
+* Column 1: the memory size directly applied by the function, in MB
+* Column 4: the total memory size of the function and all the functions it
calls.
+* The second column and the fifth column are the proportion values of the
first column and the fourth column respectively.
+* The third column is the cumulative value of the second column.
+
+Of course, it can also generate call relation pictures, which is more
convenient for analysis. For example, the following command can generate a call
graph in SVG format.
+
+```
+pprof --svg lib/doris_be /tmp/doris_be.hprof.0012.heap > heap.svg
+```
**NOTE: turning on this option will affect the execution performance of the
program. Please be careful to turn on the online instance.**
-**NOTE: if the LSAN switch is turned on, the TCMalloc will be automatically
turned off**
+##### pprof remote server
-#### JEMALLOC HEAP PROFILE
+Although heapprofile can get all the memory usage information, it has some
limitations. 1. Restart be. 2. You need to enable this command all the time,
which will affect the performance of the whole process.
-##### 1. runtime heap dump by http
-Add `,prof:true,lg_prof_sample:10` to `JEMALLOC_CONF` in `start_be.sh` and
restart BE, then use the jemalloc heap dump http interface to generate a heap
dump file on the corresponding BE machine.
+For Doris be, you can also use the way of opening and closing the heap profile
dynamically to analyze the memory application of the process. Doris supports
the [remote server debugging of
gperftools](https://gperftools.github.io/gperftools/pprof_remote_servers.html).
Then you can use 'pprof' to directly perform dynamic head profile on the remote
running Doris be. For example, we can check the memory usage increment of Doris
through the following command
-The directory where the heap dump file is located can be configured through
the ``jeprofile_dir`` variable in ``be.conf``, and the default is
``${DORIS_HOME}/log``
+```
+$ pprof --text --seconds=60 http://be_host:be_webport/pprof/heap
-```shell
-curl http://be_host:be_webport/jeheap/dump
+Total: 1296.4 MB
+ 484.9 37.4% 37.4% 484.9 37.4% doris::StorageByteBuffer::create
+ 272.2 21.0% 58.4% 273.3 21.1% doris::RowBlock::init
+ 157.5 12.1% 70.5% 157.5 12.1% doris::RowBatch::RowBatch
+ 90.7 7.0% 77.5% 90.7 7.0%
doris::SystemAllocator::allocate_via_malloc
+ 66.6 5.1% 82.7% 66.6 5.1% doris::IntegerColumnReader::init
+ 47.9 3.7% 86.4% 47.9 3.7% __gnu_cxx::new_allocator::allocate
+ 20.8 1.6% 88.0% 35.4 2.7% doris::SegmentReader::_load_index
+ 12.7 1.0% 89.0% 12.7 1.0% doris::DecimalColumnReader::init
+ 12.7 1.0% 89.9% 12.7 1.0% doris::LargeIntColumnReader::init
+ 12.7 1.0% 90.9% 12.7 1.0% doris::StringColumnDirectReader::init
+ 12.3 0.9% 91.9% 12.3 0.9% std::__cxx11::basic_string::_M_mutate
+ 10.4 0.8% 92.7% 10.4 0.8%
doris::VectorizedRowBatch::VectorizedRowBatch
+ 10.0 0.8% 93.4% 10.0 0.8%
doris::PlainTextLineReader::PlainTextLineReader
```
-`prof`: After opening, jemalloc will generate a heap dump file according to
the current memory usage. There is a small amount of performance loss in heap
profile sampling, which can be turned off during performance testing.
-`lg_prof_sample`: heap profile sampling interval, the default value is 19,
that is, the default sampling interval is 512K (2^19 B), which will result in
only 10% of the memory recorded by the heap profile, `lg_prof_sample:10` can
reduce the sampling interval to 1K (2^10 B), more frequent sampling will make
the heap profile close to real memory, but this will bring greater performance
loss.
+The output of this command is the same as the output and view mode of heap
profile, which will not be described in detail here. Statistics will be enabled
only during execution of this command, which has a limited impact on process
performance compared with heap profile.
-For detailed parameter description, refer to
https://linux.die.net/man/3/jemalloc.
+#### LSAN
-#### 2. jemalloc heap dump profiling
+[LSAN](https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer)
is an address checking tool, GCC has been integrated. When we compile the
code, we can enable this function by turning on the corresponding compilation
options. When the program has a determinable memory leak, it prints the leak
stack. Doris be has integrated this tool, only need to compile with the
following command to generate be binary with memory leak detection version.
-3.1 Generating plain text analysis results
-```shell
-jeprof lib/doris_be --base=heap_dump_file_1 heap_dump_file_2
+```
+BUILD_TYPE=LSAN ./build.sh
```
-3.2 Generate call relationship picture
-
- Install dependencies required for plotting
- ```shell
- yum install ghostscript graphviz
- ```
- Multiple dump files can be generated by running the above command multiple
times in a short period of time, and the first dump file can be selected as the
baseline for diff comparison analysis
+When the system detects a memory leak, it will output the corresponding
information in be. Out. For the following demonstration, we intentionally
insert a memory leak code into the code. We insert the following code into the
`open` function of `StorageEngine`.
- ```shell
- jeprof --dot lib/doris_be --base=heap_dump_file_1 heap_dump_file_2
- ```
- After executing the above command, the terminal will output a diagram of dot
syntax, and paste it to [online dot drawing
website](http://www.webgraphviz.com/), generate a memory allocation diagram,
and then analyze it. This method can Drawing directly through the terminal
output results is more suitable for servers where file transfer is not very
convenient.
+```
+ char* leak_buf = new char[1024];
+ strcpy(leak_buf, "hello world");
+ LOG(INFO) << leak_buf;
+```
- You can also use the following command to directly generate the call
relationship result.pdf file and transfer it to the local for viewing
- ```shell
- jeprof --pdf lib/doris_be --base=heap_dump_file_1 heap_dump_file_2 >
result.pdf
- ```
+We get the following output in be.out
-In the above jeprof related commands, remove the `--base` option to analyze
only a single heap dump file
+```
+=================================================================
+==24732==ERROR: LeakSanitizer: detected memory leaks
-##### 3. heap dump by JEMALLOC_CONF
-Perform heap dump by restarting BE after changing the `JEMALLOC_CONF` variable
in `start_be.sh`
+Direct leak of 1024 byte(s) in 1 object(s) allocated from:
+ #0 0xd10586 in operator new[](unsigned long)
../../../../gcc-7.3.0/libsanitizer/lsan/lsan_interceptors.cc:164
+ #1 0xe333a2 in doris::StorageEngine::open(doris::EngineOptions const&,
doris::StorageEngine**)
/home/ssd0/zc/palo/doris/core/be/src/olap/storage_engine.cpp:104
+ #2 0xd3cc96 in main
/home/ssd0/zc/palo/doris/core/be/src/service/doris_main.cpp:159
+ #3 0x7f573b5eebd4 in __libc_start_main
(/opt/compiler/gcc-4.8.2/lib64/libc.so.6+0x21bd4)
-1. Dump every 1MB:
+SUMMARY: LeakSanitizer: 1024 byte(s) leaked in 1 allocation(s).
+```
- Two new variable settings `prof:true,lg_prof_interval:20` have been added
to the `JEMALLOC_CONF` variable, where `prof:true` is to enable profiling, and
`lg_prof_interval:20` means that a dump is generated every 1MB (2^20)
-2. Dump each time a new high is reached:
+From the above output, we can see that 1024 bytes have been leaked, and the
stack information of memory application has been printed out.
- Added two variable settings `prof:true,prof_gdump:true` in the
`JEMALLOC_CONF` variable, where `prof:true` is to enable profiling, and
`prof_gdump:true` means to generate a dump when the memory usage reaches a new
high
-3. Memory leak dump when the program exits:
+**NOTE: turning on this option will affect the execution performance of the
program. Please be careful to turn on the online instance.**
- Added three new variable settings `prof_leak: true, lg_prof_sample: 0,
prof_final: true` in the `JEMALLOC_CONF` variable
+**NOTE: if the LSAN switch is turned on, the TCMalloc will be automatically
turned off**
#### ASAN
@@ -556,6 +477,8 @@ When the CPU idle of the system is very low, it means that
the CPU of the system
#### pprof
+[pprof](https://github.com/google/pprof): from gperftools, it is used to
transform the content generated by gperftools into a format that is easy for
people to read, such as PDF, SVG, text, etc.
+
Because Doris has integrated and compatible with GPERF rest interface, users
can analyze remote Doris be through the 'pprof' tool. The specific usage is as
follows:
```
@@ -570,6 +493,10 @@ In this way, a CPU consumption graph of be execution can
be generated.
This is a quite common CPU analysis method. Compared with `pprof`, this method
must be able to log in to the physical machine of the analysis object. However,
compared with pprof, which can only collect points on time, perf can collect
stack information through different events. The specific usage is as follows:
+[perf](https://perf.wiki.kernel.org/index.php/main_page): Linux kernel comes
with performance analysis tool. [here](http://www.brendangregg.com/perf.html)
there are some examples of perf usage.
+
+[flamegraph](https://github.com/brendangregg/flamegraph): a visualization tool
used to show the output of perf in the form of flame graph.
+
```
perf record -g -p be_pid -- sleep 60
```
diff --git
a/docs/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
b/docs/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
index 48f54edcc59..cdaf1f1b367 100644
---
a/docs/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
+++
b/docs/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
@@ -24,7 +24,17 @@ specific language governing permissions and limitations
under the License.
-->
-Heap Profile supports real-time viewing of process memory usage and call
stacks, so this usually requires some understanding of the code. Doris uses
Jemalloc as the default Allocator. For the usage of Jemalloc Heap Profile,
refer to [Jemalloc Heap
Profile](https://doris.apache.org/community/developer-guide/debug-tool/?_highlight=debug#jemalloc-1).
It should be noted that Heap Profile records virtual memory. You need to
modify the configuration and restart the Doris BE process, and the ph [...]
+Heap Profile supports real-time viewing of process memory usage and call
stacks, so this usually requires some understanding of the code. It should be
noted that Heap Profile records virtual memory. You need to modify the
configuration and restart the Doris BE process, and the phenomenon can be
reproduced.
+
+Doris uses Jemalloc as the default Allocator. Refer to the following method to
use Heap Profile.
+
+1. Change `prof_active:false` of `JEMALLOC_CONF` in `be.conf` to
`prof_active:true` and restart Doris BE.
+
+2. After executing `curl http://be_host:8040/jeheap/dump`, you will see the
generated `profile` file in the `${DORIS_HOME}/log` directory.
+
+3. After executing `jeprof --dot ${DORIS_HOME}/lib/doris_be
${DORIS_HOME}/log/profile_file`, paste the text output by the terminal to the
[online dot drawing website](http://www.webgraphviz.com/) to generate a memory
allocation graph.
+
+The above process is based on Doris 2.1.8 and 3.0.4 and later versions, which
are used for real-time memory analysis. If you need to observe memory for a
long time or observe the cumulative value of memory application, please refer
to [Jemalloc Heap
Profile](https://doris.apache.org/community/developer-guide/debug-tool/?_highlight=debug#jemalloc-1)
for more information about the use of Jemalloc Heap Profile.
If you see the `Segment`, `TabletSchema`, and `ColumnReader` fields in the
call stack with a large memory share of Heap Profile, it means that the
metadata occupies a large amount of memory.
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/developer-guide/debug-tool.md
b/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/developer-guide/debug-tool.md
index 26e9018daeb..9d8c15a514b 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/developer-guide/debug-tool.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/developer-guide/debug-tool.md
@@ -89,147 +89,19 @@ FE 是 Java 进程。这里只列举一下简单常用的 java 调试命令。
## BE 调试
-### 环境准备
-
-[pprof](https://github.com/google/pprof):
来自gperftools，用于将gperftools所产生的内容转化成便于人可以阅读的格式，比如pdf, svg, text等.
-
-[graphviz](http://www.graphviz.org/):
在没有这个库的时候pprof只可以转化为text格式，但这种方式不易查看。那么安装这个库后，pprof可以转化为svg、pdf等格式，对于调用关系则更加清晰明了。
-
-[perf](https://perf.wiki.kernel.org/index.php/Main_Page):
linux内核自带性能分析工具。[这里](http://www.brendangregg.com/perf.html)有一些perf的使用例子。
-
-[FlameGraph](https://github.com/brendangregg/FlameGraph):
可视化工具，用于将perf的输出以火焰图的形式展示出来。
-
### 内存
对于内存的调试一般分为两个方面。一个是内存使用的总量是否合理，内存使用量过大一方面可能是由于系统存在内存泄露，另一方面可能是因为程序内存使用不当。其次就是是否存在内存越界、非法访问的问题，比如程序访问一个非法地址的内存，使用了未初始化内存等。对于内存方面的调试我们一般使用如下几种方式来进行问题追踪。
-Doris 1.2.1 及之前版本使用 TCMalloc，Doris 1.2.2 版本开始默认使用 Jemalloc，根据使用的 Doris
版本选择内存调试方法，如需切换 TCMalloc 可以这样编译 `USE_JEMALLOC=OFF sh build.sh --be`。
-
-#### 查看日志
-
-当发现内存使用量过大的时候，我们可以先查看 BE 日志，看看是否有大内存申请。
-
-###### TCMalloc
-
-当使用 TCMalloc 时，遇到大内存申请会将申请的堆栈打印到be.out文件中，一般的表现形式如下：
-
-```
-tcmalloc: large alloc 1396277248 bytes == 0x3f3488000 @ 0x2af6f63 0x2c4095b
0x134d278 0x134bdcb 0x133d105 0x133d1d0 0x19930ed
-```
-
-这个表示在Doris BE在这个堆栈上尝试申请`1396277248
bytes`的内存。我们可以通过`addr2line`命令去把堆栈还原成我们能够看懂的信，具体的例子如下所示。
-
-```
-$ addr2line -e lib/doris_be 0x2af6f63 0x2c4095b 0x134d278 0x134bdcb 0x133d105
0x133d1d0 0x19930ed
-
-/home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1335
-/home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1357
-/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.cpp:267
-/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.hpp:86
-/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:239
-/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:213
-thread.cpp:?
-```
-
-##### JEMALLOC
-
-Doris绝大多数的大内存申请都使用 Allocator，比如
HashTable、数据序列化，这部分内存申请是预期中的，会被有效管理起来，除此之外的大内存申请不被预期，会将申请的堆栈打印到 be.INFO
文件中，这通常用于调试，一般的表现形式如下：
-```
-MemHook alloc large memory: 8.2GB, stacktrace:
-Alloc Stacktrace:
- @ 0x55a6a5cf6b4d doris::ThreadMemTrackerMgr::consume()
- @ 0x55a6a5cf99bf malloc
- @ 0x55a6ae0caf98 operator new()
- @ 0x55a6a57cb013 doris::segment_v2::PageIO::read_and_decompress_page()
- @ 0x55a6a57719c0 doris::segment_v2::ColumnReader::read_page()
- ……
-```
-
-#### HEAP PROFILE
-
-##### TCMalloc
-
-有时内存的申请并不是大内存的申请导致，而是通过小内存不断的堆积导致的。那么就没有办法通过查看日志定位到具体的申请信息，那么就需要通过其他方式来获得信息。
-
-这个时候我们可以利用TCMalloc的[HEAP
PROFILE](https://gperftools.github.io/gperftools/heapprofile.html)的功能。如果设置了HEAPPROFILE功能，那么我们可以获得进程整体的内存申请使用情况。使用方式是在启动Doris
BE前设置`HEAPPROFILE`环境变量。比如：
-
-```
-export TCMALLOC_SAMPLE_PARAMETER=64000 HEAP_PROFILE_ALLOCATION_INTERVAL=-1
HEAP_PROFILE_INUSE_INTERVAL=-1 HEAP_PROFILE_TIME_INTERVAL=5
HEAPPROFILE=/tmp/doris_be.hprof
-./bin/start_be.sh --daemon
-```
-
-> 需要注意，HEAPPROFILE 需要是绝对路径，且已经存在。
-
-这样，当满足HEAPPROFILE的dump条件时，就会将内存的整体使用情况写到指定路径的文件中。后续我们就可以通过使用`pprof`工具来对输出的内容进行分析。
-
-```
-$ pprof --text lib/doris_be /tmp/doris_be.hprof.0012.heap | head -30
-
-Using local file lib/doris_be.
-Using local file /tmp/doris_be.hprof.0012.heap.
-Total: 668.6 MB
- 610.6 91.3% 91.3% 610.6 91.3%
doris::SystemAllocator::allocate_via_malloc (inline)
- 18.1 2.7% 94.0% 18.1 2.7% _objalloc_alloc
- 5.6 0.8% 94.9% 63.4 9.5% doris::RowBatch::RowBatch
- 5.1 0.8% 95.6% 7.1 1.1% butil::ResourcePool::add_block (inline)
- 3.7 0.5% 96.2% 3.7 0.5% butil::iobuf::create_block (inline)
- 3.4 0.5% 96.7% 3.4 0.5% butil::FlatMap::init
- 3.2 0.5% 97.2% 5.2 0.8% butil::ObjectPool::add_block (inline)
- 2.6 0.4% 97.6% 2.6 0.4% __gnu_cxx::new_allocator::allocate
(inline)
- 2.0 0.3% 97.9% 2.0 0.3% butil::ObjectPool::add_block_group
(inline)
- 2.0 0.3% 98.2% 2.0 0.3% butil::ResourcePool::add_block_group
(inline)
- 1.7 0.3% 98.4% 1.7 0.3% doris::SegmentReader::_load_index
-```
-
-上述文件各个列的内容：
-
-* 第一列：函数直接申请的内存大小，单位MB
-* 第四列：函数以及函数所有调用的函数总共内存大小。
-* 第二列、第五列分别是第一列与第四列的比例值。
-* 第三列是个第二列的累积值。
-
-当然也可以生成调用关系图片，更加方便分析。比如下面的命令就能够生成SVG格式的调用关系图。
-
-```
-pprof --svg lib/doris_be /tmp/doris_be.hprof.0012.heap > heap.svg
-```
-
-**注意：开启这个选项是要影响程序的执行性能的，请慎重对线上的实例开启**
-
-###### pprof remote server
-
-HEAP PROFILE虽然能够获得全部的内存使用信息，但是也有比较受限的地方。1. 需要重启BE进行。2.
需要一直开启这个命令，导致对整个进程的性能造成影响。
-
-对Doris BE也可以使用动态开启、关闭heap
profile的方式来对进程进行内存申请分析。Doris内部支持了GPerftools的[远程server调试](https://gperftools.github.io/gperftools/pprof_remote_servers.html)。那么可以通过`pprof`直接对远程运行的Doris
BE进行动态的HEAP PROFILE。比如我们可以通过以下命令来查看Doris的内存的使用增量
-
-```
-$ pprof --text --seconds=60 http://be_host:be_webport/pprof/heap
-
-Total: 1296.4 MB
- 484.9 37.4% 37.4% 484.9 37.4% doris::StorageByteBuffer::create
- 272.2 21.0% 58.4% 273.3 21.1% doris::RowBlock::init
- 157.5 12.1% 70.5% 157.5 12.1% doris::RowBatch::RowBatch
- 90.7 7.0% 77.5% 90.7 7.0%
doris::SystemAllocator::allocate_via_malloc
- 66.6 5.1% 82.7% 66.6 5.1% doris::IntegerColumnReader::init
- 47.9 3.7% 86.4% 47.9 3.7% __gnu_cxx::new_allocator::allocate
- 20.8 1.6% 88.0% 35.4 2.7% doris::SegmentReader::_load_index
- 12.7 1.0% 89.0% 12.7 1.0% doris::DecimalColumnReader::init
- 12.7 1.0% 89.9% 12.7 1.0% doris::LargeIntColumnReader::init
- 12.7 1.0% 90.9% 12.7 1.0% doris::StringColumnDirectReader::init
- 12.3 0.9% 91.9% 12.3 0.9% std::__cxx11::basic_string::_M_mutate
- 10.4 0.8% 92.7% 10.4 0.8%
doris::VectorizedRowBatch::VectorizedRowBatch
- 10.0 0.8% 93.4% 10.0 0.8%
doris::PlainTextLineReader::PlainTextLineReader
-```
-
-这个命令的输出与HEAP PROFILE的输出及查看方式一样，这里就不再详细说明。这个命令只有在执行的过程中才会开启统计，相比HEAP
PROFILE对于进程性能的影响有限。
+#### Jemalloc HEAP PROFILE
-##### JEMALLOC
+> Doris 1.2.2 版本开始默认使用 Jemalloc 作为内存分配器.
有关 Heap Profile 的原理解析参考 [Heap Profiling
原理解析](https://cn.pingcap.com/blog/an-explanation-of-the-heap-profiling-principle/)，需要注意的是
Heap Profile 记录的是虚拟内存
支持实时和定期两种方式 Heap Dump，然后使用 `jeprof` 解析生成的 Heap Profile。
-###### 1. 实时 Heap Dump
+##### 1. 实时 Heap Dump，用于分析实时内存
将 `be.conf` 中 `JEMALLOC_CONF` 的 `prof:false` 修改为 `prof:true`，将
`prof_active:false` 修改为 `prof_active:true` 并重启 Doris BE，然后使用 Jemalloc Heap Dump
HTTP 接口，在对应的BE机器上生成 Heap Profile 文件。
@@ -246,9 +118,9 @@ Heap Profile 文件所在目录可以在 `be.conf` 中通过 `jeprofile_dir` 变
如果你在做性能测试，保持 `prof:false` 来避免 Heap Dump 的性能损耗。
-###### 2. 定期 Heap Dump
+##### 2. 定期 Heap Dump，用于长时间观测内存
-首先将 `be.conf` 中 `JEMALLOC_CONF` 的 `prof:false` 修改为 `prof:true`，Heap Profile
文件所在目录默认为 `${DORIS_HOME}/log`, 文件名前缀是 `be.conf` 中的 `JEMALLOC_PROF_PRFIX`，默认是
`jemalloc_heap_profile_`。
+将 `be.conf` 中 `JEMALLOC_CONF` 的 `prof:false` 修改为 `prof:true`，Heap Profile
文件所在目录默认为 `${DORIS_HOME}/log`, 文件名前缀是 `be.conf` 中的 `JEMALLOC_PROF_PRFIX`，默认是
`jemalloc_heap_profile_`。
> 在 Doris 2.1.6 之前，`JEMALLOC_PROF_PRFIX` 为空，需要修改为任意值作为 profile 文件名
@@ -283,22 +155,24 @@ Heap Profile 文件所在目录可以在 `be.conf` 中通过 `jeprofile_dir` 变
1. 分析单个 Heap Profile 文件
```shell
- jeprof --dot lib/doris_be heap_dump_file_1
+ jeprof --dot ${DORIS_HOME}/lib/doris_be ${DORIS_HOME}/log/profile_file
```
-
执行完上述命令，终端中会输出dot语法的图，将其贴到[在线dot绘图网站](http://www.webgraphviz.com/)，生成内存分配图，然后进行分析。
+ 执行完上述命令后将终端输出的文本贴到[在线dot绘图网站](http://www.webgraphviz.com/)，生成内存分配图，然后进行分析。
如果服务器方便传输文件，也可以通过如下命令直接生成调用关系图 result.pdf 文件传输到本地后进行查看，需要安装绘图所需的依赖项。
```shell
yum install ghostscript graphviz
- jeprof --pdf lib/doris_be heap_dump_file_1 > result.pdf
+ jeprof --pdf ${DORIS_HOME}/lib/doris_be ${DORIS_HOME}/log/profile_file >
result.pdf
```
+ [graphviz](http://www.graphviz.org/):
在没有这个库的时候pprof只可以转化为text格式，但这种方式不易查看。那么安装这个库后，pprof可以转化为svg、pdf等格式，对于调用关系则更加清晰明了。
+
2. 分析两个 Heap Profile 文件的diff
```shell
- jeprof --dot lib/doris_be --base=heap_dump_file_1 heap_dump_file_2
+ jeprof --dot ${DORIS_HOME}/lib/doris_be
--base=${DORIS_HOME}/log/profile_file ${DORIS_HOME}/log/profile_file2
```
通过在一段时间内多次运行上述命令可以生成多个 heap 文件，可以选取较早时间的 heap 文件作为 baseline，与较晚时间的 heap
文件对比分析它们的diff，生成调用关系图的方法同上。
@@ -408,6 +282,104 @@ fi
- 上面的操作后，如果 Heap 栈依然是代码的内存地址，尝试 `USE_JEMALLOC=OFF ./build.sh --be` 编译使用
TCMalloc 的 Doris BE，然后参考上面的章节使用 TCMalloc Heap Profile 分析内存。
+#### TCMalloc HEAP PROFILE
+
+> Doris 1.2.1 及之前版本使用 TCMalloc，Doris 1.2.2 版本开始默认使用 Jemalloc，如需切换 TCMalloc
可以这样编译 `USE_JEMALLOC=OFF sh build.sh --be`。
+
+当使用 TCMalloc 时，遇到大内存申请会将申请的堆栈打印到be.out文件中，一般的表现形式如下：
+
+```
+tcmalloc: large alloc 1396277248 bytes == 0x3f3488000 @ 0x2af6f63 0x2c4095b
0x134d278 0x134bdcb 0x133d105 0x133d1d0 0x19930ed
+```
+
+这个表示在Doris BE在这个堆栈上尝试申请`1396277248
bytes`的内存。我们可以通过`addr2line`命令去把堆栈还原成我们能够看懂的信，具体的例子如下所示。
+
+```
+$ addr2line -e lib/doris_be 0x2af6f63 0x2c4095b 0x134d278 0x134bdcb 0x133d105
0x133d1d0 0x19930ed
+
+/home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1335
+/home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1357
+/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.cpp:267
+/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.hpp:86
+/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:239
+/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:213
+thread.cpp:?
+```
+
+有时内存的申请并不是大内存的申请导致，而是通过小内存不断的堆积导致的。那么就没有办法通过查看日志定位到具体的申请信息，那么就需要通过其他方式来获得信息。
+
+这个时候我们可以利用TCMalloc的[HEAP
PROFILE](https://gperftools.github.io/gperftools/heapprofile.html)的功能。如果设置了HEAPPROFILE功能，那么我们可以获得进程整体的内存申请使用情况。使用方式是在启动Doris
BE前设置`HEAPPROFILE`环境变量。比如：
+
+```
+export TCMALLOC_SAMPLE_PARAMETER=64000 HEAP_PROFILE_ALLOCATION_INTERVAL=-1
HEAP_PROFILE_INUSE_INTERVAL=-1 HEAP_PROFILE_TIME_INTERVAL=5
HEAPPROFILE=/tmp/doris_be.hprof
+./bin/start_be.sh --daemon
+```
+
+> 需要注意，HEAPPROFILE 需要是绝对路径，且已经存在。
+
+这样，当满足HEAPPROFILE的dump条件时，就会将内存的整体使用情况写到指定路径的文件中。后续我们就可以通过使用`pprof`工具来对输出的内容进行分析。
+
+```
+$ pprof --text lib/doris_be /tmp/doris_be.hprof.0012.heap | head -30
+
+Using local file lib/doris_be.
+Using local file /tmp/doris_be.hprof.0012.heap.
+Total: 668.6 MB
+ 610.6 91.3% 91.3% 610.6 91.3%
doris::SystemAllocator::allocate_via_malloc (inline)
+ 18.1 2.7% 94.0% 18.1 2.7% _objalloc_alloc
+ 5.6 0.8% 94.9% 63.4 9.5% doris::RowBatch::RowBatch
+ 5.1 0.8% 95.6% 7.1 1.1% butil::ResourcePool::add_block (inline)
+ 3.7 0.5% 96.2% 3.7 0.5% butil::iobuf::create_block (inline)
+ 3.4 0.5% 96.7% 3.4 0.5% butil::FlatMap::init
+ 3.2 0.5% 97.2% 5.2 0.8% butil::ObjectPool::add_block (inline)
+ 2.6 0.4% 97.6% 2.6 0.4% __gnu_cxx::new_allocator::allocate
(inline)
+ 2.0 0.3% 97.9% 2.0 0.3% butil::ObjectPool::add_block_group
(inline)
+ 2.0 0.3% 98.2% 2.0 0.3% butil::ResourcePool::add_block_group
(inline)
+ 1.7 0.3% 98.4% 1.7 0.3% doris::SegmentReader::_load_index
+```
+
+上述文件各个列的内容：
+
+* 第一列：函数直接申请的内存大小，单位MB
+* 第四列：函数以及函数所有调用的函数总共内存大小。
+* 第二列、第五列分别是第一列与第四列的比例值。
+* 第三列是个第二列的累积值。
+
+当然也可以生成调用关系图片，更加方便分析。比如下面的命令就能够生成SVG格式的调用关系图。
+
+```
+pprof --svg lib/doris_be /tmp/doris_be.hprof.0012.heap > heap.svg
+```
+
+**注意：开启这个选项是要影响程序的执行性能的，请慎重对线上的实例开启**
+
+##### pprof remote server
+
+HEAP PROFILE虽然能够获得全部的内存使用信息，但是也有比较受限的地方。1. 需要重启BE进行。2.
需要一直开启这个命令，导致对整个进程的性能造成影响。
+
+对Doris BE也可以使用动态开启、关闭heap
profile的方式来对进程进行内存申请分析。Doris内部支持了GPerftools的[远程server调试](https://gperftools.github.io/gperftools/pprof_remote_servers.html)。那么可以通过`pprof`直接对远程运行的Doris
BE进行动态的HEAP PROFILE。比如我们可以通过以下命令来查看Doris的内存的使用增量
+
+```
+$ pprof --text --seconds=60 http://be_host:be_webport/pprof/heap
+
+Total: 1296.4 MB
+ 484.9 37.4% 37.4% 484.9 37.4% doris::StorageByteBuffer::create
+ 272.2 21.0% 58.4% 273.3 21.1% doris::RowBlock::init
+ 157.5 12.1% 70.5% 157.5 12.1% doris::RowBatch::RowBatch
+ 90.7 7.0% 77.5% 90.7 7.0%
doris::SystemAllocator::allocate_via_malloc
+ 66.6 5.1% 82.7% 66.6 5.1% doris::IntegerColumnReader::init
+ 47.9 3.7% 86.4% 47.9 3.7% __gnu_cxx::new_allocator::allocate
+ 20.8 1.6% 88.0% 35.4 2.7% doris::SegmentReader::_load_index
+ 12.7 1.0% 89.0% 12.7 1.0% doris::DecimalColumnReader::init
+ 12.7 1.0% 89.9% 12.7 1.0% doris::LargeIntColumnReader::init
+ 12.7 1.0% 90.9% 12.7 1.0% doris::StringColumnDirectReader::init
+ 12.3 0.9% 91.9% 12.3 0.9% std::__cxx11::basic_string::_M_mutate
+ 10.4 0.8% 92.7% 10.4 0.8%
doris::VectorizedRowBatch::VectorizedRowBatch
+ 10.0 0.8% 93.4% 10.0 0.8%
doris::PlainTextLineReader::PlainTextLineReader
+```
+
+这个命令的输出与HEAP PROFILE的输出及查看方式一样，这里就不再详细说明。这个命令只有在执行的过程中才会开启统计，相比HEAP
PROFILE对于进程性能的影响有限。
+
#### LSAN
[LSAN](https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer)是一个地址检查工具，GCC已经集成。在我们编译代码的时候开启相应的编译选项，就能够开启这个功能。当程序发生可以确定的内存泄露时，会将泄露堆栈打印。Doris
BE已经集成了这个工具，只需要在编译的时候使用如下的命令进行编译就能够生成带有内存泄露检测版本的BE二进制
@@ -504,6 +476,8 @@ cat be.out | python asan_symbolize.py | c++filt
#### pprof
+[pprof](https://github.com/google/pprof):
来自gperftools，用于将gperftools所产生的内容转化成便于人可以阅读的格式，比如pdf, svg, text等.
+
由于Doris内部已经集成了并兼容了GPerf的REST接口，那么用户可以通过`pprof`工具来分析远程的Doris BE。具体的使用方式如下：
```
@@ -518,6 +492,10 @@ pprof --svg --seconds=60
http://be_host:be_webport/pprof/profile > be.svg
这个是相当通用的一种CPU分析方式，相比于`pprof`，这种方式必须要求能够登陆到分析对象的物理机上。但是相比于pprof只能定时采点，perf是能够通过不同的事件来完成堆栈信息采集的。具体的使用方式如下：
+[perf](https://perf.wiki.kernel.org/index.php/Main_Page):
linux内核自带性能分析工具。[这里](http://www.brendangregg.com/perf.html)有一些perf的使用例子。
+
+[FlameGraph](https://github.com/brendangregg/FlameGraph):
可视化工具，用于将perf的输出以火焰图的形式展示出来。
+
```
perf record -g -p be_pid -- sleep 60
```
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
index 07d89d1ec5f..351a930fa39 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
@@ -24,7 +24,17 @@ specific language governing permissions and limitations
under the License.
-->
-Heap Profile 支持实时查看进程内存使用，并可以看到调用栈，所以这通常需要对代码有一些了解，Doris 使用 Jemalloc 作为默认的
Allocator，有关 Jemalloc Heap Profile 的使用方法参考 [Jemalloc Heap
Profile](https://doris.apache.org/community/developer-guide/debug-tool/?_highlight=debug#jemalloc-1)，需要注意的是
Heap Profile 记录的是虚拟内存，需要修改配置后重启 Doris BE 进程，并且现象可以被复现。
+Heap Profile 支持实时查看进程内存使用，并可以看到调用栈，所以这通常需要对代码有一些了解，需要注意的是 Heap Profile
记录的是虚拟内存，需要修改配置后重启 Doris BE 进程，并且现象可以被复现。
+
+Doris 使用 Jemalloc 作为默认的 Allocator，参照下面的方法使用 Heap Profile。
+
+1. 将 `be.conf` 中 `JEMALLOC_CONF` 的 `prof_active:false` 修改为 `prof_active:true`
并重启 Doris BE。
+
+2. 执行 `curl http://be_host:8040/jeheap/dump` 后会在 `${DORIS_HOME}/log` 目录看到生成的
`profile` 文件。
+
+3. 执行 `jeprof --dot ${DORIS_HOME}/lib/doris_be ${DORIS_HOME}/log/profile_file`
后将终端输出的文本贴到[在线dot绘图网站](http://www.webgraphviz.com/)，生成内存分配图。
+
+以上流程基于 Doris 2.1.8 和 3.0.4 及之后的版本，用于实时的分析内存，如需长时间观测内存，或观测内存申请的累积值，更多有关
Jemalloc Heap Profile 的使用方法参考 [Jemalloc Heap
Profile](https://doris.apache.org/community/developer-guide/debug-tool/?_highlight=debug#jemalloc-1)
如果在 Heap Profile 内存占比大的调用栈中看到 `Segment`，`TabletSchema`、`ColumnReader`
字段，说明元数据占用内存大。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
index 07d89d1ec5f..351a930fa39 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
@@ -24,7 +24,17 @@ specific language governing permissions and limitations
under the License.
-->
-Heap Profile 支持实时查看进程内存使用，并可以看到调用栈，所以这通常需要对代码有一些了解，Doris 使用 Jemalloc 作为默认的
Allocator，有关 Jemalloc Heap Profile 的使用方法参考 [Jemalloc Heap
Profile](https://doris.apache.org/community/developer-guide/debug-tool/?_highlight=debug#jemalloc-1)，需要注意的是
Heap Profile 记录的是虚拟内存，需要修改配置后重启 Doris BE 进程，并且现象可以被复现。
+Heap Profile 支持实时查看进程内存使用，并可以看到调用栈，所以这通常需要对代码有一些了解，需要注意的是 Heap Profile
记录的是虚拟内存，需要修改配置后重启 Doris BE 进程，并且现象可以被复现。
+
+Doris 使用 Jemalloc 作为默认的 Allocator，参照下面的方法使用 Heap Profile。
+
+1. 将 `be.conf` 中 `JEMALLOC_CONF` 的 `prof_active:false` 修改为 `prof_active:true`
并重启 Doris BE。
+
+2. 执行 `curl http://be_host:8040/jeheap/dump` 后会在 `${DORIS_HOME}/log` 目录看到生成的
`profile` 文件。
+
+3. 执行 `jeprof --dot ${DORIS_HOME}/lib/doris_be ${DORIS_HOME}/log/profile_file`
后将终端输出的文本贴到[在线dot绘图网站](http://www.webgraphviz.com/)，生成内存分配图。
+
+以上流程基于 Doris 2.1.8 和 3.0.4 及之后的版本，用于实时的分析内存，如需长时间观测内存，或观测内存申请的累积值，更多有关
Jemalloc Heap Profile 的使用方法参考 [Jemalloc Heap
Profile](https://doris.apache.org/community/developer-guide/debug-tool/?_highlight=debug#jemalloc-1)
如果在 Heap Profile 内存占比大的调用栈中看到 `Segment`，`TabletSchema`、`ColumnReader`
字段，说明元数据占用内存大。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
index 07d89d1ec5f..351a930fa39 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
@@ -24,7 +24,17 @@ specific language governing permissions and limitations
under the License.
-->
-Heap Profile 支持实时查看进程内存使用，并可以看到调用栈，所以这通常需要对代码有一些了解，Doris 使用 Jemalloc 作为默认的
Allocator，有关 Jemalloc Heap Profile 的使用方法参考 [Jemalloc Heap
Profile](https://doris.apache.org/community/developer-guide/debug-tool/?_highlight=debug#jemalloc-1)，需要注意的是
Heap Profile 记录的是虚拟内存，需要修改配置后重启 Doris BE 进程，并且现象可以被复现。
+Heap Profile 支持实时查看进程内存使用，并可以看到调用栈，所以这通常需要对代码有一些了解，需要注意的是 Heap Profile
记录的是虚拟内存，需要修改配置后重启 Doris BE 进程，并且现象可以被复现。
+
+Doris 使用 Jemalloc 作为默认的 Allocator，参照下面的方法使用 Heap Profile。
+
+1. 将 `be.conf` 中 `JEMALLOC_CONF` 的 `prof_active:false` 修改为 `prof_active:true`
并重启 Doris BE。
+
+2. 执行 `curl http://be_host:8040/jeheap/dump` 后会在 `${DORIS_HOME}/log` 目录看到生成的
`profile` 文件。
+
+3. 执行 `jeprof --dot ${DORIS_HOME}/lib/doris_be ${DORIS_HOME}/log/profile_file`
后将终端输出的文本贴到[在线dot绘图网站](http://www.webgraphviz.com/)，生成内存分配图。
+
+以上流程基于 Doris 2.1.8 和 3.0.4 及之后的版本，用于实时的分析内存，如需长时间观测内存，或观测内存申请的累积值，更多有关
Jemalloc Heap Profile 的使用方法参考 [Jemalloc Heap
Profile](https://doris.apache.org/community/developer-guide/debug-tool/?_highlight=debug#jemalloc-1)
如果在 Heap Profile 内存占比大的调用栈中看到 `Segment`，`TabletSchema`、`ColumnReader`
字段，说明元数据占用内存大。
diff --git
a/versioned_docs/version-2.1/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
b/versioned_docs/version-2.1/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
index 48f54edcc59..cdaf1f1b367 100644
---
a/versioned_docs/version-2.1/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
+++
b/versioned_docs/version-2.1/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
@@ -24,7 +24,17 @@ specific language governing permissions and limitations
under the License.
-->
-Heap Profile supports real-time viewing of process memory usage and call
stacks, so this usually requires some understanding of the code. Doris uses
Jemalloc as the default Allocator. For the usage of Jemalloc Heap Profile,
refer to [Jemalloc Heap
Profile](https://doris.apache.org/community/developer-guide/debug-tool/?_highlight=debug#jemalloc-1).
It should be noted that Heap Profile records virtual memory. You need to
modify the configuration and restart the Doris BE process, and the ph [...]
+Heap Profile supports real-time viewing of process memory usage and call
stacks, so this usually requires some understanding of the code. It should be
noted that Heap Profile records virtual memory. You need to modify the
configuration and restart the Doris BE process, and the phenomenon can be
reproduced.
+
+Doris uses Jemalloc as the default Allocator. Refer to the following method to
use Heap Profile.
+
+1. Change `prof_active:false` of `JEMALLOC_CONF` in `be.conf` to
`prof_active:true` and restart Doris BE.
+
+2. After executing `curl http://be_host:8040/jeheap/dump`, you will see the
generated `profile` file in the `${DORIS_HOME}/log` directory.
+
+3. After executing `jeprof --dot ${DORIS_HOME}/lib/doris_be
${DORIS_HOME}/log/profile_file`, paste the text output by the terminal to the
[online dot drawing website](http://www.webgraphviz.com/) to generate a memory
allocation graph.
+
+The above process is based on Doris 2.1.8 and 3.0.4 and later versions, which
are used for real-time memory analysis. If you need to observe memory for a
long time or observe the cumulative value of memory application, please refer
to [Jemalloc Heap
Profile](https://doris.apache.org/community/developer-guide/debug-tool/?_highlight=debug#jemalloc-1)
for more information about the use of Jemalloc Heap Profile.
If you see the `Segment`, `TabletSchema`, and `ColumnReader` fields in the
call stack with a large memory share of Heap Profile, it means that the
metadata occupies a large amount of memory.
diff --git
a/versioned_docs/version-3.0/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
b/versioned_docs/version-3.0/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
index 48f54edcc59..cdaf1f1b367 100644
---
a/versioned_docs/version-3.0/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
+++
b/versioned_docs/version-3.0/admin-manual/trouble-shooting/memory-management/memory-analysis/heap-profile-memory-analysis.md
@@ -24,7 +24,17 @@ specific language governing permissions and limitations
under the License.
-->
-Heap Profile supports real-time viewing of process memory usage and call
stacks, so this usually requires some understanding of the code. Doris uses
Jemalloc as the default Allocator. For the usage of Jemalloc Heap Profile,
refer to [Jemalloc Heap
Profile](https://doris.apache.org/community/developer-guide/debug-tool/?_highlight=debug#jemalloc-1).
It should be noted that Heap Profile records virtual memory. You need to
modify the configuration and restart the Doris BE process, and the ph [...]
+Heap Profile supports real-time viewing of process memory usage and call
stacks, so this usually requires some understanding of the code. It should be
noted that Heap Profile records virtual memory. You need to modify the
configuration and restart the Doris BE process, and the phenomenon can be
reproduced.
+
+Doris uses Jemalloc as the default Allocator. Refer to the following method to
use Heap Profile.
+
+1. Change `prof_active:false` of `JEMALLOC_CONF` in `be.conf` to
`prof_active:true` and restart Doris BE.
+
+2. After executing `curl http://be_host:8040/jeheap/dump`, you will see the
generated `profile` file in the `${DORIS_HOME}/log` directory.
+
+3. After executing `jeprof --dot ${DORIS_HOME}/lib/doris_be
${DORIS_HOME}/log/profile_file`, paste the text output by the terminal to the
[online dot drawing website](http://www.webgraphviz.com/) to generate a memory
allocation graph.
+
+The above process is based on Doris 2.1.8 and 3.0.4 and later versions, which
are used for real-time memory analysis. If you need to observe memory for a
long time or observe the cumulative value of memory application, please refer
to [Jemalloc Heap
Profile](https://doris.apache.org/community/developer-guide/debug-tool/?_highlight=debug#jemalloc-1)
for more information about the use of Jemalloc Heap Profile.
If you see the `Segment`, `TabletSchema`, and `ColumnReader` fields in the
call stack with a large memory share of Heap Profile, it means that the
metadata occupies a large amount of memory.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
