This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 5f7a2284f7b [fix] Fix memory and arrow flight (#2126)
5f7a2284f7b is described below
commit 5f7a2284f7b60b92f1572f5b6cfde6a721089da7
Author: Xinyi Zou <zouxiny...@gmail.com>
AuthorDate: Fri Feb 28 14:44:46 2025 +0800
[fix] Fix memory and arrow flight (#2126)
## Versions
fix #2053
- [x] dev
- [x] 3.0
- [x] 2.1
- [x] 2.0
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
---
...in-apache-doris-for-10x-faster-data-transfer.md | 4 ++
docs/db-connect/arrow-flight-sql-connect.md | 6 +-
.../memory-analysis/doris-cache-memory-analysis.md | 64 +++++++++++-----------
.../memory-analysis/jemalloc-memory-analysis.md | 2 +-
.../current/db-connect/arrow-flight-sql-connect.md | 6 +-
.../db-connect/arrow-flight-sql-connect.md | 6 +-
.../db-connect/arrow-flight-sql-connect.md | 6 +-
.../db-connect/arrow-flight-sql-connect.md | 6 +-
.../db-connect/arrow-flight-sql-connect.md | 6 +-
9 files changed, 61 insertions(+), 45 deletions(-)
diff --git
a/blog/arrow-flight-sql-in-apache-doris-for-10x-faster-data-transfer.md
b/blog/arrow-flight-sql-in-apache-doris-for-10x-faster-data-transfer.md
index 4aeefadc0ab..22054a2a4a9 100644
--- a/blog/arrow-flight-sql-in-apache-doris-for-10x-faster-data-transfer.md
+++ b/blog/arrow-flight-sql-in-apache-doris-for-10x-faster-data-transfer.md
@@ -99,6 +99,10 @@ Configure parameters for Doris frontend (FE) and backend
(BE):
- In `be/conf/be.conf`, set `arrow_flight_sql_port ` to an available port,
such as 9091.
+`Note: The arrow_flight_sql_port port number configured in fe.conf and be.conf
is different`
+
+After modifying the configuration and restarting the cluster, searching for
`Arrow Flight SQL service is started` in the fe/log/fe.log file indicates that
the Arrow Flight Server of FE has been successfully started; searching for
`Arrow Flight Service bind to host` in the be/log/be.INFO file indicates that
the Arrow Flight Server of BE has been successfully started.
+
Suppose that the Arrow Flight SQL services for the Doris instance will run on
ports 9090 and 9091 for FE and BE respectively, and the Doris username/password
is "user" and "pass", the connection process would be:
```C++
diff --git a/docs/db-connect/arrow-flight-sql-connect.md
b/docs/db-connect/arrow-flight-sql-connect.md
index d3d99682e7e..a122ff16b8d 100644
--- a/docs/db-connect/arrow-flight-sql-connect.md
+++ b/docs/db-connect/arrow-flight-sql-connect.md
@@ -68,7 +68,9 @@ Modify the configuration parameters of Doris FE and BE:
- Modify arrow_flight_sql_port in fe/conf/fe.conf to an available port, such
as 9090.
- Modify arrow_flight_sql_port in be/conf/be.conf to an available port, such
as 9091.
-`Note: arrow_flight_sql_port configured in fe.conf and be.conf are different`
+`Note: The arrow_flight_sql_port port number configured in fe.conf and be.conf
is different`
+
+After modifying the configuration and restarting the cluster, searching for
`Arrow Flight SQL service is started` in the fe/log/fe.log file indicates that
the Arrow Flight Server of FE has been successfully started; searching for
`Arrow Flight Service bind to host` in the be/log/be.INFO file indicates that
the Arrow Flight Server of BE has been successfully started.
Assuming that the Arrow Flight SQL services of FE and BE in the Doris instance
will run on ports 9090 and 9091 respectively, and the Doris username/password
is "user"/"pass", the connection process is as follows:
@@ -281,7 +283,7 @@ The open source JDBC driver of Arrow Flight SQL protocol is
compatible with the
POM dependency:
```Java
<properties>
- <arrow.version>15.0.1</arrow.version>
+ <arrow.version>17.0.0</arrow.version>
</properties>
<dependencies>
<dependency>
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/doris-cache-memory-analysis.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/doris-cache-memory-analysis.md
index 2967e7036a5..7060d112944 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/doris-cache-memory-analysis.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/doris-cache-memory-analysis.md
@@ -1,6 +1,6 @@
---
{
- "title": "查询报错 Memory Tracker Limit Exceeded",
+ "title": "Cache 内存分析",
"language": "zh-CN"
}
---
@@ -24,49 +24,49 @@ specific language governing permissions and limitations
under the License.
-->
-当查询或导入的报错信息中出现 `MEM_LIMIT_EXCEEDED` 且包含 `memory tracker limit exceeded`
时，说明任务超过单次执行内存限制。
+Doris 自己管理的 Cache 目前均为 LRU 淘汰策略，均支持单独通过参数控制容量和淘汰时长。
-## 错误信息解析
+## Doris Cache 类型
-```sql
-ERROR 1105 (HY000): errCode = 2, detailMessage =
(10.16.10.8)[MEM_LIMIT_EXCEEDED]PreCatch error code:11, [E11] Allocator mem
tracker check failed, [MEM_LIMIT_EXCEEDED]failed alloc size 1.03 MB, memory
tracker limit exceeded, tracker
label:Query#Id=f78208b15e064527-a84c5c0b04c04fcf, type:query, limit 100.00 MB,
peak used 99.29 MB, current used 99.25 MB. backend 10.16.10.8, process memory
used 2.65 GB. exec node:<execute:<ExecNode:VHASH_JOIN_NODE (id=4)>>, can `set
exec_mem_limit=8G` to ch [...]
-```
+1. Page Cache
-错误信息分为两部分：
+用于加速数据扫描。
-1. `failed alloc size 1.03 MB, memory tracker limit exceeded, tracker
label:Query#Id=f78208b15e064527-a84c5c0b04c04fcf, type:query, limit 100.00 MB,
peak used 99.29 MB, current used 99.25 MB`：当前正在执行 Query
`f78208b15e064527-a84c5c0b04c04fcf` 在尝试申请 1.03 MB 内存的过程中发现查询超过单次执行的内存上限，查询内存上限是
100 MB（Session Variables 中的 `exec_mem_limit`），当前已经使用 99.25 MB，内存峰值是 99.29 MB。
+```
+- DataPageCache: 缓存数据 Page。
+- IndexPageCache: 缓存数据 Page 的索引。
+- PKIndexPageCache: 缓存 Page 的主键索引。
+```
-2. `backend 10.16.10.8, process memory used 2.65 GB. exec
node:<execute:<ExecNode:VHASH_JOIN_NODE (id=4)>>, can set exec_mem_limit=8G to
change limit, details see be.INFO.`：本次内存申请的位置是`VHASH_JOIN_NODE (id=4)`，并提示可通过
`set exec_mem_limit` 来调高单次查询的内存上限。
+2. Metadata Cache
-## 单次执行内存限制和内存超发
+用于加速元数据读取。
-`show variables;` 可以查看 Doris Session Veriable，其中的 `exec_mem_limit`
是单次查询和导入的执行内存限制，但从 Doris 1.2 开始支持查询内存超发
(overcommit)，旨在允许查询设置更灵活的内存限制，内存充足时即使查询内存超过上限也不会被
Cancel，所以通常用户无需关注查询内存使用。直到内存不足时，查询会在尝试分配新内存时等待一段时间，此时会基于一定规则优先 Cancel
`mem_used` 与 `exec_mem_limit` 比值大的
Query。如果等待过程中内存释放的大小满足需求，查询将继续执行，否则将抛出异常并终止查询。
+```
+- SegmentCache: 缓存已打开的 Segment，如索引信息。
+- SchemaCache: 缓存 Rowset Schema。
+- TabletSchemaCache: 缓存 Tablet Schema。
+- CreateTabletRRIdxCache: 缓存 Create Tabelt 索引。
+- MowTabletVersionCache: 缓存 Mow Tablet Version。
+- MowDeleteBitmapAggCache: 缓存 Mow DeleteBitmap。
+```
-如果希望关闭查询内存超发，参考 [BE 配置项](../../../config/be-config)，在 `conf/be.conf` 中增加
`enable_query_memory_overcommit=false`，此时单次查询和导入的内存超过 `exec_mem_limit` 即会被
Cancel。如果你希望避免大查询对集群稳定性造成的负面影响，或者希望准确控制集群上的任务执行来保证足够的稳定性，那么可以考虑关闭查询内存超发。
+3. Cloud Cache
-## 查询内存分析
+云上专用的缓存。
-如果需要分析查询的内存使用，参考 [查询内存分析](./query-memory-analysis.md)。
+```
+- CloudTabletCache: Cloud 上缓存 Tablet。
+- CloudTxnDeleteBitmapCache: Cloud 上缓存 DeleteBitmap。
+```
-`set enable_profile=true` 开启 Query Profile 后，在任务超过单次执行的内存上限时，在
`be/log/be.INFO` 将打印查询正在申请内存的调用栈，并可以看到查询每个算子当前使用的内存和峰值，参考
[内存日志分析](./memory-log-analysis.md) 分析 `Process Memory Summary` 和 `Memory
Tracker Summary`，帮助确认当前查询内存使用是否符合预期。
+4. Inverted Index Cache
-```sql
-Allocator mem tracker check failed, [MEM_LIMIT_EXCEEDED]failed alloc size
32.00 MB, memory tracker limit exceeded, tracker label:Query#I
-d=41363cb6ba734ad5-bc8720bdf9b3090d, type:query, limit 100.00 MB, peak used
75.32 MB, current used 72.62 MB. backend 10.16.10.8, process memory used 2.33
GB. exec node:<>, can `set exec_mem_limit=8G`
- to change limit, details see be.INFO.
-Process Memory Summary:
- os physical memory 375.81 GB. process memory used 2.33 GB(= 2.60
GB[vm/rss] - 280.53 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]),
limit 338.23 GB, soft limit 304.41 GB. sys availab
-le memory 337.33 GB(= 337.33 GB[proc/available] - 0[reserved] -
0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
-Memory Tracker Summary: MemTrackerLimiter
Label=Query#Id=41363cb6ba734ad5-bc8720bdf9b3090d, Type=query, Limit=100.00
MB(104857600 B), Used=72.62 MB(76146688 B), Peak=75.32 MB(78981248 B)
- MemTracker Label=HASH_JOIN_SINK_OPERATOR, Parent
Label=Query#Id=41363cb6ba734ad5-bc8720bdf9b3090d, Used=122.00 B(122 B),
Peak=122.00 B(122 B)
- MemTracker Label=VDataStreamRecvr:41363cb6ba734ad5-bc8720bdf9b309fe,
Parent Label=Query#Id=41363cb6ba734ad5-bc8720bdf9b3090d, Used=0(0 B),
Peak=384.00 B(384 B)
- MemTracker Label=local data queue mem tracker, Parent
Label=Query#Id=41363cb6ba734ad5-bc8720bdf9b3090d, Used=0(0 B), Peak=384.00
B(384 B)
- MemTracker Label=HASH_JOIN_SINK_OPERATOR, Parent
Label=Query#Id=41363cb6ba734ad5-bc8720bdf9b3090d, Used=21.73 MB(22790276 B),
Peak=21.73 MB(22790276 B)
- MemTracker Label=VDataStreamRecvr:41363cb6ba734ad5-bc8720bdf9b309fe,
Parent Label=Query#Id=41363cb6ba734ad5-bc8720bdf9b3090d, Used=0(0 B), Peak=2.23
MB(2342912 B)
- MemTracker Label=local data queue mem tracker, Parent
Label=Query#Id=41363cb6ba734ad5-bc8720bdf9b3090d, Used=0(0 B), Peak=2.23
MB(2342912 B)
- MemTracker Label=HASH_JOIN_SINK_OPERATOR, Parent
Label=Query#Id=41363cb6ba734ad5-bc8720bdf9b3090d, Used=24.03 MB(25201284 B),
Peak=24.03 MB(25201284 B)
- MemTracker Label=VDataStreamRecvr:41363cb6ba734ad5-bc8720bdf9b309fe,
Parent Label=Query#Id=41363cb6ba734ad5-bc8720bdf9b3090d, Used=1.08 MB(1130496
B), Peak=7.17 MB(7520256 B)
- MemTracker Label=local data queue mem tracker, Parent
Label=Query#Id=41363cb6ba734ad5-bc8720bdf9b3090d, Used=1.08 MB(1130496 B),
Peak=7.17 MB(7520256 B)
+加速倒排索引。
+
+```
+- InvertedIndexSearcherCache
+- InvertedIndexQueryCache
```
5. Point Query Cache
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md
index eaad76419f0..87146f95487 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md
@@ -123,6 +123,6 @@ Jemalloc Metadata 大小和进程虚拟内存大小正相关，通常 Doris BE
如果频繁出现上述问题，参考下面的方法。
-1. 一个根本解决方法是关闭 Jemalloc Retained 缓存虚拟内存映射，在 `be.conf` 中 `JEMALLOC_CONF` 后面增加
`retain:false` 后重启 BE。但查询性能可能会明显降低，测试 TPC-H Benchmark 性能会降低 3 倍左右。
+1. 一个根本解决方法是关闭 Jemalloc Retained 缓存虚拟内存映射，在 `be.conf` 中 `JEMALLOC_CONF` 后面增加
`retain:false` 后重启 BE，但查询性能可能会降低，关闭后观察一段时间集群的性能变化。
2. Doris 2.1 上可以关闭 Pipelinex 和 Pipeline，执行 `set global
experimental_enable_pipeline_engine=false; set global
experimental_enable_pipeline_x_engine=false;`，因为 pipelinex 和 pipeline
会申请更多的虚拟内存。这同样会导致查询性能降低。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/db-connect/arrow-flight-sql-connect.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/db-connect/arrow-flight-sql-connect.md
index 4245a577955..21f0b1eb3f2 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/db-connect/arrow-flight-sql-connect.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/db-connect/arrow-flight-sql-connect.md
@@ -68,7 +68,9 @@ import adbc_driver_flightsql.dbapi as flight_sql
- 修改 fe/conf/fe.conf 中 arrow_flight_sql_port 为一个可用端口，如 9090。
- 修改 be/conf/be.conf中 arrow_flight_sql_port 为一个可用端口，如 9091。
-`注: fe.conf 与 be.conf 中配置的 arrow_flight_sql_port 不相同`
+`注: fe.conf 与 be.conf 中配置的 arrow_flight_sql_port 端口号不相同`
+
+修改配置并重启集群后，在 fe/log/fe.log 文件中搜索到 `Arrow Flight SQL service is started` 表明 FE
的 Arrow Flight Server 启动成功；在 be/log/be.INFO 文件中搜索到 `Arrow Flight Service bind
to host` 表明 BE 的 Arrow Flight Server 启动成功。
假设 Doris 实例中 FE 和 BE 的 Arrow Flight SQL 服务将分别在端口 9090 和 9091 上运行，且 Doris
用户名/密码为“user”/“pass”，那么连接过程如下所示：
@@ -281,7 +283,7 @@ Arrow Flight SQL 协议的开源 JDBC 驱动兼容标准的 JDBC API，可用于
POM dependency:
```Java
<properties>
- <arrow.version>15.0.1</arrow.version>
+ <arrow.version>17.0.0</arrow.version>
</properties>
<dependencies>
<dependency>
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/db-connect/arrow-flight-sql-connect.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/db-connect/arrow-flight-sql-connect.md
index 839f585c745..df5765507da 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/db-connect/arrow-flight-sql-connect.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/db-connect/arrow-flight-sql-connect.md
@@ -68,7 +68,9 @@ import adbc_driver_flightsql.dbapi as flight_sql
- 修改 fe/conf/fe.conf 中 arrow_flight_sql_port 为一个可用端口，如 9090。
- 修改 be/conf/be.conf中 arrow_flight_sql_port 为一个可用端口，如 9091。
-`注: fe.conf 与 be.conf 中配置的 arrow_flight_sql_port 不相同`
+`注: fe.conf 与 be.conf 中配置的 arrow_flight_sql_port 端口号不相同`
+
+修改配置并重启集群后，在 fe/log/fe.log 文件中搜索到 `Arrow Flight SQL service is started` 表明 FE
的 Arrow Flight Server 启动成功；在 be/log/be.INFO 文件中搜索到 `Arrow Flight Service bind
to host` 表明 BE 的 Arrow Flight Server 启动成功。
假设 Doris 实例中 FE 和 BE 的 Arrow Flight SQL 服务将分别在端口 9090 和 9091 上运行，且 Doris
用户名/密码为“user”/“pass”，那么连接过程如下所示：
@@ -281,7 +283,7 @@ Arrow Flight SQL 协议的开源 JDBC 驱动兼容标准的 JDBC API，可用于
POM dependency:
```Java
<properties>
- <arrow.version>15.0.1</arrow.version>
+ <arrow.version>17.0.0</arrow.version>
</properties>
<dependencies>
<dependency>
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/db-connect/arrow-flight-sql-connect.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/db-connect/arrow-flight-sql-connect.md
index 839f585c745..df5765507da 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/db-connect/arrow-flight-sql-connect.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/db-connect/arrow-flight-sql-connect.md
@@ -68,7 +68,9 @@ import adbc_driver_flightsql.dbapi as flight_sql
- 修改 fe/conf/fe.conf 中 arrow_flight_sql_port 为一个可用端口，如 9090。
- 修改 be/conf/be.conf中 arrow_flight_sql_port 为一个可用端口，如 9091。
-`注: fe.conf 与 be.conf 中配置的 arrow_flight_sql_port 不相同`
+`注: fe.conf 与 be.conf 中配置的 arrow_flight_sql_port 端口号不相同`
+
+修改配置并重启集群后，在 fe/log/fe.log 文件中搜索到 `Arrow Flight SQL service is started` 表明 FE
的 Arrow Flight Server 启动成功；在 be/log/be.INFO 文件中搜索到 `Arrow Flight Service bind
to host` 表明 BE 的 Arrow Flight Server 启动成功。
假设 Doris 实例中 FE 和 BE 的 Arrow Flight SQL 服务将分别在端口 9090 和 9091 上运行，且 Doris
用户名/密码为“user”/“pass”，那么连接过程如下所示：
@@ -281,7 +283,7 @@ Arrow Flight SQL 协议的开源 JDBC 驱动兼容标准的 JDBC API，可用于
POM dependency:
```Java
<properties>
- <arrow.version>15.0.1</arrow.version>
+ <arrow.version>17.0.0</arrow.version>
</properties>
<dependencies>
<dependency>
diff --git a/versioned_docs/version-2.1/db-connect/arrow-flight-sql-connect.md
b/versioned_docs/version-2.1/db-connect/arrow-flight-sql-connect.md
index d3d99682e7e..a122ff16b8d 100644
--- a/versioned_docs/version-2.1/db-connect/arrow-flight-sql-connect.md
+++ b/versioned_docs/version-2.1/db-connect/arrow-flight-sql-connect.md
@@ -68,7 +68,9 @@ Modify the configuration parameters of Doris FE and BE:
- Modify arrow_flight_sql_port in fe/conf/fe.conf to an available port, such
as 9090.
- Modify arrow_flight_sql_port in be/conf/be.conf to an available port, such
as 9091.
-`Note: arrow_flight_sql_port configured in fe.conf and be.conf are different`
+`Note: The arrow_flight_sql_port port number configured in fe.conf and be.conf
is different`
+
+After modifying the configuration and restarting the cluster, searching for
`Arrow Flight SQL service is started` in the fe/log/fe.log file indicates that
the Arrow Flight Server of FE has been successfully started; searching for
`Arrow Flight Service bind to host` in the be/log/be.INFO file indicates that
the Arrow Flight Server of BE has been successfully started.
Assuming that the Arrow Flight SQL services of FE and BE in the Doris instance
will run on ports 9090 and 9091 respectively, and the Doris username/password
is "user"/"pass", the connection process is as follows:
@@ -281,7 +283,7 @@ The open source JDBC driver of Arrow Flight SQL protocol is
compatible with the
POM dependency:
```Java
<properties>
- <arrow.version>15.0.1</arrow.version>
+ <arrow.version>17.0.0</arrow.version>
</properties>
<dependencies>
<dependency>
diff --git a/versioned_docs/version-3.0/db-connect/arrow-flight-sql-connect.md
b/versioned_docs/version-3.0/db-connect/arrow-flight-sql-connect.md
index d3d99682e7e..a122ff16b8d 100644
--- a/versioned_docs/version-3.0/db-connect/arrow-flight-sql-connect.md
+++ b/versioned_docs/version-3.0/db-connect/arrow-flight-sql-connect.md
@@ -68,7 +68,9 @@ Modify the configuration parameters of Doris FE and BE:
- Modify arrow_flight_sql_port in fe/conf/fe.conf to an available port, such
as 9090.
- Modify arrow_flight_sql_port in be/conf/be.conf to an available port, such
as 9091.
-`Note: arrow_flight_sql_port configured in fe.conf and be.conf are different`
+`Note: The arrow_flight_sql_port port number configured in fe.conf and be.conf
is different`
+
+After modifying the configuration and restarting the cluster, searching for
`Arrow Flight SQL service is started` in the fe/log/fe.log file indicates that
the Arrow Flight Server of FE has been successfully started; searching for
`Arrow Flight Service bind to host` in the be/log/be.INFO file indicates that
the Arrow Flight Server of BE has been successfully started.
Assuming that the Arrow Flight SQL services of FE and BE in the Doris instance
will run on ports 9090 and 9091 respectively, and the Doris username/password
is "user"/"pass", the connection process is as follows:
@@ -281,7 +283,7 @@ The open source JDBC driver of Arrow Flight SQL protocol is
compatible with the
POM dependency:
```Java
<properties>
- <arrow.version>15.0.1</arrow.version>
+ <arrow.version>17.0.0</arrow.version>
</properties>
<dependencies>
<dependency>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
