This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 444836f1ae8fd236ea9c2c630ba28aadf8d5ed13
Author: Kang <kxiao.ti...@gmail.com>
AuthorDate: Sat Sep 2 01:14:14 2023 +0800
[improvement](config) add a specific be config for segment_cache_capacity
(#23701)
* add segment_cache_capacity config istead of fd limit * 2/5
* default -1 for backward compatibility
---
be/src/common/config.cpp | 3 +++
be/src/common/config.h | 3 +++
be/src/runtime/exec_env_init.cpp | 7 +++++--
docs/en/docs/admin-manual/config/be-config.md | 12 +++++-------
docs/zh-CN/docs/admin-manual/config/be-config.md | 12 +++++-------
5 files changed, 21 insertions(+), 16 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 9bc477c27a..3ac4535966 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -1021,6 +1021,9 @@ DEFINE_Bool(enable_shrink_memory, "false");
DEFINE_mInt32(schema_cache_capacity, "1024");
DEFINE_mInt32(schema_cache_sweep_time_sec, "100");
+// max number of segment cache, default -1 for backward compatibility
fd_number*2/5
+DEFINE_mInt32(segment_cache_capacity, "-1");
+
// enable feature binlog, default false
DEFINE_Bool(enable_feature_binlog, "false");
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 4a1a4ca18d..5525745fe4 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -1059,6 +1059,9 @@ DECLARE_Bool(enable_shrink_memory);
DECLARE_mInt32(schema_cache_capacity);
DECLARE_mInt32(schema_cache_sweep_time_sec);
+// max number of segment cache
+DECLARE_mInt32(segment_cache_capacity);
+
// enable binlog
DECLARE_Bool(enable_feature_binlog);
diff --git a/be/src/runtime/exec_env_init.cpp b/be/src/runtime/exec_env_init.cpp
index 5fca5e325e..7f88efe689 100644
--- a/be/src/runtime/exec_env_init.cpp
+++ b/be/src/runtime/exec_env_init.cpp
@@ -281,8 +281,11 @@ Status ExecEnv::_init_mem_env() {
}
// SegmentLoader caches segments in rowset granularity. So the size of
// opened files will greater than segment_cache_capacity.
- uint64_t segment_cache_capacity = fd_number * 2 / 5;
- LOG(INFO) << "segment_cache_capacity = fd_number * 2 / 5, fd_number: " <<
fd_number
+ int64_t segment_cache_capacity = config::segment_cache_capacity;
+ if (segment_cache_capacity < 0 || segment_cache_capacity > fd_number * 2 /
5) {
+ segment_cache_capacity = fd_number * 2 / 5;
+ }
+ LOG(INFO) << "segment_cache_capacity <= fd_number * 2 / 5, fd_number: " <<
fd_number
<< " segment_cache_capacity: " << segment_cache_capacity;
SegmentLoader::create_global_instance(segment_cache_capacity);
diff --git a/docs/en/docs/admin-manual/config/be-config.md
b/docs/en/docs/admin-manual/config/be-config.md
index 88a044fdfd..ec85dc41f1 100644
--- a/docs/en/docs/admin-manual/config/be-config.md
+++ b/docs/en/docs/admin-manual/config/be-config.md
@@ -1012,13 +1012,6 @@ BaseCompaction:546859:
- Increasing this value can reduce the number of calls to read remote data,
but it will increase memory overhead.
* Default value: 16MB
-#### `segment_cache_capacity`
-
-* Type: int32
-* Description: The maximum number of Segments cached by Segment Cache.
- - The default value is currently only an empirical value, and may need to be
modified according to actual scenarios. Increasing this value can cache more
segments and avoid some IO. Decreasing this value will reduce memory usage.
-* Default value: 1000000
-
#### `file_cache_type`
* Type: string
@@ -1181,6 +1174,11 @@ BaseCompaction:546859:
* Description: Index page cache as a percentage of total storage page cache,
value range is [0, 100]
* Default value: 10
+#### `segment_cache_capacity`
+* Type: int32
+* Description: Max number of segment cache (the key is rowset id) entries. -1
is for backward compatibility as fd_number * 2/5.
+* Default value: -1
+
#### `storage_strict_check_incompatible_old_format`
* Type: bool
diff --git a/docs/zh-CN/docs/admin-manual/config/be-config.md
b/docs/zh-CN/docs/admin-manual/config/be-config.md
index 5e48d603f6..109bdd5462 100644
--- a/docs/zh-CN/docs/admin-manual/config/be-config.md
+++ b/docs/zh-CN/docs/admin-manual/config/be-config.md
@@ -1037,13 +1037,6 @@ BaseCompaction:546859:
- 增大这个值，可以减少远端数据读取的调用次数，但会增加内存开销。
* 默认值: 16MB
-#### `segment_cache_capacity`
-
-* 类型: int32
-* 描述: Segment Cache 缓存的 Segment 最大数量
- - 默认值目前只是一个经验值，可能需要根据实际场景修改。增大该值可以缓存更多的segment从而避免一些IO。减少该值则会降低内存使用。
-* 默认值: 1000000
-
#### `file_cache_type`
* 类型：string
@@ -1206,6 +1199,11 @@ BaseCompaction:546859:
* 描述：索引页缓存占总页面缓存的百分比，取值为[0, 100]。
* 默认值：10
+#### `segment_cache_capacity`
+* Type: int32
+* Description: segment元数据缓存（以rowset id为key）的最大rowset个数. -1代表向后兼容取值为fd_number *
2/5
+* Default value: -1
+
#### `storage_strict_check_incompatible_old_format`
* 类型：bool
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
