pengxiangyu commented on code in PR #11006:
URL: https://github.com/apache/doris/pull/11006#discussion_r926229647
##########
be/src/common/config.h:
##########
@@ -798,6 +798,11 @@ CONF_Int32(s3_transfer_executor_pool_size, "2");
CONF_Bool(enable_time_lut, "true");
+// number of s3 scanner thread pool size
+CONF_Int32(doris_s3_scanner_thread_pool_thread_num, "48");
+// number of s3 scanner thread pool queue size
+CONF_Int32(doris_s3_scanner_thread_pool_queue_size, "102400");
Review Comment:
102400 is too big,
##########
be/src/runtime/exec_env.h:
##########
@@ -200,6 +201,7 @@ class ExecEnv {
// TODO(cmy): find a better way to unify these 2 pools.
PriorityThreadPool* _scan_thread_pool = nullptr;
+ PriorityThreadPool* _s3_scan_thread_pool = nullptr;
Review Comment:
_s3_scan_thread_pool need to be _remote_scan_thread_pool
##########
be/src/exec/olap_scan_node.cpp:
##########
@@ -1503,6 +1503,7 @@ void OlapScanNode::transfer_thread(RuntimeState* state) {
* 4. Regularly increase the priority of the remaining tasks in the queue
to avoid starvation for large queries
*********************************/
PriorityThreadPool* thread_pool = state->exec_env()->scan_thread_pool();
+ PriorityThreadPool* s3_thread_pool =
state->exec_env()->s3_scan_thread_pool();
Review Comment:
Use remote_thread_pool, it is not only for S3
##########
be/src/common/config.h:
##########
@@ -798,6 +798,11 @@ CONF_Int32(s3_transfer_executor_pool_size, "2");
CONF_Bool(enable_time_lut, "true");
+// number of s3 scanner thread pool size
+CONF_Int32(doris_s3_scanner_thread_pool_thread_num, "48");
Review Comment:
DEFAULT VALUE 48 is too big, it will cost too many cpu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
