Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-28 Thread via GitHub
yiguolei merged PR #32039: URL: https://github.com/apache/doris/pull/32039 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apa

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-28 Thread via GitHub
xinyiZzz commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1543996383 ## be/src/exec/rowid_fetcher.cpp: ## @@ -327,6 +327,8 @@ struct IteratorItem { Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-28 Thread via GitHub
github-actions[bot] commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2024893986 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-28 Thread via GitHub
github-actions[bot] commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2024893935 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-28 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1542682122 ## be/src/exec/rowid_fetcher.cpp: ## @@ -327,6 +327,8 @@ struct IteratorItem { Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-27 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2024365852 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit cfd6a72eaba9516cd62df0378bba8f0ae8cbe924 with default session variables Stream

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-27 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2024360160 TPC-DS: Total hot run time: 181516 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-27 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2024351995 TPC-H: Total hot run time: 38344 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-27 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2024304732 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-27 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2024290255 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-23 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2016691209 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 19e87b450f680b0f65a1f5ee5fb510a5b857a440 with default session variables Stream

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-23 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2016690742 TPC-DS: Total hot run time: 187317 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-23 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2016688844 TPC-H: Total hot run time: 38398 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-23 Thread via GitHub
github-actions[bot] commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1536734396 ## be/src/runtime/fragment_mgr.cpp: ## @@ -976,6 +979,16 @@ void FragmentMgr::_set_scan_concurrency(const Param& params, QueryContext* query #endif } +

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-23 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2016685675 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-23 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2016665088 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 7952dd7d83a7fb924bcd3188fdfc757b7199293b with default session variables Stream

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-23 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2016664411 TPC-DS: Total hot run time: 186421 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-23 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2016662063 TPC-H: Total hot run time: 37693 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-23 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2016653053 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-22 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2015641138 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 1325bb3dc479352546344e8a1d6446e6ff7fab89 with default session variables Stream

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-22 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2015636394 TPC-DS: Total hot run time: 187716 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-22 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2015615120 TPC-H: Total hot run time: 39998 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-22 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2015483988 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-22 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2014801548 TPC-H: Total hot run time: 38550 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-22 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2014768158 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-22 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2014765874 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-22 Thread via GitHub
xinyiZzz closed pull request #32039: [opt](memory) Optimize mem tracker accuracy URL: https://github.com/apache/doris/pull/32039 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-22 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2014765552 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-22 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2014643439 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-21 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2014392118 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 23480aa8bd2f26eaf689c37d27e2c2a67e959934 with default session variables Stream

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-21 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2014388206 TPC-DS: Total hot run time: 181968 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-21 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2014379579 TPC-H: Total hot run time: 38334 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-21 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2014340564 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-21 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2014171102 TPC-H: Total hot run time: 37438 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-21 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2014117222 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-21 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2013337037 TPC-DS: Total hot run time: 181554 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-21 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2013295546 TPC-H: Total hot run time: 38211 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-21 Thread via GitHub
github-actions[bot] commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1534365788 ## be/src/runtime/thread_context.cpp: ## @@ -23,20 +23,39 @@ namespace doris { class MemTracker; +QueryThreadContext ThreadContext::query_thread_context

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-21 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2013153302 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-21 Thread via GitHub
xinyiZzz commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1534351705 ## be/src/olap/base_tablet.cpp: ## @@ -414,6 +414,8 @@ void BaseTablet::generate_tablet_meta_copy_unlocked(TabletMeta& new_tablet_meta) Status BaseTablet::calc_delet

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-21 Thread via GitHub
yiguolei commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2012528482 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-20 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1532306278 ## be/src/olap/base_tablet.cpp: ## @@ -414,6 +414,8 @@ void BaseTablet::generate_tablet_meta_copy_unlocked(TabletMeta& new_tablet_meta) Status BaseTablet::calc_delet

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-20 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2008938097 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-19 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2008729186 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-19 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2008541254 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-19 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2007644697 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-19 Thread via GitHub
github-actions[bot] commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1530487097 ## be/src/runtime/group_commit_mgr.cpp: ## @@ -228,7 +229,9 @@ Status GroupCommitTable::get_first_block_load_queue(

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-19 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2007298211 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-19 Thread via GitHub
github-actions[bot] commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1530021239 ## be/src/runtime/query_context.cpp: ## @@ -46,48 +47,53 @@ QueryContext::QueryContext(TUniqueId query_id, int total_fragment_num, ExecEnv* _is_

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-19 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2006469597 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-18 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2005927585 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-18 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2005728331 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-18 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2005603700 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-18 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2004667355 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-18 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2004175903 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-18 Thread via GitHub
github-actions[bot] commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1528422230 ## be/src/runtime/query_context.cpp: ## @@ -46,48 +47,53 @@ QueryContext::QueryContext(TUniqueId query_id, int total_fragment_num, ExecEnv* _is_

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-18 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2003715733 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-18 Thread via GitHub
github-actions[bot] commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1528341979 ## be/src/runtime/memory/mem_tracker_limiter.cpp: ## @@ -83,33 +84,63 @@ MemTrackerLimiter::MemTrackerLimiter(Type type, const std::string& label, int64_

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-18 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2003614711 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-17 Thread via GitHub
xinyiZzz commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1527680816 ## be/src/vec/columns/subcolumn_tree.h: ## @@ -72,8 +90,10 @@ class SubcolumnsTree { } void add_child(std::string_view key, std::shared_ptr next_no

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-17 Thread via GitHub
xinyiZzz commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1527680456 ## be/src/olap/memtable.cpp: ## @@ -161,6 +155,15 @@ MemTable::~MemTable() { << std::endl << MemTracker::log_usage(_insert_mem_tracker->make_

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-17 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1527672977 ## be/src/olap/memtable.cpp: ## @@ -161,6 +155,15 @@ MemTable::~MemTable() { << std::endl << MemTracker::log_usage(_insert_mem_tracker->make_

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-17 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1527670203 ## be/src/vec/exec/scan/scanner_context.h: ## @@ -100,7 +108,15 @@ class ScannerContext : public std::enable_shared_from_this, int64_t max_bytes_i

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-17 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1527669355 ## be/src/vec/columns/subcolumn_tree.h: ## @@ -72,8 +90,10 @@ class SubcolumnsTree { } void add_child(std::string_view key, std::shared_ptr next_no

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-17 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2002612034 TPC-H: Total hot run time: 38501 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-17 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2002592846 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-17 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2002515175 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-17 Thread via GitHub
github-actions[bot] commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1527531457 ## be/src/runtime/thread_context.cpp: ## @@ -23,20 +23,39 @@ namespace doris { class MemTracker; +QueryThreadContext ThreadContext::query_thread_context

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-17 Thread via GitHub
github-actions[bot] commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1527469695 ## be/src/runtime/memory/cache_policy.h: ## @@ -93,15 +94,30 @@ class CachePolicy { __builtin_unreachable(); } -CachePolicy(CacheType ty

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-17 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-2002398357 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-14 Thread via GitHub
github-actions[bot] commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1525667098 ## be/src/runtime/thread_context.cpp: ## @@ -23,11 +23,17 @@ namespace doris { class MemTracker; -AttachTask::AttachTask(const std::shared_ptr& mem_trac

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-14 Thread via GitHub
xinyiZzz commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1525663869 ## be/src/runtime/load_stream_writer.cpp: ## @@ -154,6 +154,8 @@ Status LoadStreamWriter::close_segment(uint32_t segid) { Status LoadStreamWriter::add_segment(uint3

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-13 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-1994644561 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-13 Thread via GitHub
xinyiZzz commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1523439719 ## be/src/common/config.cpp: ## @@ -131,6 +131,8 @@ DEFINE_mBool(disable_memory_gc, "false"); DEFINE_mInt64(large_memory_check_bytes, "2147483648"); +DEFINE_mBool

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-13 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1523374448 ## be/src/runtime/load_stream_writer.cpp: ## @@ -154,6 +154,8 @@ Status LoadStreamWriter::close_segment(uint32_t segid) { Status LoadStreamWriter::add_segment(uint3

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-13 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1523365248 ## be/src/common/config.cpp: ## @@ -131,6 +131,8 @@ DEFINE_mBool(disable_memory_gc, "false"); DEFINE_mInt64(large_memory_check_bytes, "2147483648"); +DEFINE_mBool

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-13 Thread via GitHub
Gabriel39 commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1523347478 ## be/src/common/config.cpp: ## @@ -131,6 +131,8 @@ DEFINE_mBool(disable_memory_gc, "false"); DEFINE_mInt64(large_memory_check_bytes, "2147483648"); +DEFINE_mBoo

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-13 Thread via GitHub
github-actions[bot] commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1523251983 ## be/src/runtime/thread_context.cpp: ## @@ -23,6 +23,12 @@ namespace doris { class MemTracker; +QueryThreadContext ThreadContext::query_thread_context(

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-13 Thread via GitHub
xinyiZzz commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-1994378495 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-13 Thread via GitHub
xinyiZzz commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1523112505 ## be/src/vec/exec/scan/scanner_scheduler.cpp: ## @@ -133,6 +133,7 @@ void ScannerScheduler::submit(std::shared_ptr ctx, // Submit scanners to thread pool /

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-13 Thread via GitHub
xinyiZzz commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1523109968 ## be/src/vec/exec/scan/scanner_context.h: ## @@ -103,7 +111,15 @@ class ScannerContext : public std::enable_shared_from_this, int64_t max_bytes_i

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-13 Thread via GitHub
xinyiZzz commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1523031493 ## be/src/runtime/thread_context.h: ## @@ -305,7 +356,18 @@ class SwitchThreadMemTrackerLimiter { explicit SwitchThreadMemTrackerLimiter(const std::shared_ptr& m

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-11 Thread via GitHub
xinyiZzz commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519700787 ## be/src/cloud/cloud_tablet_mgr.cpp: ## @@ -154,7 +154,7 @@ Result> CloudTabletMgr::get_tablet(int64_t tablet_i auto tablet_id_str = std::to_string(tablet_id)

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519087510 ## be/src/runtime/memory/lru_cache_policy.h: ## @@ -19,20 +19,40 @@ #include +#include + #include "olap/lru_cache.h" #include "runtime/memory/cache_policy.h"

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519087367 ## be/src/runtime/memory/lru_cache_policy.h: ## @@ -19,20 +19,40 @@ #include +#include + #include "olap/lru_cache.h" #include "runtime/memory/cache_policy.h"

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
xinyiZzz commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519078302 ## tools/clickbench-tools/conf/doris-cluster.conf: ## @@ -20,11 +20,11 @@ export FE_HOST='127.0.0.1' # BE host export BE_HOST='127.0.0.1' # http_port in fe.conf -ex

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519067044 ## be/src/runtime/thread_context.h: ## @@ -305,7 +356,18 @@ class SwitchThreadMemTrackerLimiter { explicit SwitchThreadMemTrackerLimiter(const std::shared_ptr& m

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519059132 ## be/src/cloud/cloud_tablet_mgr.cpp: ## @@ -154,7 +154,7 @@ Result> CloudTabletMgr::get_tablet(int64_t tablet_i auto tablet_id_str = std::to_string(tablet_id)

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519050501 ## be/src/vec/exec/scan/scanner_context.h: ## @@ -103,7 +111,15 @@ class ScannerContext : public std::enable_shared_from_this, int64_t max_bytes_i

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519050083 ## be/src/vec/exec/scan/scanner_scheduler.cpp: ## @@ -141,7 +142,10 @@ void ScannerScheduler::submit(std::shared_ptr ctx, scanner_delegate->_scanner->start

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519049847 ## be/src/vec/exec/scan/scanner_scheduler.cpp: ## @@ -133,6 +133,7 @@ void ScannerScheduler::submit(std::shared_ptr ctx, // Submit scanners to thread pool /

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519048527 ## tools/tpch-tools/bin/load-tpch-data.sh: ## @@ -30,7 +30,7 @@ ROOT=$( ) CURDIR="${ROOT}" -TPCH_DATA_DIR="${CURDIR}/tpch-data" Review Comment: 这些不要动了 -- T

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519048462 ## tools/clickbench-tools/conf/doris-cluster.conf: ## @@ -20,11 +20,11 @@ export FE_HOST='127.0.0.1' # BE host export BE_HOST='127.0.0.1' # http_port in fe.conf -ex

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519047398 ## be/src/olap/schema_cache.h: ## @@ -86,7 +86,7 @@ class SchemaCache : public LRUCachePolicy { if (!instance() || key.empty()) { return;

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519046009 ## be/src/olap/memtable_memory_limiter.h: ## @@ -45,7 +45,8 @@ class MemTableMemoryLimiter { void refresh_mem_tracker(); -MemTrackerLimiter* mem_tracker()

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
yiguolei commented on code in PR #32039: URL: https://github.com/apache/doris/pull/32039#discussion_r1519039797 ## be/src/runtime/thread_context.h: ## @@ -270,6 +293,32 @@ static ThreadContext* thread_context() { __builtin_unreachable(); } +class QueryThreadContext { Re

Re: [PR] [opt](memory) Optimize mem tracker accuracy [doris]

2024-03-10 Thread via GitHub
doris-robot commented on PR #32039: URL: https://github.com/apache/doris/pull/32039#issuecomment-1987334009 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+y