Re: [PR] [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block [doris]
HappenLee merged PR #28850: URL: https://github.com/apache/doris/pull/28850 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-1.2-lts updated: [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block (#28850)
This is an automated email from the ASF dual-hosted git repository. lihaopeng pushed a commit to branch branch-1.2-lts in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-1.2-lts by this push: new 120d69a37b1 [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block (#28850) 120d69a37b1 is described below commit 120d69a37b1fa53ebd3c7ec7f7d761be4f47e634 Author: caiconghui <55968745+caicong...@users.noreply.github.com> AuthorDate: Fri Dec 22 16:00:36 2023 +0800 [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block (#28850) Co-authored-by: caiconghui1 --- be/src/vec/exec/scan/scanner_scheduler.cpp | 4 +++- be/src/vec/exec/vunion_node.cpp| 11 +-- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/be/src/vec/exec/scan/scanner_scheduler.cpp b/be/src/vec/exec/scan/scanner_scheduler.cpp index 7b7e3aeac50..32cefe6e822 100644 --- a/be/src/vec/exec/scan/scanner_scheduler.cpp +++ b/be/src/vec/exec/scan/scanner_scheduler.cpp @@ -269,7 +269,9 @@ void ScannerScheduler::_scanner_scan(ScannerScheduler* scheduler, ScannerContext ctx->return_free_block(block); } else { if (!blocks.empty() && blocks.back()->rows() + block->rows() <= state->batch_size()) { -vectorized::MutableBlock(blocks.back()).merge(*block); +vectorized::MutableBlock mutable_block(blocks.back()); +mutable_block.merge(*block); + blocks.back()->set_columns(std::move(mutable_block.mutable_columns())); ctx->return_free_block(block); } else { blocks.push_back(block); diff --git a/be/src/vec/exec/vunion_node.cpp b/be/src/vec/exec/vunion_node.cpp index 8afe5bce8df..199782b577b 100644 --- a/be/src/vec/exec/vunion_node.cpp +++ b/be/src/vec/exec/vunion_node.cpp @@ -171,9 +171,9 @@ Status VUnionNode::get_next_materialized(RuntimeState* state, Block* block) { } } -if (!mem_reuse) { -block->swap(mblock.to_block()); -} + +block->swap(mblock.to_block()); + DCHECK_LE(_child_idx, _children.size()); return Status::OK(); @@ -204,9 +204,8 @@ Status VUnionNode::get_next_const(RuntimeState* state, Block* block) { } } -if (!mem_reuse) { -block->swap(mblock.to_block()); -} + +block->swap(mblock.to_block()); // some insert query like "insert into string_test select 1, repeat('a', 1024 * 1024);" // the const expr will be in output expr cause the union node return a empty block. so here we - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](group commit) refector replay wal function [doris]
doris-robot commented on PR #28756: URL: https://github.com/apache/doris/pull/28756#issuecomment-1867361742 TeamCity be ut coverage result: Function Coverage: 36.48% (8538/23404) Line Coverage: 28.59% (69433/242838) Region Coverage: 27.61% (35927/130119) Branch Coverage: 24.36% (18371/75406) Coverage Report: http://coverage.selectdb-in.cc/coverage/b41679ab54ad3eec3c9af8c9db7ad269a1550c57_b41679ab54ad3eec3c9af8c9db7ad269a1550c57/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](inverted index) add lowercase option for inverted index analyzer [doris]
doris-robot commented on PR #28704: URL: https://github.com/apache/doris/pull/28704#issuecomment-1867363611 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 43.94 seconds stream load tsv: 584 seconds loaded 74807831229 Bytes, about 122 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.7 seconds inserted 1000 Rows, about 348K ops/s storage size: 17183622321 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] 2.0.3 查询异常 [doris]
learner1212 commented on issue #28741: URL: https://github.com/apache/doris/issues/28741#issuecomment-1867364326 > Enable the nereids planner and the query will execute correctly. 这个有效果,多谢 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](pipelineX) fix use global rf when there no shared_scans [doris]
Mryange opened a new pull request, #28869: URL: https://github.com/apache/doris/pull/28869 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](pipelineX) fix use global rf when there no shared_scans [doris]
Mryange commented on PR #28869: URL: https://github.com/apache/doris/pull/28869#issuecomment-1867366733 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [only test] [doris]
Mryange commented on PR #28870: URL: https://github.com/apache/doris/pull/28870#issuecomment-1867368525 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [only test] [doris]
Mryange opened a new pull request, #28870: URL: https://github.com/apache/doris/pull/28870 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](pipelineX)do not override dependency() function in pipelineX [doris]
Mryange commented on PR #28848: URL: https://github.com/apache/doris/pull/28848#issuecomment-1867370077 run tpch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](group commit) Add an option of enable relay wal on all regression test [doris]
hust-hhb commented on PR #28757: URL: https://github.com/apache/doris/pull/28757#issuecomment-1867371191 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](pipelineX) fix use global rf when there no shared_scans [doris]
github-actions[bot] commented on PR #28869: URL: https://github.com/apache/doris/pull/28869#issuecomment-1867371564 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](inverted index) add lowercase option for inverted index analyzer [doris]
doris-robot commented on PR #28704: URL: https://github.com/apache/doris/pull/28704#issuecomment-1867373469 TeamCity be ut coverage result: Function Coverage: 36.47% (8536/23404) Line Coverage: 28.59% (69406/242802) Region Coverage: 27.60% (35913/130113) Branch Coverage: 24.34% (18352/75406) Coverage Report: http://coverage.selectdb-in.cc/coverage/490c05e0613f7144282dca56d2abade4013743e1_490c05e0613f7144282dca56d2abade4013743e1/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](group commit) refector replay wal function [doris]
doris-robot commented on PR #28756: URL: https://github.com/apache/doris/pull/28756#issuecomment-1867374535 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 43.58 seconds stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17183676197 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [only test] [doris]
github-actions[bot] commented on PR #28870: URL: https://github.com/apache/doris/pull/28870#issuecomment-1867374844 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) "not is null" stats estimation fix [doris]
doris-robot commented on PR #28860: URL: https://github.com/apache/doris/pull/28860#issuecomment-1867376292 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.38 seconds stream load tsv: 582 seconds loaded 74807831229 Bytes, about 122 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.3 seconds inserted 1000 Rows, about 341K ops/s storage size: 17184032145 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [hotfix](jdbc catalog) fix load table and column names npe [doris]
doris-robot commented on PR #28865: URL: https://github.com/apache/doris/pull/28865#issuecomment-1867378600 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.01 seconds stream load tsv: 584 seconds loaded 74807831229 Bytes, about 122 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17183873661 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (#28851) [doris]
jacktengg opened a new pull request, #28871: URL: https://github.com/apache/doris/pull/28871 ## Proposed changes Pick from #28851 ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](log) regularise some BE error type and fix a load task check [doris]
github-actions[bot] commented on PR #28729: URL: https://github.com/apache/doris/pull/28729#issuecomment-1867379816 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](jdbc catalog) Change the connection pool parameters of Jdbc Catalog to be configurable [doris]
doris-robot commented on PR #28779: URL: https://github.com/apache/doris/pull/28779#issuecomment-1867380818 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 49.25 seconds stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.6 seconds inserted 1000 Rows, about 337K ops/s storage size: 17163179241 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [exec](load) change default parallel num from 1 to 8 in no pipeline exec engine [doris]
HappenLee commented on PR #28864: URL: https://github.com/apache/doris/pull/28864#issuecomment-1867380941 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] [doris]
xlostpath commented on issue #28866: URL: https://github.com/apache/doris/issues/28866#issuecomment-1867382218 If the data is inserted in a single batch, the ordering will be correct. However, if the insertion is done in multiple batches, the ordering will be interrupted, resulting in incorrect sorting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [nereids] fix join fd computing bug [doris]
doris-robot commented on PR #28849: URL: https://github.com/apache/doris/pull/28849#issuecomment-1867383420 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.63 seconds stream load tsv: 585 seconds loaded 74807831229 Bytes, about 121 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.6 seconds inserted 1000 Rows, about 349K ops/s storage size: 17183763647 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block [doris]
github-actions[bot] commented on PR #27943: URL: https://github.com/apache/doris/pull/27943#issuecomment-1867385779 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block [doris]
github-actions[bot] commented on PR #27943: URL: https://github.com/apache/doris/pull/27943#issuecomment-1867385732 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](pipelineX) fix use global rf when there no shared_scans [doris]
doris-robot commented on PR #28869: URL: https://github.com/apache/doris/pull/28869#issuecomment-1867386158 TeamCity be ut coverage result: Function Coverage: 36.48% (8537/23403) Line Coverage: 28.59% (69416/242816) Region Coverage: 27.60% (35912/130112) Branch Coverage: 24.34% (18353/75408) Coverage Report: http://coverage.selectdb-in.cc/coverage/cd467df94f2ed4a92a8007ac5582fcf1a05e29bf_cd467df94f2ed4a92a8007ac5582fcf1a05e29bf/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] 2.0.3 查询异常 [doris]
learner1212 closed issue #28741: [Bug] 2.0.3 查询异常 URL: https://github.com/apache/doris/issues/28741 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](scannercore) scanner will core in deconstructor during collect profile [doris]
yiguolei commented on PR #28727: URL: https://github.com/apache/doris/pull/28727#issuecomment-1867390937 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](pipelineX) control exchange sink by memory usage [doris]
Mryange commented on PR #28814: URL: https://github.com/apache/doris/pull/28814#issuecomment-1867392342 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [only test] [doris]
doris-robot commented on PR #28870: URL: https://github.com/apache/doris/pull/28870#issuecomment-1867393006 TeamCity be ut coverage result: Function Coverage: 36.48% (8538/23403) Line Coverage: 28.59% (69421/242816) Region Coverage: 27.61% (35919/130112) Branch Coverage: 24.34% (18357/75408) Coverage Report: http://coverage.selectdb-in.cc/coverage/cc90734f922f0b5576bc8d45a6ce5cde9ee46610_cc90734f922f0b5576bc8d45a6ce5cde9ee46610/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [only test] [doris]
Mryange commented on PR #28870: URL: https://github.com/apache/doris/pull/28870#issuecomment-1867395385 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [optimize](invert index) optimize inverted index metadata into file cache [doris]
zzzxl1993 opened a new pull request, #28872: URL: https://github.com/apache/doris/pull/28872 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [docs] (DebugPoints) Update docs about Debug Points [doris]
HowardQin commented on PR #28347: URL: https://github.com/apache/doris/pull/28347#issuecomment-1867397594 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](invert index) Empty strings are not written to the index in the… [doris]
zzzxl1993 commented on PR #28822: URL: https://github.com/apache/doris/pull/28822#issuecomment-1867398396 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [optimize](invert index) optimize inverted index metadata into file cache [doris]
zzzxl1993 commented on PR #28872: URL: https://github.com/apache/doris/pull/28872#issuecomment-1867398513 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](pipelineX) control exchange sink by memory usage [doris]
github-actions[bot] commented on PR #28814: URL: https://github.com/apache/doris/pull/28814#issuecomment-1867399432 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [only test] [doris]
github-actions[bot] commented on PR #28870: URL: https://github.com/apache/doris/pull/28870#issuecomment-1867407902 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [optimize](invert index) optimize inverted index metadata into file cache [doris]
github-actions[bot] commented on code in PR #28872: URL: https://github.com/apache/doris/pull/28872#discussion_r1434851121 ## be/src/olap/rowset/beta_rowset.cpp: ## @@ -318,7 +319,8 @@ Status BetaRowset::copy_files_to(const std::string& dir, const RowsetId& new_row return Status::OK(); } -Status BetaRowset::upload_to(io::RemoteFileSystem* dest_fs, const RowsetId& new_rowset_id) { +Status BetaRowset::upload_to(const io::RemoteFileSystemSPtr& dest_fs, Review Comment: warning: function 'upload_to' exceeds recommended size/complexity thresholds [readability-function-size] ```cpp Status BetaRowset::upload_to(const io::RemoteFileSystemSPtr& dest_fs, ^ ``` Additional context **be/src/olap/rowset/beta_rowset.cpp:321:** 83 lines including whitespace and comments (threshold 80) ```cpp Status BetaRowset::upload_to(const io::RemoteFileSystemSPtr& dest_fs, ^ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](pipelineX) fix use global rf when there no shared_scans [doris]
doris-robot commented on PR #28869: URL: https://github.com/apache/doris/pull/28869#issuecomment-1867409829 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 43.88 seconds stream load tsv: 583 seconds loaded 74807831229 Bytes, about 122 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17183964101 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [FIX](complextype)fix count func with complex type [doris]
amorynan opened a new pull request, #28873: URL: https://github.com/apache/doris/pull/28873 ## Proposed changes Issue Number: close #xxx when we use count func with old planner which defined function segnature in fe code with hard code, and will not match coplex type with variable nested type. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](invert index) Empty strings are not written to the index in the… [doris]
github-actions[bot] commented on PR #28822: URL: https://github.com/apache/doris/pull/28822#issuecomment-1867410759 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [FIX](complextype) fix complextype has empty literal may make be core [doris]
amorynan commented on PR #28820: URL: https://github.com/apache/doris/pull/28820#issuecomment-1867414506 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [exec](load) change default parallel num from 1 to 8 in no pipeline exec engine [doris]
doris-robot commented on PR #28864: URL: https://github.com/apache/doris/pull/28864#issuecomment-1867416653 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 43.14 seconds stream load tsv: 581 seconds loaded 74807831229 Bytes, about 122 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17184071816 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [FIX](complextype)fix count func with complex type [doris]
amorynan commented on PR #28873: URL: https://github.com/apache/doris/pull/28873#issuecomment-1867419009 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](paimon)fix `like` predicate [doris]
wuwenchi commented on PR #28803: URL: https://github.com/apache/doris/pull/28803#issuecomment-1867421100 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [only test] [doris]
doris-robot commented on PR #28870: URL: https://github.com/apache/doris/pull/28870#issuecomment-1867421553 TeamCity be ut coverage result: Function Coverage: 36.48% (8538/23403) Line Coverage: 28.59% (69421/242816) Region Coverage: 27.60% (35911/130112) Branch Coverage: 24.34% (18357/75408) Coverage Report: http://coverage.selectdb-in.cc/coverage/c7bebd930ee5cdcab8ccfa61e2b389d1be9c3dab_c7bebd930ee5cdcab8ccfa61e2b389d1be9c3dab/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](ci) tpch pipeline should not re-load data [doris]
hello-stephen opened a new pull request, #28874: URL: https://github.com/apache/doris/pull/28874 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [improvement](log) add txn log [doris]
yujun777 opened a new pull request, #28875: URL: https://github.com/apache/doris/pull/28875 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](nereids) Fix query rewrite fail when query cache build quickly [doris]
seawinde opened a new pull request, #28876: URL: https://github.com/apache/doris/pull/28876 ## Proposed changes Fix query rewrite fail by mv when mv build interval is short and cache update quickly. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](log) add txn log [doris]
yujun777 commented on PR #28875: URL: https://github.com/apache/doris/pull/28875#issuecomment-1867422868 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ci) tpch pipeline should not re-load data [doris]
hello-stephen commented on PR #28874: URL: https://github.com/apache/doris/pull/28874#issuecomment-1867422892 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Fix query rewrite fail when query cache build quickly [doris]
seawinde commented on PR #28876: URL: https://github.com/apache/doris/pull/28876#issuecomment-1867422950 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](invert index) Empty strings are not written to the index in the… [doris]
doris-robot commented on PR #28822: URL: https://github.com/apache/doris/pull/28822#issuecomment-1867424793 TeamCity be ut coverage result: Function Coverage: 36.48% (8537/23403) Line Coverage: 28.59% (69423/242808) Region Coverage: 27.61% (35921/130100) Branch Coverage: 24.35% (18362/75410) Coverage Report: http://coverage.selectdb-in.cc/coverage/e50e9ed2c5ca3fa3c3bcac7af26451059f1ffe84_e50e9ed2c5ca3fa3c3bcac7af26451059f1ffe84/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](executor)remove scan group [doris]
yiguolei merged PR #28847: URL: https://github.com/apache/doris/pull/28847 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [refactor](executor)remove scan group #28847
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new aca8406e319 [refactor](executor)remove scan group #28847 aca8406e319 is described below commit aca8406e319530030e065a34f19a496fab86e85b Author: wangbo AuthorDate: Fri Dec 22 17:05:50 2023 +0800 [refactor](executor)remove scan group #28847 --- be/src/common/config.cpp | 2 - be/src/common/config.h | 2 - be/src/runtime/task_group/task_group.cpp | 5 - be/src/runtime/task_group/task_group.h | 6 - be/src/vec/exec/scan/scan_task_queue.cpp | 221 - be/src/vec/exec/scan/scan_task_queue.h | 99 - be/src/vec/exec/scan/scanner_context.cpp | 4 - be/src/vec/exec/scan/scanner_context.h | 1 - be/src/vec/exec/scan/scanner_scheduler.cpp | 44 -- be/src/vec/exec/scan/scanner_scheduler.h | 13 +- 10 files changed, 1 insertion(+), 396 deletions(-) diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp index 03eaee7b23c..ecc44a08e47 100644 --- a/be/src/common/config.cpp +++ b/be/src/common/config.cpp @@ -970,8 +970,6 @@ DEFINE_Bool(enable_fuzzy_mode, "false"); DEFINE_Bool(enable_debug_points, "false"); DEFINE_Int32(pipeline_executor_size, "0"); -DEFINE_Bool(enable_workload_group_for_scan, "false"); -DEFINE_mInt64(workload_group_scan_task_wait_timeout_ms, "1"); // 128 MB DEFINE_mInt64(local_exchange_buffer_mem_limit, "134217728"); diff --git a/be/src/common/config.h b/be/src/common/config.h index e011073d44d..a9508c6e8af 100644 --- a/be/src/common/config.h +++ b/be/src/common/config.h @@ -1024,8 +1024,6 @@ DECLARE_Bool(enable_fuzzy_mode); DECLARE_Bool(enable_debug_points); DECLARE_Int32(pipeline_executor_size); -DECLARE_Bool(enable_workload_group_for_scan); -DECLARE_mInt64(workload_group_scan_task_wait_timeout_ms); // Temp config. True to use optimization for bitmap_index apply predicate except leaf node of the and node. // Will remove after fully test. diff --git a/be/src/runtime/task_group/task_group.cpp b/be/src/runtime/task_group/task_group.cpp index 137f5ea2345..9e86f8b831b 100644 --- a/be/src/runtime/task_group/task_group.cpp +++ b/be/src/runtime/task_group/task_group.cpp @@ -33,7 +33,6 @@ #include "runtime/memory/mem_tracker_limiter.h" #include "util/mem_info.h" #include "util/parse_util.h" -#include "vec/exec/scan/scan_task_queue.h" #include "vec/exec/scan/scanner_scheduler.h" namespace doris { @@ -102,7 +101,6 @@ std::string TaskGroupEntity::debug_string() const { } template class TaskGroupEntity>; -template class TaskGroupEntity; TaskGroup::TaskGroup(const TaskGroupInfo& tg_info) : _id(tg_info.id), @@ -112,7 +110,6 @@ TaskGroup::TaskGroup(const TaskGroupInfo& tg_info) _enable_memory_overcommit(tg_info.enable_memory_overcommit), _cpu_share(tg_info.cpu_share), _task_entity(this, "pipeline task entity"), - _local_scan_entity(this, "local scan entity"), _mem_tracker_limiter_pool(MEM_TRACKER_GROUP_NUM), _cpu_hard_limit(tg_info.cpu_hard_limit) {} @@ -150,8 +147,6 @@ void TaskGroup::check_and_update(const TaskGroupInfo& tg_info) { } ExecEnv::GetInstance()->pipeline_task_group_scheduler()->task_queue()->update_tg_cpu_share( tg_info, &_task_entity); - ExecEnv::GetInstance()->scanner_scheduler()->local_scan_task_queue()->update_tg_cpu_share( -tg_info, &_local_scan_entity); } int64_t TaskGroup::memory_used() { diff --git a/be/src/runtime/task_group/task_group.h b/be/src/runtime/task_group/task_group.h index f1c8523664e..04dbf518f0d 100644 --- a/be/src/runtime/task_group/task_group.h +++ b/be/src/runtime/task_group/task_group.h @@ -43,7 +43,6 @@ namespace taskgroup { class TaskGroup; struct TaskGroupInfo; -class ScanTaskQueue; template class TaskGroupEntity { @@ -88,9 +87,6 @@ private: using TaskGroupPipelineTaskEntity = TaskGroupEntity>; using TGPTEntityPtr = TaskGroupPipelineTaskEntity*; -using TaskGroupScanTaskEntity = TaskGroupEntity; -using TGSTEntityPtr = TaskGroupScanTaskEntity*; - struct TgTrackerLimiterGroup { std::unordered_set> trackers; std::mutex group_lock; @@ -101,7 +97,6 @@ public: explicit TaskGroup(const TaskGroupInfo& tg_info); TaskGroupPipelineTaskEntity* task_entity() { return &_task_entity; } -TGSTEntityPtr local_scan_task_entity() { return &_local_scan_entity; } int64_t version() const { return _version; } @@ -155,7 +150,6 @@ private: bool _enable_memory_overcommit; std::atomic _cpu_share; TaskGroupPipelineTaskEntity _task_entity; -TaskGroupScanTaskEntity _local_scan_entity; std::vector _mem_tracker_limiter_pool; std::atomic _cpu_hard_limit; }; diff --git a/be/src/vec/exec/scan/scan_task_
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (enhance)(S3) Change s3 metric from bvar adder to latency recorder [doris]
ByteYue commented on PR #28861: URL: https://github.com/apache/doris/pull/28861#issuecomment-1867428473 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](pipelineX) control exchange sink by memory usage [doris]
doris-robot commented on PR #28814: URL: https://github.com/apache/doris/pull/28814#issuecomment-1867429749 TeamCity be ut coverage result: Function Coverage: 36.48% (8540/23408) Line Coverage: 28.60% (69451/242835) Region Coverage: 27.62% (35933/130117) Branch Coverage: 24.37% (18373/75406) Coverage Report: http://coverage.selectdb-in.cc/coverage/e45918b43776ee1038d98ab8f06ff17bec8d445b_e45918b43776ee1038d98ab8f06ff17bec8d445b/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](log) add txn log [doris]
github-actions[bot] commented on PR #28875: URL: https://github.com/apache/doris/pull/28875#issuecomment-1867430299 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](jdbc catalog) Optimize connection pool caching logic [doris]
doris-robot commented on PR #28859: URL: https://github.com/apache/doris/pull/28859#issuecomment-1867431272 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit 20119d0346e194fe037ac709de2612736462db0a, data reload: false run tpch-sf100 query with default conf and session variables q1 4730443344204420 q2 378 138 138 138 q3 1471126512341234 q4 1109901 925 901 q5 3182317031813170 q6 249 130 132 130 q7 1013496 486 486 q8 2206228522142214 q9 6720670866526652 q10 3219328232753275 q11 315 187 187 187 q12 357 214 209 209 q13 4542377337953773 q14 245 214 215 214 q15 577 532 521 521 q16 449 383 388 383 q17 1008641 600 600 q18 7175699269346934 q19 1553143014591430 q20 533 331 292 292 q21 3123269727232697 q22 359 282 290 282 Total cold run time: 44513 ms Total hot run time: 40142 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4378439143334333 q2 266 170 175 170 q3 3518351834943494 q4 2381237623782376 q5 5710573257215721 q6 240 126 124 124 q7 2343186618551855 q8 3533352835273527 q9 9052901689868986 q10 3938400740234007 q11 481 367 388 367 q12 774 594 603 594 q13 4298353735873537 q14 285 260 254 254 q15 584 532 536 532 q16 521 451 472 451 q17 1880186418551855 q18 8474823580318031 q19 1752175017441744 q20 2265195019451945 q21 6485618261416141 q22 495 418 424 418 Total cold run time: 63653 ms Total hot run time: 60462 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [FIX](map)fix map with rowstore table [doris]
amorynan opened a new pull request, #28877: URL: https://github.com/apache/doris/pull/28877 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[I] [Bug] Subquery has bug [doris]
SinyoWong opened a new issue, #28878: URL: https://github.com/apache/doris/issues/28878 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version Doris 2.0.3 ### What's Wrong? SQL 错误 [1105] [HY000]: errCode = 2, detailMessage = (192.168.219.30)[INTERNAL_ERROR]couldn't resolve slot descriptor 5, desc: tuples: Tuple(id=20 slots=[Slot(id=624 type=VARCHAR col=-1, colname=, nullable=1), Slot(id=625 type=VARCHAR col=-1, colname=, nullable=1), Slot(id=626 type=VARCHAR col=-1, colname=, nullable=1)] has_varlen_slots=1) Tuple(id=19 slots=[Slot(id=623 type=DECIMAL128(38, 9) col=-1, colname=cost_130533_t1, nullable=1)] has_varlen_slots=0) Tuple(id=17 slots=[Slot(id=416 type=DECIMALV2(22, 5) col=-1, colname=cost, nullable=1), Slot(id=424 type=DATEV2 col=-1, colname=dt, nullable=1), Slot(id=444 type=VARCHAR col=-1, colname=, nullable=1), Slot(id=445 type=VARCHAR col=-1, colname=, nullable=0), Slot(id=447 type=VARCHAR col=-1, colname=brand_code, nullable=1), Slot(id=460 type=DATEV2 col=-1, colname=dt, nullable=1), Slot(id=480 type=VARCHAR col=-1, colname=brand_code, nullable=1), Slot(id=493 type=DATEV2 col=-1, colname=dt, nullable=1)] has_varlen_slots=1) Tuple(id=16 slots=[Slot(id=410 type=DECIMAL128(38, 9) col=-1, colname=cost_130533_t1_1, nullable=1)] has_varlen_slots=0) Tuple(id=14 slots=[Slot(id=394 type=VARCHAR col=-1, colname=, nullable=1), Slot(id=395 type=VARCHAR col=-1, colname=, nullable=1), Slot(id=396 type=VARCHAR col=-1, colname=, nullable=1), Slot(id=397 type=DECIMAL128(38, 9) col=-1, colname=, nullable=0)] has_varlen_slots=1) Tuple(id=21 slots=[Slot(id=627 type=VARCHAR col=-1, colname=, nullable=1), Slot(id=628 type=VARCHAR col=-1, colname=, nullable=1), Slot(id=629 type=VARCHAR col=-1, colname=, nullable=1), Slot(id=630 type=DECIMAL128(38, 9) col=-1, colname=, nullable=1)] has_varlen_slots=1) Tuple(id=8 slots=[Slot(id=184 type=VARCHAR col=-1, colname=brand_code, nullable=1), Slot(id=189 type=DECIMALV2(22, 5) col=-1, colname=cost, nullable=1), Slot(id=197 type=DATEV2 col=-1, colname=dt, nullable=1)] has_varlen_slots=1) Tuple(id=18 slots=[Slot(id=518 type=DECIMALV2(22, 5) col=-1, colname=cost, nullable=1), Slot(id=526 type=DATEV2 col=-1, colname=dt, nullable=1), Slot(id=546 type=VARCHAR col=-1, colname=, nullable=1), Slot(id=547 type=VARCHAR col=-1, colname=, nullable=0), Slot(id=548 type=DECIMALV2(22, 5) col=-1, colname=cost, nullable=1), Slot(id=549 type=VARCHAR col=-1, colname=brand_code, nullable=1), Slot(id=554 type=DECIMALV2(22, 5) col=-1, colname=cost, nullable=1), Slot(id=562 type=DATEV2 col=-1, colname=dt, nullable=1), Slot(id=582 type=VARCHAR col=-1, colname=brand_code, nullable=1), Slot(id=587 type=DECIMALV2(22, 5) col=-1, colname=cost, nullable=1), Slot(id=595 type=DATEV2 col=-1, colname=dt, nullable=1), Slot(id=615 type=DATEV2 col=-1, colname=, nullable=1)] has_varlen_slots=1) Tuple(id=5 slots=[Slot(id=172 type=VARCHAR col=-1, colname=, nullable=1), Slot(id=173 type=VARCHAR col=-1, colname=, nullable=1), Slot(id=174 type=VARCHAR col=-1, colname=, nullable=1)] has_varlen_slots=1) Tuple(id=0 slots=[Slot(id=0 type=VARCHAR col=-1, colname=brand_code, nullable=1), Slot(id=13 type=DATEV2 col=-1, colname=dt, nullable=1)] has_varlen_slots=1) ### What You Expected? This BUG need fixed. ### How to Reproduce? select * from ( select (select a1,a2,sum(a3) from table_1 group by a1,a2) as tmp1 left join (select a1,a2,sum(a3) from table_2 group by a1,a2) as tmp2 on tmp1.a1=tmp2.a1 and tmp1.a2=tmp2.a2 ) as tmp3 ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [FIX](map)fix map with rowstore table [doris]
amorynan commented on PR #28877: URL: https://github.com/apache/doris/pull/28877#issuecomment-1867432558 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (#28851) [doris]
yiguolei merged PR #28871: URL: https://github.com/apache/doris/pull/28871 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (enhance)(S3) Change s3 metric from bvar adder to latency recorder [doris]
github-actions[bot] commented on code in PR #28861: URL: https://github.com/apache/doris/pull/28861#discussion_r1434872534 ## be/src/util/s3_util.h: ## @@ -19,6 +19,7 @@ #include Review Comment: warning: 'aws/core/Aws.h' file not found [clang-diagnostic-error] ```cpp #include ^ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [FIX](map)fix map with rowstore table [doris]
amorynan commented on PR #28877: URL: https://github.com/apache/doris/pull/28877#issuecomment-1867435674 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-1.2-lts updated: [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (#28871)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch branch-1.2-lts in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-1.2-lts by this push: new ed4d5bf951f [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (#28871) ed4d5bf951f is described below commit ed4d5bf951f8c827fb0e778010775a2a20bce7c1 Author: TengJianPing <18241664+jackte...@users.noreply.github.com> AuthorDate: Fri Dec 22 17:13:38 2023 +0800 [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (#28871) --- be/src/vec/columns/column_vector.cpp | 3 ++- be/src/vec/exec/join/vhash_join_node.cpp | 11 ++- be/src/vec/exec/join/vhash_join_node.h | 1 + be/src/vec/functions/function_case.h | 20 ++-- be/src/vec/functions/function_string.cpp | 28 +--- be/src/vec/functions/multiply.cpp| 3 ++- 6 files changed, 46 insertions(+), 20 deletions(-) diff --git a/be/src/vec/columns/column_vector.cpp b/be/src/vec/columns/column_vector.cpp index e656b97f2e7..93b59bce7ad 100644 --- a/be/src/vec/columns/column_vector.cpp +++ b/be/src/vec/columns/column_vector.cpp @@ -479,7 +479,8 @@ ColumnPtr ColumnVector::replicate(const IColumn::Offsets& offsets) const { res_data.reserve(offsets.back()); // vectorized this code to speed up -IColumn::Offset counts[size]; +auto counts_uptr = std::unique_ptr(new IColumn::Offset[size]); +IColumn::Offset* counts = counts_uptr.get(); for (ssize_t i = 0; i < size; ++i) { counts[i] = offsets[i] - offsets[i - 1]; } diff --git a/be/src/vec/exec/join/vhash_join_node.cpp b/be/src/vec/exec/join/vhash_join_node.cpp index 3efea5d8c6a..b3c7794965e 100644 --- a/be/src/vec/exec/join/vhash_join_node.cpp +++ b/be/src/vec/exec/join/vhash_join_node.cpp @@ -656,7 +656,7 @@ Status HashJoinNode::_materialize_build_side(RuntimeState* state) { RETURN_IF_ERROR(child(1)->open(state)); SCOPED_TIMER(_build_timer); -MutableBlock mutable_block(child(1)->row_desc().tuple_descriptors()); +MutableBlock mutable_block; uint8_t index = 0; int64_t last_mem_used = 0; @@ -669,6 +669,7 @@ Status HashJoinNode::_materialize_build_side(RuntimeState* state) { Block block; // If eos or have already met a null value using short-circuit strategy, we do not need to pull // data from data. +_build_col_ids.resize(_build_expr_ctxs.size()); while (!eos && !_short_circuit_for_null_in_probe_side) { block.clear_column_data(); RETURN_IF_CANCELLED(state); @@ -679,6 +680,8 @@ Status HashJoinNode::_materialize_build_side(RuntimeState* state) { _mem_used += block.allocated_bytes(); if (block.rows() != 0) { +RETURN_IF_ERROR(_do_evaluate(block, _build_expr_ctxs, *_build_expr_call_timer, + _build_col_ids)); SCOPED_TIMER(_build_side_merge_block_timer); RETURN_IF_CATCH_BAD_ALLOC(mutable_block.merge(block)); } @@ -889,8 +892,6 @@ Status HashJoinNode::_process_build_block(RuntimeState* state, Block& block, uin ColumnRawPtrs raw_ptrs(_build_expr_ctxs.size()); ColumnUInt8::MutablePtr null_map_val; -std::vector res_col_ids(_build_expr_ctxs.size()); -RETURN_IF_ERROR(_do_evaluate(block, _build_expr_ctxs, *_build_expr_call_timer, res_col_ids)); if (_join_op == TJoinOp::LEFT_OUTER_JOIN || _join_op == TJoinOp::FULL_OUTER_JOIN) { _convert_block_to_null(block); } @@ -898,7 +899,7 @@ Status HashJoinNode::_process_build_block(RuntimeState* state, Block& block, uin // so we have to initialize this flag by the first build block. if (!_has_set_need_null_map_for_build) { _has_set_need_null_map_for_build = true; -_set_build_ignore_flag(block, res_col_ids); +_set_build_ignore_flag(block, _build_col_ids); } if (_short_circuit_for_null_in_build_side || _build_side_ignore_null) { null_map_val = ColumnUInt8::create(); @@ -906,7 +907,7 @@ Status HashJoinNode::_process_build_block(RuntimeState* state, Block& block, uin } // Get the key column that needs to be built -Status st = _extract_join_column(block, null_map_val, raw_ptrs, res_col_ids); +Status st = _extract_join_column(block, null_map_val, raw_ptrs, _build_col_ids); st = std::visit( Overload { diff --git a/be/src/vec/exec/join/vhash_join_node.h b/be/src/vec/exec/join/vhash_join_node.h index 97af3818b8d..376ed8f377c 100644 --- a/be/src/vec/exec/join/vhash_join_node.h +++ b/be/src/vec/exec/join/vhash_join_node.h @@ -341,6 +341,7 @@ private: std::unordered_map> _inserted_rows; std::vector _runtime_filters; +std::vector _build_col_ids; }; } // namespace vectorized
Re: [PR] [Feature](inverted index) add lowercase option for inverted index analyzer [doris]
doris-robot commented on PR #28704: URL: https://github.com/apache/doris/pull/28704#issuecomment-1867438132 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit 490c05e0613f7144282dca56d2abade4013743e1, data reload: false run tpch-sf100 query with default conf and session variables q1 4709442644144414 q2 377 151 158 151 q3 1463126712621262 q4 1129896 934 896 q5 3193321031863186 q6 248 130 129 129 q7 981 488 492 488 q8 2206220821912191 q9 6677668966886688 q10 3204325932653259 q11 310 188 186 186 q12 360 214 209 209 q13 4553378338313783 q14 240 220 225 220 q15 578 521 520 520 q16 453 395 387 387 q17 1020587 547 547 q18 7072700069016901 q19 1549140514911405 q20 543 310 286 286 q21 3112268426652665 q22 347 280 295 280 Total cold run time: 44324 ms Total hot run time: 40053 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4383436543714365 q2 269 167 176 167 q3 3517353035023502 q4 2384237123822371 q5 5723574057205720 q6 237 123 123 123 q7 2369188118581858 q8 3526352735183518 q9 9002903090299029 q10 3931402040034003 q11 497 371 358 358 q12 764 599 588 588 q13 4301357735713571 q14 299 265 250 250 q15 577 520 529 520 q16 509 467 440 440 q17 1895186818731868 q18 8594805081458050 q19 1741174017551740 q20 2291196519551955 q21 6516617061766170 q22 503 440 429 429 Total cold run time: 63828 ms Total hot run time: 60595 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[I] [Enhancement] Catalog配置JDBC权限能否和设置用户权限一致 [doris]
MRYOG opened a new issue, #28880: URL: https://github.com/apache/doris/issues/28880 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description Catalog通过JDBC 方式连接的MySQL数据库,执行delete操作时,提示errCode = 2, detailMessage = External catalog 'up_mysql' is not allowed in 'DeleteStmt' 但是配置的用户有对表进行增删改查的权限 ### Solution catalog 权限能否与用户本身权限一致,不对删除等操作进行限制,可以通过本身数据库对用户的权限限制。或者doris中也可以对catalog 进行权限限制 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](invert index) Empty strings are not written to the index in the… [doris]
doris-robot commented on PR #28822: URL: https://github.com/apache/doris/pull/28822#issuecomment-1867441762 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 43.49 seconds stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.1 seconds inserted 1000 Rows, about 343K ops/s storage size: 17184198005 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [FIX](map)fix map with rowstore table [doris]
github-actions[bot] commented on code in PR #28877: URL: https://github.com/apache/doris/pull/28877#discussion_r1434875915 ## be/test/vec/jsonb/serialize_test.cpp: ## @@ -177,6 +181,159 @@ TEST(BlockSerializeTest, Array) { EXPECT_EQ(block.dump_data(), new_block.dump_data()); } +TEST(BlockSerializeTest, Map) { Review Comment: warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size] ```cpp TEST(BlockSerializeTest, Map) { ^ ``` Additional context **be/test/vec/jsonb/serialize_test.cpp:183:** 82 lines including whitespace and comments (threshold 80) ```cpp TEST(BlockSerializeTest, Map) { ^ ``` ## be/test/vec/jsonb/serialize_test.cpp: ## @@ -177,6 +181,159 @@ EXPECT_EQ(block.dump_data(), new_block.dump_data()); } +TEST(BlockSerializeTest, Map) { +TabletSchema schema; +TabletColumn map; +map.set_name("m"); +map.set_unique_id(1); +map.set_type(FieldType::OLAP_FIELD_TYPE_MAP); +schema.append_column(map); +// map string string +DataTypePtr s = std::make_shared(std::make_shared()); +DataTypePtr d = std::make_shared(std::make_shared()); +DataTypePtr m = std::make_shared(s, d); +Array k1, k2, v1, v2; +k1.push_back("null"); +k1.push_back("doris"); +k1.push_back("clever amory"); +v1.push_back("ss"); +v1.push_back(Null()); +v1.push_back("NULL"); +k2.push_back("hello amory"); +k2.push_back("NULL"); +k2.push_back("cute amory"); +k2.push_back("doris"); +v2.push_back("s"); +v2.push_back("0"); +v2.push_back("sf"); +v2.push_back(Null()); +Map m1, m2; +m1.push_back(k1); +m1.push_back(v1); +m2.push_back(k2); +m2.push_back(v2); +MutableColumnPtr map_column = m->create_column(); +map_column->reserve(2); +map_column->insert(m1); +map_column->insert(m2); +vectorized::ColumnWithTypeAndName type_and_name(map_column->get_ptr(), m, "test_map"); +vectorized::Block block; +block.insert(type_and_name); + +MutableColumnPtr col = ColumnString::create(); +// serialize +std::cout << "serialize to jsonb" << std::endl; +JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast(*col.get()), + block.columns(), + create_data_type_serdes(block.get_data_types())); +// deserialize +TupleDescriptor read_desc(PTupleDescriptor(), true); +// slot +TSlotDescriptor tslot; +tslot.__set_colName("m"); +tslot.nullIndicatorBit = -1; +tslot.nullIndicatorByte = 0; +TypeDescriptor type_desc(TYPE_MAP); +type_desc.children.push_back(TypeDescriptor(TYPE_STRING)); +type_desc.children.push_back(TypeDescriptor(TYPE_INT)); +type_desc.contains_nulls.push_back(true); +type_desc.contains_nulls.push_back(true); +tslot.__set_col_unique_id(1); +tslot.__set_slotType(type_desc.to_thrift()); +SlotDescriptor* slot = new SlotDescriptor(tslot); +read_desc.add_slot(slot); + +Block new_block = block.clone_empty(); +std::unordered_map col_uid_to_idx; +std::vector default_values; +default_values.resize(read_desc.slots().size()); +for (int i = 0; i < read_desc.slots().size(); ++i) { +col_uid_to_idx[read_desc.slots()[i]->col_unique_id()] = i; +default_values[i] = read_desc.slots()[i]->col_default_value(); +std::cout << "uid " << read_desc.slots()[i]->col_unique_id() << ":" << i << std::endl; +} +std::cout << block.dump_data() << std::endl; +std::cout << new_block.dump_data() << std::endl; +std::cout << "deserialize from jsonb" << std::endl; + JsonbSerializeUtil::jsonb_to_block(create_data_type_serdes(read_desc.slots()), + static_cast(*col.get()), col_uid_to_idx, + new_block, default_values); +std::cout << block.dump_data() << std::endl; +std::cout << new_block.dump_data() << std::endl; +EXPECT_EQ(block.dump_data(), new_block.dump_data()); +} + +TEST(BlockSerializeTest, Struct) { +TabletSchema schema; +TabletColumn struct_col; +struct_col.set_name("struct"); +struct_col.set_unique_id(1); +struct_col.set_type(FieldType::OLAP_FIELD_TYPE_STRUCT); +schema.append_column(struct_col); +vectorized::Block block; +{ +DataTypePtr s = std::make_shared(std::make_shared()); +DataTypePtr d = std::make_shared(std::make_shared()); +DataTypePtr m = std::make_shared(std::make_shared()); +DataTypePtr st = std::make_shared(std::vector {s, d, m}); +Tuple t1, t2; +t1.push_back(String("amory cute")); +t1.push_back(__int128_t(37)); +t1.push_back(true); +t2.push_back("null"); +t2.push_back(__int128_t(26)); +t2.push_back(false); +MutableColumnPtr struct_column = st->create_colum
Re: [PR] [FIX](map)fix map with rowstore table [doris]
github-actions[bot] commented on code in PR #28877: URL: https://github.com/apache/doris/pull/28877#discussion_r1434877793 ## be/test/vec/jsonb/serialize_test.cpp: ## @@ -177,6 +181,159 @@ TEST(BlockSerializeTest, Array) { EXPECT_EQ(block.dump_data(), new_block.dump_data()); } +TEST(BlockSerializeTest, Map) { +TabletSchema schema; +TabletColumn map; +map.set_name("m"); +map.set_unique_id(1); +map.set_type(FieldType::OLAP_FIELD_TYPE_MAP); +schema.append_column(map); +// map string string +DataTypePtr s = std::make_shared(std::make_shared()); +DataTypePtr d = std::make_shared(std::make_shared()); +DataTypePtr m = std::make_shared(s, d); +Array k1, k2, v1, v2; +k1.push_back("null"); +k1.push_back("doris"); +k1.push_back("clever amory"); +v1.push_back("ss"); +v1.push_back(Null()); +v1.push_back("NULL"); +k2.push_back("hello amory"); +k2.push_back("NULL"); +k2.push_back("cute amory"); +k2.push_back("doris"); +v2.push_back("s"); +v2.push_back("0"); +v2.push_back("sf"); +v2.push_back(Null()); +Map m1, m2; +m1.push_back(k1); +m1.push_back(v1); +m2.push_back(k2); +m2.push_back(v2); +MutableColumnPtr map_column = m->create_column(); +map_column->reserve(2); +map_column->insert(m1); +map_column->insert(m2); +vectorized::ColumnWithTypeAndName type_and_name(map_column->get_ptr(), m, "test_map"); +vectorized::Block block; +block.insert(type_and_name); + +MutableColumnPtr col = ColumnString::create(); +// serialize +std::cout << "serialize to jsonb" << std::endl; +JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast(*col.get()), + block.columns(), + create_data_type_serdes(block.get_data_types())); +// deserialize +TupleDescriptor read_desc(PTupleDescriptor(), true); +// slot +TSlotDescriptor tslot; +tslot.__set_colName("m"); +tslot.nullIndicatorBit = -1; +tslot.nullIndicatorByte = 0; +TypeDescriptor type_desc(TYPE_MAP); +type_desc.children.push_back(TypeDescriptor(TYPE_STRING)); +type_desc.children.push_back(TypeDescriptor(TYPE_INT)); +type_desc.contains_nulls.push_back(true); +type_desc.contains_nulls.push_back(true); +tslot.__set_col_unique_id(1); +tslot.__set_slotType(type_desc.to_thrift()); +SlotDescriptor* slot = new SlotDescriptor(tslot); Review Comment: warning: use auto when initializing with new to avoid duplicating the type name [modernize-use-auto] ```suggestion auto* slot = new SlotDescriptor(tslot); ``` ## be/test/vec/jsonb/serialize_test.cpp: ## @@ -177,6 +181,159 @@ EXPECT_EQ(block.dump_data(), new_block.dump_data()); } +TEST(BlockSerializeTest, Map) { +TabletSchema schema; +TabletColumn map; +map.set_name("m"); +map.set_unique_id(1); +map.set_type(FieldType::OLAP_FIELD_TYPE_MAP); +schema.append_column(map); +// map string string +DataTypePtr s = std::make_shared(std::make_shared()); +DataTypePtr d = std::make_shared(std::make_shared()); +DataTypePtr m = std::make_shared(s, d); +Array k1, k2, v1, v2; +k1.push_back("null"); +k1.push_back("doris"); +k1.push_back("clever amory"); +v1.push_back("ss"); +v1.push_back(Null()); +v1.push_back("NULL"); +k2.push_back("hello amory"); +k2.push_back("NULL"); +k2.push_back("cute amory"); +k2.push_back("doris"); +v2.push_back("s"); +v2.push_back("0"); +v2.push_back("sf"); +v2.push_back(Null()); +Map m1, m2; +m1.push_back(k1); +m1.push_back(v1); +m2.push_back(k2); +m2.push_back(v2); +MutableColumnPtr map_column = m->create_column(); +map_column->reserve(2); +map_column->insert(m1); +map_column->insert(m2); +vectorized::ColumnWithTypeAndName type_and_name(map_column->get_ptr(), m, "test_map"); +vectorized::Block block; +block.insert(type_and_name); + +MutableColumnPtr col = ColumnString::create(); +// serialize +std::cout << "serialize to jsonb" << std::endl; +JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast(*col.get()), + block.columns(), + create_data_type_serdes(block.get_data_types())); +// deserialize +TupleDescriptor read_desc(PTupleDescriptor(), true); +// slot +TSlotDescriptor tslot; +tslot.__set_colName("m"); +tslot.nullIndicatorBit = -1; +tslot.nullIndicatorByte = 0; +TypeDescriptor type_desc(TYPE_MAP); +type_desc.children.push_back(TypeDescriptor(TYPE_STRING)); +type_desc.children.push_back(TypeDescriptor(TYPE_INT)); +type_desc.contains_nulls.push_back(true); +type_desc.contains_nulls.push_back(true); +tslot.__set_col_unique_id(1); +tslot.__set_slotType(type_desc.to_
Re: [PR] [test](regression)Add select before analyze. [doris]
Jibing-Li closed pull request #28714: [test](regression)Add select before analyze. URL: https://github.com/apache/doris/pull/28714 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](group commit) refector replay wal function [doris]
doris-robot commented on PR #28756: URL: https://github.com/apache/doris/pull/28756#issuecomment-1867445230 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit b41679ab54ad3eec3c9af8c9db7ad269a1550c57, data reload: false run tpch-sf100 query with default conf and session variables q1 4928465646484648 q2 375 159 198 159 q3 1521131212411241 q4 1175100010391000 q5 3283325032653250 q6 273 132 130 130 q7 1050527 558 527 q8 2271235523062306 q9 6858683868786838 q10 3247331432963296 q11 325 187 185 185 q12 392 221 215 215 q13 4641398439353935 q14 247 212 212 212 q15 572 528 530 528 q16 448 391 396 391 q17 1017646 508 508 q18 7140702268436843 q19 1630149915271499 q20 563 331 304 304 q21 3322288129012881 q22 358 283 288 283 Total cold run time: 45636 ms Total hot run time: 41179 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4579456745764567 q2 293 190 198 190 q3 3601358335743574 q4 2455244124382438 q5 5827584458315831 q6 261 124 124 124 q7 2412188018461846 q8 3655363536473635 q9 9063903290019001 q10 3968405840564056 q11 520 395 400 395 q12 789 586 592 586 q13 4396367837183678 q14 295 258 256 256 q15 587 538 539 538 q16 514 459 482 459 q17 1971190318821882 q18 8633835883368336 q19 1852188318741874 q20 2274197919591959 q21 6734630362946294 q22 509 423 429 423 Total cold run time: 65188 ms Total hot run time: 61942 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipelineX](refactor) rename functions [doris]
yiguolei merged PR #28846: URL: https://github.com/apache/doris/pull/28846 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [pipelineX](refactor) rename functions (#28846)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 9e0a2e861cf [pipelineX](refactor) rename functions (#28846) 9e0a2e861cf is described below commit 9e0a2e861cf05fcb7cb22fbcfe95955b16ab7f73 Author: Gabriel AuthorDate: Fri Dec 22 17:24:39 2023 +0800 [pipelineX](refactor) rename functions (#28846) --- be/src/pipeline/exec/aggregation_sink_operator.h | 4 ++-- be/src/pipeline/exec/analytic_sink_operator.h | 4 ++-- be/src/pipeline/exec/assert_num_rows_operator.h| 2 +- be/src/pipeline/exec/exchange_source_operator.h| 2 +- be/src/pipeline/exec/hashjoin_build_sink.h | 2 +- be/src/pipeline/exec/hashjoin_probe_operator.h | 2 +- be/src/pipeline/exec/nested_loop_join_build_operator.h | 2 +- be/src/pipeline/exec/nested_loop_join_probe_operator.h | 2 +- be/src/pipeline/exec/partition_sort_sink_operator.h| 4 ++-- be/src/pipeline/exec/scan_operator.h | 2 +- be/src/pipeline/exec/set_probe_sink_operator.h | 2 +- be/src/pipeline/exec/set_sink_operator.h | 2 +- be/src/pipeline/exec/sort_sink_operator.h | 4 ++-- be/src/pipeline/exec/streaming_aggregation_sink_operator.h | 2 +- be/src/pipeline/pipeline.h | 2 +- be/src/pipeline/pipeline_x/operator.h | 4 ++-- be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp | 8 17 files changed, 25 insertions(+), 25 deletions(-) diff --git a/be/src/pipeline/exec/aggregation_sink_operator.h b/be/src/pipeline/exec/aggregation_sink_operator.h index 2cd6ef50939..97be9dcd6a3 100644 --- a/be/src/pipeline/exec/aggregation_sink_operator.h +++ b/be/src/pipeline/exec/aggregation_sink_operator.h @@ -366,12 +366,12 @@ public: Status sink(RuntimeState* state, vectorized::Block* in_block, SourceState source_state) override; -DataDistribution get_local_exchange_type() const override { +DataDistribution required_data_distribution() const override { if (_probe_expr_ctxs.empty()) { return _needs_finalize || DataSinkOperatorX::_child_x ->ignore_data_distribution() ? DataDistribution(ExchangeType::PASSTHROUGH) - : DataSinkOperatorX::get_local_exchange_type(); + : DataSinkOperatorX::required_data_distribution(); } return _is_colocate ? DataDistribution(ExchangeType::BUCKET_HASH_SHUFFLE, _partition_exprs) : DataDistribution(ExchangeType::HASH_SHUFFLE, _partition_exprs); diff --git a/be/src/pipeline/exec/analytic_sink_operator.h b/be/src/pipeline/exec/analytic_sink_operator.h index 14ed8c815b1..3e0eb85f76d 100644 --- a/be/src/pipeline/exec/analytic_sink_operator.h +++ b/be/src/pipeline/exec/analytic_sink_operator.h @@ -107,7 +107,7 @@ public: Status sink(RuntimeState* state, vectorized::Block* in_block, SourceState source_state) override; -DataDistribution get_local_exchange_type() const override { +DataDistribution required_data_distribution() const override { if (_partition_by_eq_expr_ctxs.empty()) { return {ExchangeType::PASSTHROUGH}; } else if (_order_by_eq_expr_ctxs.empty()) { @@ -115,7 +115,7 @@ public: ? DataDistribution(ExchangeType::BUCKET_HASH_SHUFFLE, _partition_exprs) : DataDistribution(ExchangeType::HASH_SHUFFLE, _partition_exprs); } -return DataSinkOperatorX::get_local_exchange_type(); +return DataSinkOperatorX::required_data_distribution(); } private: diff --git a/be/src/pipeline/exec/assert_num_rows_operator.h b/be/src/pipeline/exec/assert_num_rows_operator.h index 1e796b622dc..bb5e65168b6 100644 --- a/be/src/pipeline/exec/assert_num_rows_operator.h +++ b/be/src/pipeline/exec/assert_num_rows_operator.h @@ -57,7 +57,7 @@ public: [[nodiscard]] bool is_source() const override { return false; } -DataDistribution get_local_exchange_type() const override { +DataDistribution required_data_distribution() const override { return {ExchangeType::PASSTHROUGH}; } diff --git a/be/src/pipeline/exec/exchange_source_operator.h b/be/src/pipeline/exec/exchange_source_operator.h index 221a43779a1..b621da38072 100644 --- a/be/src/pipeline/exec/exchange_source_operator.h +++ b/be/src/pipeline/exec/exchange_source_operator.h @@ -117,7 +117,7 @@ public: return _sub_plan_query_statistics_recvr; } -DataDistribution get_local_exchange_type() const override { +DataDistribution required_data_distribution() const override { if (Oper
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (#28823) [doris]
yiguolei closed pull request #28836: [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (#28823) URL: https://github.com/apache/doris/pull/28836 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (#28823) [doris]
yiguolei commented on PR #28836: URL: https://github.com/apache/doris/pull/28836#issuecomment-1867450195 close the PR temporary, keep the branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [FIX](complextype) fix complextype has empty literal may make be core [doris]
doris-robot commented on PR #28820: URL: https://github.com/apache/doris/pull/28820#issuecomment-1867451565 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 51.73 seconds stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.7 seconds inserted 1000 Rows, about 336K ops/s storage size: 17164713746 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [exec](compress) use FragmentTransmissionCompressionCodec control the exchange compress behavior [doris]
yiguolei commented on PR #28818: URL: https://github.com/apache/doris/pull/28818#issuecomment-1867451815 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) "not is null" stats estimation fix [doris]
doris-robot commented on PR #28860: URL: https://github.com/apache/doris/pull/28860#issuecomment-1867452270 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit f149a78140b036af21fbd049036f397fd228bd8a, data reload: false run tpch-sf100 query with default conf and session variables q1 4716439344544393 q2 371 145 139 139 q3 1460131812551255 q4 1112914 881 881 q5 3154317231483148 q6 249 131 135 131 q7 1004492 490 490 q8 2192222821752175 q9 6701662366216621 q10 3224325932773259 q11 310 185 182 182 q12 349 203 200 200 q13 4528379937893789 q14 245 212 212 212 q15 581 528 531 528 q16 440 382 384 382 q17 1016636 502 502 q18 7109687068126812 q19 1548136414811364 q20 550 313 322 313 q21 3074268126772677 q22 344 277 283 277 Total cold run time: 44277 ms Total hot run time: 39730 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4336436343524352 q2 271 161 170 161 q3 3516351034933493 q4 2370237423652365 q5 5726569557135695 q6 240 123 124 123 q7 2367187118441844 q8 3517353435223522 q9 8996900289678967 q10 3905402540334025 q11 477 369 371 369 q12 774 609 607 607 q13 4313357735513551 q14 285 251 260 251 q15 590 520 527 520 q16 507 451 473 451 q17 1893185119321851 q18 8625816381728163 q19 1731178617341734 q20 2267194019471940 q21 6529613261116111 q22 490 419 412 412 Total cold run time: 63725 ms Total hot run time: 60507 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [FIX](complextype)fix count func with complex type [doris]
doris-robot commented on PR #28873: URL: https://github.com/apache/doris/pull/28873#issuecomment-1867452646 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 43.77 seconds stream load tsv: 583 seconds loaded 74807831229 Bytes, about 122 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17183949907 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](paimon)fix `like` predicate [doris]
doris-robot commented on PR #28803: URL: https://github.com/apache/doris/pull/28803#issuecomment-1867454758 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 43.58 seconds stream load tsv: 582 seconds loaded 74807831229 Bytes, about 122 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17183490722 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [nereids] fix join fd computing bug [doris]
xzj7019 commented on PR #28849: URL: https://github.com/apache/doris/pull/28849#issuecomment-1867457643 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Nereids) Pull up join from union all [doris]
xzj7019 commented on PR #28682: URL: https://github.com/apache/doris/pull/28682#issuecomment-1867458426 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [hotfix](jdbc catalog) fix load table and column names npe [doris]
doris-robot commented on PR #28865: URL: https://github.com/apache/doris/pull/28865#issuecomment-1867459505 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit a2a5adbe6d2499a5f47f604175473056e66f5975, data reload: false run tpch-sf100 query with default conf and session variables q1 4682441444284414 q2 372 143 157 143 q3 1465125212081208 q4 1114881 927 881 q5 3190318131893181 q6 250 133 131 131 q7 1002484 496 484 q8 2204222522072207 q9 6706662566596625 q10 3211329632823282 q11 315 189 184 184 q12 352 210 208 208 q13 4587383038293829 q14 241 219 217 217 q15 569 523 529 523 q16 441 374 381 374 q17 1027663 561 561 q18 7025697467556755 q19 1556143314241424 q20 510 328 283 283 q21 3093260426492604 q22 350 278 282 278 Total cold run time: 44262 ms Total hot run time: 39796 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4351434343654343 q2 272 165 173 165 q3 3534350035223500 q4 2386238223712371 q5 5740572557155715 q6 243 125 126 125 q7 2379185918951859 q8 3512352835453528 q9 9055902590479025 q10 3937400040044000 q11 480 372 364 364 q12 775 606 590 590 q13 4277354635253525 q14 286 250 256 250 q15 584 525 521 521 q16 514 458 459 458 q17 1903188218431843 q18 8529796481057964 q19 1737174317581743 q20 2251195419411941 q21 6486614161186118 q22 508 426 432 426 Total cold run time: 63739 ms Total hot run time: 60374 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [FIX](map)fix map with rowstore table [doris]
github-actions[bot] commented on PR #28877: URL: https://github.com/apache/doris/pull/28877#issuecomment-1867459955 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [FIX](map)fix map with rowstore table [doris]
github-actions[bot] commented on PR #28877: URL: https://github.com/apache/doris/pull/28877#issuecomment-1867460007 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ci) tpch pipeline should not re-load data [doris]
hello-stephen commented on PR #28874: URL: https://github.com/apache/doris/pull/28874#issuecomment-1867462805 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](pipelineX) control exchange sink by memory usage [doris]
doris-robot commented on PR #28814: URL: https://github.com/apache/doris/pull/28814#issuecomment-1867463778 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.79 seconds stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 27.9 seconds inserted 1000 Rows, about 358K ops/s storage size: 17183821535 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](pipelineX)do not override dependency() function in pipelineX [doris]
doris-robot commented on PR #28848: URL: https://github.com/apache/doris/pull/28848#issuecomment-1867466558 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit 5af4df47e484d94afe251b74a67dc31c3c02c62a, data reload: false run tpch-sf100 query with default conf and session variables q1 4673440344124403 q2 367 160 158 158 q3 1464123112481231 q4 1102944 863 863 q5 3165317531773175 q6 247 132 127 127 q7 994 488 488 488 q8 2151224721892189 q9 6670665866986658 q10 3236326632713266 q11 311 188 188 188 q12 354 213 208 208 q13 4551378538073785 q14 247 222 217 217 q15 565 522 526 522 q16 437 382 388 382 q17 1020625 589 589 q18 7142684668576846 q19 1547144914581449 q20 550 325 294 294 q21 3060267326682668 q22 350 283 295 283 Total cold run time: 44203 ms Total hot run time: 39989 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4346434343354335 q2 274 165 166 165 q3 3508350434903490 q4 2387236723652365 q5 5746572157275721 q6 239 121 122 121 q7 2382188218361836 q8 3516351835233518 q9 9027895890128958 q10 3911401040474010 q11 491 368 368 368 q12 771 593 590 590 q13 4295356735243524 q14 288 257 247 247 q15 578 522 518 518 q16 505 476 461 461 q17 1875183518461835 q18 8588821281278127 q19 1728175917361736 q20 2263194119351935 q21 6512615261436143 q22 500 424 419 419 Total cold run time: 63730 ms Total hot run time: 60422 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[I] [Bug] doris be 自动退出 [doris]
haoyan19881215 opened a new issue, #28881: URL: https://github.com/apache/doris/issues/28881 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 2.0.2 ### What's Wrong? there are 8 be in my doris cluster,but 2 be Automatic stop when they running for 1~2 hours。but I cannot find any exception in logs **be.out** ` start time: Fri Dec 22 09:47:07 CST 2023 INFO: java_cmd /opt/jdk1.8.0_191/bin/java INFO: jdk_version 8 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/hadoop_hdfs/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'. ` **be.INFO** ` I1222 11:57:49.784793 24196 olap_server.cpp:1064] cooldown producer get tablet num: 0 I1222 11:57:54.344010 24947 heartbeat_server.cpp:61] get heartbeat from FE.host:10.237.22.118, port:9020, cluster id:1914286766, counter:1561, BE start time: 1703209629783 I1222 11:57:57.641873 24283 task_worker_pool.cpp:1068] successfully report TASK|host=10.237.22.118|port=9020 I1222 11:58:09.578222 24038 load_channel_mgr.cpp:250] cleaning timed out load channels I1222 11:58:09.578332 24038 load_channel_mgr.cpp:282] load mem consumption(bytes). limit: 13420866764, current: 0, peak: 0, total running load channels: 0 I1222 11:58:09.785163 24196 olap_server.cpp:1064] cooldown producer get tablet num: 0 I1222 11:58:10.643011 24283 task_worker_pool.cpp:1068] successfully report TASK|host=10.237.22.118|port=9020 I1222 11:58:23.643994 24283 task_worker_pool.cpp:1068] successfully report TASK|host=10.237.22.118|port=9020 I1222 11:58:29.785368 24196 olap_server.cpp:1064] cooldown producer get tablet num: 0 I1222 11:58:31.761829 24285 tablet_manager.cpp:1016] find expired transactions for 0 tablets I1222 11:58:31.761873 24285 tablet_manager.cpp:1048] success to build all report tablets info. tablet_count=0 I1222 11:58:31.762550 24285 task_worker_pool.cpp:1068] successfully report TABLET|host=10.237.22.118|port=9020 I1222 11:58:36.864832 23487 daemon.cpp:397] doris start to exit I1222 11:58:38.206190 24284 data_dir.cpp:810] path: /opt/apache-doris-2.0.2-bin-x64-noavx2/be/storage total capacity: 1022174953472, available capacity: 1013264003072 I1222 11:58:38.206262 24284 storage_engine.cpp:383] get root path info cost: 0 ms. tablet counter: 0 I1222 11:58:38.207151 24284 task_worker_pool.cpp:1068] successfully report DISK|host=10.237.22.118|port=9020 I1222 11:58:38.644892 24283 task_worker_pool.cpp:1068] successfully report TASK|host=10.237.22.118|port=9020 I1222 11:58:39.867292 23487 server.cpp:1167] Server[doris::PInternalServiceImpl] is going to quit I1222 11:58:39.892066 24882 thrift_server.cpp:170] ThriftServer heartbeat exited I1222 11:58:39.892594 24289 thrift_server.cpp:170] ThriftServer backend exited I1222 11:58:39.892784 23487 storage_engine.cpp:546] begin stopping storage engine I1222 11:58:39.892843 24190 olap_server.cpp:364] try to perform path gc by rowsetid! I1222 11:58:39.893136 23487 storage_engine.cpp:566] start join garbage sweeper thread I1222 11:58:39.893158 23487 storage_engine.cpp:568] end join garbage sweeper thread I1222 11:58:39.893258 23487 storage_engine.cpp:588] end stopping storage engine I1222 11:58:40.893028 24284 data_dir.cpp:810] path: /opt/apache-doris-2.0.2-bin-x64-noavx2/be/storage total capacity: 1022174953472, available capacity: 1013264003072 I1222 11:58:40.893131 24284 storage_engine.cpp:383] get root path info cost: 0 ms. tablet counter: 0 I1222 11:58:40.893143 24285 tablet_manager.cpp:1016] find expired transactions for 0 tablets I1222 11:58:40.893206 24285 tablet_manager.cpp:1048] success to build all report tablets info. tablet_count=0 I1222 11:58:40.893572 24283 task_worker_pool.cpp:1068] successfully report TASK|host=10.237.22.118|port=9020 I1222 11:58:40.8
Re: [PR] [fix](pipelineX) fix use global rf when there no shared_scans [doris]
doris-robot commented on PR #28869: URL: https://github.com/apache/doris/pull/28869#issuecomment-1867473443 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit cd467df94f2ed4a92a8007ac5582fcf1a05e29bf, data reload: false run tpch-sf100 query with default conf and session variables q1 4673441744524417 q2 377 138 157 138 q3 1451127611761176 q4 1115922 919 919 q5 3153314131693141 q6 250 128 124 124 q7 1009491 499 491 q8 2189221621982198 q9 6714665566836655 q10 3216326832533253 q11 301 191 187 187 q12 355 204 202 202 q13 4529378537763776 q14 242 217 216 216 q15 579 529 520 520 q16 440 394 385 385 q17 1013613 533 533 q18 7033674868516748 q19 1536142015301420 q20 518 317 304 304 q21 3136266926832669 q22 352 275 287 275 Total cold run time: 44181 ms Total hot run time: 39747 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4346435443364336 q2 269 166 171 166 q3 3508349234853485 q4 2377236623662366 q5 5707569657315696 q6 240 122 120 120 q7 2366186118851861 q8 3545351736523517 q9 9033902789778977 q10 3935400339783978 q11 484 363 369 363 q12 754 593 606 593 q13 4274355235443544 q14 295 256 258 256 q15 576 514 527 514 q16 506 473 452 452 q17 1896186818601860 q18 8594824079927992 q19 1747178717861786 q20 2245197219271927 q21 6497615061256125 q22 507 442 416 416 Total cold run time: 63701 ms Total hot run time: 60330 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (#28851) [doris]
jacktengg opened a new pull request, #28882: URL: https://github.com/apache/doris/pull/28882 ## Proposed changes Pick from #28851 ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (#28851) [doris]
jacktengg commented on PR #28882: URL: https://github.com/apache/doris/pull/28882#issuecomment-1867474871 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [improve](load) limit delta writer flush task parallelism [doris]
kaijchen opened a new pull request, #28883: URL: https://github.com/apache/doris/pull/28883 ## Proposed changes Add a flush task parallelism limit for each delta writer. If flush is fast enough, this limit will do nothing since there will be no task waiting. If flush is the bottleneck, this limit could reduce memory usage. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](load) limit delta writer flush task parallelism [doris]
kaijchen commented on PR #28883: URL: https://github.com/apache/doris/pull/28883#issuecomment-1867478451 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2-0](http) add show_table_data http api [doris]
nextdreamblue commented on PR #28633: URL: https://github.com/apache/doris/pull/28633#issuecomment-1867479357 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (#28851) [doris]
github-actions[bot] commented on PR #28882: URL: https://github.com/apache/doris/pull/28882#issuecomment-1867479832 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org