[GitHub] [doris] github-actions[bot] commented on pull request #16509: [Enhancement](storage) add a new hidden column __DORIS_VERSION_COL__ for unique key table
github-actions[bot] commented on PR #16509: URL: https://github.com/apache/doris/pull/16509#issuecomment-1435520183 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch sanqi created (now a3521b3667)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch sanqi in repository https://gitbox.apache.org/repos/asf/doris.git at a3521b3667 1.2.2 rc01 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 01/02: add more logs to buffer control block
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch sanqi
in repository https://gitbox.apache.org/repos/asf/doris.git
commit d7f9288617aae988f2cbaeb906d064aaf5fb481f
Author: yiguolei
AuthorDate: Sat Feb 18 16:44:41 2023 +0800
add more logs to buffer control block
---
be/src/runtime/buffer_control_block.cpp | 5 -
be/src/runtime/result_buffer_mgr.cpp| 6 --
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/be/src/runtime/buffer_control_block.cpp
b/be/src/runtime/buffer_control_block.cpp
index 03bd6d3466..b516e3ae9f 100644
--- a/be/src/runtime/buffer_control_block.cpp
+++ b/be/src/runtime/buffer_control_block.cpp
@@ -113,10 +113,12 @@ Status
BufferControlBlock::add_batch(std::unique_ptr& result)
}
if (_waiting_rpc.empty()) {
+LOG(INFO) << "wait rpc is empty add to queue";
_buffer_rows += num_rows;
_batch_queue.push_back(std::move(result));
_data_arrival.notify_one();
} else {
+LOG(INFO) << "wait rpc is not empty, call ctx directly";
auto ctx = _waiting_rpc.front();
_waiting_rpc.pop_front();
ctx->on_data(result, _packet_num);
@@ -180,7 +182,7 @@ void BufferControlBlock::get_batch(GetResultBatchCtx* ctx) {
_batch_queue.pop_front();
_buffer_rows -= result->result_batch.rows.size();
_data_removal.notify_one();
-
+LOG(INFO) << "find data return";
ctx->on_data(result, _packet_num);
_packet_num++;
return;
@@ -189,6 +191,7 @@ void BufferControlBlock::get_batch(GetResultBatchCtx* ctx) {
ctx->on_close(_packet_num, _query_statistics.get());
return;
}
+LOG(INFO) << "not data wait";
// no ready data, push ctx to waiting list
_waiting_rpc.push_back(ctx);
}
diff --git a/be/src/runtime/result_buffer_mgr.cpp
b/be/src/runtime/result_buffer_mgr.cpp
index b9d6cc7234..6f5fe451fd 100644
--- a/be/src/runtime/result_buffer_mgr.cpp
+++ b/be/src/runtime/result_buffer_mgr.cpp
@@ -60,6 +60,7 @@ Status ResultBufferMgr::init() {
Status ResultBufferMgr::create_sender(const TUniqueId& query_id, int
buffer_size,
std::shared_ptr*
sender,
int query_timeout) {
+LOG(INFO) << "create sender for query " << query_id;
*sender = find_control_block(query_id);
if (*sender != nullptr) {
LOG(WARNING) << "already have buffer control block for this instance "
<< query_id;
@@ -91,7 +92,7 @@ std::shared_ptr
ResultBufferMgr::find_control_block(const TU
if (_buffer_map.end() != iter) {
return iter->second;
}
-
+LOG(INFO) << "could not find control block for query " << query_id;
return std::shared_ptr();
}
@@ -116,13 +117,14 @@ void ResultBufferMgr::fetch_data(const PUniqueId&
finst_id, GetResultBatchCtx* c
ctx->on_failure(Status::InternalError("no result for this query"));
return;
}
+LOG(INFO) << "try to get batch for " << tid;
cb->get_batch(ctx);
}
Status ResultBufferMgr::cancel(const TUniqueId& query_id) {
std::lock_guard l(_lock);
BufferMap::iterator iter = _buffer_map.find(query_id);
-
+LOG(INFO) << "begin to cancel query " << query_id;
if (_buffer_map.end() != iter) {
iter->second->cancel();
_buffer_map.erase(iter);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 02/02: f
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch sanqi
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 24cbcdc7b20d507cee0a389ef4c10a74a90f565e
Author: yiguolei
AuthorDate: Sat Feb 18 17:16:30 2023 +0800
f
---
be/src/vec/exec/scan/scanner_scheduler.cpp | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/be/src/vec/exec/scan/scanner_scheduler.cpp
b/be/src/vec/exec/scan/scanner_scheduler.cpp
index d62909b5bd..3b6defafc3 100644
--- a/be/src/vec/exec/scan/scanner_scheduler.cpp
+++ b/be/src/vec/exec/scan/scanner_scheduler.cpp
@@ -71,9 +71,9 @@ Status ScannerScheduler::init(ExecEnv* env) {
}
// 2. local scan thread pool
-_local_scan_thread_pool.reset(new PriorityWorkStealingThreadPool(
-config::doris_scanner_thread_pool_thread_num,
env->store_paths().size(),
-config::doris_scanner_thread_pool_queue_size, "local_scan"));
+_local_scan_thread_pool.reset(
+new
PriorityThreadPool(config::doris_scanner_thread_pool_thread_num,
+
config::doris_scanner_thread_pool_queue_size, "local_scan"));
// 3. remote scan thread pool
_remote_scan_thread_pool.reset(
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch sanqi updated (a3521b3667 -> 24cbcdc7b2)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch sanqi in repository https://gitbox.apache.org/repos/asf/doris.git from a3521b3667 1.2.2 rc01 new d7f9288617 add more logs to buffer control block new 24cbcdc7b2 f The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/runtime/buffer_control_block.cpp| 5 - be/src/runtime/result_buffer_mgr.cpp | 6 -- be/src/vec/exec/scan/scanner_scheduler.cpp | 6 +++--- 3 files changed, 11 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16862: [fix](planner) `Nullable` of slot descriptor is mistaken and cause BE crash
github-actions[bot] commented on PR #16862: URL: https://github.com/apache/doris/pull/16862#issuecomment-1435627897 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16862: [fix](planner) `Nullable` of slot descriptor is mistaken and cause BE crash
github-actions[bot] commented on PR #16862: URL: https://github.com/apache/doris/pull/16862#issuecomment-1435627959 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue opened a new issue, #16899: [Bug] unique table can not alter table ENABLE FEATURE "SEQUENCE_LOAD" WITH PROPERTIES ( "function_column.sequence_col" = "col_name")
nextdreamblue opened a new issue, #16899: URL: https://github.com/apache/doris/issues/16899 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? MySQL [test]> CREATE TABLE test ( -> c1 int(11) NULL, -> v1 int(11) NULL COMMENT "", -> v2 datetime NULL COMMENT "" -> ) ENGINE=OLAP -> unique KEY(c1) -> COMMENT "OLAP" -> DISTRIBUTED BY HASH(c1) BUCKETS 3 -> PROPERTIES ( -> "replication_allocation" = "tag.location.default: 1", -> "in_memory" = "false", -> "storage_format" = "V2" -> ); Query OK, 0 rows affected (0.01 sec) MySQL [test]> ALTER TABLE test ENABLE FEATURE "SEQUENCE_LOAD" WITH PROPERTIES ( "function_column.sequence_col" = "v2" ); ERROR 1105 (HY000): errCode = 2, detailMessage = errCode = 2, detailMessage = unknown sequence column type it can not alter table ENABLE FEATURE "SEQUENCE_LOAD" WITH PROPERTIES ( "function_column.sequence_col" = "col_name") ### What You Expected? it can alter table ENABLE FEATURE "SEQUENCE_LOAD" WITH PROPERTIES ( "function_column.sequence_col" = "col_name") like WITH PROPERTIES ( "function_column.sequence_type" = "type"). ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangGuoqiang666 opened a new pull request, #16900: Enhance teamcity pipeline trigger
ZhangGuoqiang666 opened a new pull request, #16900: URL: https://github.com/apache/doris/pull/16900 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ no] Does it affect the original behavior * [ no] Has unit tests been added * [ yes ] Has document been added or modified * [ no ] Does it need to update dependencies * [ yes ] Is this PR support rollback (If NO, please explain WHY) ## Further comments Added compatible code synchronization delay issues with failures and updates needed to trigger the pipeline -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangGuoqiang666 closed pull request #16900: Enhance teamcity pipeline trigger
ZhangGuoqiang666 closed pull request #16900: Enhance teamcity pipeline trigger URL: https://github.com/apache/doris/pull/16900 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SaintBacchus opened a new pull request, #16901: [Fix](MySQLLoad) Fix load a big local file bug since bytebuffer from mysql packet using the same byte array
SaintBacchus opened a new pull request, #16901: URL: https://github.com/apache/doris/pull/16901 # Proposed changes Issue Number: close #xxx ## Problem summary Loading a big local file will cause `INTERNAL_ERROR]too many filtered rows` issue since the bytebuffer from mysql client always use the same byte array. And the later bytes will overwrite the previous one and make wrong bytes order among the network. Copy the byte array and then fill it into network. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangGuoqiang666 opened a new pull request, #16902: Added compatible code synchronization delay issues and run key world
ZhangGuoqiang666 opened a new pull request, #16902: URL: https://github.com/apache/doris/pull/16902 …updates needed to trigger the pipeline # Proposed changes ## Problem summary Describe your changes. ## Checklist(Required) * [ no] Does it affect the original behavior * [ no] Has unit tests been added * [ yes] Has document been added or modified * [ no] Does it need to update dependencies * [ yes] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... Added compatible code synchronization delay issues with failures and updates needed to trigger the pipeline now trigger teamcity pipeline need comment run and specific key word,like run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon opened a new pull request, #16903: Simdjson
eldenmoon opened a new pull request, #16903: URL: https://github.com/apache/doris/pull/16903 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16903: (Enhancement)[load-json] support simdjson in new json reader
github-actions[bot] commented on PR #16903: URL: https://github.com/apache/doris/pull/16903#issuecomment-1435652431 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16903: (Enhancement)[load-json] support simdjson in new json reader
github-actions[bot] commented on PR #16903: URL: https://github.com/apache/doris/pull/16903#issuecomment-1435652885 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16901: [Fix](MySQLLoad) Fix load a big local file bug since bytebuffer from mysql packet using the same byte array
hello-stephen commented on PR #16901: URL: https://github.com/apache/doris/pull/16901#issuecomment-1435653863 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.43 seconds stream load tsv: 471 seconds loaded 74807831229 Bytes, about 151 MB/s stream load json: 37 seconds loaded 2358488459 Bytes, about 60 MB/s stream load orc: 69 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230218120152_clickbench_pr_99569.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16896: [bugfix](topn) fix topn read_orderby_key_columns nullptr
github-actions[bot] commented on PR #16896: URL: https://github.com/apache/doris/pull/16896#issuecomment-1435654599 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16903: (Enhancement)[load-json] support simdjson in new json reader
github-actions[bot] commented on PR #16903: URL: https://github.com/apache/doris/pull/16903#issuecomment-1435655122 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16903: (Enhancement)[load-json] support simdjson in new json reader
github-actions[bot] commented on PR #16903: URL: https://github.com/apache/doris/pull/16903#issuecomment-1435657470 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16902: Added compatible code synchronization delay issues and run key world
github-actions[bot] commented on PR #16902: URL: https://github.com/apache/doris/pull/16902#issuecomment-1435658373 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16902: Added compatible code synchronization delay issues and run key world
github-actions[bot] commented on PR #16902: URL: https://github.com/apache/doris/pull/16902#issuecomment-1435658367 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #16902: [improvement](pipeline) Added compatible code synchronization delay issues and run key world
dataroaring merged PR #16902: URL: https://github.com/apache/doris/pull/16902 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (2d7d8102c7 -> a4e42b1e94)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 2d7d8102c7 [fix](doc) fix mal-format doc #16898 add a4e42b1e94 [improvement](pipeline) Added compatible code synchronization delay issues with failures and updates needed to trigger the pipeline (#16902) No new revisions were added by this update. Summary of changes: .github/workflows/auto_trigger_teamcity.yml | 50 +++-- 1 file changed, 40 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16883: [fix](Nereids): fix SemiJoinLogicalJoinTransposeProject.
hello-stephen commented on PR #16883: URL: https://github.com/apache/doris/pull/16883#issuecomment-1435660692 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.76 seconds stream load tsv: 468 seconds loaded 74807831229 Bytes, about 152 MB/s stream load json: 35 seconds loaded 2358488459 Bytes, about 64 MB/s stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 29 seconds loaded 861443392 Bytes, about 28 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230218123727_clickbench_pr_99609.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16903: (Enhancement)[load-json] support simdjson in new json reader
github-actions[bot] commented on PR #16903: URL: https://github.com/apache/doris/pull/16903#issuecomment-1435660851 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16895: [typo](docs)fix dynamic Table version label
yiguolei merged PR #16895: URL: https://github.com/apache/doris/pull/16895 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (a4e42b1e94 -> 4bf778c6cd)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from a4e42b1e94 [improvement](pipeline) Added compatible code synchronization delay issues with failures and updates needed to trigger the pipeline (#16902) add 4bf778c6cd [typo](docs)fix dynamic Table version label (#16895) No new revisions were added by this update. Summary of changes: docs/en/docs/data-table/dynamic-schema-table.md| 4 docs/zh-CN/docs/data-table/dynamic-schema-table.md | 3 +++ 2 files changed, 7 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (4bf778c6cd -> 861e4bc64a)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 4bf778c6cd [typo](docs)fix dynamic Table version label (#16895)
add 861e4bc64a [fix](planner) Nullable of slot descriptor is mistaken and
cause BE crash #16862
No new revisions were added by this update.
Summary of changes:
.../org/apache/doris/planner/SetOperationNode.java | 12 +++
.../operator/test_set_operator.out}| 12 ++-
.../query_p0/operator/test_set_operator.groovy | 91 ++
3 files changed, 113 insertions(+), 2 deletions(-)
copy regression-test/data/{correctness_p0/test_outer_join_with_cross_join.out
=> query_p0/operator/test_set_operator.out} (84%)
create mode 100644
regression-test/suites/query_p0/operator/test_set_operator.groovy
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16862: [fix](planner) `Nullable` of slot descriptor is mistaken and cause BE crash
yiguolei merged PR #16862: URL: https://github.com/apache/doris/pull/16862 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16840: [fix](dbt)fix dbt incremental
yiguolei merged PR #16840: URL: https://github.com/apache/doris/pull/16840 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (861e4bc64a -> 45dbd4d872)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 861e4bc64a [fix](planner) Nullable of slot descriptor is mistaken and
cause BE crash #16862
add 45dbd4d872 [fix](dbt)fix dbt incremental #16840
No new revisions were added by this update.
Summary of changes:
.clangd => extension/dbt-doris/.flake8 | 20 +-
extension/dbt-doris/.pre-commit-config.yaml| 72 +
.../dbt-doris/{test/__init__.py => MANIFEST.in}|2 +
extension/dbt-doris/dbt/adapters/doris/__init__.py | 11 +-
.../dbt-doris/dbt/adapters/doris/__version__.py|2 +-
.../dbt-doris/dbt/adapters/doris/connections.py| 73 +-
extension/dbt-doris/dbt/adapters/doris/impl.py | 32 +-
.../dbt-doris/dbt/include/doris/dbt_project.yml|2 +-
.../dbt/include/doris/macros/adapters/columns.sql |4 +
.../dbt/include/doris/macros/adapters/metadata.sql |6 +-
.../dbt/include/doris/macros/adapters/relation.sql | 39 +-
.../dbt/include/doris/macros/adapters/schema.sql | 18 +-
.../macros/get_custom_name/get_custom_database.sql | 20 -
.../materializations/incremental/incremental.sql | 21 +-
.../doris/macros/materializations/seed/helpers.sql |2 +-
.../macros/materializations/snapshot/snapshot.sql | 200 +--
.../materializations/snapshot/snapshot_merge.sql | 35 -
.../materializations/table/create_table_as.sql |2 +-
.../dbt/include/doris/profile_template.yml |8 +-
extension/dbt-doris/dev-requirements.txt | 23 +
.../mysql-5.7.env => extension/dbt-doris/mypy.ini |5 +-
extension/dbt-doris/poetry.lock| 1744
extension/dbt-doris/pyproject.toml | 64 -
.clangd => extension/dbt-doris/pytest.ini | 18 +-
.../{dbt/adapters/doris/__init__.py => setup.py} | 30 +-
.../include/doris/__init__.py => test/conftest.py} | 21 +-
extension/dbt-doris/test/doris.dbtspec | 34 -
.../test/functional/adapter/test_basic.py | 115 ++
dist/README => extension/dbt-doris/tox.ini | 31 +-
29 files changed, 519 insertions(+), 2135 deletions(-)
copy .clangd => extension/dbt-doris/.flake8 (82%)
create mode 100644 extension/dbt-doris/.pre-commit-config.yaml
copy extension/dbt-doris/{test/__init__.py => MANIFEST.in} (94%)
delete mode 100644
extension/dbt-doris/dbt/include/doris/macros/get_custom_name/get_custom_database.sql
delete mode 100644
extension/dbt-doris/dbt/include/doris/macros/materializations/snapshot/snapshot_merge.sql
create mode 100644 extension/dbt-doris/dev-requirements.txt
copy docker/thirdparties/docker-compose/mysql/mysql-5.7.env =>
extension/dbt-doris/mypy.ini (92%)
delete mode 100644 extension/dbt-doris/poetry.lock
delete mode 100644 extension/dbt-doris/pyproject.toml
copy .clangd => extension/dbt-doris/pytest.ini (76%)
copy extension/dbt-doris/{dbt/adapters/doris/__init__.py => setup.py} (51%)
copy extension/dbt-doris/{dbt/include/doris/__init__.py => test/conftest.py}
(59%)
delete mode 100644 extension/dbt-doris/test/doris.dbtspec
create mode 100644 extension/dbt-doris/test/functional/adapter/test_basic.py
copy dist/README => extension/dbt-doris/tox.ini (56%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dependabot[bot] commented on pull request #14941: Bump certifi from 2021.10.8 to 2022.12.7 in /extension/dbt-doris
dependabot[bot] commented on PR #14941: URL: https://github.com/apache/doris/pull/14941#issuecomment-1435661511 Looks like certifi is no longer a dependency, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dependabot[bot] commented on pull request #15990: Bump future from 0.18.2 to 0.18.3 in /extension/dbt-doris
dependabot[bot] commented on PR #15990: URL: https://github.com/apache/doris/pull/15990#issuecomment-1435661509 Looks like future is no longer a dependency, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dependabot[bot] commented on pull request #15793: Bump gitpython from 3.1.27 to 3.1.30 in /extension/dbt-doris
dependabot[bot] commented on PR #15793: URL: https://github.com/apache/doris/pull/15793#issuecomment-1435661513 Looks like gitpython is no longer a dependency, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dependabot[bot] closed pull request #14941: Bump certifi from 2021.10.8 to 2022.12.7 in /extension/dbt-doris
dependabot[bot] closed pull request #14941: Bump certifi from 2021.10.8 to 2022.12.7 in /extension/dbt-doris URL: https://github.com/apache/doris/pull/14941 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dependabot[bot] closed pull request #15990: Bump future from 0.18.2 to 0.18.3 in /extension/dbt-doris
dependabot[bot] closed pull request #15990: Bump future from 0.18.2 to 0.18.3 in /extension/dbt-doris URL: https://github.com/apache/doris/pull/15990 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dependabot[bot] closed pull request #15793: Bump gitpython from 3.1.27 to 3.1.30 in /extension/dbt-doris
dependabot[bot] closed pull request #15793: Bump gitpython from 3.1.27 to 3.1.30 in /extension/dbt-doris URL: https://github.com/apache/doris/pull/15793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16790: [regression](struct-type) add more regression tests for struct and map type
yiguolei merged PR #16790: URL: https://github.com/apache/doris/pull/16790 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (45dbd4d872 -> 45427b86be)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 45dbd4d872 [fix](dbt)fix dbt incremental #16840
add 45427b86be [regression](struct-type) add more regression tests for
struct and map type (#16790)
No new revisions were added by this update.
Summary of changes:
.../org/apache/doris/catalog/PrimitiveType.java| 14 +++-
.../doris/alter/MaterializedViewHandler.java | 7 +-
.../apache/doris/common/util/PropertyAnalyzer.java | 2 +-
.../rules/implementation/AggregateStrategies.java | 3 +-
.../apache/doris/planner/SingleNodePlanner.java| 2 +-
.../bloom_filter_p0/test_bloom_filter.groovy | 76 --
.../rollup_p0/test_materialized_view_array.groovy | 2 +-
...groovy => test_materialized_view_struct.groovy} | 46 ++---
8 files changed, 112 insertions(+), 40 deletions(-)
copy regression-test/suites/rollup_p0/{test_materialized_view_array.groovy =>
test_materialized_view_struct.groovy} (58%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16621: [Enhancement](func)Introduce non_nullable extraction function.
yiguolei merged PR #16621: URL: https://github.com/apache/doris/pull/16621 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (45427b86be -> d6a841409f)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 45427b86be [regression](struct-type) add more regression tests for
struct and map type (#16790)
add d6a841409f [Enhancement](func)Introduce non_nullable extraction
function. #16621
No new revisions were added by this update.
Summary of changes:
be/src/vec/CMakeLists.txt | 1 +
be/src/vec/functions/function_nonnullable.cpp | 76 ++
be/src/vec/functions/simple_function_factory.h | 2 +
gensrc/script/doris_builtins_functions.py | 42 +++-
.../cast_function/test_non_nullable_function.out | 22 +++
...on.groovy => test_non_nullable_function.groovy} | 32 +
run-regression-test.sh | 2 +-
7 files changed, 160 insertions(+), 17 deletions(-)
create mode 100644 be/src/vec/functions/function_nonnullable.cpp
create mode 100644
regression-test/data/query_p0/sql_functions/cast_function/test_non_nullable_function.out
copy
regression-test/suites/query_p0/sql_functions/cast_function/{test_cast_array_function.groovy
=> test_non_nullable_function.groovy} (55%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16886: [Enchancement](function) remove unused params on aggregate function
github-actions[bot] commented on PR #16886: URL: https://github.com/apache/doris/pull/16886#issuecomment-1435667810 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16903: (Enhancement)[load-json] support simdjson in new json reader
hello-stephen commented on PR #16903: URL: https://github.com/apache/doris/pull/16903#issuecomment-1435668691 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.44 seconds stream load tsv: 468 seconds loaded 74807831229 Bytes, about 152 MB/s stream load json: 40 seconds loaded 2358488459 Bytes, about 56 MB/s stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230218132205_clickbench_pr_99630.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16883: [fix](Nereids): fix SemiJoinLogicalJoinTransposeProject.
github-actions[bot] commented on PR #16883: URL: https://github.com/apache/doris/pull/16883#issuecomment-1435673689 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16883: [fix](Nereids): fix SemiJoinLogicalJoinTransposeProject.
github-actions[bot] commented on PR #16883: URL: https://github.com/apache/doris/pull/16883#issuecomment-1435673697 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15818: [doc](typo) Update doris-join-optimization.md
github-actions[bot] commented on PR #15818: URL: https://github.com/apache/doris/pull/15818#issuecomment-1435686260 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15818: [doc](typo) Update doris-join-optimization.md
github-actions[bot] commented on PR #15818: URL: https://github.com/apache/doris/pull/15818#issuecomment-1435686309 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #16904: [improvement](help-doc) Add help doc format unit test
morningman opened a new pull request, #16904: URL: https://github.com/apache/doris/pull/16904 # Proposed changes Issue Number: close #xxx ## Problem summary So that we don't need to run p0 if only modify document. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luzhijing commented on a diff in pull request #12235: [typo](docs) Update docker-doris in developer
luzhijing commented on code in PR #12235: URL: https://github.com/apache/doris/pull/12235#discussion_r042262 ## docs/zh-CN/community/developer-guide/docker-dev.md: ## @@ -29,84 +29,27 @@ under the License. ## 相关详细文档导航 -- [使用 Docker 开发镜像编译](/docs/install/source-install/compilation) -- [部署](/docs/install/install-deploy) -- [VSCode Be 开发调试](./be-vscode-dev) +- [使用 Docker 开发镜像编译](../../docs/install/source-install/compilation.md) +- [部署](../../docs/install/install-deploy.md) +- [VSCode Be 开发调试](./be-vscode-dev.md) Review Comment: It's right -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16904: [improvement](help-doc) Add help doc format unit test
github-actions[bot] commented on PR #16904:
URL: https://github.com/apache/doris/pull/16904#issuecomment-1435687556
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/4211586487";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In run-fe-ut.sh line 94:
cp build/help-resource.zip
${DORIS_HOME}/fe/fe-core/src/test/resources/real-help-resource.zip
^---^ SC2086 (info): Double quote to
prevent globbing and word splitting.
Did you mean:
cp build/help-resource.zip
"${DORIS_HOME}"/fe/fe-core/src/test/resources/real-help-resource.zip
For more information:
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing
...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' found no issues.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [chore](workflow) Fix the BE UT (Clang) workflow (#16847)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new 9980a0a5f9 [chore](workflow) Fix the BE UT (Clang) workflow (#16847)
9980a0a5f9 is described below
commit 9980a0a5f95df946b7c2ad977c328e81458cbf94
Author: Adonis Ling
AuthorDate: Fri Feb 17 10:34:57 2023 +0800
[chore](workflow) Fix the BE UT (Clang) workflow (#16847)
Fix the BE UT (Clang) workflow
---
.github/workflows/be-ut-clang.yml | 1 -
1 file changed, 1 deletion(-)
diff --git a/.github/workflows/be-ut-clang.yml
b/.github/workflows/be-ut-clang.yml
index 927154e463..6afe9280a6 100644
--- a/.github/workflows/be-ut-clang.yml
+++ b/.github/workflows/be-ut-clang.yml
@@ -67,7 +67,6 @@ jobs:
bash /tmp/ldb_toolchain_gen.sh "${DEFAULT_DIR}/ldb-toolchain"
sudo apt update
- sudo apt upgrade --yes
sudo DEBIAN_FRONTEND=noninteractive apt install --yes tzdata byacc
# set timezone
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luzhijing merged pull request #15818: [doc](typo) Update doris-join-optimization.md
luzhijing merged PR #15818: URL: https://github.com/apache/doris/pull/15818 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: Update doris-join-optimization.md (#15818)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 1ac5b23e40 Update doris-join-optimization.md (#15818) 1ac5b23e40 is described below commit 1ac5b23e4036625cc2c4c77a035a78b1c5ff1dff Author: liwei AuthorDate: Sat Feb 18 22:24:51 2023 +0800 Update doris-join-optimization.md (#15818) 修改文档错误 --- docs/zh-CN/docs/advanced/join-optimization/doris-join-optimization.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/zh-CN/docs/advanced/join-optimization/doris-join-optimization.md b/docs/zh-CN/docs/advanced/join-optimization/doris-join-optimization.md index cf0755894c..b90e3bc4d0 100644 --- a/docs/zh-CN/docs/advanced/join-optimization/doris-join-optimization.md +++ b/docs/zh-CN/docs/advanced/join-optimization/doris-join-optimization.md @@ -59,7 +59,7 @@ Doris 支持 4 种 Shuffle 方式  - 左右表数据根据分区，计算的记过发送到不同的分区节点上。 + 左右表数据根据分区，计算的结果发送到不同的分区节点上。 3. Bucket Shuffle Join @@ -170,7 +170,7 @@ set enable_cost_based_join_reorder = true 这次耗时从 14 秒降到了 4 秒，性能提升了 3 倍多。 -此时再 Check Profile 的时候，左右表的顺序已经调整正确，即右表是大表，左表是小表。基于小表去构建哈希表，开销是很小的，这就是典型的一个利用 Join Reorder 去提升 Join 性能的一个场景 +此时再 Check Profile 的时候，左右表的顺序已经调整正确，即右表是小表，左表是大表。基于小表去构建哈希表，开销是很小的，这就是典型的一个利用 Join Reorder 去提升 Join 性能的一个场景  - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16891: [improvement](scan) separate scanner into local and remote scanner pool
github-actions[bot] commented on PR #16891: URL: https://github.com/apache/doris/pull/16891#issuecomment-1435690392 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16802: [Refactor](auth)(step-2) Add AccessController to support customized authorization
github-actions[bot] commented on PR #16802: URL: https://github.com/apache/doris/pull/16802#issuecomment-1435690537 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16903: (Enhancement)[load-json] support simdjson in new json reader
github-actions[bot] commented on PR #16903: URL: https://github.com/apache/doris/pull/16903#issuecomment-1435691397 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16903: (Enhancement)[load-json] support simdjson in new json reader
github-actions[bot] commented on PR #16903: URL: https://github.com/apache/doris/pull/16903#issuecomment-1435691761 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch luzhijing-patch-1 created (now a7d23ffd20)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a change to branch luzhijing-patch-1 in repository https://gitbox.apache.org/repos/asf/doris.git at a7d23ffd20 [doc][community]correct the number of committers This branch includes the following new commits: new a7d23ffd20 [doc][community]correct the number of committers The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 01/01: [doc][community]correct the number of committers
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a commit to branch luzhijing-patch-1 in repository https://gitbox.apache.org/repos/asf/doris.git commit a7d23ffd2072657d0451f24208c8f13858c5c515 Author: Luzhijing <82810928+luzhij...@users.noreply.github.com> AuthorDate: Sat Feb 18 22:48:50 2023 +0800 [doc][community]correct the number of committers --- docs/en/community/team.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/community/team.md b/docs/en/community/team.md index a81a5bceea..461aa96a55 100644 --- a/docs/en/community/team.md +++ b/docs/en/community/team.md @@ -50,7 +50,7 @@ We'd like to thank the following committers to the Apache Doris project who have | [yangzhg](https://people.apache.org/committer-index.html#yangzhg) | yangzhg | Zhengguo Yang | | [weizuo](https://people.apache.org/committer-index.html#weizuo) | weizuo93 | Zuo Wei | -## Committers (53) +## Committers (52) (the listing excludes PMC members above) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luzhijing opened a new pull request, #16905: [doc][community]correct the number of committers
luzhijing opened a new pull request, #16905: URL: https://github.com/apache/doris/pull/16905 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16891: [improvement](scan) separate scanner into local and remote scanner pool
hello-stephen commented on PR #16891: URL: https://github.com/apache/doris/pull/16891#issuecomment-1435693837 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.61 seconds stream load tsv: 467 seconds loaded 74807831229 Bytes, about 152 MB/s stream load json: 35 seconds loaded 2358488459 Bytes, about 64 MB/s stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230218145526_clickbench_pr_99665.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16897: [Enhance](ComputeNode) K8sDeployManager support domain
github-actions[bot] commented on PR #16897: URL: https://github.com/apache/doris/pull/16897#issuecomment-1435694135 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16897: [Enhance](ComputeNode) K8sDeployManager support domain
github-actions[bot] commented on PR #16897: URL: https://github.com/apache/doris/pull/16897#issuecomment-1435694148 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 opened a new pull request, #16906: [improvement](ui) execute selected code in sql editor.
ZhangYu0123 opened a new pull request, #16906: URL: https://github.com/apache/doris/pull/16906 Increase usability of UI： 1. Support to execute selected code in sql editor. 2. In log contents page, support the scroll bar and automatically display the content at the end. # Proposed changes Issue Number: close #xxx ## Problem summary ## Checklist(Required) * [] Does it affect the original behavior * [] Has unit tests been added * [] Has document been added or modified * [] Does it need to update dependencies * [] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16036: [feature] Support mutable property for partition
github-actions[bot] commented on PR #16036: URL: https://github.com/apache/doris/pull/16036#issuecomment-1435696584 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16036: [feature] Support mutable property for partition
github-actions[bot] commented on PR #16036: URL: https://github.com/apache/doris/pull/16036#issuecomment-1435696593 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #16036: [feature] Support mutable property for partition
morningman merged PR #16036: URL: https://github.com/apache/doris/pull/16036 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Feature](load) Support mutable property for partition (#16036)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e2e6a0dd83 [Feature](load) Support mutable property for partition
(#16036)
e2e6a0dd83 is described below
commit e2e6a0dd8387801f9a9c951a710bc29277db318f
Author: zhengshengjun
AuthorDate: Sat Feb 18 23:09:34 2023 +0800
[Feature](load) Support mutable property for partition (#16036)
The background is described in this issue: #15723,
where users used Apache Druid to satisfy such lambada requirements before.
We will not make Doris dropping data not belonged to current time window
automatically like Druid,
which is not flexible. We demand a ability to support mutable/immutable
partition, the PR works this way:
1. Support mutable property for a partition.
2. The mutable property of a partition is passed from FE to BE in a load
procedure
3. If a record's partition is immutable, we mark this row as "un selected"
which will not be included in computation of 'max_filter_ratio',
so that data write to immutable partition will be neglected and not
cause load failure.
Use Example:
1. Add immutable partition or modify an partition to be immutable:
- alter table test_tbl add [temporary] partition xxx values less than
('xxx') ('mutable' = 'true');
- alter table test_tbl modify partition xx set ('mutable' = 'false');
2. Write 5 records into table, two of then belongs to immutable partition
---
be/src/exec/tablet_info.cpp| 1 +
be/src/exec/tablet_info.h | 1 +
be/src/vec/sink/vtablet_sink.cpp | 6 +++
be/src/vec/sink/vtablet_sink.h | 1 +
fe/fe-core/src/main/cup/sql_parser.cup | 12 ++---
.../main/java/org/apache/doris/alter/Alter.java| 12 ++---
.../doris/analysis/ModifyPartitionClause.java | 23 --
.../apache/doris/analysis/SinglePartitionDesc.java | 8
.../java/org/apache/doris/backup/RestoreJob.java | 6 ++-
.../apache/doris/catalog/CatalogRecycleBin.java| 18 ++--
.../org/apache/doris/catalog/DataProperty.java | 18 +++-
.../java/org/apache/doris/catalog/OlapTable.java | 14 +++---
.../org/apache/doris/catalog/PartitionInfo.java| 20 +++--
.../doris/common/proc/PartitionsProcDir.java | 3 ++
.../org/apache/doris/common/util/NetUtils.java | 6 ++-
.../apache/doris/common/util/PropertyAnalyzer.java | 7 ++-
.../apache/doris/datasource/InternalCatalog.java | 29 ++--
.../doris/load/loadv2/LoadingTaskPlanner.java | 6 +++
.../apache/doris/persist/ModifyPartitionInfo.java | 1 +
.../apache/doris/persist/PartitionPersistInfo.java | 12 -
.../org/apache/doris/planner/OlapTableSink.java| 2 +
.../org/apache/doris/clone/DiskRebalanceTest.java | 2 +-
.../java/org/apache/doris/clone/RebalanceTest.java | 2 +-
.../org/apache/doris/common/util/UnitTestUtil.java | 1 +
.../apache/doris/load/loadv2/SparkLoadJobTest.java | 4 +-
.../apache/doris/planner/OlapTableSinkTest.java| 4 ++
gensrc/thrift/Descriptors.thrift | 2 +
.../stream_load/test_immutable_partition.csv | 11 +
.../data/load_p0/stream_load/test_stream_load.out | 8
.../load_p0/stream_load/test_stream_load.groovy| 52 ++
30 files changed, 253 insertions(+), 39 deletions(-)
diff --git a/be/src/exec/tablet_info.cpp b/be/src/exec/tablet_info.cpp
index b833831b23..ca65f5c58b 100644
--- a/be/src/exec/tablet_info.cpp
+++ b/be/src/exec/tablet_info.cpp
@@ -231,6 +231,7 @@ Status VOlapTablePartitionParam::init() {
const TOlapTablePartition& t_part = _t_param.partitions[i];
auto part = _obj_pool.add(new VOlapTablePartition(&_partition_block));
part->id = t_part.id;
+part->is_mutable = t_part.is_mutable;
if (!_is_in_partition) {
if (t_part.__isset.start_keys) {
diff --git a/be/src/exec/tablet_info.h b/be/src/exec/tablet_info.h
index c1138245e1..38244d30ff 100644
--- a/be/src/exec/tablet_info.h
+++ b/be/src/exec/tablet_info.h
@@ -102,6 +102,7 @@ struct VOlapTablePartition {
std::vector in_keys;
int64_t num_buckets = 0;
std::vector indexes;
+bool is_mutable;
VOlapTablePartition(vectorized::Block* partition_block)
: start_key {partition_block, -1}, end_key {partition_block, -1} {}
diff --git a/be/src/vec/sink/vtablet_sink.cpp b/be/src/vec/sink/vtablet_sink.cpp
index 5bedc35d8a..c535392a31 100644
--- a/be/src/vec/sink/vtablet_sink.cpp
+++ b/be/src/vec/sink/vtablet_sink.cpp
@@ -1020,6 +1020,11 @@ Status VOlapTableSink::find_tablet(RuntimeState* state,
vectorized::Block* block
is_continue = true;
return status;
}
+if (!(*partition)->is_mutable) {
+_number
[GitHub] [doris] github-actions[bot] commented on pull request #16903: (Enhancement)[load-json] support simdjson in new json reader
github-actions[bot] commented on PR #16903: URL: https://github.com/apache/doris/pull/16903#issuecomment-1435697029 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener merged pull request #16883: [fix](Nereids): fix SemiJoinLogicalJoinTransposeProject.
jackwener merged PR #16883: URL: https://github.com/apache/doris/pull/16883 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](Nereids): fix SemiJoinLogicalJoinTransposeProject. (#16883)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d4cebb39ba [fix](Nereids): fix SemiJoinLogicalJoinTransposeProject.
(#16883)
d4cebb39ba is described below
commit d4cebb39ba3d7c227947daa95a494282b60de857
Author: jakevin
AuthorDate: Sat Feb 18 23:12:34 2023 +0800
[fix](Nereids): fix SemiJoinLogicalJoinTransposeProject. (#16883)
---
.../rules/exploration/join/OuterJoinLAsscom.java | 2 +-
.../join/SemiJoinLogicalJoinTransposeProject.java | 95 +-
.../nereids/trees/plans/logical/LogicalJoin.java | 1 -
3 files changed, 56 insertions(+), 42 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscom.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscom.java
index dda781f327..a23c3f0015 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscom.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscom.java
@@ -63,8 +63,8 @@ public class OuterJoinLAsscom extends
OneExplorationRuleFactory {
return logicalJoin(logicalJoin(), group())
.when(join ->
VALID_TYPE_PAIR_SET.contains(Pair.of(join.left().getJoinType(),
join.getJoinType(
.when(topJoin -> checkReorder(topJoin, topJoin.left()))
-.when(topJoin -> checkCondition(topJoin,
topJoin.left().right().getOutputExprIdSet()))
.whenNot(join -> join.hasJoinHint() ||
join.left().hasJoinHint())
+.when(topJoin -> checkCondition(topJoin,
topJoin.left().right().getOutputExprIdSet()))
.then(topJoin -> {
LogicalJoin bottomJoin =
topJoin.left();
GroupPlan a = bottomJoin.left();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinLogicalJoinTransposeProject.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinLogicalJoinTransposeProject.java
index 76fda42fe5..89a843fffc 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinLogicalJoinTransposeProject.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinLogicalJoinTransposeProject.java
@@ -17,11 +17,13 @@
package org.apache.doris.nereids.rules.exploration.join;
+import org.apache.doris.common.Pair;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.rules.exploration.OneExplorationRuleFactory;
import org.apache.doris.nereids.trees.expressions.ExprId;
import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.plans.GroupPlan;
import org.apache.doris.nereids.trees.plans.JoinHint;
@@ -33,9 +35,12 @@ import org.apache.doris.nereids.util.Utils;
import com.google.common.base.Preconditions;
-import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
/**
*
@@ -64,7 +69,6 @@ public class SemiJoinLogicalJoinTransposeProject extends
OneExplorationRuleFacto
.whenNot(topJoin ->
topJoin.left().child().getJoinType().isSemiOrAntiJoin())
.whenNot(join -> join.hasJoinHint() ||
join.left().child().hasJoinHint())
.when(join -> JoinReorderUtils.checkProject(join.left()))
-.when(this::conditionChecker)
.then(topSemiJoin -> {
LogicalProject> project
= topSemiJoin.left();
LogicalJoin bottomJoin =
project.child();
@@ -72,17 +76,17 @@ public class SemiJoinLogicalJoinTransposeProject extends
OneExplorationRuleFacto
GroupPlan b = bottomJoin.right();
GroupPlan c = topSemiJoin.right();
-Set aOutputExprIdSet = a.getOutputExprIdSet();
-
-List hashJoinConjuncts =
topSemiJoin.getHashJoinConjuncts();
-
-boolean lasscom = false;
-for (Expression hashJoinConjunct : hashJoinConjuncts) {
-Set usedSlotExprIdSet =
hashJoinConjunct.getInputSlotExprIds();
-lasscom = Utils.isIntersecting(usedSlotExprIdSet,
aOutputExprIdSet) || lasscom;
+// push topSemiJoin down project, so we need replace
conjuncts by project.
+Pair, List> conjuncts =
replaceConjuncts(topSemiJoin, project);
+Set conjunc
[GitHub] [doris] morningman opened a new pull request, #16907: [improvement](doc) change some version from dev to 1.2.2
morningman opened a new pull request, #16907: URL: https://github.com/apache/doris/pull/16907 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jianghuzai opened a new issue, #16908: [Enhancement] [Multi-Catalog] Limit metadata display in datagrip if database is specified in jdbcurl
jianghuzai opened a new issue, #16908: URL: https://github.com/apache/doris/issues/16908 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description A company had multi mysql instance， and multi database each instance. such as, instance_a(db_a1,db_a2) and instance_b(db_b1,db_b2). when create two catalog for instance_a and instance_b, all of database will appear at datagrip, but only db_a1, db_b1 what I want. so, I think it's better to define an option for limit the database. ### Solution _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] FreeOnePlus opened a new pull request, #16909: [fix](docker)k8s docker
FreeOnePlus opened a new pull request, #16909: URL: https://github.com/apache/doris/pull/16909 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jianghuzai commented on issue #16908: [Enhancement] [Multi-Catalog] Limit metadata display in datagrip which database is specified
jianghuzai commented on issue #16908: URL: https://github.com/apache/doris/issues/16908#issuecomment-1435715820 look like I make a mistake？ because I found out I could create a user at mysql and grant privileges which database I want. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #16910: [refactor](Nereids): refactor code of Nereids.
jackwener opened a new pull request, #16910: URL: https://github.com/apache/doris/pull/16910 # Proposed changes Issue Number: close #xxx ## Problem summary polish code ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [x] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] gavinchou commented on a diff in pull request #16340: [feature](mysql) Support secure MySQL connection to FE
gavinchou commented on code in PR #16340:
URL: https://github.com/apache/doris/pull/16340#discussion_r136930
##
fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlChannel.java:
##
@@ -197,14 +285,42 @@ protected void realNetSend(ByteBuffer buffer) throws
IOException {
isSend = true;
}
+protected void encryptData(ByteBuffer dstBuf) throws SSLException {
+if (isSslMode) {
+ByteBuffer netData =
ByteBuffer.allocate(sslEngine.getSession().getPacketBufferSize());
+sslEngine.wrap(dstBuf, netData);
+netData.flip();
+dstBuf.clear();
+dstBuf.put(netData);
+dstBuf.flip();
+}
+}
+
public void flush() throws IOException {
-if (null == sendBuffer || sendBuffer.position() == 0) {
+ByteBuffer sendData ;
+if(isSslMode|isHandshaking){
+sendData = sendSslBuffer;
Review Comment:
Use ternary expression to reduce lines of code, e.g.
```
sendData = x ? a : b;
```
##
fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlChannel.java:
##
@@ -197,14 +285,42 @@ protected void realNetSend(ByteBuffer buffer) throws
IOException {
isSend = true;
}
+protected void encryptData(ByteBuffer dstBuf) throws SSLException {
+if (isSslMode) {
+ByteBuffer netData =
ByteBuffer.allocate(sslEngine.getSession().getPacketBufferSize());
+sslEngine.wrap(dstBuf, netData);
+netData.flip();
+dstBuf.clear();
+dstBuf.put(netData);
+dstBuf.flip();
+}
+}
+
public void flush() throws IOException {
-if (null == sendBuffer || sendBuffer.position() == 0) {
+ByteBuffer sendData ;
+if(isSslMode|isHandshaking){
+sendData = sendSslBuffer;
Review Comment:
Use a ternary expression to reduce lines of code, e.g.
```
sendData = x ? a : b;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] gavinchou commented on a diff in pull request #16340: [feature](mysql) Support secure MySQL connection to FE
gavinchou commented on code in PR #16340:
URL: https://github.com/apache/doris/pull/16340#discussion_r137049
##
fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlHandshakePacket.java:
##
@@ -25,11 +25,12 @@ public class MysqlHandshakePacket extends MysqlPacket {
// JDBC uses this version to check which protocol the server support
// Set the patch version to 99 to prevent the vulnerability scanning tool
from
// falsely reporting MySQL vulnerabilities
-public static final String SERVER_VERSION = "5.7.99";
+public static final String SERVER_VERSION = "8.0.31";
Review Comment:
This version may not be updated due to TLS does not complete all the
features of 8.0.31.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] gavinchou commented on a diff in pull request #16340: [feature](mysql) Support secure MySQL connection to FE
gavinchou commented on code in PR #16340:
URL: https://github.com/apache/doris/pull/16340#discussion_r138827
##
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##
@@ -1969,6 +1969,12 @@ public class Config extends ConfigBase {
@ConfField(masterOnly = true, mutable = true)
public static int max_error_tablet_of_broker_load = 3;
+/**
+ * If set to ture, doris will establish an encrypted channel based on the
SSL protocol with mysql.
+ */
+@ConfField(mutable = false, masterOnly = false)
+public static boolean enable_ssl = true;
+
Review Comment:
Should we add a conf filed for certificate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] gavinchou commented on a diff in pull request #16340: [feature](mysql) Support secure MySQL connection to FE
gavinchou commented on code in PR #16340:
URL: https://github.com/apache/doris/pull/16340#discussion_r138827
##
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##
@@ -1969,6 +1969,12 @@ public class Config extends ConfigBase {
@ConfField(masterOnly = true, mutable = true)
public static int max_error_tablet_of_broker_load = 3;
+/**
+ * If set to ture, doris will establish an encrypted channel based on the
SSL protocol with mysql.
+ */
+@ConfField(mutable = false, masterOnly = false)
+public static boolean enable_ssl = true;
+
Review Comment:
Should we add a conf filed for certificate which is needed by SSL?
##
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##
@@ -1969,6 +1969,12 @@ public class Config extends ConfigBase {
@ConfField(masterOnly = true, mutable = true)
public static int max_error_tablet_of_broker_load = 3;
+/**
+ * If set to ture, doris will establish an encrypted channel based on the
SSL protocol with mysql.
+ */
+@ConfField(mutable = false, masterOnly = false)
+public static boolean enable_ssl = true;
+
Review Comment:
Should we add a conf filed for certificate that is needed by SSL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16817: [feature-wip](MTMV) Support alter statement
github-actions[bot] commented on PR #16817: URL: https://github.com/apache/doris/pull/16817#issuecomment-1435838886 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16817: [feature-wip](MTMV) Support alter statement
github-actions[bot] commented on PR #16817: URL: https://github.com/apache/doris/pull/16817#issuecomment-1435838902 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 merged pull request #16817: [feature-wip](MTMV) Support alter statement
adonis0147 merged PR #16817: URL: https://github.com/apache/doris/pull/16817 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature-wip](MTMV) Support alter statement (#16817)
This is an automated email from the ASF dual-hosted git repository.
adonisling pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 96a3c60d3b [feature-wip](MTMV) Support alter statement (#16817)
96a3c60d3b is described below
commit 96a3c60d3b9c415cad263fcdfc95dc0ca30db848
Author: huangzhaowei
AuthorDate: Sun Feb 19 12:15:17 2023 +0800
[feature-wip](MTMV) Support alter statement (#16817)
Steps:
1. drop the old MTMV jobs
2. clear the old task records and clean the running and pending tasks
3. set the new scheduler info in MTMV and replay it in followers.
4. create a job in the master node.
Note that if you change the refresh info of MTMV, the old MTMV tasks will
be cleaned.
---
.../main/java/org/apache/doris/alter/Alter.java| 43 +--
.../main/java/org/apache/doris/catalog/Env.java| 4 +-
.../org/apache/doris/catalog/MaterializedView.java | 4 ++
.../org/apache/doris/journal/JournalEntity.java| 6 +++
.../java/org/apache/doris/mtmv/MTMVJobManager.java | 15 +-
.../org/apache/doris/mtmv/MTMVTaskManager.java | 3 ++
.../doris/persist/AlterMultiMaterializedView.java | 61 ++
.../java/org/apache/doris/persist/EditLog.java | 9
.../org/apache/doris/persist/OperationType.java| 2 +
.../suites/mtmv_p0/test_create_mtmv.groovy | 12 +
10 files changed, 152 insertions(+), 7 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
index 16298fff41..8b256a0ebb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
@@ -19,7 +19,6 @@ package org.apache.doris.alter;
import org.apache.doris.analysis.AddPartitionClause;
import org.apache.doris.analysis.AlterClause;
-import org.apache.doris.analysis.AlterMaterializedViewStmt;
import org.apache.doris.analysis.AlterSystemStmt;
import org.apache.doris.analysis.AlterTableStmt;
import org.apache.doris.analysis.AlterViewStmt;
@@ -47,6 +46,7 @@ import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.DataProperty;
import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.MaterializedView;
import org.apache.doris.catalog.MysqlTable;
import org.apache.doris.catalog.OdbcTable;
import org.apache.doris.catalog.OlapTable;
@@ -60,13 +60,17 @@ import org.apache.doris.catalog.TableIf.TableType;
import org.apache.doris.catalog.Tablet;
import org.apache.doris.catalog.View;
import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.MetaNotFoundException;
import org.apache.doris.common.UserException;
import org.apache.doris.common.util.DynamicPartitionUtil;
import org.apache.doris.common.util.MetaLockUtils;
import org.apache.doris.common.util.PropertyAnalyzer;
+import org.apache.doris.mtmv.MTMVJobFactory;
import org.apache.doris.mtmv.MTMVUtils.TaskSubmitStatus;
+import org.apache.doris.mtmv.metadata.MTMVJob;
+import org.apache.doris.persist.AlterMultiMaterializedView;
import org.apache.doris.persist.AlterViewInfo;
import org.apache.doris.persist.BatchModifyPartitionsInfo;
import org.apache.doris.persist.ModifyCommentOperationLog;
@@ -507,10 +511,39 @@ public class Alter {
}
}
-public void processAlterMaterializedView(AlterMaterializedViewStmt stmt)
throws UserException {
-TableName tbl = stmt.getTable();
-Env.getCurrentEnv().getInternalCatalog().getDb(tbl.getDb());
-throw new DdlException("ALTER MATERIALIZED VIEW is not implemented: "
+ stmt.toSql());
+public void processAlterMaterializedView(AlterMultiMaterializedView
alterView, boolean isReplay)
+throws UserException {
+TableName tbl = alterView.getMvName();
+MaterializedView olapTable = null;
+try {
+// 1. check mv exist
+Database db =
Env.getCurrentInternalCatalog().getDbOrDdlException(tbl.getDb());
+olapTable = (MaterializedView)
db.getTableOrMetaException(tbl.getTbl(), TableType.MATERIALIZED_VIEW);
+
+// 2. drop old job and kill the associated tasks
+Env.getCurrentEnv().getMTMVJobManager().dropJobByName(tbl.getDb(),
tbl.getTbl());
+
+// 3. overwrite the refresh info in the memory of fe.
+olapTable.writeLock();
+olapTable.setRefreshInfo(alterView.getInfo());
+
+// 4. log it and replay it in the follower
+if (!isReplay) {
+Env.getCurrentEnv().getEditLog().logAlterMTMV(alterView);
+// 5. master node generate new jobs
+if (Config.enable_mtmv_scheduler_framework &&
MTMVJobFactor
[GitHub] [doris] github-actions[bot] commented on pull request #16810: [enhance](cooldown)skip follow cooldown once failed to do follow cooldown
github-actions[bot] commented on PR #16810: URL: https://github.com/apache/doris/pull/16810#issuecomment-1435873650 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangtianyi2004 opened a new pull request, #16911: Update spark-load-manual.md
wangtianyi2004 opened a new pull request, #16911: URL: https://github.com/apache/doris/pull/16911 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16776: [Feature](map-type) Support stream load and fix some bugs for map type
github-actions[bot] commented on PR #16776: URL: https://github.com/apache/doris/pull/16776#issuecomment-1435912173 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 merged pull request #16776: [Feature](map-type) Support stream load and fix some bugs for map type
xy720 merged PR #16776: URL: https://github.com/apache/doris/pull/16776 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Feature](map-type) Support stream load and fix some bugs for map type (#16776)
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8b70bfdc31 [Feature](map-type) Support stream load and fix some bugs
for map type (#16776)
8b70bfdc31 is described below
commit 8b70bfdc31141332cbb72008083a423abcb7e741
Author: amory
AuthorDate: Sun Feb 19 15:11:54 2023 +0800
[Feature](map-type) Support stream load and fix some bugs for map type
(#16776)
1、support stream load with json, csv format for map
2、fix olap convertor when compaction action in map column which has null
3、support select outToFile for map
4、add some regression-test
---
be/src/olap/rowset/segment_v2/column_writer.cpp| 13 +-
be/src/olap/rowset/segment_v2/column_writer.h | 2 +-
be/src/vec/data_types/data_type_factory.cpp| 5 +-
be/src/vec/data_types/data_type_factory.hpp| 3 +-
be/src/vec/data_types/data_type_map.cpp| 171 +++--
be/src/vec/exprs/vexpr.cpp | 9 ++
be/src/vec/functions/function_cast.h | 12 ++
be/src/vec/olap/olap_data_convertor.h | 1 -
be/src/vec/runtime/vfile_result_writer.cpp | 4 +
be/src/vec/sink/vmysql_result_writer.cpp | 6 +
regression-test/data/export/test_map_export.out| 8 +
regression-test/data/load/insert/test_map_dml.out | 8 +
.../data/load_p0/stream_load/test_map.csv | 15 ++
.../stream_load/test_map_load_and_function.out | 37 +
regression-test/data/map_p0/test_map_dml.out | 11 ++
.../data/query_p0/show/test_map_show_create.out| 4 +
.../suites/export/test_map_export.groovy | 131
.../suites/load/insert/test_map_dml.groovy | 114 ++
.../stream_load/test_map_load_and_function.groovy | 74 +
.../query_p0/show/test_map_show_create.groovy | 69 +
20 files changed, 643 insertions(+), 54 deletions(-)
diff --git a/be/src/olap/rowset/segment_v2/column_writer.cpp
b/be/src/olap/rowset/segment_v2/column_writer.cpp
index 583becb2b0..a35c840eb5 100644
--- a/be/src/olap/rowset/segment_v2/column_writer.cpp
+++ b/be/src/olap/rowset/segment_v2/column_writer.cpp
@@ -1000,7 +1000,15 @@ Status MapColumnWriter::finish() {
return Status::OK();
}
-// todo. make keys and values write
+Status MapColumnWriter::append_nullable(const uint8_t* null_map, const
uint8_t** ptr,
+size_t num_rows) {
+if (is_nullable()) {
+RETURN_IF_ERROR(_null_writer->append_data(&null_map, num_rows));
+}
+RETURN_IF_ERROR(append_data(ptr, num_rows));
+return Status::OK();
+}
+
Status MapColumnWriter::append_data(const uint8_t** ptr, size_t num_rows) {
auto kv_ptr = reinterpret_cast(*ptr);
for (size_t i = 0; i < 2; ++i) {
@@ -1008,9 +1016,6 @@ Status MapColumnWriter::append_data(const uint8_t** ptr,
size_t num_rows) {
const uint8_t* val_ptr = (const uint8_t*)data;
RETURN_IF_ERROR(_kv_writers[i]->append_data(&val_ptr, num_rows));
}
-if (is_nullable()) {
-return write_null_column(num_rows, false);
-}
return Status::OK();
}
diff --git a/be/src/olap/rowset/segment_v2/column_writer.h
b/be/src/olap/rowset/segment_v2/column_writer.h
index 7d140324dd..036c4d3baf 100644
--- a/be/src/olap/rowset/segment_v2/column_writer.h
+++ b/be/src/olap/rowset/segment_v2/column_writer.h
@@ -382,7 +382,7 @@ public:
Status init() override;
Status append_data(const uint8_t** ptr, size_t num_rows) override;
-
+Status append_nullable(const uint8_t* null_map, const uint8_t** ptr,
size_t num_rows) override;
uint64_t estimate_buffer_size() override;
Status finish() override;
diff --git a/be/src/vec/data_types/data_type_factory.cpp
b/be/src/vec/data_types/data_type_factory.cpp
index 65510dd9f2..91fc51187b 100644
--- a/be/src/vec/data_types/data_type_factory.cpp
+++ b/be/src/vec/data_types/data_type_factory.cpp
@@ -169,9 +169,10 @@ DataTypePtr DataTypeFactory::create_data_type(const
TypeDescriptor& col_desc, bo
break;
case TYPE_MAP:
DCHECK(col_desc.children.size() == 2);
+// todo. (Amory) Support Map contains_nulls in FE MapType.java Later PR
nested = std::make_shared(
-create_data_type(col_desc.children[0],
col_desc.contains_nulls[0]),
-create_data_type(col_desc.children[1],
col_desc.contains_nulls[1]));
+create_data_type(col_desc.children[0], true),
+create_data_type(col_desc.children[1], true));
break;
case TYPE_STRUCT: {
DCHECK(col_desc.children.size() >= 1);
diff --git a/be/src/vec/data_types/data_type_factory.hpp
b/be/src/vec/data_types/data_type_factory.hpp
index 879418a326..b2623b06da 100644
--- a/be/src/vec/data_type
[GitHub] [doris] xy720 merged pull request #16626: [fix](struct-type) forbid struct-type to be distributed key/aggregation key and add more tests
xy720 merged PR #16626: URL: https://github.com/apache/doris/pull/16626 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](struct-type) forbid struct-type to be distributed key/aggregation key and add more tests (#16626)
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 73f7979b73 [fix](struct-type) forbid struct-type to be distributed
key/aggregation key and add more tests (#16626)
73f7979b73 is described below
commit 73f7979b73b5cad140fbe6e9d28f05b62f77ac5d
Author: xy720 <22125576+xy...@users.noreply.github.com>
AuthorDate: Sun Feb 19 15:16:36 2023 +0800
[fix](struct-type) forbid struct-type to be distributed key/aggregation key
and add more tests (#16626)
This commits forbid struct and map type to be distributed key/aggregation
key.
The sql such as:
select distinct stuct_col from struct_table
will report an error.
---
.../main/java/org/apache/doris/catalog/Type.java | 9 +
.../org/apache/doris/analysis/CreateTableStmt.java | 2 +-
.../doris/analysis/HashDistributionDesc.java | 6 ++
.../analysis/CheckExpressionLegalityTest.java | 2 +-
.../aggregate_group_by_metric_type.groovy | 22 ++
5 files changed, 35 insertions(+), 6 deletions(-)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java
b/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java
index 56fbb20a1e..61e22fb32a 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java
@@ -382,12 +382,13 @@ public abstract class Type {
// 3. don't support group by
// 4. don't support index
public boolean isOnlyMetricType() {
-return isObjectStored() || isArrayType();
+return isObjectStored() || isComplexType();
}
public static final String OnlyMetricTypeErrorMsg =
-"Doris hll, bitmap and array column must use with specific
function, and don't support filter or group by."
-+ "please run 'help hll' or 'help bitmap' or 'help array'
in your mysql client.";
+"Doris hll, bitmap, array, map, struct column must use with
specific function, and don't"
++ " support filter or group by. please run 'help hll' or
'help bitmap' or 'help array'"
++ " or 'help map' or 'help struct' in your mysql client.";
public boolean isHllType() {
return isScalarType(PrimitiveType.HLL);
@@ -480,7 +481,7 @@ public abstract class Type {
}
public boolean isCollectionType() {
-return isMapType() || isArrayType() || isMultiRowType() ||
isStructType();
+return isMapType() || isArrayType() || isMultiRowType();
}
public boolean isMapType() {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
index dc699cde00..9a727e5de0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
@@ -372,7 +372,7 @@ public class CreateTableStmt extends DdlStmt {
if (columnDef.getType().getPrimitiveType() ==
PrimitiveType.JSONB) {
break;
}
-if (columnDef.getType().isCollectionType()) {
+if (columnDef.getType().isComplexType()) {
break;
}
if (columnDef.getType().getPrimitiveType() ==
PrimitiveType.VARCHAR) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/HashDistributionDesc.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/HashDistributionDesc.java
index 8b312a6ba7..d54049f793 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/analysis/HashDistributionDesc.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/analysis/HashDistributionDesc.java
@@ -120,6 +120,12 @@ public class HashDistributionDesc extends DistributionDesc
{
} else if (column.getType().isArrayType()) {
throw new DdlException("Array Type should not be used
in distribution column["
+ column.getName() + "].");
+} else if (column.getType().isMapType()) {
+throw new DdlException("Map Type should not be used in
distribution column["
++ column.getName() + "].");
+} else if (column.getType().isStructType()) {
+throw new DdlException("Struct Type should not be used
in distribution column["
++ column.getName() + "].");
} else if (column.getType().isFloatingPointType()) {
throw new DdlException("Floating point type should not
be used in distribution column["
[GitHub] [doris] itsallsame opened a new issue, #16912: [Bug] after use Mutil-Catalog JDBC connect to clickhouse, simple select query random report two exception.
itsallsame opened a new issue, #16912: URL: https://github.com/apache/doris/issues/16912 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris: 1.2.2 clickhouse: 21.3.6 ### What's Wrong? Clickhouse-catalog: CREATE CATALOG clickhouse PROPERTIES ( "type"="jdbc", "user"="someuser", "password"="xxx", "jdbc_url" = "jdbc:clickhouse://clickhouse21:8123/dim", "driver_url" = "clickhouse-jdbc-0.3.2-patch11-all.jar", "driver_class" = "com.clickhouse.jdbc.ClickHouseDriver" ); show catalogs: https://user-images.githubusercontent.com/115528312/219934390-00b82d07-c7f6-41e8-b07c-69edeb87170a.png";> ; however, when i use select query, random report two exception: 1. [HY000][1105] errCode = 2, detailMessage = Fail to convert jdbc type of com.clickhouse.data.value.UnsignedInteger to doris type BIGINT on column: uid. You need to check this column type between external table and doris table. 2. [HY000][1105] errCode = 2, detailMessage = NoClassDefFoundError: com/clickhouse/client/ClickHouseClientBuilder$DummyClient ### What You Expected? select query run without exception ### How to Reproduce? _No response_ ### Anything Else? i try jdbc driver clickhouse-jdbc-0.4.0-all.jar, but it does not work. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
