[GitHub] [doris] catpineapple commented on a diff in pull request #13772: [feature](planner) add multi partition
catpineapple commented on code in PR #13772: URL: https://github.com/apache/doris/pull/13772#discussion_r1011282923 ## fe/fe-core/src/main/cup/sql_parser.cup: ## @@ -349,6 +349,7 @@ terminal String KW_ENTER, KW_ERRORS, KW_EVENTS, +KW_EVERY, Review Comment: Reusing keywords is a good idea, I'll give it a try -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] smallhibiscus opened a new pull request, #13897: [regression-test](query) Add the regression case of the query under the large wide table.
smallhibiscus opened a new pull request, #13897: URL: https://github.com/apache/doris/pull/13897 # Proposed changes Issue Number: close #xxx ## Problem summary Add the regression case of the query under the large wide table. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang commented on pull request #13763: [feature](function)add search functions: multi_search_all_positions & multi_match_any
xiaokang commented on PR #13763: URL: https://github.com/apache/doris/pull/13763#issuecomment-1299680688 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new pull request, #13898: [fix](Nereids) throw NPE when call getOutputExprIds in LogicalProperties
morrySnow opened a new pull request, #13898: URL: https://github.com/apache/doris/pull/13898 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13894: [enhancement](decommission) speed up decommission process (#13579)
hello-stephen commented on PR #13894: URL: https://github.com/apache/doris/pull/13894#issuecomment-1299686785 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.02 seconds load time: 581 seconds storage size: 17154810782 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102072225_clickbench_pr_38053.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13898: [fix](Nereids) throw NPE when call getOutputExprIds in LogicalProperties
github-actions[bot] commented on PR #13898: URL: https://github.com/apache/doris/pull/13898#issuecomment-1299688188 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13898: [fix](Nereids) throw NPE when call getOutputExprIds in LogicalProperties
github-actions[bot] commented on PR #13898: URL: https://github.com/apache/doris/pull/13898#issuecomment-1299688225 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 closed issue #13573: [Bug] Inconsistent behavior for string comparison in FE and BE
Gabriel39 closed issue #13573: [Bug] Inconsistent behavior for string comparison in FE and BE URL: https://github.com/apache/doris/issues/13573 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](fe) Inconsistent behavior for string comparison in FE and BE (#13604)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f2a0adf34e [fix](fe) Inconsistent behavior for string comparison in FE
and BE (#13604)
f2a0adf34e is described below
commit f2a0adf34e6a5170fc4ccf2d3ba56a577d26e7c6
Author: xueweizhang
AuthorDate: Wed Nov 2 15:32:13 2022 +0800
[fix](fe) Inconsistent behavior for string comparison in FE and BE (#13604)
---
fe/fe-core/src/main/java/org/apache/doris/analysis/StringLiteral.java | 4 ++--
regression-test/data/query_p0/sort/sort.out | 3 +++
regression-test/suites/query_p0/sort/sort.groovy | 1 +
3 files changed, 6 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/StringLiteral.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/StringLiteral.java
index 3a6dcd6094..9761a43725 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/StringLiteral.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/StringLiteral.java
@@ -96,9 +96,9 @@ public class StringLiteral extends LiteralExpr {
int minLength = Math.min(thisBytes.length, otherBytes.length);
int i = 0;
for (i = 0; i < minLength; i++) {
-if (thisBytes[i] < otherBytes[i]) {
+if (Byte.toUnsignedInt(thisBytes[i]) <
Byte.toUnsignedInt(otherBytes[i])) {
return -1;
-} else if (thisBytes[i] > otherBytes[i]) {
+} else if (Byte.toUnsignedInt(thisBytes[i]) >
Byte.toUnsignedInt(otherBytes[i])) {
return 1;
}
}
diff --git a/regression-test/data/query_p0/sort/sort.out
b/regression-test/data/query_p0/sort/sort.out
index b4a5c9ee01..4e898a3f62 100644
--- a/regression-test/data/query_p0/sort/sort.out
+++ b/regression-test/data/query_p0/sort/sort.out
@@ -7,3 +7,6 @@
2022-01-01 1
汇总 1
+-- !sort_string_on_fe --
+true
+
diff --git a/regression-test/suites/query_p0/sort/sort.groovy
b/regression-test/suites/query_p0/sort/sort.groovy
index b967c93137..2d41adf192 100644
--- a/regression-test/suites/query_p0/sort/sort.groovy
+++ b/regression-test/suites/query_p0/sort/sort.groovy
@@ -22,4 +22,5 @@
suite("sort") {
qt_sort_string_single_column """ select * from ( select '汇总' as a union
all select '2022-01-01' as a ) a order by 1 """
qt_sort_string_multiple_columns """ select * from ( select '汇总' as a,1 as
b union all select '2022-01-01' as a,1 as b ) a order by 1,2 """
+qt_sort_string_on_fe """ select '汇总' > '2022-01-01' """
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 merged pull request #13604: [fix](fe) Inconsistent behavior for string comparison in FE and BE
Gabriel39 merged PR #13604: URL: https://github.com/apache/doris/pull/13604 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13895: [fix](storage) evaluate_and of ComparisonPredicateBase has logical error
hello-stephen commented on PR #13895: URL: https://github.com/apache/doris/pull/13895#issuecomment-1299704739 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.66 seconds load time: 585 seconds storage size: 17154865190 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102074410_clickbench_pr_38063.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13897: [regression-test](query) Add the regression case of the query under the large wide table.
github-actions[bot] commented on PR #13897: URL: https://github.com/apache/doris/pull/13897#issuecomment-1299718442 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13897: [regression-test](query) Add the regression case of the query under the large wide table.
github-actions[bot] commented on PR #13897: URL: https://github.com/apache/doris/pull/13897#issuecomment-1299718325 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13896: [refactor](crossjoin) refactor cross join
hello-stephen commented on PR #13896: URL: https://github.com/apache/doris/pull/13896#issuecomment-1299723965 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.41 seconds load time: 574 seconds storage size: 17154699356 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102075236_clickbench_pr_38076.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13847: [fix](sequence) fix that update table core dump with sequence column
github-actions[bot] commented on PR #13847: URL: https://github.com/apache/doris/pull/13847#issuecomment-1299752073 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13847: [fix](sequence) fix that update table core dump with sequence column
github-actions[bot] commented on PR #13847: URL: https://github.com/apache/doris/pull/13847#issuecomment-1299752183 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen opened a new pull request, #13899: [improvement](exec) add more debug info on fragment exec error
zhannngchen opened a new pull request, #13899: URL: https://github.com/apache/doris/pull/13899 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13869: [Vectorized](function) support topn_array function
github-actions[bot] commented on PR #13869: URL: https://github.com/apache/doris/pull/13869#issuecomment-1299776899 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13869: [Vectorized](function) support topn_array function
github-actions[bot] commented on PR #13869: URL: https://github.com/apache/doris/pull/13869#issuecomment-1299776991 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] qi4coder commented on issue #13322: [Feature] S3 Load 针对CSV文件导入数据，增加忽略首行选项
qi4coder commented on issue #13322: URL: https://github.com/apache/doris/issues/13322#issuecomment-1299777097 后面自己写spark任务实现了。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname opened a new pull request, #13900: [Enhancement](Nereids)optimize merge group in memo
sohardforaname opened a new pull request, #13900: URL: https://github.com/apache/doris/pull/13900 # Proposed changes Issue Number: close #xxx ## Problem summary optimize mergeGroup, now it's no need to travel the map groupExpressions. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #13746: [enhancement](multi-catalog)Persist external catalog related metadata.
morningman commented on code in PR #13746:
URL: https://github.com/apache/doris/pull/13746#discussion_r1011352564
##
fe/fe-core/src/main/java/org/apache/doris/catalog/external/EsExternalDatabase.java:
##
@@ -60,13 +60,24 @@ public EsExternalDatabase(ExternalCatalog extCatalog, long
id, String name) {
super(extCatalog, id, name);
}
-public void addTable(ExternalTable table) {
-if (!(table instanceof EsExternalTable)) {
-LOG.warn("Table {} is not EsExternalTable.", table.getName());
-return;
+public synchronized void replayInitDb(InitDatabaseLog log, ExternalCatalog
catalog) {
Review Comment:
Do we need `synchronized`?
##
fe/fe-core/src/main/java/org/apache/doris/catalog/external/EsExternalDatabase.java:
##
@@ -135,12 +145,14 @@ public Set getTableNamesWithLock() {
@Override
public List getTables() {
makeSureInitialized();
+MasterCatalogExecutor.waitJournalReplay(replayToJournalId);
Review Comment:
Why not move `MasterCatalogExecutor.waitJournalReplay(replayToJournalId);`
into `makeSureInitialized()`?
##
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:
##
@@ -85,12 +93,20 @@ public abstract class ExternalCatalog implements
CatalogIf, Wr
public abstract void makeSureInitialized();
-public abstract void addDb(ExternalDatabase db);
-
public void setInitialized(boolean initialized) {
this.initialized = initialized;
}
+protected void waitJournalReplay() {
+if (!Env.getCurrentEnv().isMaster()) {
+try {
+
ConnectContext.get().getEnv().getJournalObservable().waitOn(replayToJournalId,
3000);
Review Comment:
I think this should be done in `MasterCatalogExecutor`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #13899: [improvement](exec) add more debug info on fragment exec error
morningman commented on code in PR #13899:
URL: https://github.com/apache/doris/pull/13899#discussion_r1011347456
##
be/src/runtime/fragment_mgr.cpp:
##
@@ -388,6 +388,10 @@ void FragmentExecState::coordinator_callback(const Status&
status, RuntimeProfil
VLOG_DEBUG << "reportExecStatus params is "
<< apache::thrift::ThriftDebugString(params).c_str();
+if (!exec_status.ok()) {
+LOG(INFO) << "report error status: " << exec_status.to_string()
Review Comment:
```suggestion
LOG(WARNING) << "report error status: " << exec_status.to_string()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann opened a new pull request, #13902: [Feat](Nereids) add graph simplifier
XieJiann opened a new pull request, #13902: URL: https://github.com/apache/doris/pull/13902 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann opened a new issue, #13903: [Feature] Add graph simplifier
XieJiann opened a new issue, #13903: URL: https://github.com/apache/doris/issues/13903 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description Now that the hypergraph is ready, we need to further simplify the graph. Therefore, a graph simplifier is needed that support to decide the order of join according the cost. ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue opened a new pull request, #13904: [feature-wip](recover) modify recover doc
nextdreamblue opened a new pull request, #13904: URL: https://github.com/apache/doris/pull/13904 Signed-off-by: nextdreamblue # Proposed changes Issue Number: close #8421 https://github.com/apache/doris/pull/13067 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [x] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #13725: [enhancement](Nereids) generate correct distribution spec after project
morrySnow merged PR #13725: URL: https://github.com/apache/doris/pull/13725 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](Nereids) generate correct distribution spec after project (#13725)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 699ffbca0e [enhancement](Nereids) generate correct distribution spec
after project (#13725)
699ffbca0e is described below
commit 699ffbca0e171a64bf1b9d032fb6dcd05603e1e5
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Wed Nov 2 16:50:44 2022 +0800
[enhancement](Nereids) generate correct distribution spec after project
(#13725)
after project, some Slot maybe project to another one. So we need to
replace ExprId in DistributionSpecHash to the new one. if we do project other
than Alias, We need to return DistributionSpecAny other than child's
DistributionSpec.
---
.../properties/ChildOutputPropertyDeriver.java | 40 +++-
.../nereids/properties/DistributionSpecHash.java | 42 +++--
.../apache/doris/nereids/properties/OrderSpec.java | 1 +
.../properties/DistributionSpecHashTest.java | 44 ++
4 files changed, 122 insertions(+), 5 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildOutputPropertyDeriver.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildOutputPropertyDeriver.java
index e1ff5168ad..b4772dad73 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildOutputPropertyDeriver.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildOutputPropertyDeriver.java
@@ -20,6 +20,10 @@ package org.apache.doris.nereids.properties;
import org.apache.doris.nereids.PlanContext;
import org.apache.doris.nereids.memo.GroupExpression;
import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalAggregate;
import org.apache.doris.nereids.trees.plans.physical.PhysicalAssertNumRows;
@@ -37,9 +41,14 @@ import
org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
import org.apache.doris.nereids.util.JoinUtils;
import com.google.common.base.Preconditions;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
import java.util.List;
+import java.util.Map;
import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
/**
* Used for property drive.
@@ -117,8 +126,37 @@ public class ChildOutputPropertyDeriver extends
PlanVisitor project, PlanContext context) {
+// TODO: order spec do not process since we do not use it.
Preconditions.checkState(childrenOutputProperties.size() == 1);
-return childrenOutputProperties.get(0);
+PhysicalProperties childProperties = childrenOutputProperties.get(0);
+DistributionSpec childDistributionSpec =
childProperties.getDistributionSpec();
+OrderSpec childOrderSpec = childProperties.getOrderSpec();
+DistributionSpec outputDistributionSpec;
+if (childDistributionSpec instanceof DistributionSpecHash) {
+Map projections = Maps.newHashMap();
+Set obstructions = Sets.newHashSet();
+for (NamedExpression namedExpression : project.getProjects()) {
+if (namedExpression instanceof Alias) {
+Alias alias = (Alias) namedExpression;
+if (alias.child() instanceof SlotReference) {
+projections.put(((SlotReference)
alias.child()).getExprId(), alias.getExprId());
+} else {
+obstructions.addAll(
+alias.child().getInputSlots().stream()
+.map(NamedExpression::getExprId)
+.collect(Collectors.toSet()));
+}
+}
+}
+if (projections.entrySet().stream().allMatch(kv ->
kv.getKey().equals(kv.getValue( {
+return childrenOutputProperties.get(0);
+}
+outputDistributionSpec = ((DistributionSpecHash)
childDistributionSpec).project(projections, obstructions);
+return new PhysicalProperties(outputDistributionSpec,
childOrderSpec);
+} else {
+return childrenOutputProperties.get(0);
+}
+
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpecHash.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpecHash.java
index 2cd74ebb1e..407c40f812 100644
---
a/fe/fe-
[GitHub] [doris] morrySnow merged pull request #13898: [fix](Nereids) throw NPE when call getOutputExprIds in LogicalProperties
morrySnow merged PR #13898: URL: https://github.com/apache/doris/pull/13898 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](Nereids) throw NPE when call getOutputExprIds in LogicalProperties (#13898)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1bafb26217 [fix](Nereids) throw NPE when call getOutputExprIds in
LogicalProperties (#13898)
1bafb26217 is described below
commit 1bafb26217d5ed9fb197e09c75ff567d2613a6a5
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Wed Nov 2 16:52:18 2022 +0800
[fix](Nereids) throw NPE when call getOutputExprIds in LogicalProperties
(#13898)
---
.../java/org/apache/doris/nereids/properties/LogicalProperties.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/LogicalProperties.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/LogicalProperties.java
index c0cae91f7c..cf8921e843 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/LogicalProperties.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/LogicalProperties.java
@@ -71,7 +71,7 @@ public class LogicalProperties {
public List getOutputExprIds() {
if (outputExprIds == null) {
-outputExprIds =
outputExprIdSet.stream().map(Id.class::cast).collect(Collectors.toList());
+outputExprIds =
getOutputExprIdSet().stream().map(Id.class::cast).collect(Collectors.toList());
}
return outputExprIds;
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13897: [regression-test](query) Add the regression case of the query under the large wide table.
hello-stephen commented on PR #13897: URL: https://github.com/apache/doris/pull/13897#issuecomment-1299876356 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 40.1 seconds load time: 585 seconds storage size: 17154810743 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102085544_clickbench_pr_38122.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju opened a new pull request, #13905: [refractor](array) refractor DataTypeArray from_string
cambyzju opened a new pull request, #13905: URL: https://github.com/apache/doris/pull/13905 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. 1. refractor DataTypeArray from_string, make it more clear; 2. support ',' and ']' inside string element, for example: ['hello,,,', 'world][]'] ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue opened a new pull request, #13906: [fix](ctas) text column type len = 1 when create table as select
nextdreamblue opened a new pull request, #13906: URL: https://github.com/apache/doris/pull/13906 Signed-off-by: nextdreamblue # Proposed changes Issue Number: close #13891 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [x] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: fix
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 13f972d6da8 fix
13f972d6da8 is described below
commit 13f972d6da8a2f53761e787a40a53655df927f4d
Author: jiafeng.zhang
AuthorDate: Wed Nov 2 17:08:43 2022 +0800
fix
---
.../current/admin-manual/multi-tenant.md | 232 +
1 file changed, 232 insertions(+)
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/multi-tenant.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/multi-tenant.md
new file mode 100644
index 000..acc1775c0a6
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/multi-tenant.md
@@ -0,0 +1,232 @@
+---
+{
+"title": "多租户和资源划分",
+"language": "zh-CN"
+}
+---
+
+
+
+# 多租户和资源划分
+
+Doris 的多租户和资源隔离方案，主要目的是为了多用户在同一 Doris 集群内进行数据操作时，减少相互之间的干扰，能够将集群资源更合理的分配给各用户。
+
+该方案主要分为两部分，一是集群内节点级别的资源组划分，二是针对单个查询的资源限制。
+
+## Doris 中的节点
+
+首先先简单介绍一下 Doris 的节点组成。一个 Doris 集群中有两类节点：Frontend(FE) 和 Backend(BE)。
+
+FE 主要负责元数据管理、集群管理、用户请求的接入和查询计划的解析等工作。
+
+BE 主要负责数据存储、查询计划的执行等工作。
+
+FE 不参与用户数据的处理计算等工作，因此是一个资源消耗较低的节点。而 BE
负责所有的数据计算、任务处理，属于资源消耗型的节点。因此，本文所介绍的资源划分及资源限制方案，都是针对 BE 节点的。FE
节点因为资源消耗相对较低，并且还可以横向扩展，因此通常无需做资源上的隔离和限制，FE 节点由所有用户共享即可。
+
+## 节点资源划分
+
+节点资源划分，是指将一个 Doris 集群内的 BE 节点设置标签（Tag），标签相同的 BE 节点组成一个资源组（Resource
Group）。资源组可以看作是数据存储和计算的一个管理单元。下面我们通过一个具体示例，来介绍资源组的使用方式。
+
+1. 为 BE 节点设置标签
+
+ 假设当前 Doris 集群有 6 个 BE 节点。分别为 host[1-6]。在初始情况下，所有节点都属于一个默认资源组（Default）。
+
+ 我们可以使用以下命令将这6个节点划分成3个资源组：group_a、group_b、group_c：
+
+ ```sql
+ alter system modify backend "host1:9050" set ("tag.location" = "group_a");
+ alter system modify backend "host2:9050" set ("tag.location" = "group_a");
+ alter system modify backend "host3:9050" set ("tag.location" = "group_b");
+ alter system modify backend "host4:9050" set ("tag.location" = "group_b");
+ alter system modify backend "host5:9050" set ("tag.location" = "group_c");
+ alter system modify backend "host6:9050" set ("tag.location" = "group_c");
+ ```
+
+ 这里我们将 `host[1-2]` 组成资源组 `group_a`，`host[3-4]` 组成资源组 `group_b`，`host[5-6]`
组成资源组 `group_c`。
+
+ > 注：一个 BE 只支持设置一个 Tag。
+
+2. 按照资源组分配数据分布
+
+ 资源组划分好后。我们可以将用户数据的不同副本分布在不同资源组内。假设一张用户表
UserTable。我们希望在3个资源组内各存放一个副本，则可以通过如下建表语句实现：
+
+ ```sql
+ create table UserTable
+ (k1 int, k2 int)
+ distributed by hash(k1) buckets 1
+ properties(
+ "replication_allocation"="tag.location.group_a:1,
tag.location.group_b:1, tag.location.group_c:1"
+ )
+ ```
+
+ 这样一来，表 UserTable 中的数据，将会以3副本的形式，分别存储在资源组 group_a、group_b、group_c所在的节点中。
+
+ 下图展示了当前的节点划分和数据分布：
+
+ ```text
+┌┐
+││
+│ ┌──┐ ┌──┐ │
+│ │ host1│ │ host2│ │
+│ │ ┌─┐ │ │ │ │
+│ group_a │ │ replica1 │ │ │ │ │
+│ │ └─┘ │ │ │ │
+│ │ │ │ │ │
+│ └──┘ └──┘ │
+││
+├┤
+├┤
+││
+│ ┌──┐ ┌──┐ │
+│ │ host3│ │ host4│ │
+│ │ │ │ ┌─┐ │ │
+│ group_b │ │ │ │ replica2 │ │ │
+│ │ │ │ └─┘ │ │
+│ │ │ │ │ │
+│ └──┘ └──┘ │
+││
+├┤
+├┤
+││
+│ ┌──┐ ┌──┐ │
+│ │ host5│ │ host6│ │
+│ │ │ │ ┌─┐ │ │
+│ group_c │ │ │ │ replica3 │ │ │
+│ │ │ │ └─┘ │ │
+│ │ │ │ │ │
+│ └──┘ └──┘ │
+││
+└┘
+ ```
+
+3. 使用不同资源组进行数据查询
+
+ 在前两步执行完成后，我们就可以通过设置用户的资源使用权限，来限制某一用户的查询，只能使用指定资源组中的节点来执行。
+
+ 比如我们可以通过以下语句，限制 user1 只能使用 `group_a` 资源组中的节点进行数据查询，user2 只能使用 `group_b`
资源组，而 user3 可以同时使用 3 个资源组：
+
+ ```sql
+
[GitHub] [doris] englefly commented on a diff in pull request #13873: [feature](nereids) runtime filter prune
englefly commented on code in PR #13873:
URL: https://github.com/apache/doris/pull/13873#discussion_r1011415967
##
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java:
##
@@ -131,6 +137,41 @@ public PhysicalPlan
visitPhysicalHashJoin(PhysicalHashJoin leftSlots = equalTo.child(0).getInputSlots();
+if (leftSlots.size() > 1) {
+return false;
+}
+Set rightSlots = equalTo.child(1).getInputSlots();
+if (rightSlots.size() > 1) {
+return false;
+}
+Slot leftSlot = leftSlots.iterator().next();
+Slot rightSlot = rightSlots.iterator().next();
+ColumnStat probeColumnStat = leftStats.getColumnStatsBySlot(leftSlot);
+ColumnStat buildColumnStat =
rightStats.getColumnStatsBySlot(rightSlot);
+if (probeColumnStat == null || buildColumnStat == null) {
+probeColumnStat = leftStats.getColumnStatsBySlot(rightSlot);
+buildColumnStat = rightStats.getColumnStatsBySlot(leftSlot);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hongli-my commented on issue #4589: [show frontends cost 12s]
hongli-my commented on issue #4589: URL: https://github.com/apache/doris/issues/4589#issuecomment-1299933425 some problems, have fixed? @francisoliverlee -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #13046: [Fix](Nereids)add comments to CostAndEnforcerJob and fix some bugs
morrySnow commented on code in PR #13046:
URL: https://github.com/apache/doris/pull/13046#discussion_r1011467239
##
fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java:
##
@@ -109,6 +114,14 @@ public void setChildren(ImmutableList children) {
this.children = children;
}
+public boolean isHasCalculateCost() {
+return hasCalculateCost;
+}
+
+public void setHasCalculateCost(boolean hasCalculateCost) {
+this.hasCalculateCost = hasCalculateCost;
+}
+
Review Comment:
remove it in this PR
##
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/CostAndEnforcerJob.java:
##
@@ -164,74 +167,96 @@ public void execute() {
if (curTotalCost > context.getCostUpperBound()) {
break;
}
+// the request child properties will be covered by the output
properties
+// that corresponding to the request properties. so if we run
a costAndEnforceJob of the same
+// group expression, that request child properties will be
different of this.
}
// This mean that we successfully optimize all child groups.
+// if break when running the loop above, the condition must be
false.
if (curChildIndex == groupExpression.arity()) {
-
-// to ensure distributionSpec has been added sufficiently.
-ChildrenPropertiesRegulator regulator = new
ChildrenPropertiesRegulator(groupExpression,
-lowestCostChildren, requestChildrenProperties,
requestChildrenProperties, context);
-double enforceCost = regulator.adjustChildrenProperties();
-if (enforceCost < 0) {
-// invalid enforce, return.
-return;
-}
-curTotalCost += enforceCost;
-
-// Not need to do pruning here because it has been done when
we get the
-// best expr from the child group
-ChildOutputPropertyDeriver childOutputPropertyDeriver
-= new
ChildOutputPropertyDeriver(requestChildrenProperties);
-PhysicalProperties outputProperty =
childOutputPropertyDeriver.getOutputProperties(groupExpression);
-
-// update current group statistics and re-compute costs.
-if (groupExpression.children().stream().anyMatch(group ->
group.getStatistics() == null)) {
-// if we come here, mean that we have some error in stats
calculator and should fix it.
+if (!calculateEnforce(requestChildrenProperties)) {
return;
}
-StatsCalculator.estimate(groupExpression);
-curTotalCost -= curNodeCost;
-curNodeCost = CostCalculator.calculateCost(groupExpression);
-groupExpression.setCost(curNodeCost);
-curTotalCost += curNodeCost;
-
-// record map { outputProperty -> outputProperty }, { ANY ->
outputProperty },
-recordPropertyAndCost(groupExpression, outputProperty,
PhysicalProperties.ANY,
-requestChildrenProperties);
-recordPropertyAndCost(groupExpression, outputProperty,
outputProperty, requestChildrenProperties);
-enforce(outputProperty, requestChildrenProperties);
-
if (curTotalCost < context.getCostUpperBound()) {
context.setCostUpperBound(curTotalCost);
}
}
-
clear();
}
}
+private boolean calculateEnforce(List
requestChildrenProperties) {
Review Comment:
add some comment to explain return value
##
fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java:
##
@@ -168,25 +182,24 @@ public List
getInputPropertiesList(PhysicalProperties requir
/**
* Add a (outputProperties) -> (cost, childrenInputProperties) in
lowestCostTable.
+ * if the outputProperties exists, will be covered.
+ * @return true if lowest cost table change.
*/
public boolean updateLowestCostTable(PhysicalProperties outputProperties,
List childrenInputProperties, double cost) {
if (lowestCostTable.containsKey(outputProperties)) {
if (lowestCostTable.get(outputProperties).first > cost) {
lowestCostTable.put(outputProperties, Pair.of(cost,
childrenInputProperties));
return true;
-} else {
-return false;
}
-} else {
-lowestCostTable.put(outputProperties, Pair.of(cost,
childrenInputProperties));
-return true;
+return false;
Review Comment:
do not remove else block. it is easy to do a accidentally modifying with out
it
--
This is an automat
[GitHub] [doris] morningman opened a new pull request, #13907: [fix](keyword) add BIN as keyword
morningman opened a new pull request, #13907: URL: https://github.com/apache/doris/pull/13907 # Proposed changes Issue Number: close #xxx ## Problem summary This keyword BIN is introduced in #13067 , but forgot to add it to `keywords` region. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 merged pull request #13402: [Improve](Nereids): refactor eliminate outer join
924060929 merged PR #13402: URL: https://github.com/apache/doris/pull/13402 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (1bafb26217 -> a871fef815)
This is an automated email from the ASF dual-hosted git repository. huajianlan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 1bafb26217 [fix](Nereids) throw NPE when call getOutputExprIds in LogicalProperties (#13898) add a871fef815 [Improve](Nereids): refactor eliminate outer join (#13402) No new revisions were added by this update. Summary of changes: .../org/apache/doris/nereids/rules/RuleSet.java| 4 +- .../org/apache/doris/nereids/rules/RuleType.java | 2 +- .../rules/rewrite/logical/EliminateOuter.java | 91 .../rules/rewrite/logical/EliminateOuterJoin.java | 98 ++ .../nereids/trees/plans/logical/LogicalJoin.java | 4 + .../apache/doris/nereids/util/ExpressionUtils.java | 14 .../rules/rewrite/logical/EliminateOuterTest.java | 20 ++--- 7 files changed, 129 insertions(+), 104 deletions(-) delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EliminateOuter.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EliminateOuterJoin.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] gnehil opened a new pull request, #13908: [doc] fix 404 link
gnehil opened a new pull request, #13908: URL: https://github.com/apache/doris/pull/13908 # Proposed changes Issue Number: close #xxx ## Problem summary fix tablet_rebalancer_type link to fe-config.md 404 error ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 merged pull request #13764: [feature](Nereids) Support lots of scalar function and fix some bug
924060929 merged PR #13764: URL: https://github.com/apache/doris/pull/13764 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Pace2Car commented on issue #13889: [Enhancement] `forEach()` replace `stream().forEach()`
Pace2Car commented on issue #13889: URL: https://github.com/apache/doris/issues/13889#issuecomment-1299976888 I will work on it, please assignee it to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju opened a new pull request, #13909: [enhancement](chore) remove debug log which is really too frequent
cambyzju opened a new pull request, #13909: URL: https://github.com/apache/doris/pull/13909 # Proposed changes Issue Number: close #xxx ## Problem summary Every min_compaction_failure_interval_sec, each tablet will print one debug log, it is really too frequent even as debug log. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13792: [doc](jsonb type)add documents for JSONB datatype
hello-stephen commented on PR #13792: URL: https://github.com/apache/doris/pull/13792#issuecomment-125061 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.08 seconds load time: 577 seconds storage size: 17154810724 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102102140_clickbench_pr_38220.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13899: [improvement](exec) add more debug info on fragment exec error
github-actions[bot] commented on PR #13899: URL: https://github.com/apache/doris/pull/13899#issuecomment-138747 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13899: [improvement](exec) add more debug info on fragment exec error
github-actions[bot] commented on PR #13899: URL: https://github.com/apache/doris/pull/13899#issuecomment-138823 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13719: [Bugfix](MV) Fixed load negative values into bitmap type materialized views successfully under non-vectorization
github-actions[bot] commented on PR #13719: URL: https://github.com/apache/doris/pull/13719#issuecomment-1300067395 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13719: [Bugfix](MV) Fixed load negative values into bitmap type materialized views successfully under non-vectorization
github-actions[bot] commented on PR #13719: URL: https://github.com/apache/doris/pull/13719#issuecomment-1300067241 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13899: [improvement](exec) add more debug info on fragment exec error
hello-stephen commented on PR #13899: URL: https://github.com/apache/doris/pull/13899#issuecomment-1300076444 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.23 seconds load time: 598 seconds storage size: 17154810471 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102110816_clickbench_pr_38322.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13907: [fix](keyword) add BIN as keyword
github-actions[bot] commented on PR #13907: URL: https://github.com/apache/doris/pull/13907#issuecomment-1300080479 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13907: [fix](keyword) add BIN as keyword
github-actions[bot] commented on PR #13907: URL: https://github.com/apache/doris/pull/13907#issuecomment-1300080564 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13900: [Enhancement](Nereids)optimize merge group in memo
hello-stephen commented on PR #13900: URL: https://github.com/apache/doris/pull/13900#issuecomment-1300142240 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.22 seconds load time: 572 seconds storage size: 17154644827 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102112922_clickbench_pr_38263.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #13836: [fix](rpc) The proxy removed when rpc exception occurs is not an abnormal proxy
morningman merged PR #13836: URL: https://github.com/apache/doris/pull/13836 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (6eea855e78 -> b26d8f284c)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 6eea855e78 [feature](Nereids) Support lots of scalar function and fix some bug (#13764) add b26d8f284c [fix](rpc) The proxy removed when rpc exception occurs is not an abnormal proxy (#13836) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/qe/Coordinator.java | 50 +- .../org/apache/doris/rpc/BackendServiceProxy.java | 2 +- 2 files changed, 30 insertions(+), 22 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13909: [enhancement](chore) remove debug log which is really too frequent
github-actions[bot] commented on PR #13909: URL: https://github.com/apache/doris/pull/13909#issuecomment-1300188741 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13909: [enhancement](chore) remove debug log which is really too frequent
github-actions[bot] commented on PR #13909: URL: https://github.com/apache/doris/pull/13909#issuecomment-1300188871 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee merged pull request #13869: [Vectorized](function) support topn_array function
HappenLee merged PR #13869: URL: https://github.com/apache/doris/pull/13869 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Vectorized](function) support topn_array function (#13869)
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 374303186c [Vectorized](function) support topn_array function (#13869)
374303186c is described below
commit 374303186c996342c1c1bbbefd01e13a2cc5112f
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Wed Nov 2 19:49:23 2022 +0800
[Vectorized](function) support topn_array function (#13869)
---
.../aggregate_function_topn.cpp| 61 ++
.../aggregate_functions/aggregate_function_topn.h | 133 +
be/test/vec/aggregate_functions/agg_test.cpp | 2 +-
.../aggregate_functions/vec_window_funnel_test.cpp | 7 +-
.../aggregate-functions/topn_array.md | 61 ++
docs/sidebars.json | 1 +
.../aggregate-functions/topn_array.md | 60 ++
.../java/org/apache/doris/catalog/FunctionSet.java | 7 ++
.../main/java/org/apache/doris/catalog/Type.java | 2 +
.../test_select_stddev_variance_window.out | 132 ++--
.../aggregate_functions/test_aggregate_collect.out | 12 ++
.../test_select_stddev_variance_window.out | 132 ++--
.../test_select_stddev_variance_window.groovy | 12 +-
.../test_aggregate_collect.groovy | 24
.../test_select_stddev_variance_window.groovy | 12 +-
15 files changed, 482 insertions(+), 176 deletions(-)
diff --git a/be/src/vec/aggregate_functions/aggregate_function_topn.cpp
b/be/src/vec/aggregate_functions/aggregate_function_topn.cpp
index 19f52fbff8..74bb154d6a 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_topn.cpp
+++ b/be/src/vec/aggregate_functions/aggregate_function_topn.cpp
@@ -17,6 +17,8 @@
#include
+#include "vec/aggregate_functions/helpers.h"
+#include "vec/core/types.h"
namespace doris::vectorized {
AggregateFunctionPtr create_aggregate_function_topn(const std::string& name,
@@ -38,8 +40,67 @@ AggregateFunctionPtr create_aggregate_function_topn(const
std::string& name,
return nullptr;
}
+template
+AggregateFunctionPtr create_topn_array(const DataTypes& argument_types) {
+auto type = argument_types[0].get();
+if (type->is_nullable()) {
+type = assert_cast(type)->get_nested_type().get();
+}
+
+WhichDataType which(*type);
+
+#define DISPATCH(TYPE)
\
+if (which.idx == TypeIndex::TYPE)
\
+return AggregateFunctionPtr(
\
+new AggregateFunctionTopNArray<
\
+AggregateFunctionTopNImplArray, TYPE>( \
+argument_types));
+FOR_NUMERIC_TYPES(DISPATCH)
+#undef DISPATCH
+if (which.is_string_or_fixed_string()) {
+return AggregateFunctionPtr(new AggregateFunctionTopNArray<
+
AggregateFunctionTopNImplArray,
+std::string>(argument_types));
+}
+if (which.is_decimal()) {
+return AggregateFunctionPtr(
+new AggregateFunctionTopNArray<
+AggregateFunctionTopNImplArray, Decimal128>(
+argument_types));
+}
+if (which.is_date_or_datetime() || which.is_date_time_v2()) {
+return AggregateFunctionPtr(
+new AggregateFunctionTopNArray<
+AggregateFunctionTopNImplArray, Int64>(
+argument_types));
+}
+if (which.is_date_v2()) {
+return AggregateFunctionPtr(
+new AggregateFunctionTopNArray<
+AggregateFunctionTopNImplArray, UInt32>(
+argument_types));
+}
+
+LOG(WARNING) << fmt::format("Illegal argument type for aggregate function
topn_array is: {}",
+type->get_name());
+return nullptr;
+}
+
+AggregateFunctionPtr create_aggregate_function_topn_array(const std::string&
name,
+ const DataTypes&
argument_types,
+ const Array&
parameters,
+ const bool
result_is_nullable) {
+bool has_default_param = (argument_types.size() == 3);
+if (has_default_param) {
+return create_topn_array(argument_types);
+} else {
+return create_topn_array(argument_types);
+}
+}
+
void register_aggregate_function_topn(AggregateFunctionSimpleFactory& factory)
{
factory.register_function("topn", create_aggregate_function_topn);
+factory.register
[GitHub] [doris] github-actions[bot] commented on pull request #13900: [Enhancement](Nereids)optimize merge group in memo
github-actions[bot] commented on PR #13900: URL: https://github.com/apache/doris/pull/13900#issuecomment-1300206515 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13900: [Enhancement](Nereids)optimize merge group in memo
github-actions[bot] commented on PR #13900: URL: https://github.com/apache/doris/pull/13900#issuecomment-130020 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13902: [Feat](Nereids) add graph simplifier
hello-stephen commented on PR #13902: URL: https://github.com/apache/doris/pull/13902#issuecomment-1300216438 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.2 seconds load time: 571 seconds storage size: 17154655393 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102115816_clickbench_pr_38296.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13904: [feature-wip](recover) modify recover doc
hello-stephen commented on PR #13904: URL: https://github.com/apache/doris/pull/13904#issuecomment-1300248997 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.25 seconds load time: 578 seconds storage size: 17154810035 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102121224_clickbench_pr_38289.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hongli-my commented on issue #4589: [show frontends cost 12s]
hongli-my commented on issue #4589: URL: https://github.com/apache/doris/issues/4589#issuecomment-1300283529 in my case: com.sleepycat.je.rep.elections.Learner$3:execute() will send query master info to fe which have been droped. so restart fe will recovery. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] lljqy opened a new issue, #13910: arm和x86混合部署，并发查询不稳定
lljqy opened a new issue, #13910: URL: https://github.com/apache/doris/issues/13910 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris 版本 1.1.3 操作系统：EulerOS release 2.0 (SP5) 单节点内存：1T 单节点CPU核心数： 96 ~ 128 不等 单节点磁盘 ：18T ~ 48T不等 ### What's Wrong? 6个BE（doris100, doris101, doris102, doris103, doris104，doris105），其中doris103为arm架构，其他的为x86；当单个查询SQL返回结果的数据到达百万行并且查询并发数50左右，会导致一至两个be挂死而且挂死的be始终是（doris100和doris101）；如果把arm架构的doris103剔除掉，再去测试就是正常的 ### What You Expected? doris官方介绍的支持点查询高并发和高，希望能够支持x86和arm混合部署同时又能并发查询大量数据 ### How to Reproduce? 单表数据量20亿行，13列 6个BE（doris100, doris101, doris102, doris103, doris104，doris105），其中doris103为arm架构，其他的为x86；当单个查询SQL返回结果的数据到达百万行并且查询并发数50左右，会导致一至两个be挂死而且挂死的be始终是（doris100和doris101） ### Anything Else? 下面是gdb调试core文件的部分报错信息： __throw_system_error(int) () at /mnt/disk2/ygl/installs/ldbtools/include/c++/11/ext/new_allocator.h:89 be.out日志  ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yinzhijian opened a new pull request, #13911: [fix](runtime-filter) build thread destruct first may cause probe thread coredump
yinzhijian opened a new pull request, #13911: URL: https://github.com/apache/doris/pull/13911 # Proposed changes Issue Number: close #xxx ## Problem summary hash join with runtime filter may cause BE coredump ### coredump ``` *** SIGSEGV unkown detail explain (@0x0) received by PID 63967 (TID 0x7f634afe8700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at be/src/common/signal_handler.h:420 1# 0x7F65244C4920 in /lib64/libc.so.6 2# doris::ScopedTimer >::UpdateCounter() at be/src/util/runtime_profile.h:556 3# doris::ScopedTimer >::~ScopedTimer() at be/src/util/runtime_profile.h:566 4# doris::IRuntimeFilter::await() at be/src/exprs/runtime_filter.cpp:1200 5# doris::OlapScanNode::open(doris::RuntimeState*) at be/src/exec/olap_scan_node.cpp:255 6# doris::HashJoinNode::probe_side_open_thread(doris::RuntimeState*, std::promise*) at be/src/exec/hash_join_node.cpp:188 7# void std::__invoke_impl*), doris::HashJoinNode*&, doris::RuntimeState*&, std::promise*&>(std::__invoke_memfun_deref, void (doris::HashJoinNode::*&)(doris::RuntimeState*, std::promise*), doris::HashJoinNode*&, doris::RuntimeState*&, std::promise*&) at palo-toolchain/ldb_toolchain/include/c++/11/bits/invoke.h:74 8# std::__invoke_result*), doris::HashJoinNode*&, doris::RuntimeState*&, std::promise*&>::type std::__invoke*), doris::HashJoinNode*&, doris::RuntimeState*&, std::promise*&>(void (doris::HashJoinNode::*&)(doris::RuntimeState*, std::promise*), doris::HashJoinNode*&, doris::RuntimeState*&, std::promise*&) at palo-toolchain/ldb_toolchain/include/c++/11/bits/invoke.h:97 ``` ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener merged pull request #13900: [Enhancement](Nereids)optimize merge group in memo
jackwener merged PR #13900: URL: https://github.com/apache/doris/pull/13900 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Enhancement](Nereids)optimize merge group in memo #13900
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 53814e466b [Enhancement](Nereids)optimize merge group in memo #13900
53814e466b is described below
commit 53814e466b67e40a7f8656484c6fe38c9040d369
Author: mch_ucchi <41606806+sohardforan...@users.noreply.github.com>
AuthorDate: Wed Nov 2 20:42:55 2022 +0800
[Enhancement](Nereids)optimize merge group in memo #13900
---
.../apache/doris/nereids/memo/GroupExpression.java | 23 +++---
.../java/org/apache/doris/nereids/memo/Memo.java | 23 +-
2 files changed, 25 insertions(+), 21 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java
index 92067f607d..1972e71961 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java
@@ -28,6 +28,7 @@ import org.apache.doris.statistics.StatsDeriveResult;
import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableList.Builder;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
@@ -106,7 +107,9 @@ public class GroupExpression {
}
public void setChildren(ImmutableList children) {
+this.children.forEach(g -> g.removeParentExpression(this));
this.children = children;
+this.children.forEach(g -> g.addParentExpression(this));
}
/**
@@ -117,22 +120,36 @@ public class GroupExpression {
*/
public void replaceChild(Group originChild, Group newChild) {
originChild.removeParentExpression(this);
+List groups =
Lists.newArrayListWithCapacity(this.children.size());
for (int i = 0; i < children.size(); i++) {
if (children.get(i) == originChild) {
-children.set(i, newChild);
-newChild.addParentExpression(this);
+groups.add(newChild);
+} else {
+groups.add(child(i));
}
}
+children = ImmutableList.copyOf(groups);
+newChild.addParentExpression(this);
}
public void setChild(int index, Group group) {
-this.children.set(index, group);
+this.children.get(index).removeParentExpression(this);
+setChildByIndex(index, group);
}
public boolean hasApplied(Rule rule) {
return ruleMasks.get(rule.getRuleType().ordinal());
}
+private void setChildByIndex(int index, Group group) {
+ImmutableList.Builder builder = new Builder<>();
+builder.addAll(children.subList(0, index));
+builder.add(group);
+builder.addAll(children.subList(index + 1, children.size()));
+children = builder.build();
+group.addParentExpression(this);
+}
+
public boolean notApplied(Rule rule) {
return !hasApplied(rule);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
index f5616a71c7..3583db9f4e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
@@ -375,26 +375,13 @@ public class Memo {
if (source.equals(destination)) {
return source;
}
-List needReplaceChild = Lists.newArrayList();
-for (GroupExpression groupExpression : groupExpressions.values()) {
-if (groupExpression.children().contains(source)) {
-if (groupExpression.getOwnerGroup().equals(destination)) {
-// cycle, we should not merge
-return null;
-}
-needReplaceChild.add(groupExpression);
-}
+if (source.getParentGroupExpressions().stream()
+.anyMatch(e -> e.getOwnerGroup().equals(destination))) {
+return null;
}
-for (GroupExpression groupExpression : needReplaceChild) {
+for (GroupExpression groupExpression :
source.getParentGroupExpressions()) {
groupExpressions.remove(groupExpression);
-List children = new ArrayList<>(groupExpression.children());
-// TODO: use a better way to replace child, avoid traversing all
groupExpression
-for (int i = 0; i < children.size(); i++) {
-if (children.get(i).equals(source)) {
-children.set(i, destination);
-}
-}
-groupExpression.setChildren(ImmutableList.copyOf(children));
+groupExpression.replaceChild(source, destinat
[GitHub] [doris] hello-stephen commented on pull request #13867: [feature-wip](multi-catalog) iceberg v2
hello-stephen commented on PR #13867: URL: https://github.com/apache/doris/pull/13867#issuecomment-1300334359 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.05 seconds load time: 571 seconds storage size: 17154644860 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102124710_clickbench_pr_38569.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 opened a new pull request, #13912: [enhancement](load) Increase batch size of node channel to improve import performance
liaoxin01 opened a new pull request, #13912: URL: https://github.com/apache/doris/pull/13912 # Proposed changes Issue Number: close #xxx ## Problem summary change the _batch_size of node channel from 2048 to 8192. The stream load performance is improved by about two times when import ssb lineorder table. Test with tools/ssb-tools for single node. 600MB data load time: 14s -> 7s. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #13913: [fix](Nereids): make GroupExpression Children Immutable.
jackwener opened a new pull request, #13913: URL: https://github.com/apache/doris/pull/13913 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] DongLiang-0 commented on a diff in pull request #78: [fix] Add log in abort phase
DongLiang-0 commented on code in PR #78:
URL:
https://github.com/apache/doris-flink-connector/pull/78#discussion_r1011715574
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/DorisStreamLoad.java:
##
@@ -269,7 +271,7 @@ private void abortTransaction(long txnID) throws Exception {
ObjectMapper mapper = new ObjectMapper();
String loadResult = EntityUtils.toString(response.getEntity());
Map res = mapper.readValue(loadResult, new
TypeReference>(){});
-if (FAIL.equals(res.get("status"))) {
+if (FAIL.equals(res.get("status")) ||
INTERNAL_ERROR.equals(res.get("status"))) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13905: [refractor](array) refractor DataTypeArray from_string
hello-stephen commented on PR #13905: URL: https://github.com/apache/doris/pull/13905#issuecomment-1300355659 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.63 seconds load time: 573 seconds storage size: 17154699291 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102125658_clickbench_pr_38523.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener closed pull request #13913: [fix](Nereids): make GroupExpression Children Immutable.
jackwener closed pull request #13913: [fix](Nereids): make GroupExpression Children Immutable. URL: https://github.com/apache/doris/pull/13913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang opened a new pull request, #13914: split_part函数支持负数索引参数
Yulei-Yang opened a new pull request, #13914: URL: https://github.com/apache/doris/pull/13914 # Proposed changes Issue Number: close #13712 split_part函数支持负数索引参数，方便倒着取数，兼容Mysql 类似用法 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13906: [fix](ctas) text column type len = 1 when create table as select
hello-stephen commented on PR #13906: URL: https://github.com/apache/doris/pull/13906#issuecomment-1300379166 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.01 seconds load time: 571 seconds storage size: 17154655426 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102130815_clickbench_pr_38346.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on issue #13712: [Feature] split_part函数支持用负数倒取字段
Yulei-Yang commented on issue #13712: URL: https://github.com/apache/doris/issues/13712#issuecomment-1300379157 try this https://github.com/apache/doris/pull/13914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #13915: [fix](Nereids): make GroupExpression Children Immutable.
jackwener opened a new pull request, #13915: URL: https://github.com/apache/doris/pull/13915 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ByteYue opened a new pull request, #13916: [BugFix](Concat) output of string concat function exceeds UINT makes crash
ByteYue opened a new pull request, #13916: URL: https://github.com/apache/doris/pull/13916 # Proposed changes The previously logic of concat function assumes all input string could be concat inside INT size buffer, user might accidentally input string parameters which could sum up to exceed the size. Issue Number: close #xxx ## Problem summary When input such query below be would coredump. ``` select concat( cast(substr( cast(ref_1.`cp_type` as varchar), cast( max( cast(ref_1.`cp_catalog_page_number` as int)) over (partition by ref_1.`cp_end_date_sk` order by ref_1.`cp_catalog_page_number`) as int), cast(ref_1.`cp_end_date_sk` as int)) as varchar), cast(substring( cast(ref_1.`cp_department` as varchar), cast(ref_1.`cp_end_date_sk` as int), cast(ref_1.`cp_end_date_sk` as int)) as varchar), cast(rpad( cast(ref_1.`cp_type` as varchar), cast(ref_1.`cp_start_date_sk` as int), cast(ref_1.`cp_description` as varchar)) as varchar)) as c1 from regression_test_tpcds_sf1_p1.catalog_page as ref_1 ``` Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12742: [feature](nereids) support common table expression
github-actions[bot] commented on PR #12742: URL: https://github.com/apache/doris/pull/12742#issuecomment-1300428709 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12742: [feature](nereids) support common table expression
github-actions[bot] commented on PR #12742: URL: https://github.com/apache/doris/pull/12742#issuecomment-1300428786 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #13915: [fix](Nereids): make GroupExpression Children Immutable.
924060929 commented on code in PR #13915:
URL: https://github.com/apache/doris/pull/13915#discussion_r1011790554
##
fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java:
##
@@ -106,50 +130,47 @@ public List children() {
return children;
}
-public void setChildren(ImmutableList children) {
-this.children.forEach(g -> g.removeParentExpression(this));
-this.children = children;
-this.children.forEach(g -> g.addParentExpression(this));
-}
-
/**
- * replaceChild.
+ * use newChild to replace oldChild in Children of this GroupExpr.
+ * We replace something:
+ * - Memo: groupExpressions
+ * - Child Group: parentGroupExpr
+ * - Parent Group:
+ * - - children: logical|physical
+ * - - lowestCostPlans
+ * - Plan: groupExpression
*
- * @param originChild origin child group
+ * @param oldChild origin child group
* @param newChild new child group
*/
-public void replaceChild(Group originChild, Group newChild) {
-originChild.removeParentExpression(this);
-List groups =
Lists.newArrayListWithCapacity(this.children.size());
-for (int i = 0; i < children.size(); i++) {
-if (children.get(i) == originChild) {
-groups.add(newChild);
-} else {
-groups.add(child(i));
-}
+public GroupExpression replaceChild(Map
groupExpressions,
+Group oldChild, Group newChild) {
+GroupExpression newGroupExpression =
withChildren(Utils.replaceListWithNew(children, oldChild, newChild));
+
+// replace childrenGroup {parent}
+newGroupExpression.children().forEach(childGroup ->
+Utils.replaceList(childGroup.getParentGroupExpressions(),
this, newGroupExpression));
+
+// replace ownerGroup {children} (logical|physical, lowestCostPlans)
+// TODO: maybe no need replace physical.
+newGroupExpression.getOwnerGroup().replaceGroupExpression(this,
newGroupExpression);
+
+// replace Memo {groupExpressions}
+groupExpressions.remove(this);
+groupExpressions.put(newGroupExpression, newGroupExpression);
+
+// replace Plan {GroupExpr}
+if (plan.getGroupExpression().isPresent()) {
+this.plan.setGroupExpression(newGroupExpression);
Review Comment:
plan should be immutable
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/AbstractPlan.java:
##
@@ -77,6 +77,10 @@ public Optional getGroupExpression() {
return groupExpression;
}
+public void setGroupExpression(GroupExpression groupExpression) {
+this.groupExpression = Optional.of(groupExpression);
+}
+
Review Comment:
plan should be immutable so can not support setXxx. You can invoke
`withGroupExpression()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12866: [enhancement](compaction) introduce segment compaction (#12609)
github-actions[bot] commented on PR #12866: URL: https://github.com/apache/doris/pull/12866#issuecomment-1300435444 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] AshinGau opened a new pull request, #13917: [feature-wip](multi-catalog) lazy read for ParquetReader
AshinGau opened a new pull request, #13917: URL: https://github.com/apache/doris/pull/13917 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13907: [fix](keyword) add BIN as keyword
hello-stephen commented on PR #13907: URL: https://github.com/apache/doris/pull/13907#issuecomment-1300449377 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.35 seconds load time: 567 seconds storage size: 17154699349 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102135448_clickbench_pr_38454.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener closed pull request #13915: [fix](Nereids): make GroupExpression Children Immutable.
jackwener closed pull request #13915: [fix](Nereids): make GroupExpression Children Immutable. URL: https://github.com/apache/doris/pull/13915 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #13918: [revert](Nereids): revert GroupExpression Children ImmutableList.
jackwener opened a new pull request, #13918: URL: https://github.com/apache/doris/pull/13918 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](scan) speed up inserting strings into ColumnString (#13397)
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 62f765b7f5 [improvement](scan) speed up inserting strings into
ColumnString (#13397)
62f765b7f5 is described below
commit 62f765b7f530c5af2bb292dab820e0b8077b64d1
Author: Jerry Hu
AuthorDate: Wed Nov 2 22:19:02 2022 +0800
[improvement](scan) speed up inserting strings into ColumnString (#13397)
---
be/src/olap/rowset/segment_v2/binary_plain_page.h | 14 ++-
be/src/vec/columns/column.h | 12 ++-
be/src/vec/columns/column_complex.h | 11 +++
be/src/vec/columns/column_dictionary.h| 17 ++--
be/src/vec/columns/column_jsonb.h | 26 ++
be/src/vec/columns/column_nullable.h | 9 ++
be/src/vec/columns/column_string.h| 101 +++---
be/src/vec/columns/predicate_column.h | 30 ++-
8 files changed, 150 insertions(+), 70 deletions(-)
diff --git a/be/src/olap/rowset/segment_v2/binary_plain_page.h
b/be/src/olap/rowset/segment_v2/binary_plain_page.h
index 659df55fee..96cfc392a5 100644
--- a/be/src/olap/rowset/segment_v2/binary_plain_page.h
+++ b/be/src/olap/rowset/segment_v2/binary_plain_page.h
@@ -248,16 +248,14 @@ public:
return Status::OK();
}
const size_t max_fetch = std::min(*n, static_cast(_num_elems -
_cur_idx));
-uint32_t len_array[max_fetch];
-uint32_t start_offset_array[max_fetch];
uint32_t last_offset = guarded_offset(_cur_idx);
+uint32_t offsets[max_fetch + 1];
+offsets[0] = last_offset;
for (int i = 0; i < max_fetch - 1; i++, _cur_idx++) {
const uint32_t start_offset = last_offset;
last_offset = guarded_offset(_cur_idx + 1);
-uint32_t len = last_offset - start_offset;
-len_array[i] = len;
-start_offset_array[i] = start_offset;
+offsets[i + 1] = last_offset;
if constexpr (Type == OLAP_FIELD_TYPE_OBJECT) {
if (_options.need_check_bitmap) {
RETURN_IF_ERROR(BitmapTypeCode::validate(*(_data.data +
start_offset)));
@@ -265,15 +263,13 @@ public:
}
}
_cur_idx++;
-len_array[max_fetch - 1] = offset(_cur_idx) - last_offset;
-start_offset_array[max_fetch - 1] = last_offset;
+offsets[max_fetch] = offset(_cur_idx);
if constexpr (Type == OLAP_FIELD_TYPE_OBJECT) {
if (_options.need_check_bitmap) {
RETURN_IF_ERROR(BitmapTypeCode::validate(*(_data.data +
last_offset)));
}
}
-dst->insert_many_binary_data(_data.mutable_data(), len_array,
start_offset_array,
- max_fetch);
+dst->insert_many_continuous_binary_data(_data.data, offsets,
max_fetch);
*n = max_fetch;
return Status::OK();
diff --git a/be/src/vec/columns/column.h b/be/src/vec/columns/column.h
index ec794bf747..4b89a002af 100644
--- a/be/src/vec/columns/column.h
+++ b/be/src/vec/columns/column.h
@@ -243,6 +243,14 @@ public:
LOG(FATAL) << "Method insert_many_binary_data is not supported for "
<< get_name();
}
+/// Insert binary data into column from a continuous buffer, the
implementation maybe copy all binary data
+/// in one single time.
+virtual void insert_many_continuous_binary_data(const char* data, const
uint32_t* offsets,
+const size_t num) {
+LOG(FATAL) << "Method insert_many_continuous_binary_data is not
supported for "
+ << get_name();
+}
+
virtual void insert_many_strings(const StringRef* strings, size_t num) {
LOG(FATAL) << "Method insert_many_binary_data is not supported for "
<< get_name();
}
@@ -271,10 +279,6 @@ public:
}
}
-virtual void insert_elements(void* elements, size_t num) {
-LOG(FATAL) << "Method insert_elements is not supported for " <<
get_name();
-}
-
/** Removes last n elements.
* Is used to support exception-safety of several operations.
* For example, sometimes insertion should be reverted if we catch an
exception during operation processing.
diff --git a/be/src/vec/columns/column_complex.h
b/be/src/vec/columns/column_complex.h
index 260729f736..ec15c65df7 100644
--- a/be/src/vec/columns/column_complex.h
+++ b/be/src/vec/columns/column_complex.h
@@ -79,6 +79,17 @@ public:
}
}
+void insert_many_continuous_binary_data(const char* data, const uint32_t*
offsets,
+const size_t num) override {
+if (UNLIKELY(num == 0)) {
+return;
+}
+
+for (size_t i = 0; i != num; ++i) {
+
[GitHub] [doris] HappenLee merged pull request #13397: [improvement](scan) speed up inserting strings into ColumnString
HappenLee merged PR #13397: URL: https://github.com/apache/doris/pull/13397 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 merged pull request #13887: [Opt](function) opt the function of ndv
Gabriel39 merged PR #13887: URL: https://github.com/apache/doris/pull/13887 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (62f765b7f5 -> fbc8b7311f)
This is an automated email from the ASF dual-hosted git repository. gabriellee pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 62f765b7f5 [improvement](scan) speed up inserting strings into ColumnString (#13397) add fbc8b7311f [Opt](function) opt the function of ndv (#13887) No new revisions were added by this update. Summary of changes: .../aggregate_function_approx_count_distinct.h | 17 + be/src/vec/columns/column_decimal.h| 4 be/src/vec/columns/columns_number.h| 18 ++ 3 files changed, 31 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](keyword) add BIN as keyword (#13907)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0ea7f85986 [fix](keyword) add BIN as keyword (#13907)
0ea7f85986 is described below
commit 0ea7f85986b6a37ea3263630e28882613c67ddfa
Author: Mingyu Chen
AuthorDate: Wed Nov 2 22:30:43 2022 +0800
[fix](keyword) add BIN as keyword (#13907)
---
fe/fe-core/src/main/cup/sql_parser.cup | 2 +
.../data/correctness_p0/test_keywords.out | 5 +++
.../suites/correctness_p0/test_keywords.groovy | 43 ++
3 files changed, 50 insertions(+)
diff --git a/fe/fe-core/src/main/cup/sql_parser.cup
b/fe/fe-core/src/main/cup/sql_parser.cup
index c9c6260c60..a9cf6b3dea 100644
--- a/fe/fe-core/src/main/cup/sql_parser.cup
+++ b/fe/fe-core/src/main/cup/sql_parser.cup
@@ -6349,6 +6349,8 @@ keyword ::=
{: RESULT = id; :}
| KW_BEGIN:id
{: RESULT = id; :}
+| KW_BIN:id
+{: RESULT = id; :}
| KW_BITMAP:id
{: RESULT = id; :}
| KW_QUANTILE_STATE:id
diff --git a/regression-test/data/correctness_p0/test_keywords.out
b/regression-test/data/correctness_p0/test_keywords.out
new file mode 100644
index 00..512c06bec3
--- /dev/null
+++ b/regression-test/data/correctness_p0/test_keywords.out
@@ -0,0 +1,5 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !select --
+1
+10
+
diff --git a/regression-test/suites/correctness_p0/test_keywords.groovy
b/regression-test/suites/correctness_p0/test_keywords.groovy
new file mode 100644
index 00..e4af2a6edf
--- /dev/null
+++ b/regression-test/suites/correctness_p0/test_keywords.groovy
@@ -0,0 +1,43 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Test some keywords that may conflict.
+// For example, "bin" is used for function "bin",
+// and also used "show catalog recycle bin"
+suite("test_keywords") {
+def table = "test_keywords"
+sql """
+drop table if exists $table
+"""
+
+sql """
+create table $table ( k1 int, k2 varchar(1024) )
+DISTRIBUTED BY HASH(k1) BUCKETS 3
+PROPERTIES (
+"replication_allocation" = "tag.location.default: 1"
+);
+"""
+
+sql """
+insert into $table values(1, "abc"), (2, "xyz");
+"""
+
+sql "sync"
+order_qt_select """
+select bin(k1) from $table
+"""
+}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener merged pull request #13907: [fix](keyword) add BIN as keyword
jackwener merged PR #13907: URL: https://github.com/apache/doris/pull/13907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13909: [enhancement](chore) remove debug log which is really too frequent
hello-stephen commented on PR #13909: URL: https://github.com/apache/doris/pull/13909#issuecomment-1300552198 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 38.26 seconds load time: 576 seconds storage size: 17154644877 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221102143808_clickbench_pr_38503.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee opened a new pull request, #13919: [Load](Sink) remove validate the column data when data is NULL
HappenLee opened a new pull request, #13919: URL: https://github.com/apache/doris/pull/13919 # Proposed changes Do not need to validate the column data when data is NULL ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #13872: [Enhancement](sequence-column) optimize the use of sequence column
morningman commented on code in PR #13872:
URL: https://github.com/apache/doris/pull/13872#discussion_r1011875780
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##
@@ -4125,6 +4131,11 @@ private void renameColumn(Database db, OlapTable table,
String colName,
}
}
+// 5. modify sequence map col
+if (table.hasSequenceCol() &&
table.getSequenceMapCol().equalsIgnoreCase(colName)) {
Review Comment:
Please add regression test to cover this "rename column" case
##
fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java:
##
@@ -893,6 +893,21 @@ public void setBloomFilterInfo(Set bfColumns,
double bfFpp) {
this.bfFpp = bfFpp;
}
+public String getSequenceMapCol() {
+if (tableProperty == null) {
+return null;
+}
+return tableProperty.getSequenceMapCol();
+}
+
+// map the sequence column to other column
+public void setSequenceMapCol(String colName) {
+if (tableProperty == null) {
+tableProperty = new TableProperty(new HashMap<>());
+}
+tableProperty.setSequenceMapCol(colName);
Review Comment:
Forget to set `this.hasSequenceCol`?
##
fe/fe-core/src/main/java/org/apache/doris/planner/BrokerScanNode.java:
##
@@ -277,9 +278,13 @@ private void initColumns(ParamCreateContext context)
throws UserException {
columnDescs.descs.add(ImportColumnDesc.newDeleteSignImportColumnDesc(new
IntLiteral(1)));
}
// add columnExpr for sequence column
-if (context.fileGroup.hasSequenceCol()) {
+if (targetTable instanceof OlapTable && ((OlapTable)
targetTable).hasSequenceCol()) {
Review Comment:
Please also modify `LoadScanProvider`, line 194
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring commented on a diff in pull request #13872: [Enhancement](sequence-column) optimize the use of sequence column
dataroaring commented on code in PR #13872: URL: https://github.com/apache/doris/pull/13872#discussion_r1011384252 ## docs/zh-CN/docs/data-operate/update-delete/sequence-column-manual.md: ## @@ -64,15 +64,24 @@ Sequence列建表时在property中增加了一个属性，用来标识`__DORIS_S **建表** -创建Uniq表时，可以指定sequence列类型 +方法一：创建Uniq表时，可以指定sequence列类型。 ```text PROPERTIES ( "function_column.sequence_type" = 'Date', ); ``` -sequence_type用来指定sequence列的类型，可以为整型和时间类型（DATE、DATETIME）。 +sequence_type用来指定sequence列的类型，可以为整型和时间类型（DATE、DATETIME）。导入时需要指定映射列。 + +方法二：创建Uniq表时，可以指定sequence列到其他column的映射。 + +```text +PROPERTIES ( +"function_column.sequence_col" = 'column_name', +); +``` +sequence_col用来指定sequence列到其他column的映射，该列可以为整型和时间类型（DATE、DATETIME），创建后不能更改该列的类型。导入时不用再指定映射列。 Review Comment: Refer to comment in en doc. ## docs/en/docs/data-operate/update-delete/sequence-column-manual.md: ## @@ -60,13 +60,21 @@ The principle is the same as the reading process during Base Compaction. When the Sequence column creates a table, an attribute is added to the property, which is used to identify the type import of `__DORIS_SEQUENCE_COL__`. The grammar design is mainly to add a mapping from the sequence column to other columns. The settings of each seed method will be described below introduce. Create Table -When you create the Uniq table, you can specify the sequence column type. +1. When you create the Uniq table, you can specify the sequence column type. ```text PROPERTIES ( "function_column.sequence_type" = 'Date', ); ``` -The sequence_type is used to specify the type of the sequence column, which can be integral and time (DATE / DATETIME). +The sequence_type is used to specify the type of the sequence column, which can be integral and time (DATE / DATETIME). The mapping column needs to be specified when importing. + +2. When you create the Uniq table, you can specify the mapping of sequence column to other columns. +```text +PROPERTIES ( +"function_column.sequence_col" = 'column_name', +); +``` +The sequence_col is used to specify the mapping of the sequence column to other columns, which can be integral and time (DATE, DATETIME). The type of this column cannot be changed after creation. You no longer need to specify mapping column when importing. Review Comment: If function_column.sequence_col is set, function_column.sequence_type is ignored. ## docs/en/docs/data-operate/update-delete/sequence-column-manual.md: ## @@ -60,13 +60,21 @@ The principle is the same as the reading process during Base Compaction. When the Sequence column creates a table, an attribute is added to the property, which is used to identify the type import of `__DORIS_SEQUENCE_COL__`. The grammar design is mainly to add a mapping from the sequence column to other columns. The settings of each seed method will be described below introduce. Create Table -When you create the Uniq table, you can specify the sequence column type. +1. When you create the Uniq table, you can specify the sequence column type. ```text PROPERTIES ( "function_column.sequence_type" = 'Date', ); ``` -The sequence_type is used to specify the type of the sequence column, which can be integral and time (DATE / DATETIME). +The sequence_type is used to specify the type of the sequence column, which can be integral and time (DATE / DATETIME). The mapping column needs to be specified when importing. + +2. When you create the Uniq table, you can specify the mapping of sequence column to other columns. Review Comment: Put this method as the 1st one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed issue #13765: [Feature] add functions: regexp_replace_one, regexp_extract_all
morningman closed issue #13765: [Feature] add functions: regexp_replace_one, regexp_extract_all URL: https://github.com/apache/doris/issues/13765 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #13766: [feature](function)add regexp functions: regexp_replace_one, regexp_extract_all
morningman merged PR #13766: URL: https://github.com/apache/doris/pull/13766 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature](function)add regexp functions: regexp_replace_one, regexp_extract_all (#13766)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b83744d2f6 [feature](function)add regexp functions:
regexp_replace_one, regexp_extract_all (#13766)
b83744d2f6 is described below
commit b83744d2f6a581e4c03b430e218d7905e7e09373
Author: qiye
AuthorDate: Wed Nov 2 23:15:57 2022 +0800
[feature](function)add regexp functions: regexp_replace_one,
regexp_extract_all (#13766)
---
be/src/vec/functions/function_regexp.cpp | 263 +++--
be/test/vec/function/function_like_test.cpp| 61 +
.../string-functions/regexp/regexp_extract.md | 2 +-
.../string-functions/regexp/regexp_extract_all.md | 61 +
.../string-functions/regexp/regexp_replace_one.md | 54 +
docs/sidebars.json | 2 +
.../string-functions/regexp/regexp_extract_all.md | 59 +
.../string-functions/regexp/regexp_replace_one.md | 54 +
gensrc/script/doris_builtins_functions.py | 8 +
.../test_string_function_regexp.out| 15 ++
.../test_string_function_regexp.groovy | 6 +
11 files changed, 512 insertions(+), 73 deletions(-)
diff --git a/be/src/vec/functions/function_regexp.cpp
b/be/src/vec/functions/function_regexp.cpp
index c99bb84d31..0fc3681a1e 100644
--- a/be/src/vec/functions/function_regexp.cpp
+++ b/be/src/vec/functions/function_regexp.cpp
@@ -29,92 +29,51 @@
#include "vec/utils/util.hpp"
namespace doris::vectorized {
-template
-class FunctionRegexp : public IFunction {
-public:
-static constexpr auto name = Impl::name;
-
-static FunctionPtr create() { return std::make_shared(); }
-
-String get_name() const override { return name; }
-
-bool use_default_implementation_for_constants() const override { return
false; }
-
-bool use_default_implementation_for_nulls() const override { return false;
}
-
-size_t get_number_of_arguments() const override { return 3; }
+struct RegexpReplaceImpl {
+static constexpr auto name = "regexp_replace";
-DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
-return make_nullable(std::make_shared());
-}
+static Status execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
+ size_t input_rows_count, ColumnString::Chars&
result_data,
+ ColumnString::Offsets& result_offset, NullMap&
null_map) {
+const auto* str_col =
check_and_get_column(argument_columns[0].get());
+const auto* pattern_col =
check_and_get_column(argument_columns[1].get());
+const auto* replace_col =
check_and_get_column(argument_columns[2].get());
-Status prepare(FunctionContext* context,
FunctionContext::FunctionStateScope scope) override {
-if (scope == FunctionContext::THREAD_LOCAL) {
-if (context->is_col_constant(1)) {
-DCHECK(!context->get_function_state(scope));
-const auto pattern_col =
context->get_constant_col(1)->column_ptr;
-const auto& pattern =
pattern_col->get_data_at(0).to_string_val();
-if (pattern.is_null) {
-return Status::OK();
-}
+for (int i = 0; i < input_rows_count; ++i) {
+if (null_map[i]) {
+StringOP::push_null_string(i, result_data, result_offset,
null_map);
+continue;
+}
+re2::RE2* re = reinterpret_cast(
+
context->get_function_state(FunctionContext::THREAD_LOCAL));
+std::unique_ptr scoped_re; // destroys re if state->re
is nullptr
+if (re == nullptr) {
std::string error_str;
-re2::RE2* re =
-StringFunctions::compile_regex(pattern, &error_str,
StringVal::null());
+const auto& pattern =
pattern_col->get_data_at(i).to_string_val();
+re = StringFunctions::compile_regex(pattern, &error_str,
StringVal::null());
if (re == nullptr) {
-context->set_error(error_str.c_str());
-return Status::InvalidArgument(error_str);
+context->add_warning(error_str.c_str());
+StringOP::push_null_string(i, result_data, result_offset,
null_map);
+continue;
}
-context->set_function_state(scope, re);
+scoped_re.reset(re);
}
-}
-return Status::OK();
-}
-Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
-size_t result, size_t input_rows_count) override {
-size_t argument_size = arguments.size();
-ColumnPtr a
