date:20220907

[GitHub] [doris] wangshuo128 commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



wangshuo128 commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964443875


##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOneRowRelation.java:
##
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.OneRowRelation;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * A physical relation that contains only one row consist of some constant 
expressions.
+ * e.g. select 100, 'value'
+ */
+public class PhysicalOneRowRelation extends PhysicalLeaf implements 
OneRowRelation {
+private List projects;

Review Comment:
   ```suggestion
   private final List projects;
   ```



##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/TypeCoercion.java:
##
@@ -141,21 +140,16 @@ public Expression visitInPredicate(InPredicate 
inPredicate, ExpressionRewriteCon
 private Expression visitImplicitCastInputTypes(Expression expr, 
ExpressionRewriteContext ctx) {
 ImplicitCastInputTypes implicitCastInputTypes = 
(ImplicitCastInputTypes) expr;
 List newChildren = 
Lists.newArrayListWithCapacity(expr.arity());
-AtomicInteger changed = new AtomicInteger(0);
+boolean changed = false;
 for (int i = 0; i < 
implicitCastInputTypes.expectedInputTypes().size(); i++) {
-newChildren.add(implicitCast(expr.child(i), 
implicitCastInputTypes.expectedInputTypes().get(i), ctx)
-.map(e -> {
-changed.incrementAndGet();
-return e;
-})
-.orElse(expr.child(0))
-);
-}
-if (changed.get() != 0) {
-return expr.withChildren(newChildren);
-} else {
-return expr;
+AbstractDataType expectedType = 
implicitCastInputTypes.expectedInputTypes().get(i);

Review Comment:
   @qzsee  please take a look.



##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalOneRowRelation.java:
##
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.On

[GitHub] [doris] wangshuo128 commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



wangshuo128 commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964458881


##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundOneRowRelation.java:
##
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.analyzer;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.UnboundLogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.OneRowRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalLeaf;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * A relation that contains only one row consist of some constant expressions.
+ * e.g. select 100, 'value'
+ */
+public class UnboundOneRowRelation extends LogicalLeaf implements Unbound, 
OneRowRelation {
+private List projects;

Review Comment:
   ```suggestion
   private final List projects;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] zhengshiJ opened a new pull request, #12420: [fix](nereids) fix uncorrelated subquery error

2022-09-07 Thread GitBox



zhengshiJ opened a new pull request, #12420:
URL: https://github.com/apache/doris/pull/12420

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   When the current non-correlated subquery is executed, an error will be 
reported that the corresponding column cannot be found.
   The reason is that the tupleID of the child obtained in 
visitPhysicalNestedLoopJoin is not consistent with the child.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] zhengshiJ closed pull request #12420: [fix](nereids) fix uncorrelated subquery error

2022-09-07 Thread GitBox



zhengshiJ closed pull request #12420: [fix](nereids) fix uncorrelated subquery 
error
URL: https://github.com/apache/doris/pull/12420


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] zhengshiJ opened a new pull request, #12421: [fix](nereids) fix uncorrelated subquery can't get the correct result

2022-09-07 Thread GitBox



zhengshiJ opened a new pull request, #12421:
URL: https://github.com/apache/doris/pull/12421

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   When the current non-correlated subquery is executed, an error will be 
reported that the corresponding column cannot be found.
   The reason is that the tupleID of the child obtained in 
visitPhysicalNestedLoopJoin is not consistent with the child.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] bingfenglai commented on issue #12278: Communications link failure. Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.

2022-09-07 Thread GitBox



bingfenglai commented on issue #12278:
URL: https://github.com/apache/doris/issues/12278#issuecomment-1238999646

   We encountered the same situation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yangzhg closed issue #11990: [Enhancement] Add elt function for doris.

2022-09-07 Thread GitBox



yangzhg closed issue #11990: [Enhancement] Add elt function for doris.
URL: https://github.com/apache/doris/issues/11990


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yangzhg merged pull request #12321: [Function](ELT)Add elt function for doris

2022-09-07 Thread GitBox



yangzhg merged PR #12321:
URL: https://github.com/apache/doris/pull/12321


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated (f2923f9180 -> 09b45f2b71)

2022-09-07 Thread yangzhg

This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


from f2923f9180 [Refactor](Nereids) Simplify get input and output slots for 
plan/expression. (#12356)
 add 09b45f2b71 [Function](ELT)Add elt function (#12321)

No new revisions were added by this update.

Summary of changes:
 be/src/exprs/string_functions.cpp  |  9 +
 be/src/exprs/string_functions.h|  2 +
 be/src/vec/functions/function_string.cpp   |  1 +
 be/src/vec/functions/function_string.h | 45 ++
 be/test/exprs/string_functions_test.cpp|  8 
 be/test/vec/function/function_string_test.cpp  | 44 +
 .../variance.md => string-functions/elt.md}| 39 +++
 .../variance.md => string-functions/elt.md}| 42 
 gensrc/script/doris_builtins_functions.py  |  6 +++
 .../string_functions/test_string_function.out  | 12 ++
 .../string_functions/test_string_function.groovy   |  5 +++
 11 files changed, 181 insertions(+), 32 deletions(-)
 copy docs/en/docs/sql-manual/sql-functions/{aggregate-functions/variance.md => 
string-functions/elt.md} (60%)
 copy docs/zh-CN/docs/sql-manual/sql-functions/{aggregate-functions/variance.md 
=> string-functions/elt.md} (59%)
 mode change 100755 => 100644


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] AshinGau opened a new pull request, #12422: [feature-wip](parquet-reader) bug fix, create compress codec before parsing dictionary

2022-09-07 Thread GitBox



AshinGau opened a new pull request, #12422:
URL: https://github.com/apache/doris/pull/12422

   # Proposed changes
   
   ## Fix three bugs:
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [x] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [x] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [x] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [x] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [x] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] dujl commented on pull request #12188: [improvement](spark-load)Spark load supports version 3.x, default supports 3.2.2

2022-09-07 Thread GitBox



dujl commented on PR #12188:
URL: https://github.com/apache/doris/pull/12188#issuecomment-1239010649

   
   > Some ideas to supoort the multi-version spark engine in spark load:
   > 
   > 1. The spark load module is relatively independent and has the 
characteristics of low update frequency.  So maybe we can just update and 
maintain the fat jar package.
   > 2. For each future version, e.g. 1.x.x, we only maintain and update the 
latest jar package.
   > 3. For the elder version(such as spark 2.x), we just package it into a jar 
and upload it to the official website.
   > 4. Based on 1 and 2, the spark-dpp build phase can skipped by default when 
building fe. Instead, the corresponding spark-dpp jar package can be download 
by enviroment variable url in `build.sh` and users can choose not to upgrade 
the spark-dpp jar package when upgrading fe.
   
   The spark runtime environment is very complex. Different user maybe has 
different version. So It would be great if we could provide flexible version 
dependencie.
   For case 1&2,  There may be multiple spark versions in for a foreseeable 
period of time，Such as current mainstream version of spark includes spark2.x 
and spark3.x.  So it is very great to provide spark 2.x and spark3.x 
dependencie with different compile profile. Like hudi/iceberg.
   For case 3, some user maybe need compile the spark-dpp with their own 
version which major version same as our the elder version(such as spark 2.x).  
It is very inconvenient if we can only provide binary packages on website.
   For case 4, agree with u to move spark-dpp out of the doris' core.  But some 
compile env can not down package from the extranet
   
   My idea is:


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Kikyou1997 closed pull request #12226: [fix](planner) Fix wrong plan of SQL with cross join and aggreagate

2022-09-07 Thread GitBox



Kikyou1997 closed pull request #12226: [fix](planner) Fix wrong plan of SQL 
with cross join and aggreagate
URL: https://github.com/apache/doris/pull/12226


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Kikyou1997 opened a new pull request, #12423: [fix](nereids) Fix the wrong logic of the binding of TimestampArithmetic

2022-09-07 Thread GitBox



Kikyou1997 opened a new pull request, #12423:
URL: https://github.com/apache/doris/pull/12423

   # Proposed changes
   
   Issue Number: noissue
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Kikyou1997 commented on pull request #12423: [fix](nereids) Fix the wrong logic of the binding of TimestampArithmetic

2022-09-07 Thread GitBox



Kikyou1997 commented on PR #12423:
URL: https://github.com/apache/doris/pull/12423#issuecomment-1239018268

   @morrySnow @englefly @jackwener 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] carlvinhust2012 opened a new pull request, #12424: [fix](array-type) fix the invalid format load for stream load

2022-09-07 Thread GitBox



carlvinhust2012 opened a new pull request, #12424:
URL: https://github.com/apache/doris/pull/12424

   # Proposed changes
   1. this pr is used to fix the invalid format load for stream load.
   2. before the change , we will get the error when we load the invalid array 
format.
   3. after this change, we will get success and the error url which report the 
error line.
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   4. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   5. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   6. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   7. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] ningbing commented on issue #12278: Communications link failure. Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.

2022-09-07 Thread GitBox



ningbing commented on issue #12278:
URL: https://github.com/apache/doris/issues/12278#issuecomment-1239032065

   @stalary @ChPi 这个问题有进展吗？我们想实现通过JDBC自定义insert into 
Doris，按照你们官网的jdbc的例子，也是提示上面的错误。。应该是驱动，或者哪个环节跟你们的不一样。。但是我们无法定位这个问题，麻烦帮忙看下如何解决。（另外，我们不熟的Doris服务是正常的，通过Dbware等客户端，是可以正常访问的）


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morrySnow commented on a diff in pull request #12421: [fix](nereids) fix uncorrelated subquery can't get the correct result

2022-09-07 Thread GitBox



morrySnow commented on code in PR #12421:
URL: https://github.com/apache/doris/pull/12421#discussion_r964499341


##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalAssertNumRows.java:
##
@@ -59,7 +60,8 @@ public AssertNumRowsElement getAssertNumRowsElement() {
 
 @Override
 public String toString() {
-return "LogicalAssertNumRows (" + assertNumRowsElement + ")";
+return Utils.toSqlString("LogicalAssertNumRows (",

Review Comment:
   ```suggestion
   return Utils.toSqlString("LogicalAssertNumRows",
   ```



##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/AssertNumRowsElement.java:
##
@@ -75,7 +75,7 @@ public AssertNumRowsElement withChildren(List 
children) {
 public String toString() {
 return Utils.toSqlString("desiredNumOfRows: ",
 Long.toString(desiredNumOfRows),
-"assertion: " + assertion);
+"assertion: ", assertion);

Review Comment:
   the first parameter is plan name, and followed by k-v pairs.



##
fe/fe-core/src/main/java/org/apache/doris/planner/CrossJoinNode.java:
##
@@ -57,6 +58,26 @@ public CrossJoinNode(PlanNodeId id, PlanNode outer, PlanNode 
inner, TableRef inn
 nullableTupleIds.addAll(inner.getNullableTupleIds());
 }
 
+/**
+ * .

Review Comment:
   comment: used by Nereids only



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris-website] branch master updated: update manifest.json (#86)

2022-09-07 Thread jiafengzheng

This is an automated email from the ASF dual-hosted git repository.

jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git


The following commit(s) were added to refs/heads/master by this push:
 new 2fc9b77fd76 update manifest.json (#86)
2fc9b77fd76 is described below

commit 2fc9b77fd761c7aeafa397006d8f5a8cc0e9991d
Author: song7788q 
AuthorDate: Wed Sep 7 15:57:18 2022 +0800

update manifest.json (#86)
---
 static/manifest.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/static/manifest.json b/static/manifest.json
index e58aba16fc9..066003cc009 100644
--- a/static/manifest.json
+++ b/static/manifest.json
@@ -4,7 +4,7 @@
 "theme_color": "#FF",
 "background_color": "#FF",
 "display": "standalone",
-"scope": "https://doris.apache.org";,
+"scope": "/",
 "start_url": "https://doris.apache.org";,
 "related_applications": [
 {


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] hf200012 merged pull request #86: update manifest.json

2022-09-07 Thread GitBox



hf200012 merged PR #86:
URL: https://github.com/apache/doris-website/pull/86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yixiutt opened a new pull request, #12427: [enhancement](tcmalloc) disable tcmalloc.aggressive_memory_decommit

2022-09-07 Thread GitBox



yixiutt opened a new pull request, #12427:
URL: https://github.com/apache/doris/pull/12427

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] 924060929 commented on a diff in pull request #12421: [fix](nereids) fix uncorrelated subquery can't get the correct result

2022-09-07 Thread GitBox



924060929 commented on code in PR #12421:
URL: https://github.com/apache/doris/pull/12421#discussion_r964502766


##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalAssertNumRows.java:
##
@@ -59,7 +60,8 @@ public AssertNumRowsElement getAssertNumRowsElement() {
 
 @Override
 public String toString() {
-return "LogicalAssertNumRows (" + assertNumRowsElement + ")";
+return Utils.toSqlString("LogicalAssertNumRows (",

Review Comment:
   ```suggestion
   return Utils.toSqlString("LogicalAssertNumRows",
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] wsjz opened a new pull request, #12428: [feature-wip](parquet-reader) page index bug fix

2022-09-07 Thread GitBox



wsjz opened a new pull request, #12428:
URL: https://github.com/apache/doris/pull/12428

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] jackwener merged pull request #12396: [fix](Nereids)lowest cost plan map do not be merged when do group merge

2022-09-07 Thread GitBox



jackwener merged PR #12396:
URL: https://github.com/apache/doris/pull/12396


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated (09b45f2b71 -> 40f481049a)

2022-09-07 Thread jakevin

This is an automated email from the ASF dual-hosted git repository.

jakevin pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


from 09b45f2b71 [Function](ELT)Add elt function (#12321)
 add 40f481049a [fix](Nereids)lowest cost plan map do not be merged when do 
group merge (#12396)

No new revisions were added by this update.

Summary of changes:
 .../java/org/apache/doris/nereids/memo/Group.java  | 32 ++-
 .../java/org/apache/doris/nereids/memo/Memo.java   |  2 +
 .../doris/nereids/stats/StatsCalculator.java   |  2 +-
 .../apache/doris/nereids/memo/MemoCopyInTest.java  | 66 ++
 4 files changed, 64 insertions(+), 38 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yangzhg merged pull request #12276: [enhancement](spark-load)support dynamic set env

2022-09-07 Thread GitBox



yangzhg merged PR #12276:
URL: https://github.com/apache/doris/pull/12276


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [enhancement](spark-load)support dynamic set env (#12276)

2022-09-07 Thread yangzhg

This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 941bda5a20 [enhancement](spark-load)support dynamic set env  (#12276)
941bda5a20 is described below

commit 941bda5a20843f81eaafbe6a634c0da908ee339f
Author: chenlinzhong <490103...@qq.com>
AuthorDate: Wed Sep 7 16:24:29 2022 +0800

[enhancement](spark-load)support dynamic set env  (#12276)

* [enhancement](spark-load)support dynamic set env and display spark appid

* [enhancement](spark-load)support dynamic set env
---
 .../org/apache/doris/catalog/SparkResource.java| 49 --
 .../doris/load/loadv2/SparkEtlJobHandler.java  | 32 --
 .../apache/doris/load/loadv2/SparkLoadJobTest.java |  2 +-
 3 files changed, 76 insertions(+), 7 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/catalog/SparkResource.java 
b/fe/fe-core/src/main/java/org/apache/doris/catalog/SparkResource.java
index c8b59bfa26..001c8ae0cd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/SparkResource.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/SparkResource.java
@@ -74,6 +74,7 @@ public class SparkResource extends Resource {
 private static final String YARN_MASTER = "yarn";
 private static final String SPARK_CONFIG_PREFIX = "spark.";
 private static final String BROKER_PROPERTY_PREFIX = "broker.";
+private static final String ENV_PREFIX = "env.";
 // spark uses hadoop configs in the form of spark.hadoop.*
 private static final String SPARK_HADOOP_CONFIG_PREFIX = "spark.hadoop.";
 private static final String SPARK_YARN_RESOURCE_MANAGER_ADDRESS = 
"spark.hadoop.yarn.resourcemanager.address";
@@ -103,19 +104,22 @@ public class SparkResource extends Resource {
 // broker username and password
 @SerializedName(value = "brokerProperties")
 private Map brokerProperties;
+@SerializedName(value = "envConfigs")
+private Map envConfigs;
 
 public SparkResource(String name) {
-this(name, Maps.newHashMap(), null, null, Maps.newHashMap());
+this(name, Maps.newHashMap(), null, null, Maps.newHashMap(), 
Maps.newHashMap());
 }
 
 // "public" for testing
 public SparkResource(String name, Map sparkConfigs, String 
workingDir, String broker,
- Map brokerProperties) {
+ Map brokerProperties, Map envConfigs) {
 super(name, ResourceType.SPARK);
 this.sparkConfigs = sparkConfigs;
 this.workingDir = workingDir;
 this.broker = broker;
 this.brokerProperties = brokerProperties;
+this.envConfigs = envConfigs;
 }
 
 public String getMaster() {
@@ -149,12 +153,25 @@ public class SparkResource extends Resource {
 return sparkConfigs;
 }
 
+public Map getEnvConfigsWithoutPrefix() {
+Map envConfig = Maps.newHashMap();
+if (envConfigs != null) {
+for (Map.Entry entry : envConfigs.entrySet()) {
+if (entry.getKey().startsWith(ENV_PREFIX)) {
+String key = entry.getKey().substring(ENV_PREFIX.length());
+envConfig.put(key, entry.getValue());
+}
+}
+}
+return envConfig;
+}
+
 public Pair getYarnResourcemanagerAddressPair() {
 return Pair.of(YARN_RESOURCE_MANAGER_ADDRESS, 
sparkConfigs.get(SPARK_YARN_RESOURCE_MANAGER_ADDRESS));
 }
 
 public SparkResource getCopiedResource() {
-return new SparkResource(name, Maps.newHashMap(sparkConfigs), 
workingDir, broker, brokerProperties);
+return new SparkResource(name, Maps.newHashMap(sparkConfigs), 
workingDir, broker, brokerProperties, envConfigs);
 }
 
 @Override
@@ -233,6 +250,15 @@ public class SparkResource extends Resource {
 broker = properties.get(BROKER);
 }
 brokerProperties.putAll(getBrokerProperties(properties));
+Map env = getEnvConfig(properties);
+if (env.size() > 0) {
+if (envConfigs == null) {
+envConfigs = env;
+} else {
+envConfigs.putAll(env);
+}
+}
+LOG.info("updateProperties,{},{}", properties, envConfigs);
 }
 
 @Override
@@ -241,6 +267,8 @@ public class SparkResource extends Resource {
 
 // get spark configs
 sparkConfigs = getSparkConfig(properties);
+envConfigs = getEnvConfig(properties);
+LOG.info("setProperties,{},{}", properties, envConfigs);
 // check master and deploy mode
 if (getMaster() == null) {
 throw new DdlException("Missing " + SPARK_MASTER + " in 
properties");
@@ -285,6 +313,16 @@ public class SparkResource extends Resource {
 return sparkConfig;
 }
 
+private Map getEn

[GitHub] [doris] TaoZex opened a new issue, #12429: [Feature] The cbrt function support like hive.

2022-09-07 Thread GitBox



TaoZex opened a new issue, #12429:
URL: https://github.com/apache/doris/issues/12429

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Description
   
   Need poseexplode function support, this function returns the cube root of a 
double value.
   
   ### Use case
   
   https://user-images.githubusercontent.com/45089228/188832371-c6f7b5ed-ad49-4e56-90c1-4020a6353c7f.png";>
   When we enter a double value,this function returns the cube root of a double 
value.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morrySnow commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



morrySnow commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964532188


##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundOneRowRelation.java:
##
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.analyzer;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.UnboundLogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.OneRowRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalLeaf;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * A relation that contains only one row consist of some constant expressions.
+ * e.g. select 100, 'value'
+ */
+public class UnboundOneRowRelation extends LogicalLeaf implements Unbound, 
OneRowRelation {
+private final List projects;
+
+public UnboundOneRowRelation(List projects) {
+this(projects, Optional.empty(), Optional.empty());
+}
+
+private UnboundOneRowRelation(List projects, 
Optional groupExpression,
+Optional logicalProperties) {
+super(PlanType.LOGICAL_UNBOUND_ONE_ROW_RELATION, groupExpression, 
logicalProperties);
+
Preconditions.checkArgument(projects.stream().allMatch(Expression::isConstant),

Review Comment:
   i doubt `isConstant` is too strictness. for example
   ```
   SELECT now(), random();
   ```



##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundOneRowRelation.java:
##
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.analyzer;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.UnboundLogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.OneRowRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalLeaf;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * A relation that contains only one row consist of some constant expressions.
+ * e.g. select 100, 'value'
+ */
+public class UnboundOneRowRelation extends Logi

[GitHub] [doris] 924060929 commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



924060929 commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964580027


##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/Substring.java:
##
@@ -54,7 +54,7 @@ public Substring(Expression str, Expression pos) {
 
 @Override
 public DataType getDataType() {
-return StringType.INSTANCE;
+return VarcharType.SYSTEM_DEFAULT;

Review Comment:
   keep the same type as OriginPlanner, or else BE can not found the function



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] 924060929 commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



924060929 commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964583186


##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundOneRowRelation.java:
##
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.analyzer;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.UnboundLogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.OneRowRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalLeaf;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * A relation that contains only one row consist of some constant expressions.
+ * e.g. select 100, 'value'
+ */
+public class UnboundOneRowRelation extends LogicalLeaf implements Unbound, 
OneRowRelation {
+private final List projects;
+
+public UnboundOneRowRelation(List projects) {
+this(projects, Optional.empty(), Optional.empty());
+}
+
+private UnboundOneRowRelation(List projects, 
Optional groupExpression,
+Optional logicalProperties) {
+super(PlanType.LOGICAL_UNBOUND_ONE_ROW_RELATION, groupExpression, 
logicalProperties);
+
Preconditions.checkArgument(projects.stream().allMatch(Expression::isConstant),

Review Comment:
   I see, thinks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] 924060929 commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



924060929 commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964585467


##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundOneRowRelation.java:
##
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.analyzer;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.UnboundLogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.OneRowRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalLeaf;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * A relation that contains only one row consist of some constant expressions.
+ * e.g. select 100, 'value'
+ */
+public class UnboundOneRowRelation extends LogicalLeaf implements Unbound, 
OneRowRelation {
+private final List projects;
+
+public UnboundOneRowRelation(List projects) {
+this(projects, Optional.empty(), Optional.empty());
+}
+
+private UnboundOneRowRelation(List projects, 
Optional groupExpression,
+Optional logicalProperties) {
+super(PlanType.LOGICAL_UNBOUND_ONE_ROW_RELATION, groupExpression, 
logicalProperties);
+
Preconditions.checkArgument(projects.stream().allMatch(Expression::isConstant),
+"OneRowRelation must consist of some constant expression");
+this.projects = ImmutableList.copyOf(projects);
+}
+
+@Override
+public  R accept(PlanVisitor visitor, C context) {
+return visitor.visitUnboundOneRowRelation(this, context);
+}
+
+@Override
+public List getProjects() {
+return projects;
+}
+
+@Override
+public List getExpressions() {
+throw new 
UnsupportedOperationException(this.getClass().getSimpleName() + " don't support 
getExpression()");

Review Comment:
   Keep same as the logic: UnboundXxx can not invoke getExpressions(), like 
UnboundRelation.
   It maybe help up to find some bug if we getExpression from an unbound plan.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] 924060929 commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



924060929 commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964585672


##
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##
@@ -227,6 +232,56 @@ public PlanFragment visitPhysicalAggregate(
 return currentFragment;
 }
 
+@Override
+public PlanFragment visitPhysicalEmptyRelation(PhysicalEmptyRelation 
emptyRelation, PlanTranslatorContext context) {
+List output = emptyRelation.getOutput();
+TupleDescriptor tupleDescriptor = generateTupleDesc(output, null, 
context);
+for (int i = 0; i < output.size(); i++) {
+SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+slotDescriptor.setIsNullable(true);

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] 924060929 commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



924060929 commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964580027


##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/Substring.java:
##
@@ -54,7 +54,7 @@ public Substring(Expression str, Expression pos) {
 
 @Override
 public DataType getDataType() {
-return StringType.INSTANCE;
+return VarcharType.SYSTEM_DEFAULT;

Review Comment:
   Keep the same type as OriginPlanner, BE can not found the function if return 
StringType.
   But I didn't try `new VarcharType(len)`, why we need a length here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] 924060929 commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



924060929 commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964585467


##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundOneRowRelation.java:
##
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.analyzer;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.UnboundLogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.OneRowRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalLeaf;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * A relation that contains only one row consist of some constant expressions.
+ * e.g. select 100, 'value'
+ */
+public class UnboundOneRowRelation extends LogicalLeaf implements Unbound, 
OneRowRelation {
+private final List projects;
+
+public UnboundOneRowRelation(List projects) {
+this(projects, Optional.empty(), Optional.empty());
+}
+
+private UnboundOneRowRelation(List projects, 
Optional groupExpression,
+Optional logicalProperties) {
+super(PlanType.LOGICAL_UNBOUND_ONE_ROW_RELATION, groupExpression, 
logicalProperties);
+
Preconditions.checkArgument(projects.stream().allMatch(Expression::isConstant),
+"OneRowRelation must consist of some constant expression");
+this.projects = ImmutableList.copyOf(projects);
+}
+
+@Override
+public  R accept(PlanVisitor visitor, C context) {
+return visitor.visitUnboundOneRowRelation(this, context);
+}
+
+@Override
+public List getProjects() {
+return projects;
+}
+
+@Override
+public List getExpressions() {
+throw new 
UnsupportedOperationException(this.getClass().getSimpleName() + " don't support 
getExpression()");

Review Comment:
   Keep same as the logic: UnboundXxx can not invoke getExpressions(), like 
UnboundRelation.
   It maybe help up to find some bug if we getExpressions from an unbound plan?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] 924060929 commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



924060929 commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964605390


##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundOneRowRelation.java:
##
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.analyzer;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.UnboundLogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.OneRowRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalLeaf;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * A relation that contains only one row consist of some constant expressions.
+ * e.g. select 100, 'value'
+ */
+public class UnboundOneRowRelation extends LogicalLeaf implements Unbound, 
OneRowRelation {
+private final List projects;
+
+public UnboundOneRowRelation(List projects) {
+this(projects, Optional.empty(), Optional.empty());
+}
+
+private UnboundOneRowRelation(List projects, 
Optional groupExpression,
+Optional logicalProperties) {
+super(PlanType.LOGICAL_UNBOUND_ONE_ROW_RELATION, groupExpression, 
logicalProperties);
+
Preconditions.checkArgument(projects.stream().allMatch(Expression::isConstant),

Review Comment:
   'not contains any slot' is correct?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] 924060929 commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



924060929 commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964583186


##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundOneRowRelation.java:
##
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.analyzer;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.UnboundLogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.OneRowRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalLeaf;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * A relation that contains only one row consist of some constant expressions.
+ * e.g. select 100, 'value'
+ */
+public class UnboundOneRowRelation extends LogicalLeaf implements Unbound, 
OneRowRelation {
+private final List projects;
+
+public UnboundOneRowRelation(List projects) {
+this(projects, Optional.empty(), Optional.empty());
+}
+
+private UnboundOneRowRelation(List projects, 
Optional groupExpression,
+Optional logicalProperties) {
+super(PlanType.LOGICAL_UNBOUND_ONE_ROW_RELATION, groupExpression, 
logicalProperties);
+
Preconditions.checkArgument(projects.stream().allMatch(Expression::isConstant),

Review Comment:
   I see, thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] zxealous opened a new issue, #12431: [Bug] The judgment logic of the cache cleaner is wrong

2022-09-07 Thread GitBox



zxealous opened a new issue, #12431:
URL: https://github.com/apache/doris/issues/12431

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Version
   
   master
   
   ### What's Wrong?
   
   The judgment logic of the cache cleaner is wrong.
   Wrong code path: be/src/io/cache/file_cache_manager.cpp:81
   
   ### What You Expected?
   
   work well
   
   ### How to Reproduce?
   
   _No response_
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] starocean999 commented on a diff in pull request #12313: [fix](grouping sets) columns in grouping sets should not be in selectstmt‘s AggFunc

2022-09-07 Thread GitBox



starocean999 commented on code in PR #12313:
URL: https://github.com/apache/doris/pull/12313#discussion_r964608877


##
fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java:
##
@@ -444,13 +444,18 @@ public void analyze(Analyzer analyzer) throws 
UserException {
 }
 }
 if (groupByClause != null && groupByClause.isGroupByExtension()) {
+ArrayList aggFnExprList = new ArrayList<>();
 for (SelectListItem item : selectList.getItems()) {
-if (item.getExpr() instanceof FunctionCallExpr && 
item.getExpr().fn instanceof AggregateFunction) {
+aggFnExprList.clear();
+getAggregateFnExpr(item.getExpr(), aggFnExprList);
+if (!aggFnExprList.isEmpty()) {

Review Comment:
   this is more clear and simple:
   `for (Expr aggFnExpr : aggFnExprList) {
   for (Expr expr : groupByClause.getGroupingExprs()) {
   
   }
   }`
   
   
   
   
  
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] starocean999 commented on a diff in pull request #12313: [fix](grouping sets) columns in grouping sets should not be in selectstmt‘s AggFunc

2022-09-07 Thread GitBox



starocean999 commented on code in PR #12313:
URL: https://github.com/apache/doris/pull/12313#discussion_r964609863


##
fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java:
##
@@ -1966,6 +1971,16 @@ private boolean checkGroupingFn(Expr expr) {
 return false;
 }
 
+private void getAggregateFnExpr(Expr expr, ArrayList aggFnExprList) {
+if (expr instanceof FunctionCallExpr && expr.fn instanceof 
AggregateFunction) {
+aggFnExprList.add(expr);
+} else if (expr.getChildren() != null && expr.getChildren().size() > 
0) {

Review Comment:
   remove else if (expr.getChildren() != null && expr.getChildren().size() > 0) 
call `for (Expr child : expr.getChildren()) {
   getAggregateFnExpr(child, aggFnExprList);
   }` directly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] zxealous opened a new pull request, #12432: [Bug](remote) Fix the judgment logic of cache cleaner

2022-09-07 Thread GitBox



zxealous opened a new pull request, #12432:
URL: https://github.com/apache/doris/pull/12432

   # Proposed changes
   
   Issue Number: close #12431 
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morrySnow opened a new pull request, #12433: [fix](Nereids) local aggregate output type is mismatch with BE

2022-09-07 Thread GitBox



morrySnow opened a new pull request, #12433:
URL: https://github.com/apache/doris/pull/12433

   ## Problem summary
   
   After aggregate disassemble, we break aggregate into two stage: LOCAL and 
GLOBAL.
   All `AggregateFunction` in both of them use `getDataType` to get the output 
data type of the function.
   But in BE, aggregate function has different output type according the stage 
it is belong to.
   This PR fix this mismatch by:
   - introduce two interface on `AggregateFunction`: `getLocalDataType` and 
`getGlobalDataType`
   - introduce a new attribute in `AggregateFunction`: `isLocal`
   - set `getDataType` as final on `AggregateFunction`
   - `AggregateFunction#getDataType` choose `getLocalDataType` or 
`getGlobalDataType` according to `isLocal`
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [x] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [x] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [x] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [x] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [x] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morrySnow commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



morrySnow commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964617443


##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/Substring.java:
##
@@ -54,7 +54,7 @@ public Substring(Expression str, Expression pos) {
 
 @Override
 public DataType getDataType() {
-return StringType.INSTANCE;
+return VarcharType.SYSTEM_DEFAULT;

Review Comment:
   if be can obtain length, so optimize in runtime could be used. Such as low 
cardinality optimization in building hash table



##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/Substring.java:
##
@@ -54,7 +54,7 @@ public Substring(Expression str, Expression pos) {
 
 @Override
 public DataType getDataType() {
-return StringType.INSTANCE;
+return VarcharType.SYSTEM_DEFAULT;

Review Comment:
   if be can obtain length, some optimize in runtime could be used. Such as low 
cardinality optimization in building hash table



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morrySnow commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



morrySnow commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964618007


##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundOneRowRelation.java:
##
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.analyzer;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.UnboundLogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.OneRowRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalLeaf;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * A relation that contains only one row consist of some constant expressions.
+ * e.g. select 100, 'value'
+ */
+public class UnboundOneRowRelation extends LogicalLeaf implements Unbound, 
OneRowRelation {
+private final List projects;
+
+public UnboundOneRowRelation(List projects) {
+this(projects, Optional.empty(), Optional.empty());
+}
+
+private UnboundOneRowRelation(List projects, 
Optional groupExpression,
+Optional logicalProperties) {
+super(PlanType.LOGICAL_UNBOUND_ONE_ROW_RELATION, groupExpression, 
logicalProperties);
+
Preconditions.checkArgument(projects.stream().allMatch(Expression::isConstant),

Review Comment:
   > 'not contains any slot' is correct?
   
   i think so



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morrySnow commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



morrySnow commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964619191


##
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundOneRowRelation.java:
##
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.analyzer;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.UnboundLogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.OneRowRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalLeaf;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * A relation that contains only one row consist of some constant expressions.
+ * e.g. select 100, 'value'
+ */
+public class UnboundOneRowRelation extends LogicalLeaf implements Unbound, 
OneRowRelation {
+private final List projects;
+
+public UnboundOneRowRelation(List projects) {
+this(projects, Optional.empty(), Optional.empty());
+}
+
+private UnboundOneRowRelation(List projects, 
Optional groupExpression,
+Optional logicalProperties) {
+super(PlanType.LOGICAL_UNBOUND_ONE_ROW_RELATION, groupExpression, 
logicalProperties);
+
Preconditions.checkArgument(projects.stream().allMatch(Expression::isConstant),
+"OneRowRelation must consist of some constant expression");
+this.projects = ImmutableList.copyOf(projects);
+}
+
+@Override
+public  R accept(PlanVisitor visitor, C context) {
+return visitor.visitUnboundOneRowRelation(this, context);
+}
+
+@Override
+public List getProjects() {
+return projects;
+}
+
+@Override
+public List getExpressions() {
+throw new 
UnsupportedOperationException(this.getClass().getSimpleName() + " don't support 
getExpression()");

Review Comment:
   just curiousness, i think current code is ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yixiutt opened a new pull request, #12434: [enhancement](generic_iterator) fix num check and add some notes

2022-09-07 Thread GitBox



yixiutt opened a new pull request, #12434:
URL: https://github.com/apache/doris/pull/12434

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xinyiZzz commented on pull request #12427: [enhancement](tcmalloc) disable tcmalloc.aggressive_memory_decommit

2022-09-07 Thread GitBox



xinyiZzz commented on PR #12427:
URL: https://github.com/apache/doris/pull/12427#issuecomment-1239164357

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] 924060929 commented on a diff in pull request #12416: [feature](Nereids) Support OneRowRelation and EmptyRelation

2022-09-07 Thread GitBox



924060929 commented on code in PR #12416:
URL: https://github.com/apache/doris/pull/12416#discussion_r964629636


##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/Substring.java:
##
@@ -54,7 +54,7 @@ public Substring(Expression str, Expression pos) {
 
 @Override
 public DataType getDataType() {
-return StringType.INSTANCE;
+return VarcharType.SYSTEM_DEFAULT;

Review Comment:
   so you mean this?
   ```java
   @Override
   public DataType getDataType() {
   if (getLength() instanceof IntegerLiteral) {
   return VarcharType.createVarcharType(((IntegerLiteral) 
getLength()).getValue());
   }
   return VarcharType.SYSTEM_DEFAULT;
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] pengxiangyu opened a new pull request, #12435: [fix](cache)fix bug for cache cleaner

2022-09-07 Thread GitBox



pengxiangyu opened a new pull request, #12435:
URL: https://github.com/apache/doris/pull/12435

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xy720 merged pull request #12353: [fix](array-type) ARRAY is not supported in bloomfilter index

2022-09-07 Thread GitBox



xy720 merged PR #12353:
URL: https://github.com/apache/doris/pull/12353


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei merged pull request #12409: Pick read by rowids

2022-09-07 Thread GitBox



yiguolei merged PR #12409:
URL: https://github.com/apache/doris/pull/12409


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated (941bda5a20 -> 184be8d13c)

2022-09-07 Thread xuyang

This is an automated email from the ASF dual-hosted git repository.

xuyang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


from 941bda5a20 [enhancement](spark-load)support dynamic set env  (#12276)
 add 184be8d13c [fix](array-type) ARRAY is not supported in bloomfilter 
index (#12353)

No new revisions were added by this update.

Summary of changes:
 .../apache/doris/common/util/PropertyAnalyzer.java |  3 +-
 .../bloom_filter_p0/test_bloom_filter.groovy   | 33 ++
 2 files changed, 35 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch dev-1.1.2 updated (59f34e5118 -> 42fcd7c7ee)

2022-09-07 Thread yiguolei

This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a change to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git


from 59f34e5118 [fix](memtracker) Fix the exceeded limit of the first query 
execution (#12338)
 add 42fcd7c7ee Pick read by rowids (#12409)

No new revisions were added by this update.

Summary of changes:
 be/src/olap/rowset/segment_v2/binary_dict_page.cpp |  32 ++
 be/src/olap/rowset/segment_v2/binary_dict_page.h   |   3 +
 be/src/olap/rowset/segment_v2/binary_plain_page.h  |  32 ++
 be/src/olap/rowset/segment_v2/bitshuffle_page.h|  26 +
 be/src/olap/rowset/segment_v2/column_reader.cpp| 125 +
 be/src/olap/rowset/segment_v2/column_reader.h  |  23 +++-
 be/src/olap/rowset/segment_v2/page_decoder.h   |   5 +
 be/src/olap/rowset/segment_v2/rle_page.h   |  32 ++
 be/src/olap/rowset/segment_v2/segment_iterator.cpp |  25 ++---
 be/src/vec/columns/column_string.h |  15 ++-
 10 files changed, 283 insertions(+), 35 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yixiutt opened a new pull request, #12436: [enhancement](tcmalloc) add aggressive_memory_decommit conf and make …

2022-09-07 Thread GitBox



yixiutt opened a new pull request, #12436:
URL: https://github.com/apache/doris/pull/12436

   …it disable
   
   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] mrhhsg opened a new pull request, #12437: [fix](agg) crash caused by failure of prepare

2022-09-07 Thread GitBox



mrhhsg opened a new pull request, #12437:
URL: https://github.com/apache/doris/pull/12437

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [x] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [x] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [x] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [x] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [x] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xinyiZzz commented on pull request #12436: [enhancement](tcmalloc) add aggressive_memory_decommit conf and make …

2022-09-07 Thread GitBox



xinyiZzz commented on PR #12436:
URL: https://github.com/apache/doris/pull/12436#issuecomment-1239206573

   a little experience:
   This modification will double the performance of some cases, but will cause 
the BE process to use more memory.
   Expect better manual gc strategies in the future to avoid excessive memory 
usage.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] zy-kkk opened a new pull request, #12438: [improvement](error info)improve the s3 path err msg

2022-09-07 Thread GitBox



zy-kkk opened a new pull request, #12438:
URL: https://github.com/apache/doris/pull/12438

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Gabriel39 commented on a diff in pull request #12407: [Opt](vectorized) Speed up bucket shuffle join hash compute

2022-09-07 Thread GitBox



Gabriel39 commented on code in PR #12407:
URL: https://github.com/apache/doris/pull/12407#discussion_r964670903


##
be/src/vec/columns/column_nullable.cpp:
##
@@ -67,6 +67,25 @@ void 
ColumnNullable::update_hashes_with_value(std::vector& hashes,
 }
 }
 
+void ColumnNullable::update_crcs_with_value(std::vector& hashes,
+doris::PrimitiveType type,
+const uint8_t* __restrict 
null_data) const {
+DCHECK(null_data == nullptr);
+auto s = hashes.size();
+DCHECK(s == size());
+auto* __restrict real_null_data = assert_cast(*null_map).get_data().data();
+if (doris::simd::count_zero_num(reinterpret_cast(real_null_data), s) == s) {

Review Comment:
   May be we can unify these two function `this->hash_null()` and 
`doris::simd::count_zero_num` ? I think it's better to use `!this->hash_null()` 
here.



##
be/src/runtime/primitive_type.h:
##
@@ -33,43 +34,6 @@ class DateTimeValue;
 class DecimalV2Value;
 struct StringValue;
 
-enum PrimitiveType {

Review Comment:
   Why delete this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Gabriel39 commented on a diff in pull request #12437: [fix](agg) crash caused by failure of prepare

2022-09-07 Thread GitBox



Gabriel39 commented on code in PR #12437:
URL: https://github.com/apache/doris/pull/12437#discussion_r964684961


##
be/src/vec/exec/vaggregation_node.cpp:
##
@@ -499,11 +508,14 @@ Status AggregationNode::close(RuntimeState* state) {
 VExpr::close(_probe_expr_ctxs, state);
 if (_executor.close) _executor.close();
 
-std::visit(
-[&](auto&& agg_method) {
-COUNTER_SET(_hash_table_size_counter, 
int64_t(agg_method.data.size()));
-},
-_agg_data._aggregated_method_variant);
+/// _hash_table_size_counter may be null if prepare failed.
+if (_hash_table_size_counter) {

Review Comment:
   If prepare failed, `close` should be never called.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] jackwener opened a new pull request, #12439: [feature](Nereids): Left deep tree join order.

2022-09-07 Thread GitBox



jackwener opened a new pull request, #12439:
URL: https://github.com/apache/doris/pull/12439

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [x] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [x] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [x] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [x] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [x] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] cambyzju commented on a diff in pull request #12438: [improvement](error info)improve the s3 path err msg

2022-09-07 Thread GitBox



cambyzju commented on code in PR #12438:
URL: https://github.com/apache/doris/pull/12438#discussion_r964699114


##
fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java:
##
@@ -244,7 +244,7 @@ public static String checkPath(String path, 
StorageBackend.StorageType type) thr
 }
 } else if (type == StorageBackend.StorageType.S3) {
 if (schema == null || !schema.equalsIgnoreCase("s3")) {
-throw new AnalysisException("Invalid export path. please use 
valid 'S3://' path.");
+throw new AnalysisException("Invalid export path. please use 
valid 's3://' path.");

Review Comment:
   !schema.equalsIgnoreCase("s3")
   
   why we need change 'S3' to 's3'?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] cambyzju commented on pull request #12435: [fix](cache)fix bug for cache cleaner

2022-09-07 Thread GitBox



cambyzju commented on PR #12435:
URL: https://github.com/apache/doris/pull/12435#issuecomment-1239240797

   same as https://github.com/apache/doris/pull/12432


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] zy-kkk commented on a diff in pull request #12438: [improvement](error info)improve the s3 path err msg

2022-09-07 Thread GitBox



zy-kkk commented on code in PR #12438:
URL: https://github.com/apache/doris/pull/12438#discussion_r964709125


##
fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java:
##
@@ -244,7 +244,7 @@ public static String checkPath(String path, 
StorageBackend.StorageType type) thr
 }
 } else if (type == StorageBackend.StorageType.S3) {
 if (schema == null || !schema.equalsIgnoreCase("s3")) {
-throw new AnalysisException("Invalid export path. please use 
valid 'S3://' path.");
+throw new AnalysisException("Invalid export path. please use 
valid 's3://' path.");

Review Comment:
   When the s3 path entered by the user is wrong, we should return the correct 
prompt information, and returning 'S3' is wrong, so I changed to the correct 
's3'



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #12423: [fix](nereids) `months_add` Function cannot be found with such expr:`'1996-01-01' + INTERVAL '3' month`

2022-09-07 Thread GitBox



github-actions[bot] commented on PR #12423:
URL: https://github.com/apache/doris/pull/12423#issuecomment-1239245064

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #12423: [fix](nereids) `months_add` Function cannot be found with such expr:`'1996-01-01' + INTERVAL '3' month`

2022-09-07 Thread GitBox



github-actions[bot] commented on PR #12423:
URL: https://github.com/apache/doris/pull/12423#issuecomment-1239245124

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] cambyzju commented on a diff in pull request #12424: [fix](array-type) fix the invalid format load for stream load

2022-09-07 Thread GitBox



cambyzju commented on code in PR #12424:
URL: https://github.com/apache/doris/pull/12424#discussion_r964713303


##
be/src/exec/base_scanner.cpp:
##
@@ -481,4 +481,35 @@ void BaseScanner::_fill_columns_from_path() {
 }
 }
 
+bool BaseScanner::is_null(const Slice& slice) {
+return slice.size == 2 && slice.data[0] == '\\' && slice.data[1] == 'N';
+}
+
+bool BaseScanner::is_array(const Slice& slice) {
+return slice.size > 0 && slice.data[0] == '[' && slice.data[slice.size - 
1] == ']';

Review Comment:
   ```suggestion
   return slice.size > 1 && slice.data[0] == '[' && slice.data[slice.size - 
1] == ']';
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] carlvinhust2012 commented on a diff in pull request #12424: [fix](array-type) fix the invalid format load for stream load

2022-09-07 Thread GitBox



carlvinhust2012 commented on code in PR #12424:
URL: https://github.com/apache/doris/pull/12424#discussion_r964717805


##
be/src/exec/base_scanner.cpp:
##
@@ -481,4 +481,35 @@ void BaseScanner::_fill_columns_from_path() {
 }
 }
 
+bool BaseScanner::is_null(const Slice& slice) {
+return slice.size == 2 && slice.data[0] == '\\' && slice.data[1] == 'N';
+}
+
+bool BaseScanner::is_array(const Slice& slice) {
+return slice.size > 0 && slice.data[0] == '[' && slice.data[slice.size - 
1] == ']';

Review Comment:
   the size at least is 2?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] carlvinhust2012 commented on a diff in pull request #12424: [fix](array-type) fix the invalid format load for stream load

2022-09-07 Thread GitBox



carlvinhust2012 commented on code in PR #12424:
URL: https://github.com/apache/doris/pull/12424#discussion_r964717805


##
be/src/exec/base_scanner.cpp:
##
@@ -481,4 +481,35 @@ void BaseScanner::_fill_columns_from_path() {
 }
 }
 
+bool BaseScanner::is_null(const Slice& slice) {
+return slice.size == 2 && slice.data[0] == '\\' && slice.data[1] == 'N';
+}
+
+bool BaseScanner::is_array(const Slice& slice) {
+return slice.size > 0 && slice.data[0] == '[' && slice.data[slice.size - 
1] == ']';

Review Comment:
   so the size at least is 2?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] pengxiangyu closed pull request #12435: [fix](cache)fix bug for cache cleaner

2022-09-07 Thread GitBox



pengxiangyu closed pull request #12435: [fix](cache)fix bug for cache cleaner
URL: https://github.com/apache/doris/pull/12435


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] mrhhsg commented on a diff in pull request #12437: [fix](agg) crash caused by failure of prepare

2022-09-07 Thread GitBox



mrhhsg commented on code in PR #12437:
URL: https://github.com/apache/doris/pull/12437#discussion_r964722684


##
be/src/vec/exec/vaggregation_node.cpp:
##
@@ -499,11 +508,14 @@ Status AggregationNode::close(RuntimeState* state) {
 VExpr::close(_probe_expr_ctxs, state);
 if (_executor.close) _executor.close();
 
-std::visit(
-[&](auto&& agg_method) {
-COUNTER_SET(_hash_table_size_counter, 
int64_t(agg_method.data.size()));
-},
-_agg_data._aggregated_method_variant);
+/// _hash_table_size_counter may be null if prepare failed.
+if (_hash_table_size_counter) {

Review Comment:
   Actually in function `Status FragmentMgr::exec_plan_fragment(const 
TExecPlanFragmentParams& params, FinishCallback cb)` when prepare was failed, 
the destructor of `PlanFragmentExecutor` will call `close`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] carlvinhust2012 commented on a diff in pull request #12424: [fix](array-type) fix the invalid format load for stream load

2022-09-07 Thread GitBox



carlvinhust2012 commented on code in PR #12424:
URL: https://github.com/apache/doris/pull/12424#discussion_r964723318


##
be/src/exec/base_scanner.cpp:
##
@@ -481,4 +481,35 @@ void BaseScanner::_fill_columns_from_path() {
 }
 }
 
+bool BaseScanner::is_null(const Slice& slice) {
+return slice.size == 2 && slice.data[0] == '\\' && slice.data[1] == 'N';
+}
+
+bool BaseScanner::is_array(const Slice& slice) {
+return slice.size > 0 && slice.data[0] == '[' && slice.data[slice.size - 
1] == ']';

Review Comment:
   done



##
be/src/exec/base_scanner.cpp:
##
@@ -481,4 +481,35 @@ void BaseScanner::_fill_columns_from_path() {
 }
 }
 
+bool BaseScanner::is_null(const Slice& slice) {
+return slice.size == 2 && slice.data[0] == '\\' && slice.data[1] == 'N';
+}
+
+bool BaseScanner::is_array(const Slice& slice) {
+return slice.size > 0 && slice.data[0] == '[' && slice.data[slice.size - 
1] == ']';

Review Comment:
   so the size at least is 2?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Gabriel39 commented on a diff in pull request #12437: [fix](agg) crash caused by failure of prepare

2022-09-07 Thread GitBox



Gabriel39 commented on code in PR #12437:
URL: https://github.com/apache/doris/pull/12437#discussion_r964724969


##
be/src/vec/exec/vaggregation_node.cpp:
##
@@ -499,11 +508,14 @@ Status AggregationNode::close(RuntimeState* state) {
 VExpr::close(_probe_expr_ctxs, state);
 if (_executor.close) _executor.close();
 
-std::visit(
-[&](auto&& agg_method) {
-COUNTER_SET(_hash_table_size_counter, 
int64_t(agg_method.data.size()));
-},
-_agg_data._aggregated_method_variant);
+/// _hash_table_size_counter may be null if prepare failed.
+if (_hash_table_size_counter) {

Review Comment:
   ```suggestion
   if (LIKELY(_hash_table_size_counter)) {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] hf200012 commented on pull request #12402: [feature](quick-compaciton) enable quick compaction by default

2022-09-07 Thread GitBox



hf200012 commented on PR #12402:
URL: https://github.com/apache/doris/pull/12402#issuecomment-1239267560

   @chenlinzhong  rebase


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] hf200012 commented on a diff in pull request #12406: [Doc](balance)add replica balance speed param

2022-09-07 Thread GitBox



hf200012 commented on code in PR #12406:
URL: https://github.com/apache/doris/pull/12406#discussion_r964727024


##
docs/zh-CN/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md:
##
@@ -637,6 +637,14 @@ TabletScheduler 在每轮调度时，都会通过 LoadBalancer 来选择一定
 * 默认值：false
 * 重要性：中
 
+以下可调整参数均为 be.conf 中可配置参数。
+
+* clone\_worker\_count

Review Comment:
   ```suggestion
   * clone_worker_count
   ```



##
docs/en/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md:
##
@@ -639,6 +639,13 @@ The following adjustable parameters are all configurable 
parameters in fe.conf.
* Default value: false
* Importance:
 
+The following adjustable parameters are all configurable parameters in be.conf.
+
+* clone\_worker\_count

Review Comment:
   ```suggestion
   * clone_worker_count
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12433: [fix](Nereids) local aggregate output type is mismatch with BE

2022-09-07 Thread GitBox



Kikyou1997 commented on code in PR #12433:
URL: https://github.com/apache/doris/pull/12433#discussion_r964734359


##
fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java:
##
@@ -1431,7 +1431,7 @@ public void finalizeImplForNereids() throws 
AnalysisException {
 // TODO: Supports type conversion to match the type of the function's 
parameters
 if (fnName.getFunction().equalsIgnoreCase("sum")) {
 // Prevent the cast type in vector exec engine
-Type childType = getChild(0).type.getMaxResolutionType();
+Type childType = getChild(0).type;

Review Comment:
   Why change this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xinyiZzz merged pull request #12427: [enhancement](tcmalloc) disable tcmalloc.aggressive_memory_decommit

2022-09-07 Thread GitBox



xinyiZzz merged PR #12427:
URL: https://github.com/apache/doris/pull/12427


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch dev-1.1.2 updated: [enhancement](tcmalloc) disable tcmalloc.aggressive_memory_decommit (#12427)

2022-09-07 Thread zouxinyi

This is an automated email from the ASF dual-hosted git repository.

zouxinyi pushed a commit to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/dev-1.1.2 by this push:
 new 0832267197 [enhancement](tcmalloc) disable 
tcmalloc.aggressive_memory_decommit (#12427)
0832267197 is described below

commit 08322671971d53a93ec55dd36d59c8b4e81c79d7
Author: yixiutt <102007456+yixi...@users.noreply.github.com>
AuthorDate: Wed Sep 7 19:41:24 2022 +0800

[enhancement](tcmalloc) disable tcmalloc.aggressive_memory_decommit (#12427)

This modification will double the performance of some cases, but will cause 
the BE process to use more memory.
Expect better manual gc strategies in the future to avoid excessive memory 
usage.
---
 be/src/runtime/bufferpool/system_allocator.cc | 7 ---
 be/src/service/doris_main.cpp | 3 ---
 2 files changed, 10 deletions(-)

diff --git a/be/src/runtime/bufferpool/system_allocator.cc 
b/be/src/runtime/bufferpool/system_allocator.cc
index 1368b93a35..3cd02b6673 100644
--- a/be/src/runtime/bufferpool/system_allocator.cc
+++ b/be/src/runtime/bufferpool/system_allocator.cc
@@ -45,13 +45,6 @@ static int64_t HUGE_PAGE_SIZE = 2LL * 1024 * 1024;
 
 SystemAllocator::SystemAllocator(int64_t min_buffer_len) : 
min_buffer_len_(min_buffer_len) {
 DCHECK(BitUtil::IsPowerOf2(min_buffer_len));
-#if !defined(ADDRESS_SANITIZER) && !defined(THREAD_SANITIZER) && 
!defined(LEAK_SANITIZER)
-// Free() assumes that aggressive decommit is enabled for TCMalloc.
-size_t aggressive_decommit_enabled;
-
MallocExtension::instance()->GetNumericProperty("tcmalloc.aggressive_memory_decommit",
-
&aggressive_decommit_enabled);
-CHECK_EQ(true, aggressive_decommit_enabled);
-#endif
 }
 
 Status SystemAllocator::Allocate(int64_t len, BufferPool::BufferHandle* 
buffer) {
diff --git a/be/src/service/doris_main.cpp b/be/src/service/doris_main.cpp
index 37d08a2a38..bce288d6de 100644
--- a/be/src/service/doris_main.cpp
+++ b/be/src/service/doris_main.cpp
@@ -315,9 +315,6 @@ int main(int argc, char** argv) {
 }
 
 #if !defined(ADDRESS_SANITIZER) && !defined(LEAK_SANITIZER) && 
!defined(THREAD_SANITIZER)
-// Aggressive decommit is required so that unused pages in the TCMalloc 
page heap are
-// not backed by physical pages and do not contribute towards memory 
consumption.
-
MallocExtension::instance()->SetNumericProperty("tcmalloc.aggressive_memory_decommit",
 1);
 // Change the total TCMalloc thread cache size if necessary.
 if (!MallocExtension::instance()->SetNumericProperty(
 "tcmalloc.max_total_thread_cache_bytes",


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] hf200012 merged pull request #12406: [Doc](balance)add replica balance speed param

2022-09-07 Thread GitBox



hf200012 merged PR #12406:
URL: https://github.com/apache/doris/pull/12406


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [Doc](balance)add replica balance speed param (#12406)

2022-09-07 Thread jiafengzheng

This is an automated email from the ASF dual-hosted git repository.

jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new c2808de867 [Doc](balance)add replica balance speed param (#12406)
c2808de867 is described below

commit c2808de8678afea48b25045db150aff3762752ae
Author: wudi <676366...@qq.com>
AuthorDate: Wed Sep 7 19:41:45 2022 +0800

[Doc](balance)add replica balance speed param (#12406)

* update balance param
---
 .../docs/admin-manual/maint-monitor/tablet-repair-and-balance.md | 9 -
 .../docs/admin-manual/maint-monitor/tablet-repair-and-balance.md | 8 
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git 
a/docs/en/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md 
b/docs/en/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md
index 709800376c..abb88053b6 100644
--- a/docs/en/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md
+++ b/docs/en/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md
@@ -212,7 +212,7 @@ Priority ensures that severely damaged fragments can be 
repaired first, and impr
 
 At the same time, in order to ensure the weight of the initial priority, we 
stipulate that the initial priority is VERY HIGH, and the lowest is lowered to 
NORMAL. When the initial priority is LOW, it is raised to HIGH at most. The 
priority adjustment here also adjusts the priority set manually by the user.
 
-## Duplicate Equilibrium
+## Replicas Balance
 
 Doris automatically balances replicas within the cluster. Currently supports 
two rebalance strategies, BeLoad and Partition. BeLoad rebalance will consider 
about the disk usage and replica count for each BE. Partition rebalance just 
aim at replica count for each partition, this helps to avoid hot spots. If you 
want high read/write performance, you may need this. Note that Partition 
rebalance do not consider about the disk usage, pay more attention to it when 
you are using Partition rebal [...]
 
@@ -639,6 +639,13 @@ The following adjustable parameters are all configurable 
parameters in fe.conf.
* Default value: false
* Importance:
 
+The following adjustable parameters are all configurable parameters in be.conf.
+
+* clone\_worker\_count
+
+ * Description: Affects the speed of copy equalization. In the case of low 
disk pressure, you can speed up replica balancing by adjusting this parameter.
+ * Default: 3
+ * Importance: Medium
 ### Unadjustable parameters
 
 The following parameters do not support modification for the time being, just 
for illustration.
diff --git 
a/docs/zh-CN/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md 
b/docs/zh-CN/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md
index cd511a4baf..c5e0a174e1 100644
--- a/docs/zh-CN/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md
+++ b/docs/zh-CN/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md
@@ -637,6 +637,14 @@ TabletScheduler 在每轮调度时，都会通过 LoadBalancer 来选择一定
 * 默认值：false
 * 重要性：中
 
+以下可调整参数均为 be.conf 中可配置参数。
+
+* clone\_worker\_count
+
+* 说明：影响副本均衡的速度。在磁盘压力不大的情况下，可以通过调整该参数来加快副本均衡。
+* 默认值：3
+* 重要性：中
+
 ### 不可调整参数
 
 以下参数暂不支持修改，仅作说明。


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] FreeOnePlus opened a new pull request, #12440: [Feature-wip](docker)Initialize and submit code for rapid development of Docker

2022-09-07 Thread GitBox



FreeOnePlus opened a new pull request, #12440:
URL: https://github.com/apache/doris/pull/12440

   # Proposed changes
   Submitted shell script code for Dockerfile, Docker-Compose and related 
registers.
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12433: [fix](Nereids) local aggregate output type is mismatch with BE

2022-09-07 Thread GitBox



Kikyou1997 commented on code in PR #12433:
URL: https://github.com/apache/doris/pull/12433#discussion_r964748000


##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/AggregateFunction.java:
##
@@ -17,31 +17,59 @@
 
 package org.apache.doris.nereids.trees.expressions.functions;
 
+import org.apache.doris.nereids.exceptions.UnboundException;
 import org.apache.doris.nereids.trees.expressions.Expression;
 import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
 import org.apache.doris.nereids.types.DataType;
 
 /** AggregateFunction. */
 public abstract class AggregateFunction extends BoundFunction {
 
-private DataType intermediate;
+/**
+ * indicate current function used in local stage or global stage.
+ */
+protected final boolean isLocal;
 
 public AggregateFunction(String name, Expression... arguments) {
+this(name, false, arguments);
+}
+
+public AggregateFunction(String name, boolean isLocal, Expression... 
arguments) {
 super(name, arguments);
+this.isLocal = isLocal;
+}
+
+public boolean isLocal() {
+return isLocal;
 }
 
 public abstract DataType getIntermediateType();

Review Comment:
   remove this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12433: [fix](Nereids) local aggregate output type is mismatch with BE

2022-09-07 Thread GitBox



Kikyou1997 commented on code in PR #12433:
URL: https://github.com/apache/doris/pull/12433#discussion_r964749153


##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/AggregateFunction.java:
##
@@ -17,31 +17,59 @@
 
 package org.apache.doris.nereids.trees.expressions.functions;
 
+import org.apache.doris.nereids.exceptions.UnboundException;
 import org.apache.doris.nereids.trees.expressions.Expression;
 import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
 import org.apache.doris.nereids.types.DataType;
 
 /** AggregateFunction. */
 public abstract class AggregateFunction extends BoundFunction {
 
-private DataType intermediate;
+/**
+ * indicate current function used in local stage or global stage.
+ */
+protected final boolean isLocal;
 
 public AggregateFunction(String name, Expression... arguments) {
+this(name, false, arguments);
+}
+
+public AggregateFunction(String name, boolean isLocal, Expression... 
arguments) {
 super(name, arguments);
+this.isLocal = isLocal;
+}
+
+public boolean isLocal() {
+return isLocal;
 }
 
 public abstract DataType getIntermediateType();
 
+/**
+ * When do two stage aggregate, the local stage's output type is different 
from the global.
+ * This method is used to get the local stage's output type.
+ */
+public abstract DataType getLocalDataType();

Review Comment:
   Why don't return the getDataType refer to the isLocal property, but use two 
distinct funciton to do this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xinyiZzz commented on pull request #12348: [enhancement](load) make default load mem limit configurable

2022-09-07 Thread GitBox



xinyiZzz commented on PR #12348:
URL: https://github.com/apache/doris/pull/12348#issuecomment-1239297059

   `runtime_state.cpp` the code below should be removed and replaced with 
`_query_options.load_mem_limit`.
   ```
   int64_t RuntimeState::get_load_mem_limit() {
   if (_query_options.__isset.load_mem_limit && 
_query_options.load_mem_limit > 0) {
   return _query_options.load_mem_limit;
   } else {
   return _query_mem_tracker->limit();
   }
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] zenoyang commented on a diff in pull request #12313: [fix](grouping sets) columns in grouping sets should not be in selectstmt‘s AggFunc

2022-09-07 Thread GitBox



zenoyang commented on code in PR #12313:
URL: https://github.com/apache/doris/pull/12313#discussion_r964765694


##
fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java:
##
@@ -444,13 +444,18 @@ public void analyze(Analyzer analyzer) throws 
UserException {
 }
 }
 if (groupByClause != null && groupByClause.isGroupByExtension()) {
+ArrayList aggFnExprList = new ArrayList<>();
 for (SelectListItem item : selectList.getItems()) {
-if (item.getExpr() instanceof FunctionCallExpr && 
item.getExpr().fn instanceof AggregateFunction) {
+aggFnExprList.clear();
+getAggregateFnExpr(item.getExpr(), aggFnExprList);
+if (!aggFnExprList.isEmpty()) {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] zenoyang commented on a diff in pull request #12313: [fix](grouping sets) columns in grouping sets should not be in selectstmt‘s AggFunc

2022-09-07 Thread GitBox



zenoyang commented on code in PR #12313:
URL: https://github.com/apache/doris/pull/12313#discussion_r964774086


##
fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java:
##
@@ -1966,6 +1971,16 @@ private boolean checkGroupingFn(Expr expr) {
 return false;
 }
 
+private void getAggregateFnExpr(Expr expr, ArrayList aggFnExprList) {
+if (expr instanceof FunctionCallExpr && expr.fn instanceof 
AggregateFunction) {
+aggFnExprList.add(expr);
+} else if (expr.getChildren() != null && expr.getChildren().size() > 
0) {

Review Comment:
   Thanks for the reminder, I have modified it, but I have kept `else if 
(expr.getChildren() != null)`, otherwise it may cause NullPointerException.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morrySnow merged pull request #12423: [fix](nereids) cast left child of TimestampArithmetic to wrong type in BindFunction

2022-09-07 Thread GitBox



morrySnow merged PR #12423:
URL: https://github.com/apache/doris/pull/12423


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [fix](nereids) cast left child of TimestampArithmetic to wrong type in BindFunction (#12423)

2022-09-07 Thread morrysnow

This is an automated email from the ASF dual-hosted git repository.

morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new bdbce77227 [fix](nereids) cast left child of TimestampArithmetic to 
wrong type in BindFunction (#12423)
bdbce77227 is described below

commit bdbce77227d16a1fe731dc4aa55b108f291fd24d
Author: Kikyou1997 <33112463+kikyou1...@users.noreply.github.com>
AuthorDate: Wed Sep 7 20:32:47 2022 +0800

[fix](nereids) cast left child of TimestampArithmetic to wrong type in 
BindFunction (#12423)
---
 .../main/java/org/apache/doris/nereids/rules/analysis/BindFunction.java | 2 +-
 .../org/apache/doris/nereids/trees/expressions/TimestampArithmetic.java | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindFunction.java
 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindFunction.java
index 567d2169ff..d1a4a4cd2c 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindFunction.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindFunction.java
@@ -175,7 +175,7 @@ public class BindFunction implements AnalysisRuleFactory {
 } catch (Exception e) {
 // ignore
 }
-if (!left.getDataType().isDateType() && 
arithmetic.getTimeUnit().isDateTimeUnit()) {
+if (!left.getDataType().isDateType() && 
!arithmetic.getTimeUnit().isDateTimeUnit()) {
 left = arithmetic.left().castTo(DateType.INSTANCE);
 }
 }
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/TimestampArithmetic.java
 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/TimestampArithmetic.java
index ec5c5ce02f..c288698451 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/TimestampArithmetic.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/TimestampArithmetic.java
@@ -36,6 +36,7 @@ import java.util.List;
  * Describes the addition and subtraction of time units from timestamps.
  * Arithmetic expressions on timestamps are syntactic sugar.
  * They are executed as function call exprs in the BE.
+ * Example: '1996-01-01' + INTERVAL '3' month;
  * TODO: we need to rethink this, and maybe need to add a new type of Interval 
then implement IntervalLiteral as others
  */
 public class TimestampArithmetic extends Expression implements 
BinaryExpression {


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] starocean999 opened a new pull request, #12441: [fix](agg)the intermediate slots should be materialized as output slots

2022-09-07 Thread GitBox



starocean999 opened a new pull request, #12441:
URL: https://github.com/apache/doris/pull/12441

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   in some case, the output slots of agg info may be materialized by call 
SlotDescriptor's materializeSrcExpr method, but not the intermediate slots. 
This pr set intermediate slots materialized info to keep consistent with output 
slots.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morrySnow commented on a diff in pull request #12433: [fix](Nereids) local aggregate output type is mismatch with BE

2022-09-07 Thread GitBox



morrySnow commented on code in PR #12433:
URL: https://github.com/apache/doris/pull/12433#discussion_r964796270


##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/AggregateFunction.java:
##
@@ -17,31 +17,59 @@
 
 package org.apache.doris.nereids.trees.expressions.functions;
 
+import org.apache.doris.nereids.exceptions.UnboundException;
 import org.apache.doris.nereids.trees.expressions.Expression;
 import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
 import org.apache.doris.nereids.types.DataType;
 
 /** AggregateFunction. */
 public abstract class AggregateFunction extends BoundFunction {
 
-private DataType intermediate;
+/**
+ * indicate current function used in local stage or global stage.
+ */
+protected final boolean isLocal;
 
 public AggregateFunction(String name, Expression... arguments) {
+this(name, false, arguments);
+}
+
+public AggregateFunction(String name, boolean isLocal, Expression... 
arguments) {
 super(name, arguments);
+this.isLocal = isLocal;
+}
+
+public boolean isLocal() {
+return isLocal;
 }
 
 public abstract DataType getIntermediateType();

Review Comment:
   this is useful in future



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [bug](NodeChannel) fix OOM caused by pending queue in sink send (#12359) (#12362)

2022-09-07 Thread dataroaring

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 569ab30556 [bug](NodeChannel) fix OOM caused by pending queue in sink 
send (#12359) (#12362)
569ab30556 is described below

commit 569ab3055699c975af5b64af01871c7597565c3e
Author: zhengyu 
AuthorDate: Wed Sep 7 20:49:08 2022 +0800

[bug](NodeChannel) fix OOM caused by pending queue in sink send (#12359) 
(#12362)

Each NodeChannel has its own queue, with size up to 1/20 exec_mem_limit.
User will crash into OOM if set exec_mem_limit high. This commit uses
fixed number to control the total max memory used by NodeChannels.

Signed-off-by: freemandealer 
---
 be/src/common/config.h   | 3 +++
 be/src/exec/tablet_sink.cpp  | 5 -
 be/src/exec/tablet_sink.h| 2 +-
 be/src/vec/sink/vtablet_sink.cpp | 4 
 4 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/be/src/common/config.h b/be/src/common/config.h
index 0d52f5f844..6a21eaf746 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -854,6 +854,9 @@ CONF_Int32(doris_remote_scanner_thread_pool_queue_size, 
"10240");
 // This config should be removed when the new scan node is ready.
 CONF_Bool(enable_new_scan_node, "true");
 
+// limit the queue of pending batches which will be sent by a single 
nodechannel
+CONF_mInt64(nodechannel_pending_queue_max_bytes, "67108864");
+
 #ifdef BE_TEST
 // test s3
 CONF_String(test_s3_resource, "resource");
diff --git a/be/src/exec/tablet_sink.cpp b/be/src/exec/tablet_sink.cpp
index cd4d8c5c8d..32bb9af088 100644
--- a/be/src/exec/tablet_sink.cpp
+++ b/be/src/exec/tablet_sink.cpp
@@ -113,7 +113,6 @@ Status NodeChannel::init(RuntimeState* state) {
 
 _rpc_timeout_ms = state->query_options().query_timeout * 1000;
 _timeout_watch.start();
-_max_pending_batches_bytes = _parent->_load_mem_limit / 20; //TODO: 
session variable percent
 
 return Status::OK();
 }
@@ -317,6 +316,10 @@ Status NodeChannel::add_row(Tuple* input_tuple, int64_t 
tablet_id) {
 //To simplify the add_row logic, postpone adding batch into req 
until the time of sending req
 _pending_batches.emplace(std::move(_cur_batch), 
_cur_add_batch_request);
 _pending_batches_num++;
+VLOG_DEBUG << "OlapTableSink:" << _parent << " NodeChannel:" << 
this
+   << " pending_batches_bytes:" << _pending_batches_bytes
+   << " jobid:" << std::to_string(_state->load_job_id())
+   << " tabletid:" << tablet_id << " loadinfo:" << 
_load_info;
 }
 
 _cur_batch.reset(new RowBatch(*_row_desc, _batch_size));
diff --git a/be/src/exec/tablet_sink.h b/be/src/exec/tablet_sink.h
index 7ba1b385a2..c1660a31f0 100644
--- a/be/src/exec/tablet_sink.h
+++ b/be/src/exec/tablet_sink.h
@@ -286,7 +286,7 @@ protected:
 
 // limit _pending_batches size
 std::atomic _pending_batches_bytes {0};
-size_t _max_pending_batches_bytes {10 * 1024 * 1024};
+size_t _max_pending_batches_bytes 
{(size_t)config::nodechannel_pending_queue_max_bytes};
 std::mutex _pending_batches_lock;  // reuse for vectorized
 std::atomic _pending_batches_num {0}; // reuse for vectorized
 
diff --git a/be/src/vec/sink/vtablet_sink.cpp b/be/src/vec/sink/vtablet_sink.cpp
index d84aceac77..025de0a035 100644
--- a/be/src/vec/sink/vtablet_sink.cpp
+++ b/be/src/vec/sink/vtablet_sink.cpp
@@ -201,6 +201,10 @@ Status VNodeChannel::add_block(vectorized::Block* block,
 _pending_batches_bytes += _cur_mutable_block->allocated_bytes();
 _pending_blocks.emplace(std::move(_cur_mutable_block), 
_cur_add_block_request);
 _pending_batches_num++;
+VLOG_DEBUG << "VOlapTableSink:" << _parent << " VNodeChannel:" << 
this
+   << " pending_batches_bytes:" << _pending_batches_bytes
+   << " jobid:" << std::to_string(_state->load_job_id())
+   << " loadinfo:" << _load_info;
 }
 
 _cur_mutable_block.reset(new vectorized::MutableBlock({_tuple_desc}));


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] dataroaring merged pull request #12362: [Bug](NodeChannel) Fix OOM caused by pending queue in sink send (#12359)

2022-09-07 Thread GitBox



dataroaring merged PR #12362:
URL: https://github.com/apache/doris/pull/12362


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] dataroaring closed issue #12359: [Bug] [Load] NodeChannel pending batches causing OOM with large exec_mem_limit and/or high load concurrency

2022-09-07 Thread GitBox



dataroaring closed issue #12359: [Bug] [Load] NodeChannel pending batches 
causing OOM with large exec_mem_limit and/or high load concurrency
URL: https://github.com/apache/doris/issues/12359


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Gabriel39 opened a new pull request, #12442: [refactor](shuffle) remove unused code

2022-09-07 Thread GitBox



Gabriel39 opened a new pull request, #12442:
URL: https://github.com/apache/doris/pull/12442

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xy720 commented on a diff in pull request #12378: [enhancement](array-type) avoid abuse of Offset and Offset64

2022-09-07 Thread GitBox



xy720 commented on code in PR #12378:
URL: https://github.com/apache/doris/pull/12378#discussion_r964818244


##
be/src/vec/columns/column_array.h:
##
@@ -45,6 +45,18 @@ class ColumnArray final : public COWHelper {
 
 ColumnArray(const ColumnArray&) = default;
 
+public:
+// offsets of array is 64bit wise
+using Offset64 = IColumn::Offset64;
+using Offsets64 = IColumn::Offsets64;
+
+private:
+// please use IColumn::Offset if we really need 32bit offset, otherwise 
use ColumnArray::Offset64
+using Offset [[deprecated("ColumnArray::Offset64 for Array, 
IColumn::Offset for String")]] =

Review Comment:
   Why should we still keep the ColumnArray::Offset, are there any other place 
still using it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman merged pull request #12392: [dev-1.1.2][fix] avoid thread blocks on wait_for_start()

2022-09-07 Thread GitBox



morningman merged PR #12392:
URL: https://github.com/apache/doris/pull/12392


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch dev-1.1.2 updated: [dev-1.1.2][fix] avoid thread blocks on wait_for_start() (#12392)

2022-09-07 Thread morningman

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/dev-1.1.2 by this push:
 new 0d873883ba [dev-1.1.2][fix] avoid thread blocks on wait_for_start() 
(#12392)
0d873883ba is described below

commit 0d873883ba76260d6d2a011a227f6b2de8696eb0
Author: Mingyu Chen 
AuthorDate: Wed Sep 7 21:34:02 2022 +0800

[dev-1.1.2][fix] avoid thread blocks on wait_for_start() (#12392)

When cancel a fragment, we should notify the "wait_for_start()" thread to 
avoid thread blocking.
Only for 1.1.2, #12411 is for master branch
---
 be/src/common/config.h|  3 +++
 be/src/runtime/fragment_mgr.cpp   | 43 +--
 be/src/runtime/plan_fragment_executor.cpp |  8 +++---
 be/src/runtime/plan_fragment_executor.h   |  5 
 be/src/runtime/query_fragments_ctx.h  | 13 +++---
 be/src/util/doris_metrics.h   |  1 +
 be/src/vec/exec/volap_scan_node.cpp   |  4 +--
 be/src/vec/sink/vdata_stream_sender.h |  3 ---
 8 files changed, 36 insertions(+), 44 deletions(-)

diff --git a/be/src/common/config.h b/be/src/common/config.h
index aa0b55bc0b..945e9631cd 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -746,6 +746,9 @@ CONF_Int32(quick_compaction_max_rows, "1000");
 CONF_Int32(quick_compaction_batch_size, "10");
 // do compaction min rowsets
 CONF_Int32(quick_compaction_min_rowsets, "10");
+// Max waiting time to wait the "plan fragment start" rpc.
+// If timeout, the fragment will be cancelled.
+CONF_mInt32(max_fragment_start_wait_time_seconds, "30");
 } // namespace config
 
 } // namespace doris
diff --git a/be/src/runtime/fragment_mgr.cpp b/be/src/runtime/fragment_mgr.cpp
index a70d77..3612ce2479 100644
--- a/be/src/runtime/fragment_mgr.cpp
+++ b/be/src/runtime/fragment_mgr.cpp
@@ -57,6 +57,7 @@ namespace doris {
 
 DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(plan_fragment_count, MetricUnit::NOUNIT);
 DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(timeout_canceled_fragment_count, 
MetricUnit::NOUNIT);
+DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(fragment_thread_pool_queue_size, 
MetricUnit::NOUNIT);
 
 std::string to_load_error_http_path(const std::string& file_name) {
 if (file_name.empty()) {
@@ -92,8 +93,6 @@ public:
 
 Status execute();
 
-Status cancel_before_execute();
-
 Status cancel(const PPlanFragmentCancelReason& reason, const std::string& 
msg = "");
 TUniqueId fragment_instance_id() const { return _fragment_instance_id; }
 
@@ -233,7 +232,12 @@ Status FragmentExecState::execute() {
 if (_need_wait_execution_trigger) {
 // if _need_wait_execution_trigger is true, which means this instance
 // is prepared but need to wait for the signal to do the rest 
execution.
-_fragments_ctx->wait_for_start();
+if (!_fragments_ctx->wait_for_start()) {
+return cancel(PPlanFragmentCancelReason::INTERNAL_ERROR, "wait 
fragment start timeout");
+}
+}
+if (_executor.runtime_state()->is_cancelled()) {
+return Status::Cancelled("cancelled before execution");
 }
 int64_t duration_ns = 0;
 {
@@ -248,24 +252,9 @@ Status FragmentExecState::execute() {
 return Status::OK();
 }
 
-Status FragmentExecState::cancel_before_execute() {
-// set status as 'abort', cuz cancel() won't effect the status arg of 
DataSink::close().
-#ifndef BE_TEST
-SCOPED_ATTACH_TASK(executor()->runtime_state());
-#endif
-_executor.set_abort();
-_executor.cancel();
-if (_pipe != nullptr) {
-_pipe->cancel("Execution aborted before start");
-}
-return Status::OK();
-}
-
 Status FragmentExecState::cancel(const PPlanFragmentCancelReason& reason, 
const std::string& msg) {
 if (!_cancelled) {
-_cancelled = true;
 std::lock_guard l(_status_lock);
-RETURN_IF_ERROR(_exec_status);
 if (reason == PPlanFragmentCancelReason::LIMIT_REACH) {
 _executor.set_is_report_on_cancel(false);
 }
@@ -273,6 +262,7 @@ Status FragmentExecState::cancel(const 
PPlanFragmentCancelReason& reason, const
 if (_pipe != nullptr) {
 _pipe->cancel(PPlanFragmentCancelReason_Name(reason));
 }
+_cancelled = true;
 }
 return Status::OK();
 }
@@ -447,11 +437,15 @@ FragmentMgr::FragmentMgr(ExecEnv* exec_env)
 .set_max_threads(config::fragment_pool_thread_num_max)
 .set_max_queue_size(config::fragment_pool_queue_size)
 .build(&_thread_pool);
+
+REGISTER_HOOK_METRIC(fragment_thread_pool_queue_size,
+ [this]() { return _thread_pool->get_queue_size(); });
 CHECK(s.ok()) << s.to_string();
 }
 
 FragmentMgr::~FragmentMgr() {
 DEREGISTER_HOOK_METRIC(plan_fragment_count);
+DEREGISTER_HOOK_METRIC(fragment_thread_pool_queue_size);
 _stop_back

[GitHub] [doris] cambyzju opened a new pull request, #12443: [enhancement](array-type) shrink column suffix zero for type ARRAY

2022-09-07 Thread GitBox



cambyzju opened a new pull request, #12443:
URL: https://github.com/apache/doris/pull/12443

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   In compute level, CHAR type will shrink suffix zeros.
   To keep the logic the same as CHAR type, we also shrink for ARRAY or 
ARRAY> types.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xinyiZzz commented on a diff in pull request #12281: [feature-wip](new-scan) Add memtracker and span for new olap scan node

2022-09-07 Thread GitBox



xinyiZzz commented on code in PR #12281:
URL: https://github.com/apache/doris/pull/12281#discussion_r964853955


##
be/src/vec/exec/scan/new_olap_scan_node.cpp:
##
@@ -36,6 +36,7 @@ NewOlapScanNode::NewOlapScanNode(ObjectPool* pool, const 
TPlanNode& tnode,
 }
 
 Status NewOlapScanNode::prepare(RuntimeState* state) {
+SCOPED_CONSUME_MEM_TRACKER(mem_tracker());

Review Comment:
   `SCOPED_CONSUME_MEM_TRACKER(mem_tracker());` should be after 
`VScanNode::prepare`
   _mem_tracker is initialized in `VScanNode::prepare -> ExecNode::prepare`,



##
be/src/vec/exec/scan/vscan_node.cpp:
##
@@ -73,7 +74,6 @@ Status VScanNode::init(const TPlanNode& tnode, RuntimeState* 
state) {
 
 Status VScanNode::prepare(RuntimeState* state) {
 RETURN_IF_ERROR(ExecNode::prepare(state));
-SCOPED_CONSUME_MEM_TRACKER(mem_tracker());

Review Comment:
   should not be removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xinyiZzz commented on pull request #12257: [fix](memory): avoid coredump when list pointer is null

2022-09-07 Thread GitBox



xinyiZzz commented on PR #12257:
URL: https://github.com/apache/doris/pull/12257#issuecomment-1239426859

   `FreePool` will be removed in the future. simple fixes to try.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xinyiZzz commented on pull request #12257: [fix](memory): avoid coredump when list pointer is null

2022-09-07 Thread GitBox



xinyiZzz commented on PR #12257:
URL: https://github.com/apache/doris/pull/12257#issuecomment-1239427694

   @levy5307 p0 case failed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] cambyzju commented on a diff in pull request #12378: [enhancement](array-type) avoid abuse of Offset and Offset64

2022-09-07 Thread GitBox



cambyzju commented on code in PR #12378:
URL: https://github.com/apache/doris/pull/12378#discussion_r964883352


##
be/src/vec/columns/column_array.h:
##
@@ -45,6 +45,18 @@ class ColumnArray final : public COWHelper {
 
 ColumnArray(const ColumnArray&) = default;
 
+public:
+// offsets of array is 64bit wise
+using Offset64 = IColumn::Offset64;
+using Offsets64 = IColumn::Offsets64;
+
+private:
+// please use IColumn::Offset if we really need 32bit offset, otherwise 
use ColumnArray::Offset64
+using Offset [[deprecated("ColumnArray::Offset64 for Array, 
IColumn::Offset for String")]] =

Review Comment:
   Do not find any method to delete typedef Offset in ColumnArray now.
   
   So we define it inside private region and mark it as deprecated.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

1 2 3 >

1 - 100 of 245 matches

Mail list logo