[GitHub] [doris] github-actions[bot] commented on pull request #10551: [fix] Fix RewriteBinaryPredicatesRule which causes wrong query results in some cases.
github-actions[bot] commented on PR #10551: URL: https://github.com/apache/doris/pull/10551#issuecomment-1175854712 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10551: [fix] Fix RewriteBinaryPredicatesRule which causes wrong query results in some cases.
github-actions[bot] commented on PR #10551: URL: https://github.com/apache/doris/pull/10551#issuecomment-1175854744 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xbyang18 opened a new issue, #10642: [Bug]failed to send brpc batch,error=Host is down
xbyang18 opened a new issue, #10642:
URL: https://github.com/apache/doris/issues/10642
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Version
0.12
### What's Wrong?
when use doris load stream into doris ,it will be some error,this error log:
W0706 14:18:10.900640 225326 alpha_rowset.cpp:170] failed to get
largest_segment_group. is null: 1. version: 0-20906. tablet: 34152065
W0706 14:18:10.958122 225312 alpha_rowset.cpp:170] failed to get
largest_segment_group. is null: 1. version: 0-21009. tablet: 34152073
W0706 14:18:10.993564 225340 alpha_rowset.cpp:170] failed to get
largest_segment_group. is null: 1. version: 0-6057. tablet: 321994
W0706 14:18:11.043638 225315 alpha_rowset.cpp:170] failed to get
largest_segment_group. is null: 1. version: 1423-1423. tablet: 74188010
W0706 14:18:11.315685 225345 alpha_rowset.cpp:170] failed to get
largest_segment_group. is null: 1. version: 0-29008. tablet: 3826530
W0706 14:18:11.819109 225310 alpha_rowset.cpp:170] failed to get
largest_segment_group. is null: 1. version: 0-6383. tablet: 231408
W0706 14:18:11.826135 225321 alpha_rowset.cpp:170] failed to get
largest_segment_group. is null: 1. version: 0-6159. tablet: 232245
W0706 14:18:11.826288 225321 alpha_rowset.cpp:170] failed to get
largest_segment_group. is null: 1. version: 6109-6154. tablet: 232281
W0706 14:18:12.164536 225298 alpha_rowset.cpp:170] failed to get
largest_segment_group. is null: 1. version: 0-14363. tablet: 69249434
W0706 14:18:12.236649 230494 data_stream_sender.cpp:141] failed to send brpc
batch, error=Host is down, error_text=[E104]Fail to read fr
om Socket{id=771 fd=769 addr=192.168.10.42:8060:34354} (0x0x6751c600):
Connection reset by peer [R1][E112]Not connected to 192.168.10.42
:8060 yet, server_id=771 [R2][E112]Not connected to 192.168.10.42:8060 yet,
server_id=771 [R3][E112]Not connected to 192.168.10.42:8060
yet, server_id=771
W0706 14:18:12.238539 230494 data_stream_sender.cpp:141] failed to send brpc
batch, error=Host is down, error_text=[E104]Fail to read fr
om Socket{id=771 fd=769 addr=192.168.10.42:8060:34354} (0x0x6751c600):
Connection reset by peer [R1][E112]Not connected to 192.168.10.42:8060 yet,
server_id=771 [R2][E112]Not connected to 192.168.10.42:8060 yet, server_id=771
[R3][E112]Not connected to 192.168.10.42:8060 yet, server_id=771
W0706 14:18:12.240651 225336 data_stream_sender.cpp:141] failed to send brpc
batch, error=Host is down, error_text=[E104]Fail to read from Socket{id=771
fd=769 addr=192.168.10.42:8060:34354} (0x0x6751c600): Connection reset by peer
[R1][E112]Not connected to 192.168.10.42:8060 yet, server_id=771 [R2][E112]Not
connected to 192.168.10.42:8060 yet, server_id=771 [R3][E112]Not connected to
192.168.10.42:8060 yet, server_id=771
### What You Expected?
Can I do something with optimize it?
and I don't need to upgrade the latest version
### How to Reproduce?
_No response_
### Anything Else?
I hope that do something with be
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on pull request #10388: [feature-wip] (array-type) add the array_distinct function
cambyzju commented on PR #10388: URL: https://github.com/apache/doris/pull/10388#issuecomment-1175860730 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on pull request #10385: [feature-wip](array-type) add function array_remove
cambyzju commented on PR #10385: URL: https://github.com/apache/doris/pull/10385#issuecomment-1175861017 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #10643: [BUG] Fix invalid return type for `left` and `right` function
Gabriel39 opened a new pull request, #10643: URL: https://github.com/apache/doris/pull/10643 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #10630: [enhancement](nereids) add betweentocompound rewrite rule for ssb
924060929 commented on code in PR #10630:
URL: https://github.com/apache/doris/pull/10630#discussion_r914489422
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/BetweenToCompoundRule.java:
##
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rewrite.rules;
+
+import
org.apache.doris.nereids.rules.expression.rewrite.AbstractExpressionRewriteRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.Between;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.GreaterThanEqual;
+import org.apache.doris.nereids.trees.expressions.LessThan;
+import org.apache.doris.nereids.trees.expressions.LessThanEqual;
+import org.apache.doris.nereids.trees.expressions.Not;
+
+/**
+ * Rewrites BetweenPredicates into an equivalent conjunctive/disjunctive
+ * CompoundPredicate.
+ * Examples:
+ * A BETWEEN X AND Y ==> A >= X AND A <= Y
+ * A NOT BETWEEN X AND Y ==> A < X OR A > Y
+ */
+public class BetweenToCompoundRule extends AbstractExpressionRewriteRule {
+
+public static BetweenToCompoundRule INSTANCE = new BetweenToCompoundRule();
+
+@Override
+public Expression visitNot(Not expr, ExpressionRewriteContext context) {
Review Comment:
In the case `Not(a between b and c)`, we can
1. rewrite `a between b and c` to ` a >= b and a <= c` in
`BetweenToCompoundRue`
2. then apply `SimplifyNotExprRule` to rewrite `Not(a >= b and a <= c)` to
`Not(a >= b) Or Not(a <= c)` by the De Morgan's laws
3. finally `SimplifyNotExprRule` recursive rewrite `Not(a >= b)` to `a < b`,
and `Not(a <= c)` to `a > c` the result is `(a < b) Or (a > c)`.
This steps make the functions independent of each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #10630: [enhancement](nereids) add betweentocompound rewrite rule for ssb
924060929 commented on code in PR #10630:
URL: https://github.com/apache/doris/pull/10630#discussion_r914489422
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/BetweenToCompoundRule.java:
##
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rewrite.rules;
+
+import
org.apache.doris.nereids.rules.expression.rewrite.AbstractExpressionRewriteRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.Between;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.GreaterThanEqual;
+import org.apache.doris.nereids.trees.expressions.LessThan;
+import org.apache.doris.nereids.trees.expressions.LessThanEqual;
+import org.apache.doris.nereids.trees.expressions.Not;
+
+/**
+ * Rewrites BetweenPredicates into an equivalent conjunctive/disjunctive
+ * CompoundPredicate.
+ * Examples:
+ * A BETWEEN X AND Y ==> A >= X AND A <= Y
+ * A NOT BETWEEN X AND Y ==> A < X OR A > Y
+ */
+public class BetweenToCompoundRule extends AbstractExpressionRewriteRule {
+
+public static BetweenToCompoundRule INSTANCE = new BetweenToCompoundRule();
+
+@Override
+public Expression visitNot(Not expr, ExpressionRewriteContext context) {
Review Comment:
In the case `Not(a between b and c)`, we can
1. rewrite `a between b and c` to ` a >= b and a <= c` in
`BetweenToCompoundRue`
2. then apply `SimplifyNotExprRule` to rewrite `Not(a >= b and a <= c)` to
`Not(a >= b) Or Not(a <= c)` by the De Morgan's laws
3. finally `SimplifyNotExprRule` recursive rewrite `Not(a >= b)` to `a < b`,
and `Not(a <= c)` to `a > c`. the result is `(a < b) Or (a > c)`.
This steps make the functions independent of each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yinzhijian commented on a diff in pull request #10630: [enhancement](nereids) add betweentocompound rewrite rule for ssb
yinzhijian commented on code in PR #10630:
URL: https://github.com/apache/doris/pull/10630#discussion_r914493288
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/BetweenToCompoundRule.java:
##
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rewrite.rules;
+
+import
org.apache.doris.nereids.rules.expression.rewrite.AbstractExpressionRewriteRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.Between;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.GreaterThanEqual;
+import org.apache.doris.nereids.trees.expressions.LessThan;
+import org.apache.doris.nereids.trees.expressions.LessThanEqual;
+import org.apache.doris.nereids.trees.expressions.Not;
+
+/**
+ * Rewrites BetweenPredicates into an equivalent conjunctive/disjunctive
+ * CompoundPredicate.
+ * Examples:
+ * A BETWEEN X AND Y ==> A >= X AND A <= Y
+ * A NOT BETWEEN X AND Y ==> A < X OR A > Y
+ */
+public class BetweenToCompoundRule extends AbstractExpressionRewriteRule {
+
+public static BetweenToCompoundRule INSTANCE = new BetweenToCompoundRule();
+
+@Override
+public Expression visitNot(Not expr, ExpressionRewriteContext context) {
Review Comment:
you're right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #10630: [enhancement](nereids) add betweentocompound rewrite rule for ssb
924060929 commented on code in PR #10630:
URL: https://github.com/apache/doris/pull/10630#discussion_r914497068
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/BetweenToCompoundRule.java:
##
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rewrite.rules;
+
+import
org.apache.doris.nereids.rules.expression.rewrite.AbstractExpressionRewriteRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.Between;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.GreaterThanEqual;
+import org.apache.doris.nereids.trees.expressions.LessThan;
+import org.apache.doris.nereids.trees.expressions.LessThanEqual;
+import org.apache.doris.nereids.trees.expressions.Not;
+
+/**
+ * Rewrites BetweenPredicates into an equivalent conjunctive/disjunctive
+ * CompoundPredicate.
+ * Examples:
+ * A BETWEEN X AND Y ==> A >= X AND A <= Y
+ * A NOT BETWEEN X AND Y ==> A < X OR A > Y
+ */
+public class BetweenToCompoundRule extends AbstractExpressionRewriteRule {
+
+public static BetweenToCompoundRule INSTANCE = new BetweenToCompoundRule();
+
+@Override
+public Expression visitNot(Not expr, ExpressionRewriteContext context) {
Review Comment:
and maybe rewrite between should be a part of a NormalizeExpressionRule?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10622: [fix] substr('', 1, 5) return empty string instead of null
github-actions[bot] commented on PR #10622: URL: https://github.com/apache/doris/pull/10622#issuecomment-1175872377 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
svn commit: r55485 - in /dev/doris: flink-connector/1.1.0/ spark-connector/1.1.0/
Author: jiafengzheng Date: Wed Jul 6 07:30:50 2022 New Revision: 55485 Log: update flink/spark connector 1.1.0 Modified: dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.11-1.1.0-src.tar.gz dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.11-1.1.0-src.tar.gz.asc dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.11-1.1.0-src.tar.gz.sha512 dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.12-1.1.0-src.tar.gz dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.12-1.1.0-src.tar.gz.asc dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.12-1.1.0-src.tar.gz.sha512 dev/doris/spark-connector/1.1.0/apache-doris-spark-connector-2.3_2.11-1.1.0-src.tar.gz dev/doris/spark-connector/1.1.0/apache-doris-spark-connector-2.3_2.11-1.1.0-src.tar.gz.asc dev/doris/spark-connector/1.1.0/apache-doris-spark-connector-2.3_2.11-1.1.0-src.tar.gz.sha512 dev/doris/spark-connector/1.1.0/apache-doris-spark-connector-3.1_2.12-1.1.0-src.tar.gz dev/doris/spark-connector/1.1.0/apache-doris-spark-connector-3.1_2.12-1.1.0-src.tar.gz.asc dev/doris/spark-connector/1.1.0/apache-doris-spark-connector-3.1_2.12-1.1.0-src.tar.gz.sha512 dev/doris/spark-connector/1.1.0/apache-doris-spark-connector-3.2_2.12-1.1.0-src.tar.gz dev/doris/spark-connector/1.1.0/apache-doris-spark-connector-3.2_2.12-1.1.0-src.tar.gz.asc dev/doris/spark-connector/1.1.0/apache-doris-spark-connector-3.2_2.12-1.1.0-src.tar.gz.sha512 Modified: dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.11-1.1.0-src.tar.gz == Binary files - no diff available. Modified: dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.11-1.1.0-src.tar.gz.asc == --- dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.11-1.1.0-src.tar.gz.asc (original) +++ dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.11-1.1.0-src.tar.gz.asc Wed Jul 6 07:30:50 2022 @@ -1,7 +1,7 @@ -BEGIN PGP SIGNATURE- -iI4EABYKADYWIQRE7MG0WSaAC++A1AnsvcKzqPARGwUCYsT6IhgcamlhZmVuZ3po -ZW5nQGFwYWNoZS5vcmcACgkQ7L3Cs6jwERsUMgD+Ps50zvpSdAoXDkCSrf3tgdBN -bXoa2jOu7Ggvyhq7RIcBANJfH7rMhq8XcPhH/HBHeWE4nOnQlTQvseLkNv6GiYcG -=Smoi +iI4EABYKADYWIQRE7MG0WSaAC++A1AnsvcKzqPARGwUCYsU3qhgcamlhZmVuZ3po +ZW5nQGFwYWNoZS5vcmcACgkQ7L3Cs6jwERsSIAEAl1yL5tBOKWSqgN+OeXlbvVc8 +5qq2ghHxxPCAXMhDeuUBAP/BsIwRPoWgPR4FQ+rbOOdl6cFVlx7JkinwzFWmop4A +=XecQ -END PGP SIGNATURE- Modified: dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.11-1.1.0-src.tar.gz.sha512 == --- dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.11-1.1.0-src.tar.gz.sha512 (original) +++ dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.11-1.1.0-src.tar.gz.sha512 Wed Jul 6 07:30:50 2022 @@ -1 +1 @@ -6a7c9f9d5d7eadef553bfd96303caed1045f3bf2d2ed301a29424bb73eafeccabe2b05c6a26f499dacfc56b15f4356c6070d53fbc85d6bd571f7c10a6dc1f17c apache-doris-flink-connector-1.14_2.11-1.1.0-src.tar.gz +879e3f3890a483bd8a52b1aa39214be88de6d8d00434d5bf983a8a72f180fd5037d7b13947dde7339f38f953d89e65315d3e7bb02a5386cae241a4870ca8db7f apache-doris-flink-connector-1.14_2.11-1.1.0-src.tar.gz Modified: dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.12-1.1.0-src.tar.gz == Binary files - no diff available. Modified: dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.12-1.1.0-src.tar.gz.asc == --- dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.12-1.1.0-src.tar.gz.asc (original) +++ dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.12-1.1.0-src.tar.gz.asc Wed Jul 6 07:30:50 2022 @@ -1,7 +1,7 @@ -BEGIN PGP SIGNATURE- -iI4EABYKADYWIQRE7MG0WSaAC++A1AnsvcKzqPARGwUCYsT6axgcamlhZmVuZ3po -ZW5nQGFwYWNoZS5vcmcACgkQ7L3Cs6jwERsuvQD/REoQkj77QvzoVYQJbDdwI6AK -JjzEZaPKcDQS/WOOnv4BALyScPjxti75DsexDgYiif0D9qxAXCsLEBFln6ro02wG -=fMzR +iI4EABYKADYWIQRE7MG0WSaAC++A1AnsvcKzqPARGwUCYsU3RBgcamlhZmVuZ3po +ZW5nQGFwYWNoZS5vcmcACgkQ7L3Cs6jwERvjFgD9GwyWmtmfH9jGoHWLSVEvj74f +PnEZ6hW975pq1UGaOMkBAKgFb2rGKGCv5VNW6wz94zqtrcM4LXV1PZkLu+N31Z4H +=mYWL -END PGP SIGNATURE- Modified: dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.12-1.1.0-src.tar.gz.sha512 == --- dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.12-1.1.0-src.tar.gz.sha512 (original) +++ dev/doris/flink-connector/1.1.0/apache-doris-flink-connector-1.14_2.12-1.1.0-src.tar.gz
[GitHub] [doris] EmmyMiao87 commented on a diff in pull request #8861: [feature-wip](statistics) step4: collect statistics by implementing statistics tasks
EmmyMiao87 commented on code in PR #8861:
URL: https://github.com/apache/doris/pull/8861#discussion_r914490772
##
fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeStmt.java:
##
@@ -60,47 +60,50 @@
* properties: properties of statistics jobs
*/
public class AnalyzeStmt extends DdlStmt {
-private static final Logger LOG = LogManager.getLogger(AnalyzeStmt.class);
-
-// time to wait for collect statistics
+/** time to wait for collect statistics */
public static final String CBO_STATISTICS_TASK_TIMEOUT_SEC =
"cbo_statistics_task_timeout_sec";
private static final ImmutableSet PROPERTIES_SET = new
ImmutableSet.Builder()
.add(CBO_STATISTICS_TASK_TIMEOUT_SEC)
.build();
-public static final Predicate DESIRED_TASK_TIMEOUT_SEC = (v) -> v >
0L;
+private static final Predicate DESIRED_TASK_TIMEOUT_SEC = (v) -> v >
0L;
private final TableName dbTableName;
+private final PartitionNames partitionNames;
private final List columnNames;
private final Map properties;
// after analyzed
private long dbId;
private final Set tblIds = Sets.newHashSet();
-public AnalyzeStmt(TableName dbTableName, List columns,
Map properties) {
+public AnalyzeStmt(TableName dbTableName,
+List columns,
+PartitionNames partitionNames,
+Map properties) {
Review Comment:
By the way, if you want to analyze statistics for a given partition, how do
you think the syntax should be designed?
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTableStatsStmt.java:
##
@@ -89,4 +92,9 @@ public ShowResultSetMetaData getMetaData() {
}
return builder.build();
}
+
+public List getPartitionNames() {
Review Comment:
same as above
##
fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeStmt.java:
##
@@ -122,12 +125,40 @@ public List getTables() throws AnalysisException {
return tables;
}
+public List getPartitionNames() {
+if (partitionNames == null) {
+return Lists.newArrayList();
+}
+return partitionNames.getPartitionNames();
+}
+
+public Map> getTableIdToPartitionName() throws
AnalysisException {
+Preconditions.checkArgument(isAnalyzed(),
+"The partitionIds must be obtained after the parsing is
complete");
+Map> tableIdToPartitionName = Maps.newHashMap();
+
+for (Table table : getTables()) {
+table.readLock();
+try {
+OlapTable olapTable = (OlapTable) table;
+List partitionNames = getPartitionNames();
Review Comment:
If you allow users to specify partitions and allow users to specify multiple
tables at the same time, the user-specified partitions and tables must
correspond to each other. So you need a map structure to store related
information, not a unified partitionnames
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java:
##
@@ -394,6 +394,18 @@ public long getDataSize(boolean singleReplica) {
return singleReplica ?
Double.valueOf(s.average().orElse(0)).longValue() : s.sum();
}
+/**
+ * Get the tablet row count.
+ *
+ * @param singleReplica whether to return the row count of all replicas
+ * @return the row count of tablet
+ */
+public long getRowCount(boolean singleReplica) {
Review Comment:
It looks like we don't need to get the rowcount sum of all partitions of the
tablet. Because this value is meaningless.
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowColumnStatsStmt.java:
##
@@ -67,4 +70,9 @@ public ShowResultSetMetaData getMetaData() {
}
return builder.build();
}
+
+public List getPartitionNames() {
Review Comment:
I suggest that if the statistics setting function of partition table data
has not been implemented at all, it is best to write todo directly, and the
function `getPartitionNames` is not needed.
Then you can directly add the syntax, constructor and get method together.
##
fe/fe-core/src/main/java/org/apache/doris/analysis/AlterTableStatsStmt.java:
##
@@ -85,4 +87,9 @@ public TableName getTableName() {
public Map getStatsTypeToValue() {
return statsTypeToValue;
}
+
+public List getPartitionNames() {
Review Comment:
Same as above
##
fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeStmt.java:
##
@@ -60,47 +60,50 @@
* properties: properties of statistics jobs
*/
public class AnalyzeStmt extends DdlStmt {
-private static final Logger LOG = LogManager.getLogger(AnalyzeStmt.class);
-
-// time to wait for collect statistics
+/** time to wait for collect statistics */
public static final String CBO_STATISTICS_TASK_TIMEOUT_SEC =
"cbo_statistics_task_timeout_sec";
p
[GitHub] [doris] EmmyMiao87 closed issue #10544: [Bug] BinaryPredicate contains DecimalLiteral in some cases can not query the data
EmmyMiao87 closed issue #10544: [Bug] BinaryPredicate contains DecimalLiteral in some cases can not query the data URL: https://github.com/apache/doris/issues/10544 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix] Fix RewriteBinaryPredicatesRule which causes wrong query results in some cases. (#10551)
This is an automated email from the ASF dual-hosted git repository.
lingmiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f758e1166a [fix] Fix RewriteBinaryPredicatesRule which causes wrong
query results in some cases. (#10551)
f758e1166a is described below
commit f758e1166a2844d507806c2ddc5376cea036d31b
Author: luozenglin <37725793+luozeng...@users.noreply.github.com>
AuthorDate: Wed Jul 6 15:39:27 2022 +0800
[fix] Fix RewriteBinaryPredicatesRule which causes wrong query results in
some cases. (#10551)
During the query planning phase, the binary predicate rewrite optimization
process converting DecimalLiteral to integers may overflow, resulting in false
values like "id = 12345678901.0" (see the issue for detailed examples).
This pr fixes a possible overflow and optimizes the case where
DecimalLiteral is not in the column type value range.
Issue Number: close #10544
---
.../org/apache/doris/analysis/DecimalLiteral.java | 6 ++
.../java/org/apache/doris/analysis/IntLiteral.java | 22
.../doris/rewrite/RewriteBinaryPredicatesRule.java | 72 +
.../org/apache/doris/analysis/SelectStmtTest.java | 40 +++
.../java/org/apache/doris/planner/PlannerTest.java | 4 +-
.../rewrite/RewriteBinaryPredicatesRuleTest.java | 118 +
6 files changed, 219 insertions(+), 43 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/DecimalLiteral.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/DecimalLiteral.java
index e938a46361..3e5bf9abc7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DecimalLiteral.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DecimalLiteral.java
@@ -246,6 +246,12 @@ public class DecimalLiteral extends LiteralExpr {
} else if (targetType.isFloatingPointType()) {
return new FloatLiteral(value.doubleValue(), targetType);
} else if (targetType.isIntegerType()) {
+// If the integer part of BigDecimal is too big to fit into long,
+// longValue() will only return the low-order 64-bit value.
+if (value.compareTo(BigDecimal.valueOf(Long.MAX_VALUE)) > 0
+|| value.compareTo(BigDecimal.valueOf(Long.MIN_VALUE)) <
0) {
+throw new AnalysisException("Integer part of " + value + "
exceeds storage range of Long Type.");
+}
return new IntLiteral(value.longValue(), targetType);
} else if (targetType.isStringType()) {
return new StringLiteral(value.toString());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/IntLiteral.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/IntLiteral.java
index 00662c5e6a..4d4f673822 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/IntLiteral.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/IntLiteral.java
@@ -172,6 +172,28 @@ public class IntLiteral extends LiteralExpr {
return new IntLiteral(value);
}
+public static IntLiteral createMaxValue(Type type) {
+long value = 0L;
+switch (type.getPrimitiveType()) {
+case TINYINT:
+value = TINY_INT_MAX;
+break;
+case SMALLINT:
+value = SMALL_INT_MAX;
+break;
+case INT:
+value = INT_MAX;
+break;
+case BIGINT:
+value = BIG_INT_MAX;
+break;
+default:
+Preconditions.checkState(false);
+}
+
+return new IntLiteral(value);
+}
+
@Override
public boolean isMinValue() {
switch (type.getPrimitiveType()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/rewrite/RewriteBinaryPredicatesRule.java
b/fe/fe-core/src/main/java/org/apache/doris/rewrite/RewriteBinaryPredicatesRule.java
index 4ed232b4b1..a18797b657 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/rewrite/RewriteBinaryPredicatesRule.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/rewrite/RewriteBinaryPredicatesRule.java
@@ -19,10 +19,13 @@ package org.apache.doris.rewrite;
import org.apache.doris.analysis.Analyzer;
import org.apache.doris.analysis.BinaryPredicate;
+import org.apache.doris.analysis.BinaryPredicate.Operator;
import org.apache.doris.analysis.BoolLiteral;
import org.apache.doris.analysis.CastExpr;
import org.apache.doris.analysis.DecimalLiteral;
import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.IntLiteral;
+import org.apache.doris.analysis.LiteralExpr;
import org.apache.doris.analysis.SlotRef;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
@@ -53,36 +56,61 @@ public class RewriteBinaryPredicatesRule implements
ExprRewriteRule {
* 3) "select
[GitHub] [doris] englefly opened a new pull request, #10644: [enhancement] if pending bytes exceeded, vtableSink wait until pending bytes consumed or task was cancelled
englefly opened a new pull request, #10644: URL: https://github.com/apache/doris/pull/10644 # Proposed changes 1. table_sink 在send block时检查 sink下所有nodeChannel的pending batch size 总和，如果pending size 超过 query mem limit 的1/3(666M), 则等待。每100ms检查一次pending size。直到pending size 低于阈值或 task 已经超时退出 2. nodechannel 增加pending batch时检查条件去掉 对memtracker的依赖，因为 dev-1.0.1 中 memtracker有bug，会导致判断不准确。 Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 4. Has unit tests been added: (Yes/No/No Need) 5. Has document been added or modified: (Yes/No/No Need) 6. Does it need to update dependencies: (Yes/No) 7. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] EmmyMiao87 merged pull request #10551: [fix] Fix RewriteBinaryPredicatesRule which causes wrong query results in some cases.
EmmyMiao87 merged PR #10551: URL: https://github.com/apache/doris/pull/10551 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #10644: [refactor] if pending bytes exceeded, vtableSink wait until pending bytes consumed or task was cancelled
yiguolei commented on code in PR #10644:
URL: https://github.com/apache/doris/pull/10644#discussion_r914536150
##
be/src/vec/sink/vtablet_sink.cpp:
##
@@ -116,20 +115,11 @@ Status VOlapTableSink::send(RuntimeState* state,
vectorized::Block* input_block)
if (findTabletMode == FindTabletMode::FIND_TABLET_EVERY_BATCH) {
_partition_to_tablet_map.clear();
}
-
-//if pending bytes is more than table_sink_pending_bytes_limitation, wait
at most 1 min
-size_t MAX_PENDING_BYTES = config::table_sink_pending_bytes_limitation;
-constexpr int max_retry = 120;
-int retry = 0;
-while (get_pending_bytes() > MAX_PENDING_BYTES && retry++ < max_retry) {
-std::this_thread::sleep_for(std::chrono::microseconds(500));
-}
-if (get_pending_bytes() > MAX_PENDING_BYTES) {
-std::stringstream str;
-str << "Load task " << _load_id
-<< ": pending bytes exceed limit
(config::table_sink_pending_bytes_limitation):"
-<< MAX_PENDING_BYTES;
-return Status::MemoryLimitExceeded(str.str());
+
+size_t MAX_PENDING_BYTES = _load_mem_limit / 3;
+while (get_pending_bytes() > MAX_PENDING_BYTES) {
Review Comment:
while (get_pending_bytes() > MAX_PENDING_BYTES && !state->is_cancelled())
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yinzhijian commented on a diff in pull request #10630: [enhancement](nereids) add betweentocompound rewrite rule for ssb
yinzhijian commented on code in PR #10630:
URL: https://github.com/apache/doris/pull/10630#discussion_r914539213
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/BetweenToCompoundRule.java:
##
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rewrite.rules;
+
+import
org.apache.doris.nereids.rules.expression.rewrite.AbstractExpressionRewriteRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.Between;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.GreaterThanEqual;
+import org.apache.doris.nereids.trees.expressions.LessThan;
+import org.apache.doris.nereids.trees.expressions.LessThanEqual;
+import org.apache.doris.nereids.trees.expressions.Not;
+
+/**
+ * Rewrites BetweenPredicates into an equivalent conjunctive/disjunctive
+ * CompoundPredicate.
+ * Examples:
+ * A BETWEEN X AND Y ==> A >= X AND A <= Y
+ * A NOT BETWEEN X AND Y ==> A < X OR A > Y
+ */
+public class BetweenToCompoundRule extends AbstractExpressionRewriteRule {
+
+public static BetweenToCompoundRule INSTANCE = new BetweenToCompoundRule();
+
+@Override
+public Expression visitNot(Not expr, ExpressionRewriteContext context) {
Review Comment:
Maybe NormalizeExpressionRule should be renamed
NormalizeBinaryPredicatesRule?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 opened a new pull request, #10645: [FIX] bottom line solution for vec outer join
starocean999 opened a new pull request, #10645: URL: https://github.com/apache/doris/pull/10645 # Proposed changes Issue Number: close #xxx ## Problem Summary: agg and hash join node should produce nullable type column correctly according to the fe planner ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #10644: [refactor] if pending bytes exceeded, vtableSink wait until pending bytes consumed or task was cancelled
englefly commented on code in PR #10644:
URL: https://github.com/apache/doris/pull/10644#discussion_r914545899
##
be/src/vec/sink/vtablet_sink.cpp:
##
@@ -116,20 +115,11 @@ Status VOlapTableSink::send(RuntimeState* state,
vectorized::Block* input_block)
if (findTabletMode == FindTabletMode::FIND_TABLET_EVERY_BATCH) {
_partition_to_tablet_map.clear();
}
-
-//if pending bytes is more than table_sink_pending_bytes_limitation, wait
at most 1 min
-size_t MAX_PENDING_BYTES = config::table_sink_pending_bytes_limitation;
-constexpr int max_retry = 120;
-int retry = 0;
-while (get_pending_bytes() > MAX_PENDING_BYTES && retry++ < max_retry) {
-std::this_thread::sleep_for(std::chrono::microseconds(500));
-}
-if (get_pending_bytes() > MAX_PENDING_BYTES) {
-std::stringstream str;
-str << "Load task " << _load_id
-<< ": pending bytes exceed limit
(config::table_sink_pending_bytes_limitation):"
-<< MAX_PENDING_BYTES;
-return Status::MemoryLimitExceeded(str.str());
+
+size_t MAX_PENDING_BYTES = _load_mem_limit / 3;
+while (get_pending_bytes() > MAX_PENDING_BYTES) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SaintBacchus commented on a diff in pull request #10512: [feature] (vectorization)parquet push down support
SaintBacchus commented on code in PR #10512:
URL: https://github.com/apache/doris/pull/10512#discussion_r914553110
##
be/src/exec/arrow/parquet_row_group_reader.cpp:
##
@@ -0,0 +1,567 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/arrow/parquet_row_group_reader.h"
+
+#include
+#include
+#include
+
+#include
+
+#define _PLAIN_DECODE(T, value, min_bytes, max_bytes, out_value, out_min,
out_max) \
+const T out_min = reinterpret_cast(min_bytes)[0];
\
+const T out_max = reinterpret_cast(max_bytes)[0];
\
+T out_value = *((T*)value);
+
+#define _PLAIN_DECODE_SINGLE(T, value, bytes, conjunct_value, out) \
+const T out = reinterpret_cast(bytes)[0];\
+T conjunct_value = *((T*)value);
+
+#define _FILTER_GROUP_BY_EQ_PRED(conjunct_value, min, max) \
+if (conjunct_value < min || conjunct_value > max) {\
+return true; \
+}
+
+#define _FILTER_GROUP_BY_GT_PRED(conjunct_value, max) \
+if (max <= conjunct_value) { \
+return true; \
+}
+
+#define _FILTER_GROUP_BY_GE_PRED(conjunct_value, max) \
+if (max < conjunct_value) { \
+return true; \
+}
+
+#define _FILTER_GROUP_BY_LT_PRED(conjunct_value, min) \
+if (min >= conjunct_value) { \
+return true; \
+}
+
+#define _FILTER_GROUP_BY_LE_PRED(conjunct_value, min) \
+if (min > conjunct_value) { \
+return true; \
+}
+
+#define _FILTER_GROUP_BY_IN(T, in_pred_values, min_bytes, max_bytes) \
+std::vector in_values;\
+for (auto val : in_pred_values) {\
+T value = reinterpret_cast(val)[0]; \
+in_values.emplace_back(value); \
+}\
+if (in_values.empty()) { \
+return false;\
+}\
+std::sort(in_values.begin(), in_values.end()); \
+T in_min = in_values.front();\
+T in_max = in_values.back(); \
+const T group_min = reinterpret_cast(min_bytes)[0];\
+const T group_max = reinterpret_cast(max_bytes)[0];\
+if (in_max < group_min || in_min > group_max) { \
+return true; \
+}
+
+namespace doris {
+
+RowGroupReader::RowGroupReader(RuntimeProfile* profile,
+ const std::vector& conjunct_ctxs,
+ std::shared_ptr&
file_metadata,
+ ParquetReaderWrap* parent)
+: _conjunct_ctxs(conjunct_ctxs),
+ _file_metadata(file_metadata),
+ _profile(profile),
+ _parent(parent) {}
+
+RowGroupReader::~RowGroupReader() {
+_slot_conjuncts.clear();
+_filter_group.clear();
+}
+
+Status RowGroupReader::init_filter_groups(const TupleDescriptor* tuple_desc,
+ const std::map&
map_column,
+ const std::vector&
include_column_ids) {
+std::unordered_set parquet_column_ids(include_column_ids.begin(),
+ include_column_ids.end());
+_init_conjuncts(tuple_desc, map_column, parquet_column_ids);
+int total_group = _file_metadata->num_row_groups();
+_parent->statistics()->total_groups = total_group;
+_parent->statistics()->total_rows = _file_metadata->num_rows();
+
+int64_t filtered_num_row_groups = 0;
+int64_t filtered_num_rows = 0;
+int64_t filtered_total_byte_size = 0;
+bool need_filter = false;
+for (int row_group_id = 0; row_group_id < total_group; row_group_id++) {
+
[GitHub] [doris] jackwener opened a new issue, #10646: [Feature] Push sort to the olapscan.
jackwener opened a new issue, #10646: URL: https://github.com/apache/doris/issues/10646 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description Now, table is order by the `Keys` like `Aggregate Key` When we use the sql include `orderby` which match `keys` of table. We can push it. like: ```sql create CREATE TABLE IF NOT EXISTS example_db.expamle_tbl ( `user_id` LARGEINT NOT NULL COMMENT "用户id", `date` DATE NOT NULL COMMENT "数据灌入日期时间", `city` VARCHAR(20) COMMENT "用户所在城市", `age` SMALLINT COMMENT "用户年龄", `sex` TINYINT COMMENT "用户性别", `last_visit_date` DATETIME REPLACE DEFAULT "1970-01-01 00:00:00" COMMENT "用户最后一次访问时间", `cost` BIGINT SUM DEFAULT "0" COMMENT "用户总消费", `max_dwell_time` INT MAX DEFAULT "0" COMMENT "用户最大停留时间", `min_dwell_time` INT MIN DEFAULT "9" COMMENT "用户最小停留时间" ) AGGREGATE KEY(`user_id`, `date`, `city`, `age`, `sex`) DISTRIBUTED BY HASH(`user_id`) BUCKETS 1 PROPERTIES ( "replication_allocation" = "tag.location.default: 1" ); query: select * from tbl order by user_id, city limit 10; ``` We can push `sort` to `olapscan`. Limit->Sort->OlapScan --> Limit->Merge->OlapScan. ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10617: [fix] fix the inaccurate comments
github-actions[bot] commented on PR #10617: URL: https://github.com/apache/doris/pull/10617#issuecomment-1175930783 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10617: [fix] fix the inaccurate comments
github-actions[bot] commented on PR #10617: URL: https://github.com/apache/doris/pull/10617#issuecomment-1175930831 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #10647: [feature](planner): push sort to olapscan.
jackwener opened a new pull request, #10647: URL: https://github.com/apache/doris/pull/10647 # Proposed changes Issue Number: close #10646. ## Problem Summary: Describe the overview of changes. push sort to olapscan. ## Checklist(Required) 1. Does it affect the original behavior: (Yes) 2. Has unit tests been added: (No) 3. Has document been added or modified: (No need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg commented on a diff in pull request #10611: [BUG] fix core dump caused by runtime filter
yangzhg commented on code in PR #10611:
URL: https://github.com/apache/doris/pull/10611#discussion_r914561867
##
be/src/vec/exec/volap_scan_node.cpp:
##
@@ -403,7 +403,6 @@ void VOlapScanNode::scanner_thread(VOlapScanner* scanner) {
scanner->set_opened();
}
-/*
// the follow code may cause double free in VExprContext,
Review Comment:
if this bug is fixed those comments should removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10643: [BUG] Fix invalid return type for `left` and `right` function
github-actions[bot] commented on PR #10643: URL: https://github.com/apache/doris/pull/10643#issuecomment-1175939414 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10643: [BUG] Fix invalid return type for `left` and `right` function
github-actions[bot] commented on PR #10643: URL: https://github.com/apache/doris/pull/10643#issuecomment-1175939459 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #10630: [enhancement](nereids) add betweentocompound rewrite rule for ssb
924060929 commented on code in PR #10630:
URL: https://github.com/apache/doris/pull/10630#discussion_r914567448
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/BetweenToCompoundRule.java:
##
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rewrite.rules;
+
+import
org.apache.doris.nereids.rules.expression.rewrite.AbstractExpressionRewriteRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.Between;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.GreaterThanEqual;
+import org.apache.doris.nereids.trees.expressions.LessThan;
+import org.apache.doris.nereids.trees.expressions.LessThanEqual;
+import org.apache.doris.nereids.trees.expressions.Not;
+
+/**
+ * Rewrites BetweenPredicates into an equivalent conjunctive/disjunctive
+ * CompoundPredicate.
+ * Examples:
+ * A BETWEEN X AND Y ==> A >= X AND A <= Y
+ * A NOT BETWEEN X AND Y ==> A < X OR A > Y
+ */
+public class BetweenToCompoundRule extends AbstractExpressionRewriteRule {
+
+public static BetweenToCompoundRule INSTANCE = new BetweenToCompoundRule();
+
+@Override
+public Expression visitNot(Not expr, ExpressionRewriteContext context) {
Review Comment:
In my opinion, SimplifyNotExprRule should be a part of
NormalizeExpressionRule. And NormalizeExpressionRule can process not only
binary expression, it can process all normalization of all type of expression.
I think we can refactor and merge all related rule(include
SimplifyNotExprRule) in the NormalizeExpressionRule in the future? @morrySnow
@EmmyMiao87 @wangshuo128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #10630: [enhancement](nereids) add betweentocompound rewrite rule for ssb
924060929 commented on code in PR #10630:
URL: https://github.com/apache/doris/pull/10630#discussion_r914567448
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/BetweenToCompoundRule.java:
##
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rewrite.rules;
+
+import
org.apache.doris.nereids.rules.expression.rewrite.AbstractExpressionRewriteRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.Between;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.GreaterThanEqual;
+import org.apache.doris.nereids.trees.expressions.LessThan;
+import org.apache.doris.nereids.trees.expressions.LessThanEqual;
+import org.apache.doris.nereids.trees.expressions.Not;
+
+/**
+ * Rewrites BetweenPredicates into an equivalent conjunctive/disjunctive
+ * CompoundPredicate.
+ * Examples:
+ * A BETWEEN X AND Y ==> A >= X AND A <= Y
+ * A NOT BETWEEN X AND Y ==> A < X OR A > Y
+ */
+public class BetweenToCompoundRule extends AbstractExpressionRewriteRule {
+
+public static BetweenToCompoundRule INSTANCE = new BetweenToCompoundRule();
+
+@Override
+public Expression visitNot(Not expr, ExpressionRewriteContext context) {
Review Comment:
In my opinion, SimplifyNotExprRule and BetweenToCompoundRule should be a
part of NormalizeExpressionRule. And NormalizeExpressionRule can process not
only binary expression, it can process all normalization of all type of
expression.
I think we can refactor and merge all related rule(include
SimplifyNotExprRule and BetweenToCompoundRule) in the NormalizeExpressionRule
in the future? @morrySnow @EmmyMiao87 @wangshuo128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 opened a new issue, #10648: [Bug] using Array aggregate function caused be coredump
xy720 opened a new issue, #10648: URL: https://github.com/apache/doris/issues/10648 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? ``` #0 get_data (this=) at /home/disk4/xuyang/work/baidu/bdg/doris/core/be/src/vec/common/cow.h:190 #1 get_offsets (this=) at /home/disk4/xuyang/work/baidu/bdg/doris/core/be/src/vec/columns/column_array.h:126 #2 execute (array=..., data_type_array=0x7fc8fdc7b630, result=2, block=...) at /home/disk4/xuyang/work/baidu/bdg/doris/core/be/src/vec/functions/array/function_array_aggregation.cpp:154 #3 doris::vectorized::FunctionArrayMapped, doris::vectorized::NameArrayAverage>::execute_impl (this=, context=, block=..., arguments=..., result=2, input_rows_count=1) at /home/disk4/xuyang/work/baidu/bdg/doris/core/be/src/vec/functions/array/function_array_mapped.h:65 #4 0x7fc8f3a6f4bd in execute_impl (input_rows_count=1, result=2, arguments=..., block=..., context=0x7fc8f8bdf0e8, this=0x7fc8f962a3d0) at /home/disk4/xuyang/work/baidu/bdg/doris/palo-toolchain/ldb_toolchain/include/c++/11/bits/shared_ptr_base.h:1290 #5 doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns (this=0x7fc8f962a3d0, context=0x7fc8f8bdf0e8, block=..., args=..., result=, input_rows_count=1, dry_run=false) at /home/disk4/xuyang/work/baidu/bdg/doris/core/be/src/vec/functions/function.cpp:253 #6 0x7fc8f3a70a29 in doris::vectorized::PreparedFunctionImpl::execute (this=, context=, block=..., args=..., result=, input_rows_count=1, dry_run=false) at /home/disk4/xuyang/work/baidu/bdg/doris/core/be/src/vec/functions/function.cpp:274 #7 0x7fc8f3105d04 in doris::vectorized::IFunctionBase::execute (this=this@entry=0x7fc8fde95280, context=0x7fc8f8bdf0e8, block=..., arguments=..., result=result@entry=2, input_rows_count=1, dry_run=false) at /home/disk4/xuyang/work/baidu/bdg/doris/palo-toolchain/ldb_toolchain/include/c++/11/bits/shared_ptr_base.h:1290 #8 0x7fc8f33c829b in doris::vectorized::VectorizedFnCall::execute (this=0x7fc8fdc96c00, context=0x7fc8fde946e0, block=0x7fc895366f80, result_column_id=0x7fc8f8bdf140) at /home/disk4/xuyang/work/baidu/bdg/doris/palo-toolchain/ldb_toolchain/include/c++/11/bits/stl_vector.h:1043 #9 0x7fc8f33cac27 in doris::vectorized::VExprContext::execute (this=0x7fc8fde946e0, block=block@entry=0x7fc895366f80, result_column_id=result_column_id@entry=0x7fc8f8bdf140) at /home/disk4/xuyang/work/baidu/bdg/doris/core/be/src/vec/exprs/vexpr_context.cpp:42 #10 0x7fc8f3097d7f in doris::vectorized::VUnionNode::get_next_const (this=this@entry=0x7fc8f9cf5680, state=state@entry=0x7fc8fde83000, block=block@entry=0x7fc8fde956d0) at /home/disk4/xuyang/work/baidu/bdg/doris/palo-toolchain/ldb_toolchain/include/c++/11/bits/stl_vector.h:1043 #11 0x7fc8f30985f6 in doris::vectorized::VUnionNode::get_next (this=0x7fc8f9cf5680, state=0x7fc8fde83000, block=0x7fc8fde956d0, eos=0x7fc8fde86149) at /home/disk4/xuyang/work/baidu/bdg/doris/core/be/src/vec/exec/vunion_node.cpp:237 #12 0x7fc8f25f9e9e in doris::PlanFragmentExecutor::get_vectorized_internal (this=this@entry=0x7fc8fde86070, block=block@entry=0x7fc8953689b8) at /home/disk4/xuyang/work/baidu/bdg/doris/palo-toolchain/ldb_toolchain/include/c++/11/bits/unique_ptr.h:421 #13 0x7fc8f25fe275 in doris::PlanFragmentExecutor::open_vectorized_internal (this=this@entry=0x7fc8fde86070) at /home/disk4/xuyang/work/baidu/bdg/doris/core/be/src/runtime/plan_fragment_executor.cpp:270 #14 0x7fc8f26001af in doris::PlanFragmentExecutor::open (this=this@entry=0x7fc8fde86070) at /home/disk4/xuyang/work/baidu/bdg/doris/core/be/src/runtime/plan_fragment_executor.cpp:228 #15 0x7fc8f25e1fe5 in doris::FragmentExecState::execute (this=0x7fc8fde86000) at /home/disk4/xuyang/work/baidu/bdg/doris/core/be/src/runtime/fragment_mgr.cpp:242 #16 0x7fc8f25e6a28 in doris::FragmentMgr::_exec_actual(std::shared_ptr, std::function) (this=0x7fc8f9298c40, exec_state=..., cb=...) at /home/disk4/xuyang/work/baidu/bdg/doris/palo-toolchain/ldb_toolchain/include/c++/11/bits/shared_ptr_base.h:1290 #17 0x7fc8f25f294c in __invoke_impl, std::function), doris::FragmentMgr*&, std::shared_ptr&, std::function&> (__f= @0x7fc8fde95720: (void (doris::FragmentMgr::*)(doris::FragmentMgr * const, std::shared_ptr, std::function)) 0x7fc8f25e64a0 , std::function)>, __t=@0x7fc8fde95760: 0x7fc8f9298c40, __f= @0x7fc8fde95720: (void (doris::FragmentMgr::*)(doris::FragmentMgr * const, std::shared_ptr, std::function)) 0x7fc8f25e64a0 , std::function)>, __t=@0x7fc8fde95760: 0x7fc8f9298c40) at /home/disk4/xuyang/work/baidu/bdg/doris/palo-toolchain/ldb_toolchain/include/c++/11/ext/atomicity.h:109
[GitHub] [doris] weizhengte commented on a diff in pull request #8861: [feature-wip](statistics) step4: collect statistics by implementing statistics tasks
weizhengte commented on code in PR #8861:
URL: https://github.com/apache/doris/pull/8861#discussion_r914574487
##
fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeStmt.java:
##
@@ -60,47 +60,50 @@
* properties: properties of statistics jobs
*/
public class AnalyzeStmt extends DdlStmt {
-private static final Logger LOG = LogManager.getLogger(AnalyzeStmt.class);
-
-// time to wait for collect statistics
+/** time to wait for collect statistics */
public static final String CBO_STATISTICS_TASK_TIMEOUT_SEC =
"cbo_statistics_task_timeout_sec";
private static final ImmutableSet PROPERTIES_SET = new
ImmutableSet.Builder()
.add(CBO_STATISTICS_TASK_TIMEOUT_SEC)
.build();
-public static final Predicate DESIRED_TASK_TIMEOUT_SEC = (v) -> v >
0L;
+private static final Predicate DESIRED_TASK_TIMEOUT_SEC = (v) -> v >
0L;
private final TableName dbTableName;
+private final PartitionNames partitionNames;
private final List columnNames;
private final Map properties;
// after analyzed
private long dbId;
private final Set tblIds = Sets.newHashSet();
-public AnalyzeStmt(TableName dbTableName, List columns,
Map properties) {
+public AnalyzeStmt(TableName dbTableName,
+List columns,
+PartitionNames partitionNames,
Review Comment:
I did not add the corresponding semantic parsing code to this PR. I plan to
add all the SQL syntax related to statistical information at the end. The
following getPartitionNames method is actually reserved because the syntax
parsing has not yet been implemented, otherwise, it may occur exceptions when
referenced elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 opened a new pull request, #10649: [BugFix](Array)Fix using Array aggregate function caused be coredump
xy720 opened a new pull request, #10649: URL: https://github.com/apache/doris/pull/10649 # Proposed changes Issue Number: close #10648 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #8861: [feature-wip](statistics) step4: collect statistics by implementing statistics tasks
weizhengte commented on code in PR #8861:
URL: https://github.com/apache/doris/pull/8861#discussion_r914575705
##
fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeStmt.java:
##
@@ -60,47 +60,50 @@
* properties: properties of statistics jobs
*/
public class AnalyzeStmt extends DdlStmt {
-private static final Logger LOG = LogManager.getLogger(AnalyzeStmt.class);
-
-// time to wait for collect statistics
+/** time to wait for collect statistics */
public static final String CBO_STATISTICS_TASK_TIMEOUT_SEC =
"cbo_statistics_task_timeout_sec";
private static final ImmutableSet PROPERTIES_SET = new
ImmutableSet.Builder()
.add(CBO_STATISTICS_TASK_TIMEOUT_SEC)
.build();
-public static final Predicate DESIRED_TASK_TIMEOUT_SEC = (v) -> v >
0L;
+private static final Predicate DESIRED_TASK_TIMEOUT_SEC = (v) -> v >
0L;
private final TableName dbTableName;
+private final PartitionNames partitionNames;
private final List columnNames;
private final Map properties;
// after analyzed
private long dbId;
private final Set tblIds = Sets.newHashSet();
-public AnalyzeStmt(TableName dbTableName, List columns,
Map properties) {
+public AnalyzeStmt(TableName dbTableName,
+List columns,
+PartitionNames partitionNames,
+Map properties) {
Review Comment:
The SQL syntax might look like this:
ANALYZE [[ db_name.tb_name ] [( column_name [, ...] )], ...] [
PARTITION(...) ][ PROPERTIES(...) ];
ANALYZE tableName(columnName...) PARTITION(partitionName...) PROPERTIES(xxx
= ...);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #10647: [feature](planner): push sort to olapscan.
morrySnow commented on code in PR #10647:
URL: https://github.com/apache/doris/pull/10647#discussion_r914581350
##
fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java:
##
@@ -693,6 +708,37 @@ private void computeTabletInfo() throws UserException {
}
}
+/**
+ * Check Parent sort node can push down to child olap scan.
Review Comment:
add more comment to explain why use prefix matching
##
fe/fe-core/src/main/java/org/apache/doris/planner/OriginalPlanner.java:
##
@@ -197,6 +197,28 @@ public void createPlanFragments(StatementBase statement,
Analyzer analyzer, TQue
fragments = distributedPlanner.createPlanFragments(singleNodePlan);
}
+// Push sort node down to the bottom of olapscan.
+for (PlanFragment fragment : fragments) {
+if (fragment.getPlanRoot() instanceof SortNode) {
Review Comment:
we need to find a pattern sort(scan), so should use recursive way to do that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on pull request #10640: [BugFix](Array) fix DataTypeArray to_string use after free
cambyzju commented on PR #10640: URL: https://github.com/apache/doris/pull/10640#issuecomment-1175962840 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #8861: [feature-wip](statistics) step4: collect statistics by implementing statistics tasks
weizhengte commented on code in PR #8861:
URL: https://github.com/apache/doris/pull/8861#discussion_r914590560
##
fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeStmt.java:
##
@@ -122,12 +125,40 @@ public List getTables() throws AnalysisException {
return tables;
}
+public List getPartitionNames() {
+if (partitionNames == null) {
+return Lists.newArrayList();
+}
+return partitionNames.getPartitionNames();
+}
+
+public Map> getTableIdToPartitionName() throws
AnalysisException {
+Preconditions.checkArgument(isAnalyzed(),
+"The partitionIds must be obtained after the parsing is
complete");
+Map> tableIdToPartitionName = Maps.newHashMap();
+
+for (Table table : getTables()) {
+table.readLock();
+try {
+OlapTable olapTable = (OlapTable) table;
+List partitionNames = getPartitionNames();
Review Comment:
Now it supports analysing all tables under the a database; supports
analysing a single table (user can specify columns, specify partitions). In the
case of specified conditions, it is not supported to analyze multiple tables
with one statement. If this is necessary, we can discuss and support this
scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] EmmyMiao87 commented on a diff in pull request #8861: [feature-wip](statistics) step4: collect statistics by implementing statistics tasks
EmmyMiao87 commented on code in PR #8861:
URL: https://github.com/apache/doris/pull/8861#discussion_r914511593
##
fe/fe-core/src/main/java/org/apache/doris/statistics/TableStats.java:
##
@@ -59,51 +56,190 @@ public class TableStats {
private long rowCount = -1;
private long dataSize = -1;
-private Map nameToColumnStats =
Maps.newConcurrentMap();
+private final Map nameToPartitionStats =
Maps.newConcurrentMap();
+private final Map nameToColumnStats =
Maps.newConcurrentMap();
+
+public long getRowCount() {
+if (rowCount == -1) {
+return nameToPartitionStats.values().stream()
+.filter(partitionStats -> partitionStats.getRowCount() !=
-1)
+.mapToLong(PartitionStats::getRowCount).sum();
+}
+return rowCount;
+}
+
+public void setRowCount(long rowCount) {
+this.rowCount = rowCount;
+}
+
+public long getDataSize() {
+if (dataSize == -1) {
+return nameToPartitionStats.values().stream()
+.filter(partitionStats -> partitionStats.getDataSize() !=
-1)
+.mapToLong(PartitionStats::getDataSize).sum();
+}
+return dataSize;
+}
+
+public Map getNameToPartitionStats() {
+return nameToPartitionStats;
+}
+
+public Map getNameToColumnStats() {
+if (nameToColumnStats.isEmpty()) {
Review Comment:
It seems that you need to update the column stats at the same time as
updating the partition stats, otherwise after setting the column stats once,
the column stats will not be updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] EmmyMiao87 commented on a diff in pull request #10630: [enhancement](nereids) add betweentocompound rewrite rule for ssb
EmmyMiao87 commented on code in PR #10630:
URL: https://github.com/apache/doris/pull/10630#discussion_r914608496
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/BetweenToCompoundRule.java:
##
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rewrite.rules;
+
+import
org.apache.doris.nereids.rules.expression.rewrite.AbstractExpressionRewriteRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.Between;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.GreaterThanEqual;
+import org.apache.doris.nereids.trees.expressions.LessThan;
+import org.apache.doris.nereids.trees.expressions.LessThanEqual;
+import org.apache.doris.nereids.trees.expressions.Not;
+
+/**
+ * Rewrites BetweenPredicates into an equivalent conjunctive/disjunctive
+ * CompoundPredicate.
+ * Examples:
+ * A BETWEEN X AND Y ==> A >= X AND A <= Y
+ * A NOT BETWEEN X AND Y ==> A < X OR A > Y
+ */
+public class BetweenToCompoundRule extends AbstractExpressionRewriteRule {
+
+public static BetweenToCompoundRule INSTANCE = new BetweenToCompoundRule();
+
+@Override
+public Expression visitNot(Not expr, ExpressionRewriteContext context) {
Review Comment:
I think the rules related to planning should be put in a ruleset. But it
doesn't need to be all merged into one Normalize rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10630: [enhancement](nereids) add betweentocompound rewrite rule for ssb
github-actions[bot] commented on PR #10630: URL: https://github.com/apache/doris/pull/10630#issuecomment-1175988771 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10630: [enhancement](nereids) add betweentocompound rewrite rule for ssb
github-actions[bot] commented on PR #10630: URL: https://github.com/apache/doris/pull/10630#issuecomment-1175988808 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] EmmyMiao87 commented on a diff in pull request #10624: [enhancement]: remove redundant field.
EmmyMiao87 commented on code in PR #10624:
URL: https://github.com/apache/doris/pull/10624#discussion_r914616469
##
gensrc/thrift/PlanNodes.thrift:
##
@@ -573,17 +573,8 @@ struct TSortNode {
// This is the number of rows to skip before returning results
3: optional i64 offset
- // TODO(lingbin): remove blew, because duplaicate with TSortInfo
Review Comment:
The upgrade process is generally a rolling upgrade. This means that there
may be phases of a higher version of be and a lower version of fe. Are your
changes compatible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10630: [enhancement](nereids) add betweentocompound rewrite rule for ssb
qzsee commented on code in PR #10630:
URL: https://github.com/apache/doris/pull/10630#discussion_r914619821
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/BetweenToCompoundRule.java:
##
@@ -0,0 +1,45 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rewrite.rules;
+
+import
org.apache.doris.nereids.rules.expression.rewrite.AbstractExpressionRewriteRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.Between;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThanEqual;
+import org.apache.doris.nereids.trees.expressions.LessThanEqual;
+
+/**
+ * Rewrites BetweenPredicates into an equivalent conjunctive CompoundPredicate,
+ * "not between" is first processed by the BetweenToCompoundRule and then by
the SimplifyNotExprRule.
+ * Examples:
+ * A BETWEEN X AND Y ==> A >= X AND A <= Y
+ */
+public class BetweenToCompoundRule extends AbstractExpressionRewriteRule {
+
+public static BetweenToCompoundRule INSTANCE = new BetweenToCompoundRule();
+
+@Override
+public Expression visitBetween(Between expr, ExpressionRewriteContext
context) {
+Expression left = new GreaterThanEqual<>(expr.getCompareExpr(),
expr.getLowerBound());
+Expression right = new LessThanEqual<>(expr.getCompareExpr(),
expr.getUpperBound());
+return new CompoundPredicate<>(NodeType.AND, left, right);
Review Comment:
use And？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 commented on a diff in pull request #10611: [BUG] fix core dump caused by runtime filter
Gabriel39 commented on code in PR #10611:
URL: https://github.com/apache/doris/pull/10611#discussion_r914629927
##
be/src/vec/exec/volap_scan_node.cpp:
##
@@ -403,7 +403,6 @@ void VOlapScanNode::scanner_thread(VOlapScanner* scanner) {
scanner->set_opened();
}
-/*
// the follow code may cause double free in VExprContext,
Review Comment:
Done. Thanks for reminding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch dev-1.0.1 updated: [refactor] if pending bytes exceeded, vtableSink wait until pending bytes consumed or task was cancelled (#10644)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/dev-1.0.1 by this push:
new e5a15b43b1 [refactor] if pending bytes exceeded, vtableSink wait until
pending bytes consumed or task was cancelled (#10644)
e5a15b43b1 is described below
commit e5a15b43b141c71bbc648e7a912f271c2e90d1ee
Author: minghong
AuthorDate: Wed Jul 6 17:52:10 2022 +0800
[refactor] if pending bytes exceeded, vtableSink wait until pending bytes
consumed or task was cancelled (#10644)
* if pending bytes exceeded, vtableSink wait until pending bytes consumed
or task cancelled
* refactor
---
be/src/common/config.h | 4
be/src/exec/tablet_sink.cpp | 6 ++
be/src/vec/sink/vtablet_sink.cpp | 24 +++-
3 files changed, 9 insertions(+), 25 deletions(-)
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 3174b445c2..2f956e3b8d 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -720,10 +720,6 @@ CONF_Int32(quick_compaction_max_rows, "1000");
CONF_Int32(quick_compaction_batch_size, "10");
// do compaction min rowsets
CONF_Int32(quick_compaction_min_rowsets, "10");
-
-//memory limitation for batches in pending queue, default 500M
-CONF_Int64(table_sink_pending_bytes_limitation, "524288000");
-
} // namespace config
} // namespace doris
diff --git a/be/src/exec/tablet_sink.cpp b/be/src/exec/tablet_sink.cpp
index adb44a5843..2d9733e9bf 100644
--- a/be/src/exec/tablet_sink.cpp
+++ b/be/src/exec/tablet_sink.cpp
@@ -260,8 +260,7 @@ Status NodeChannel::add_row(Tuple* input_tuple, int64_t
tablet_id) {
// But there is still some unfinished things, we do mem limit here
temporarily.
// _cancelled may be set by rpc callback, and it's possible that
_cancelled might be set in any of the steps below.
// It's fine to do a fake add_row() and return OK, because we will check
_cancelled in next add_row() or mark_close().
-while (!_cancelled && (_pending_batches_bytes > _max_pending_batches_bytes
|| _parent->_mem_tracker->AnyLimitExceeded(MemLimit::HARD)) &&
- _pending_batches_num > 0) {
+while (!_cancelled && _pending_batches_bytes > _max_pending_batches_bytes)
{
SCOPED_ATOMIC_TIMER(&_mem_exceeded_block_ns);
SleepFor(MonoDelta::FromMilliseconds(10));
}
@@ -310,8 +309,7 @@ Status NodeChannel::add_row(BlockRow& block_row, int64_t
tablet_id) {
// But there is still some unfinished things, we do mem limit here
temporarily.
// _cancelled may be set by rpc callback, and it's possible that
_cancelled might be set in any of the steps below.
// It's fine to do a fake add_row() and return OK, because we will check
_cancelled in next add_row() or mark_close().
-while (!_cancelled && (_pending_batches_bytes > _max_pending_batches_bytes
|| _parent->_mem_tracker->AnyLimitExceeded(MemLimit::HARD)) &&
- _pending_batches_num > 0) {
+while (!_cancelled && _pending_batches_bytes > _max_pending_batches_bytes)
{
SCOPED_ATOMIC_TIMER(&_mem_exceeded_block_ns);
SleepFor(MonoDelta::FromMilliseconds(10));
}
diff --git a/be/src/vec/sink/vtablet_sink.cpp b/be/src/vec/sink/vtablet_sink.cpp
index c90e3b65b7..f0239b23c6 100644
--- a/be/src/vec/sink/vtablet_sink.cpp
+++ b/be/src/vec/sink/vtablet_sink.cpp
@@ -58,9 +58,8 @@ Status VOlapTableSink::open(RuntimeState* state) {
size_t VOlapTableSink::get_pending_bytes() const {
size_t mem_consumption = 0;
-for (auto& indexChannel : _channels){
+for (auto& indexChannel : _channels) {
mem_consumption += indexChannel->get_pending_bytes();
-
}
return mem_consumption;
}
@@ -116,20 +115,10 @@ Status VOlapTableSink::send(RuntimeState* state,
vectorized::Block* input_block)
if (findTabletMode == FindTabletMode::FIND_TABLET_EVERY_BATCH) {
_partition_to_tablet_map.clear();
}
-
-//if pending bytes is more than table_sink_pending_bytes_limitation, wait
at most 1 min
-size_t MAX_PENDING_BYTES = config::table_sink_pending_bytes_limitation;
-constexpr int max_retry = 120;
-int retry = 0;
-while (get_pending_bytes() > MAX_PENDING_BYTES && retry++ < max_retry) {
-std::this_thread::sleep_for(std::chrono::microseconds(500));
-}
-if (get_pending_bytes() > MAX_PENDING_BYTES) {
-std::stringstream str;
-str << "Load task " << _load_id
-<< ": pending bytes exceed limit
(config::table_sink_pending_bytes_limitation):"
-<< MAX_PENDING_BYTES;
-return Status::MemoryLimitExceeded(str.str());
+
+size_t MAX_PENDING_BYTES = _load_mem_limit / 3;
+while (get_pending_bytes() > MAX_PENDING_BYTES && !state->is_cancelled()) {
+std::this_thread::sleep_for(std::chrono::microseconds(100));
}
for (int i = 0; i <
[doris] branch master updated: rename some files inside vec/sink dir (#10636)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a7df6e3dee rename some files inside vec/sink dir (#10636)
a7df6e3dee is described below
commit a7df6e3dee99c41136f86d67ae9e92d66a8e524c
Author: camby <104178...@qq.com>
AuthorDate: Wed Jul 6 17:52:47 2022 +0800
rename some files inside vec/sink dir (#10636)
Co-authored-by: cambyzju
---
be/src/exec/data_sink.cpp | 2 +-
be/src/vec/CMakeLists.txt | 4 ++--
be/src/vec/runtime/vfile_result_writer.h | 2 +-
be/src/vec/sink/{mysql_result_writer.cpp => vmysql_result_writer.cpp} | 2 +-
be/src/vec/sink/{mysql_result_writer.h => vmysql_result_writer.h} | 2 +-
be/src/vec/sink/{result_sink.cpp => vresult_sink.cpp} | 4 ++--
be/src/vec/sink/{result_sink.h => vresult_sink.h} | 2 +-
be/src/vec/sink/{result_writer.h => vresult_writer.h} | 0
8 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/be/src/exec/data_sink.cpp b/be/src/exec/data_sink.cpp
index c71e39a999..1e9b27b538 100644
--- a/be/src/exec/data_sink.cpp
+++ b/be/src/exec/data_sink.cpp
@@ -34,10 +34,10 @@
#include "runtime/result_file_sink.h"
#include "runtime/result_sink.h"
#include "runtime/runtime_state.h"
-#include "vec/sink/result_sink.h"
#include "vec/sink/vdata_stream_sender.h"
#include "vec/sink/vmysql_table_sink.h"
#include "vec/sink/vresult_file_sink.h"
+#include "vec/sink/vresult_sink.h"
#include "vec/sink/vtablet_sink.h"
namespace doris {
diff --git a/be/src/vec/CMakeLists.txt b/be/src/vec/CMakeLists.txt
index aa3c8b7ad4..c63ec7b54d 100644
--- a/be/src/vec/CMakeLists.txt
+++ b/be/src/vec/CMakeLists.txt
@@ -194,8 +194,8 @@ set(VEC_FILES
olap/vcollect_iterator.cpp
olap/block_reader.cpp
olap/olap_data_convertor.cpp
- sink/mysql_result_writer.cpp
- sink/result_sink.cpp
+ sink/vmysql_result_writer.cpp
+ sink/vresult_sink.cpp
sink/vdata_stream_sender.cpp
sink/vtablet_sink.cpp
sink/vmysql_table_writer.cpp
diff --git a/be/src/vec/runtime/vfile_result_writer.h
b/be/src/vec/runtime/vfile_result_writer.h
index c4066c75a5..3d10bb9b47 100644
--- a/be/src/vec/runtime/vfile_result_writer.h
+++ b/be/src/vec/runtime/vfile_result_writer.h
@@ -19,7 +19,7 @@
#include "io/file_writer.h"
#include "runtime/file_result_writer.h"
-#include "vec/sink/result_sink.h"
+#include "vec/sink/vresult_sink.h"
namespace doris {
diff --git a/be/src/vec/sink/mysql_result_writer.cpp
b/be/src/vec/sink/vmysql_result_writer.cpp
similarity index 99%
rename from be/src/vec/sink/mysql_result_writer.cpp
rename to be/src/vec/sink/vmysql_result_writer.cpp
index b725ac26e9..8e54e62004 100644
--- a/be/src/vec/sink/mysql_result_writer.cpp
+++ b/be/src/vec/sink/vmysql_result_writer.cpp
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-#include "vec/sink/mysql_result_writer.h"
+#include "vec/sink/vmysql_result_writer.h"
#include "runtime/buffer_control_block.h"
#include "runtime/large_int_value.h"
diff --git a/be/src/vec/sink/mysql_result_writer.h
b/be/src/vec/sink/vmysql_result_writer.h
similarity index 97%
rename from be/src/vec/sink/mysql_result_writer.h
rename to be/src/vec/sink/vmysql_result_writer.h
index f2697a6a14..49f03ee356 100644
--- a/be/src/vec/sink/mysql_result_writer.h
+++ b/be/src/vec/sink/vmysql_result_writer.h
@@ -20,7 +20,7 @@
#include "util/mysql_row_buffer.h"
#include "util/runtime_profile.h"
#include "vec/core/block.h"
-#include "vec/sink/result_writer.h"
+#include "vec/sink/vresult_writer.h"
namespace doris {
class BufferControlBlock;
diff --git a/be/src/vec/sink/result_sink.cpp b/be/src/vec/sink/vresult_sink.cpp
similarity index 98%
rename from be/src/vec/sink/result_sink.cpp
rename to be/src/vec/sink/vresult_sink.cpp
index 02da2c7326..a428eda66c 100644
--- a/be/src/vec/sink/result_sink.cpp
+++ b/be/src/vec/sink/vresult_sink.cpp
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-#include "vec/sink/result_sink.h"
+#include "vec/sink/vresult_sink.h"
#include "runtime/buffer_control_block.h"
#include "runtime/exec_env.h"
@@ -23,7 +23,7 @@
#include "runtime/result_buffer_mgr.h"
#include "runtime/runtime_state.h"
#include "vec/exprs/vexpr.h"
-#include "vec/sink/mysql_result_writer.h"
+#include "vec/sink/vmysql_result_writer.h"
namespace doris {
namespace vectorized {
diff --git a/be/src/vec/sink/result_sink.h b/be/src/vec/sink/vresult_sink.h
similarity index 98%
rename from be/src/vec/sink/result_sink.h
rename to be/src/vec/sink/vresult_sink.h
index f2dcd2a174..0ccc8ca593 100644
--- a/be/src/vec/sink/result_sink.h
+++ b/be/src/vec/sink/vresult_sink.h
@@ -17,7 +17,7 @@
[doris] branch master updated: [feature](script) add --grace option for stop_be.sh (#10626)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new bff561c0da [feature](script) add --grace option for stop_be.sh (#10626)
bff561c0da is described below
commit bff561c0da0cc94a8018f6822a41b16236e4ac42
Author: Lei Zhang <1091517...@qq.com>
AuthorDate: Wed Jul 6 17:53:01 2022 +0800
[feature](script) add --grace option for stop_be.sh (#10626)
be asan mem leak check needs exit app gracefully.
---
bin/stop_be.sh | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/bin/stop_be.sh b/bin/stop_be.sh
index 2ca1b47124..9d65d73307 100755
--- a/bin/stop_be.sh
+++ b/bin/stop_be.sh
@@ -22,6 +22,11 @@ curdir=`cd "$curdir"; pwd`
export DORIS_HOME=`cd "$curdir/.."; pwd`
export PID_DIR=`cd "$curdir"; pwd`
+signum=9
+if [ $1 = "--grace" ]; then
+signum=15
+fi
+
while read line; do
envline=`echo $line | sed 's/[[:blank:]]*=[[:blank:]]*/=/g' | sed
's/^[[:blank:]]*//g' | egrep "^[[:upper:]]([[:upper:]]|_|[[:digit:]])*="`
envline=`eval "echo $envline"`
@@ -41,7 +46,7 @@ if [ -f $pidfile ]; then
fi
if kill -0 $pid; then
-if kill -9 $pid > /dev/null 2>&1; then
+if kill -${signum} $pid > /dev/null 2>&1; then
echo "stop $pidcomm, and remove pid file. "
rm $pidfile
exit 0
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhance](*): git ignore `package-lock.json`. (#10637)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new d9ba946118 [enhance](*): git ignore `package-lock.json`. (#10637) d9ba946118 is described below commit d9ba9461184fb53285051eed7de90dab66111baf Author: jakevin <30525741+jackwe...@users.noreply.github.com> AuthorDate: Wed Jul 6 17:53:22 2022 +0800 [enhance](*): git ignore `package-lock.json`. (#10637) --- .gitignore | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index c7d9e11ce9..b8c8b7b9f5 100644 --- a/.gitignore +++ b/.gitignore @@ -20,6 +20,7 @@ derby.log dependency-reduced-pom.xml yarn.lock ut_dir +package-lock.json # ignore all sub `/bin` and `/build` directory **/build/ @@ -78,7 +79,7 @@ be/tags be/test/olap/test_data/tablet_meta_test.hdr be/.devcontainer/ - +## tools tools/ssb-tools/ssb-data/ tools/ssb-tools/ssb-dbgen/ tools/tpch-tools/TPC-H_Tools_v*.zip - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Improvement] remove redundant code of VOlapScanner (#10621)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b4c5dfc28e [Improvement] remove redundant code of VOlapScanner (#10621)
b4c5dfc28e is described below
commit b4c5dfc28ec8824b53e06d68a6cc39260a6687dc
Author: TengJianPing <18241664+jackte...@users.noreply.github.com>
AuthorDate: Wed Jul 6 17:54:10 2022 +0800
[Improvement] remove redundant code of VOlapScanner (#10621)
---
be/src/vec/exec/volap_scanner.cpp | 10 --
be/src/vec/exec/volap_scanner.h | 6 --
2 files changed, 16 deletions(-)
diff --git a/be/src/vec/exec/volap_scanner.cpp
b/be/src/vec/exec/volap_scanner.cpp
index bd2728f988..2b3093ce8b 100644
--- a/be/src/vec/exec/volap_scanner.cpp
+++ b/be/src/vec/exec/volap_scanner.cpp
@@ -204,15 +204,6 @@ Status VOlapScanner::_init_tablet_reader_params(
}
}
-// use _tablet_reader_params.return_columns, because reader use this to
merge sort
-Status res =
-_read_row_cursor.init(_tablet->tablet_schema(),
_tablet_reader_params.return_columns);
-if (!res.ok()) {
-LOG(WARNING) << "fail to init row cursor.res = " << res;
-return Status::InternalError("failed to initialize storage read row
cursor");
-}
-_read_row_cursor.allocate_memory_for_string_type(_tablet->tablet_schema());
-
// If a agg node is this scan node direct parent
// we will not call agg object finalize method in scan node,
// to avoid the unnecessary SerDe and improve query performance
@@ -240,7 +231,6 @@ Status VOlapScanner::_init_return_columns(bool
need_seq_col) {
_return_columns.push_back(index);
if (slot->is_nullable() &&
!_tablet->tablet_schema().column(index).is_nullable())
_tablet_columns_convert_to_null_set.emplace(index);
-_query_slots.push_back(slot);
}
// expand the sequence column
diff --git a/be/src/vec/exec/volap_scanner.h b/be/src/vec/exec/volap_scanner.h
index 5786a388ad..d4c40c13c1 100644
--- a/be/src/vec/exec/volap_scanner.h
+++ b/be/src/vec/exec/volap_scanner.h
@@ -89,8 +89,6 @@ public:
std::vector* mutable_runtime_filter_marks() { return
&_runtime_filter_marks; }
-const std::vector& get_query_slots() const { return
_query_slots; }
-
const std::shared_ptr& mem_tracker() const { return
_mem_tracker; }
private:
@@ -129,10 +127,6 @@ private:
std::vector _return_columns;
std::unordered_set _tablet_columns_convert_to_null_set;
-RowCursor _read_row_cursor;
-
-std::vector _query_slots;
-
// time costed and row returned statistics
int64_t _num_rows_read = 0;
int64_t _raw_rows_read = 0;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: fix the inaccurate comments (#10617)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new cff9ffa0e1 fix the inaccurate comments (#10617)
cff9ffa0e1 is described below
commit cff9ffa0e127f1eb44b02924a39731c15336cf79
Author: carlvinhust2012
AuthorDate: Wed Jul 6 17:54:43 2022 +0800
fix the inaccurate comments (#10617)
Co-authored-by: hucheng01
---
be/src/exec/broker_scan_node.cpp | 2 +-
be/src/exec/es_http_scan_node.cpp | 2 +-
be/src/vec/exec/file_scan_node.cpp | 2 +-
be/src/vec/exec/vbroker_scan_node.cpp | 2 +-
be/src/vec/exec/ves_http_scan_node.cpp | 2 +-
5 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/be/src/exec/broker_scan_node.cpp b/be/src/exec/broker_scan_node.cpp
index ed9e646397..532a9d7d06 100644
--- a/be/src/exec/broker_scan_node.cpp
+++ b/be/src/exec/broker_scan_node.cpp
@@ -360,7 +360,7 @@ Status BrokerScanNode::scanner_scan(const TBrokerScanRange&
scan_range,
// Queue size Must be smaller than _max_buffered_batches
_batch_queue.push_back(row_batch);
-// Notify reader to
+// Notify reader to process
_queue_reader_cond.notify_one();
}
}
diff --git a/be/src/exec/es_http_scan_node.cpp
b/be/src/exec/es_http_scan_node.cpp
index 81303b002b..62d219d13d 100644
--- a/be/src/exec/es_http_scan_node.cpp
+++ b/be/src/exec/es_http_scan_node.cpp
@@ -390,7 +390,7 @@ Status
EsHttpScanNode::scanner_scan(std::unique_ptr scanner,
// Queue size Must be smaller than _max_buffered_batches
_batch_queue.push_back(row_batch);
-// Notify reader to
+// Notify reader to process
_queue_reader_cond.notify_one();
}
}
diff --git a/be/src/vec/exec/file_scan_node.cpp
b/be/src/vec/exec/file_scan_node.cpp
index 10ecfde835..4105f8dc01 100644
--- a/be/src/vec/exec/file_scan_node.cpp
+++ b/be/src/vec/exec/file_scan_node.cpp
@@ -258,7 +258,7 @@ Status FileScanNode::scanner_scan(const TFileScanRange&
scan_range, ScannerCount
// Queue size Must be smaller than _max_buffered_batches
_block_queue.push_back(block);
-// Notify reader to
+// Notify reader to process
_queue_reader_cond.notify_one();
}
return Status::OK();
diff --git a/be/src/vec/exec/vbroker_scan_node.cpp
b/be/src/vec/exec/vbroker_scan_node.cpp
index 212c82d157..6f63324e6d 100644
--- a/be/src/vec/exec/vbroker_scan_node.cpp
+++ b/be/src/vec/exec/vbroker_scan_node.cpp
@@ -257,7 +257,7 @@ Status VBrokerScanNode::scanner_scan(const
TBrokerScanRange& scan_range, Scanner
// Queue size Must be smaller than _max_buffered_batches
_block_queue.push_back(block);
-// Notify reader to
+// Notify reader to process
_queue_reader_cond.notify_one();
}
return Status::OK();
diff --git a/be/src/vec/exec/ves_http_scan_node.cpp
b/be/src/vec/exec/ves_http_scan_node.cpp
index c1c2e79f37..b81511408a 100644
--- a/be/src/vec/exec/ves_http_scan_node.cpp
+++ b/be/src/vec/exec/ves_http_scan_node.cpp
@@ -320,7 +320,7 @@ Status
VEsHttpScanNode::scanner_scan(std::unique_ptr scanner) {
}
_block_queue.push_back(block);
-// Notify reader to
+// Notify reader to process
_queue_reader_cond.notify_one();
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: set strleft to always_nullable (#10496)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 6d092a6d53 set strleft to always_nullable (#10496) 6d092a6d53 is described below commit 6d092a6d534779ee01ad39bcb6cc39fbb9c3b32c Author: Pxl <952130...@qq.com> AuthorDate: Wed Jul 6 17:56:01 2022 +0800 set strleft to always_nullable (#10496) --- gensrc/script/doris_builtins_functions.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/gensrc/script/doris_builtins_functions.py b/gensrc/script/doris_builtins_functions.py index 2b916ea908..f95bc7c55b 100755 --- a/gensrc/script/doris_builtins_functions.py +++ b/gensrc/script/doris_builtins_functions.py @@ -1035,10 +1035,10 @@ visible_functions = [ '9doris_udf15FunctionContextERKNS1_9StringValERKNS1_6IntValES9_', '', '', 'vec', 'ALWAYS_NULLABLE'], [['strleft', 'left'], 'STRING', ['STRING', 'INT'], '_ZN5doris15StringFunctions4leftEPN9doris_udf' -'15FunctionContextERKNS1_9StringValERKNS1_6IntValE', '', '', 'vec', ''], +'15FunctionContextERKNS1_9StringValERKNS1_6IntValE', '', '', 'vec', 'ALWAYS_NULLABLE'], [['strright', 'right'], 'STRING', ['STRING', 'INT'], '_ZN5doris15StringFunctions5rightEPN9doris_udf' -'15FunctionContextERKNS1_9StringValERKNS1_6IntValE', '', '', 'vec', ''], +'15FunctionContextERKNS1_9StringValERKNS1_6IntValE', '', '', 'vec', 'ALWAYS_NULLABLE'], [['ends_with'], 'BOOLEAN', ['STRING', 'STRING'], '_ZN5doris15StringFunctions9ends_withEPN9doris_udf15FunctionContextERKNS1_9StringValES6_', '', '', 'vec', ''], - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (6d092a6d53 -> 416fb73621)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 6d092a6d53 set strleft to always_nullable (#10496) add 416fb73621 docs format fix for explode-json-array table function (#10613) No new revisions were added by this update. Summary of changes: .../docs/sql-manual/sql-functions/table-functions/explode-json-array.md | 2 +- .../docs/sql-manual/sql-functions/table-functions/explode-json-array.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #10627: [Bug] tpcds_sf1 q79 result is not right
yiguolei closed issue #10627: [Bug] tpcds_sf1 q79 result is not right URL: https://github.com/apache/doris/issues/10627 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on issue #10627: [Bug] tpcds_sf1 q79 result is not right
yiguolei commented on issue #10627: URL: https://github.com/apache/doris/issues/10627#issuecomment-1176027726 The reason is the order by clause is missed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10644: [refactor] if pending bytes exceeded, vtableSink wait until pending bytes consumed or task was cancelled
yiguolei merged PR #10644: URL: https://github.com/apache/doris/pull/10644 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10636: [Enhancement ] rename some files inside vec/sink dir
yiguolei merged PR #10636: URL: https://github.com/apache/doris/pull/10636 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10626: [feature](script) add --grace option for stop_be.sh
yiguolei merged PR #10626: URL: https://github.com/apache/doris/pull/10626 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10637: [enhance](*): git ignore `package-lock.json`.
yiguolei merged PR #10637: URL: https://github.com/apache/doris/pull/10637 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10617: [fix] fix the inaccurate comments
yiguolei merged PR #10617: URL: https://github.com/apache/doris/pull/10617 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10621: [Improvement] remove redundant code of VOlapScanner
yiguolei merged PR #10621: URL: https://github.com/apache/doris/pull/10621 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10496: [Bug][Vectorized] set strleft to always_nullable
yiguolei merged PR #10496: URL: https://github.com/apache/doris/pull/10496 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10613: [docs] fix explode-json-array docs format
yiguolei merged PR #10613: URL: https://github.com/apache/doris/pull/10613 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] yuruguo commented on issue #42: Flink Doris Connector Release Note 1.1.0
yuruguo commented on issue #42: URL: https://github.com/apache/doris-flink-connector/issues/42#issuecomment-1176043666 @hf200012 @JNSimba Looking forward to the release of `1.1.0`. By the way, do we have any plans to implement flink associating doris dimension tables? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] hf200012 commented on issue #42: Flink Doris Connector Release Note 1.1.0
hf200012 commented on issue #42: URL: https://github.com/apache/doris-flink-connector/issues/42#issuecomment-1176045843 > Looking forward to the release of `1.1.0`. > By the way, do we have any plans to implement flink associating doris dimension tables? @yuruguo Not yet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #10638: [Bug] using Array caused coredump in ASAN build
yiguolei closed issue #10638: [Bug] using Array caused coredump in ASAN build URL: https://github.com/apache/doris/issues/10638 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [BugFix](Array) fix DataTypeArray to_string use after free (#10640)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 29d4809c80 [BugFix](Array) fix DataTypeArray to_string use after free
(#10640)
29d4809c80 is described below
commit 29d4809c80550ab7c864b415da1d95fb28b4793a
Author: lihangyu <15605149...@163.com>
AuthorDate: Wed Jul 6 18:18:00 2022 +0800
[BugFix](Array) fix DataTypeArray to_string use after free (#10640)
ColumnArray::convert_to_full_column_if_const override the base function
and ColumnArray::create generate a temporary variable
---
be/src/vec/data_types/data_type_array.cpp | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/be/src/vec/data_types/data_type_array.cpp
b/be/src/vec/data_types/data_type_array.cpp
index 40aa07011b..4c459ed4b8 100644
--- a/be/src/vec/data_types/data_type_array.cpp
+++ b/be/src/vec/data_types/data_type_array.cpp
@@ -96,8 +96,8 @@ void DataTypeArray::to_pb_column_meta(PColumnMeta* col_meta)
const {
}
void DataTypeArray::to_string(const IColumn& column, size_t row_num,
BufferWritable& ostr) const {
-auto& data_column =
-assert_cast(*column.convert_to_full_column_if_const().get());
+auto ptr = column.convert_to_full_column_if_const();
+auto& data_column = assert_cast(*ptr.get());
auto& offsets = data_column.get_offsets();
size_t offset = offsets[row_num - 1];
@@ -115,8 +115,8 @@ void DataTypeArray::to_string(const IColumn& column, size_t
row_num, BufferWrita
}
std::string DataTypeArray::to_string(const IColumn& column, size_t row_num)
const {
-auto& data_column =
-assert_cast(*column.convert_to_full_column_if_const().get());
+auto ptr = column.convert_to_full_column_if_const();
+auto& data_column = assert_cast(*ptr.get());
auto& offsets = data_column.get_offsets();
size_t offset = offsets[row_num - 1];
@@ -183,4 +183,4 @@ Status DataTypeArray::from_string(ReadBuffer& rb, IColumn*
column) const {
return Status::OK();
}
-} // namespace doris::vectorized
\ No newline at end of file
+} // namespace doris::vectorized
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #10650: [Bug][Function] pass intermediate argument list to be
BiteThet opened a new pull request, #10650: URL: https://github.com/apache/doris/pull/10650 # Proposed changes Issue Number: close #10641 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10640: [BugFix](Array) fix DataTypeArray to_string use after free
yiguolei merged PR #10640: URL: https://github.com/apache/doris/pull/10640 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (29d4809c80 -> 5dfb59844f)
This is an automated email from the ASF dual-hosted git repository.
lingmiao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 29d4809c80 [BugFix](Array) fix DataTypeArray to_string use after free
(#10640)
add 5dfb59844f [enhancement](Nereids)refactor PlannerContext and
JobContext (#10485)
No new revisions were added by this update.
Summary of changes:
.../org/apache/doris/nereids/NereidsPlanner.java | 8 ++-
.../org/apache/doris/nereids/OptimizerContext.java | 76
.../org/apache/doris/nereids/PlannerContext.java | 84 +-
.../apache/doris/nereids/jobs/BatchRulesJob.java | 16 ++---
.../java/org/apache/doris/nereids/jobs/Job.java| 9 ++-
.../MatchingContext.java => jobs/JobContext.java} | 39 +-
.../doris/nereids/jobs/cascades/ApplyRuleJob.java | 10 +--
.../nereids/jobs/cascades/CostAndEnforcerJob.java | 4 +-
.../nereids/jobs/cascades/DeriveStatsJob.java | 6 +-
.../jobs/cascades/ExploreGroupExpressionJob.java | 6 +-
.../nereids/jobs/cascades/ExploreGroupJob.java | 6 +-
.../jobs/cascades/OptimizeGroupExpressionJob.java | 4 +-
.../nereids/jobs/cascades/OptimizeGroupJob.java| 10 +--
.../nereids/jobs/rewrite/RewriteBottomUpJob.java | 13 ++--
.../nereids/jobs/rewrite/RewriteTopDownJob.java| 12 ++--
.../nereids/jobs/scheduler/SimpleJobScheduler.java | 2 +-
.../org/apache/doris/nereids/AnalyzeSSBTest.java | 17 +++--
.../doris/nereids/jobs/RewriteTopDownJobTest.java | 14 ++--
.../LogicalProjectToPhysicalProjectTest.java | 7 +-
.../rewrite/logical/PushDownPredicateTest.java | 38 +-
20 files changed, 169 insertions(+), 212 deletions(-)
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/OptimizerContext.java
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/{pattern/MatchingContext.java
=> jobs/JobContext.java} (53%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] EmmyMiao87 merged pull request #10485: [enhancement](Nereids)refactor PlannerContext and JobContext
EmmyMiao87 merged PR #10485: URL: https://github.com/apache/doris/pull/10485 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] myfjdthink commented on issue #10452: [Feature] Doris support read iceberg table on google cloud storage
myfjdthink commented on issue #10452: URL: https://github.com/apache/doris/issues/10452#issuecomment-1176093319 this is the error logs ``` 2022-07-06 10:58:25,295 WARN (doris-mysql-nio-pool-3736|10326) [StmtExecutor.handleDdlStmt():1543] DDL statement(/* ApplicationName=DBeaver 22.1.1 - SQLEditor */ CREATE TABLE `gs_table` ENGINE = ICEBERG PROPERTIES ( "iceberg.database" = "gsdb", "iceberg.table" = "gs_table", "iceberg.hive.metastore.uris" = "thrift://10.201.0.104:9083", "iceberg.catalog.type" = "HIVE_CATALOG" )) process failed. org.apache.iceberg.exceptions.RuntimeIOException: Failed to get file system for path: gs://iceberg-spark-warehouse/datasets/gsdb.db/gs_table/metadata/1-8ee13c7f-f1ad-4ca1-b88f-1d06a0f4899b.metadata.json at org.apache.iceberg.hadoop.Util.getFs(Util.java:53) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.hadoop.HadoopInputFile.fromLocation(HadoopInputFile.java:54) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.hadoop.HadoopFileIO.newInputFile(HadoopFileIO.java:59) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:252) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:179) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:405) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:178) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:160) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:200) ~[iceberg-hive-metastore-0.12.0.jar:?] at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:94) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:77) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:93) ~[iceberg-core-0.12.0.jar:?] at org.apache.iceberg.catalog.Catalog.tableExists(Catalog.java:270) ~[iceberg-api-0.12.0.jar:?] at org.apache.doris.external.iceberg.HiveCatalog.tableExists(HiveCatalog.java:57) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.external.iceberg.IcebergCatalogMgr.getTableFromIceberg(IcebergCatalogMgr.java:151) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.external.iceberg.IcebergCatalogMgr.createIcebergTable(IcebergCatalogMgr.java:201) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.catalog.Catalog.createTable(Catalog.java:3059) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:130) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:1533) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:438) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:308) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:212) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:349) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:538) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:50) ~[palo-fe.jar:0.15-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] Caused by: java.io.IOException: No FileSystem for scheme: gs at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2798) ~[hadoop-common-2.8.0.jar:?] at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2809) ~[hadoop-common-2.8.0.jar:?] at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) ~[hadoop-common-2.8.0.jar:?] at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848) ~[hadoop-common-2.8.0.jar:?] at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830) ~[hado
[GitHub] [doris] github-actions[bot] commented on pull request #10630: [enhancement](nereids) add betweentocompound rewrite rule for ssb
github-actions[bot] commented on PR #10630: URL: https://github.com/apache/doris/pull/10630#issuecomment-1176093949 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #10650: [Bug][Function] pass intermediate argument list to be
BiteThet commented on PR #10650: URL: https://github.com/apache/doris/pull/10650#issuecomment-1176096405 > We should get same argument at two diffrent aggregate phase, then we can create same specific function. Even if we support variadic aggregate functions, we still need the correct parameter list. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 opened a new pull request, #10651: fix colocate join bug
starocean999 opened a new pull request, #10651: URL: https://github.com/apache/doris/pull/10651 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #8861: [feature-wip](statistics) step4: collect statistics by implementing statistics tasks
weizhengte commented on code in PR #8861:
URL: https://github.com/apache/doris/pull/8861#discussion_r914735322
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowColumnStatsStmt.java:
##
@@ -67,4 +70,9 @@ public ShowResultSetMetaData getMetaData() {
}
return builder.build();
}
+
+public List getPartitionNames() {
Review Comment:
In this PR, the relevant methods have been implemented, but the
corresponding syntax has not been added(to be added together in the future, and
users can not use it now). this TODO is added to avoid compilation failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #8861: [feature-wip](statistics) step4: collect statistics by implementing statistics tasks
weizhengte commented on code in PR #8861:
URL: https://github.com/apache/doris/pull/8861#discussion_r914739582
##
fe/fe-core/src/main/java/org/apache/doris/statistics/PartitionStats.java:
##
@@ -0,0 +1,129 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics;
+
+import org.apache.doris.catalog.Type;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.util.Util;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import java.util.List;
+import java.util.Map;
+import java.util.function.Predicate;
+
+
+/**
+ * There are the statistics of partition.
+ * The partition stats are mainly used to provide input for the Optimizer's
cost model.
+ * The description of partition stats are following:
+ * - @rowCount: The row count of partition.
+ * - @dataSize: The data size of partition.
+ * - @nameToColumnStats: <@String columnName, @ColumnStats columnStats>
+ *
+ * Each column in the Table will have corresponding @ColumnStats.
+ * Those @ColumnStats are recorded in @nameToColumnStats form of MAP.
+ * This facilitates the optimizer to quickly find the corresponding:
+ * - @ColumnStats: based on the column name.
+ * - @rowCount: The row count of partition.
+ * - @dataSize: The data size of partition.
+ *
+ * The granularity of the statistics is whole partition.
+ * For example: "@rowCount = 1000" means that the row count is 1000 in the
whole partition.
+ */
+public class PartitionStats {
+
+public static final StatsType DATA_SIZE = StatsType.DATA_SIZE;
+public static final StatsType ROW_COUNT = StatsType.ROW_COUNT;
+
+private static final Predicate DESIRED_ROW_COUNT_PRED = (v) -> v >=
-1L;
+private static final Predicate DESIRED_DATA_SIZE_PRED = (v) -> v >=
-1L;
+
+private long rowCount = -1;
+private long dataSize = -1;
+private final Map nameToColumnStats =
Maps.newConcurrentMap();
+
+/**
+ * Update the partition stats.
+ *
+ * @param statsTypeToValue the map of stats type to value
+ * @throws AnalysisException if the stats value is not valid
+ */
+public void updatePartitionStats(Map statsTypeToValue)
throws AnalysisException {
+for (Map.Entry entry : statsTypeToValue.entrySet())
{
Review Comment:
Here I mainly want to throw exceptions to the upper layer. Is it necessary
to use forEach to handle exceptions here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10467: [Bug][Vectorized] Support array function in where pre in volap_scan_node
github-actions[bot] commented on PR #10467: URL: https://github.com/apache/doris/pull/10467#issuecomment-1176140078 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10467: [Bug][Vectorized] Support array function in where pre in volap_scan_node
github-actions[bot] commented on PR #10467: URL: https://github.com/apache/doris/pull/10467#issuecomment-1176140029 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #8861: [feature-wip](statistics) step4: collect statistics by implementing statistics tasks
weizhengte commented on code in PR #8861:
URL: https://github.com/apache/doris/pull/8861#discussion_r914760756
##
fe/fe-core/src/main/java/org/apache/doris/statistics/TableStats.java:
##
@@ -59,51 +56,190 @@ public class TableStats {
private long rowCount = -1;
private long dataSize = -1;
-private Map nameToColumnStats =
Maps.newConcurrentMap();
+private final Map nameToPartitionStats =
Maps.newConcurrentMap();
+private final Map nameToColumnStats =
Maps.newConcurrentMap();
+
+public long getRowCount() {
+if (rowCount == -1) {
+return nameToPartitionStats.values().stream()
+.filter(partitionStats -> partitionStats.getRowCount() !=
-1)
+.mapToLong(PartitionStats::getRowCount).sum();
+}
+return rowCount;
+}
+
+public void setRowCount(long rowCount) {
+this.rowCount = rowCount;
+}
+
+public long getDataSize() {
+if (dataSize == -1) {
+return nameToPartitionStats.values().stream()
+.filter(partitionStats -> partitionStats.getDataSize() !=
-1)
+.mapToLong(PartitionStats::getDataSize).sum();
+}
+return dataSize;
+}
+
+public Map getNameToPartitionStats() {
+return nameToPartitionStats;
+}
+
+public Map getNameToColumnStats() {
+if (nameToColumnStats.isEmpty()) {
Review Comment:
Yes, in fact, I think there is no need to update the ColumnStats of the
table here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10651: [FIX] fix colocate join bug
github-actions[bot] commented on PR #10651: URL: https://github.com/apache/doris/pull/10651#issuecomment-1176194207 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #10645: [FIX] bottom line solution for vec outer join
morningman commented on code in PR #10645:
URL: https://github.com/apache/doris/pull/10645#discussion_r914895847
##
gensrc/thrift/PlanNodes.thrift:
##
@@ -465,6 +465,8 @@ struct TAggregationNode {
5: required bool need_finalize
6: optional bool use_streaming_preaggregation
7: optional bool is_update_stage
+
+ 8: required list aggregate_function_changed_flags
Review Comment:
use optional, and add some comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [BUG] Fix invalid return type for `left` and `right` function (#10643)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 3bf8c761a4 [BUG] Fix invalid return type for `left` and `right` function (#10643) 3bf8c761a4 is described below commit 3bf8c761a41b651461c0e4656e8b674cd6e96ef4 Author: Gabriel AuthorDate: Wed Jul 6 22:49:19 2022 +0800 [BUG] Fix invalid return type for `left` and `right` function (#10643) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [docs] Fixed description about networks in Quick Start (#10639)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 8de8a9571a [docs] Fixed description about networks in Quick Start (#10639) 8de8a9571a is described below commit 8de8a9571ae2c2b4b9b070c3f849295ea237b438 Author: FreeOnePlus <54164178+freeonep...@users.noreply.github.com> AuthorDate: Wed Jul 6 22:49:43 2022 +0800 [docs] Fixed description about networks in Quick Start (#10639) --- docs/en/docs/get-starting/get-starting.md| 17 - docs/zh-CN/docs/get-starting/get-starting.md | 17 - 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/docs/en/docs/get-starting/get-starting.md b/docs/en/docs/get-starting/get-starting.md index 17ab9df4a6..1ee38f7541 100644 --- a/docs/en/docs/get-starting/get-starting.md +++ b/docs/en/docs/get-starting/get-starting.md @@ -40,6 +40,9 @@ under the License. **Before creating, please prepare the compiled FE/BE file, this tutorial will not repeat the compilation process.** + +**This tutorial is a mixed distribution tutorial of single node, 1FE and 1BE, only one node is needed, which is convenient for quickly experiencing Doris.** + 1. Set the maximum number of open file handles in the system ```shell @@ -82,7 +85,9 @@ under the License. # Configure FE-Config vi fe/conf/fe.conf # Uncomment priority_networks and modify parameters - priority_networks = 127.0.0.0/24 + # For example, if the IP address of the current node is 10.10.2.21, you need to change it to 10.10.2.0/24 and fill in + # What needs to be filled in here is the IP subnet address, not the IP address + priority_networks = 10.10.2.0/24 # save and exit ``` @@ -92,7 +97,9 @@ under the License. # Configure BE-Config vi be/conf/be.conf # Uncomment priority_networks and modify parameters - priority_networks = 127.0.0.0/24 + # For example, if the IP address of the current node is 10.10.2.21, you need to change it to 10.10.2.0/24 and fill in + # What needs to be filled in here is the IP subnet address, not the IP address + priority_networks = 10.10.2.0/24 # save and exit ``` @@ -137,10 +144,10 @@ under the License. Register BE to FE (using MySQL-Client, you need to install it yourself) ```shell - # login - mysql -h 127.0.0.1 -P 9030 -uroot + # login，Since it is a single node mixed distribution, FE_IP and BE_IP are the same IP address + mysql -h FE_IP -P 9030 -uroot # Register BE - ALTER SYSTEM ADD BACKEND "127.0.0.1:9050"; + ALTER SYSTEM ADD BACKEND "BE_IP:9050"; ``` ## Apache Doris is easy to use diff --git a/docs/zh-CN/docs/get-starting/get-starting.md b/docs/zh-CN/docs/get-starting/get-starting.md index e7ff203d6a..0f799b054e 100644 --- a/docs/zh-CN/docs/get-starting/get-starting.md +++ b/docs/zh-CN/docs/get-starting/get-starting.md @@ -40,6 +40,9 @@ under the License. **在创建之前，请准备好已完成编译的FE/BE文件，此教程不再赘述编译过程。** + +**该教程为单节点、1FE 和 1BE 的混布教程，只需要一个节点，便于快速体验 Doris。** + 1. 设置系统最大打开文件句柄数 ```shell @@ -82,7 +85,9 @@ under the License. # 配置FE-Config vi fe/conf/fe.conf # 取消priority_networks的注释，修改参数 - priority_networks = 127.0.0.0/24 + # 例如当前节点的 IP 地址为 10.10.2.21，则需要更改为 10.10.2.0/24 填入 + # 这里需要填入的是 IP 子网网段地址，并非 IP 地址 + priority_networks = 10.10.2.0/24 # 保存退出 ``` @@ -92,7 +97,9 @@ under the License. # 配置FE-Config vi be/conf/be.conf # 取消priority_networks的注释，修改参数 - priority_networks = 127.0.0.0/24 + # 例如当前节点的 IP 地址为 10.10.2.21，则需要更改为 10.10.2.0/24 填入 + # 这里需要填入的是 IP 子网网段地址，并非 IP 地址 + priority_networks = 10.10.2.0/24 # 保存退出 ``` @@ -137,10 +144,10 @@ under the License. 注册 BE 至 FE（使用MySQL-Client，需自行安装） ```shell - # 登录 - mysql -h 127.0.0.1 -P 9030 -uroot + # 登录，由于是单节点混布，所以 FE_IP 和 BE_IP 为同一个 IP 地址 + mysql -h FE_IP -P 9030 -uroot # 注册BE - ALTER SYSTEM ADD BACKEND "127.0.0.1:9050"; + ALTER SYSTEM ADD BACKEND "BE_IP:9050"; ``` ## Apache Doris简单使用 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix] substr('', 1, 5) return empty string instead of null (#10622)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8012d63ea0 [fix] substr('', 1, 5) return empty string instead of null
(#10622)
8012d63ea0 is described below
commit 8012d63ea027751847826dbcb9114f4153d3a6c7
Author: Dongyang Li
AuthorDate: Wed Jul 6 22:51:02 2022 +0800
[fix] substr('', 1, 5) return empty string instead of null (#10622)
---
be/src/vec/functions/function_string.h | 4 ++--
be/test/vec/function/function_string_test.cpp | 18 +++---
.../string_functions/test_string_function.out | 2 +-
3 files changed, 18 insertions(+), 6 deletions(-)
diff --git a/be/src/vec/functions/function_string.h
b/be/src/vec/functions/function_string.h
index 4d4fdc566f..c545223a23 100644
--- a/be/src/vec/functions/function_string.h
+++ b/be/src/vec/functions/function_string.h
@@ -165,9 +165,9 @@ private:
for (int i = 0; i < size; ++i) {
auto* raw_str = reinterpret_cast(&chars[offsets[i - 1]]);
int str_size = offsets[i] - offsets[i - 1] - 1;
-// return null if start > src.length
+// return empty string if start > src.length
if (start[i] > str_size) {
-StringOP::push_null_string(i, res_chars, res_offsets,
null_map);
+StringOP::push_empty_string(i, res_chars, res_offsets);
continue;
}
// return "" if len < 0 or str == 0 or start == 0
diff --git a/be/test/vec/function/function_string_test.cpp
b/be/test/vec/function/function_string_test.cpp
index 1ed92aabe8..60cb797aae 100644
--- a/be/test/vec/function/function_string_test.cpp
+++ b/be/test/vec/function/function_string_test.cpp
@@ -39,7 +39,12 @@ TEST(function_string_test, function_string_substr_test) {
{{std::string("hello word"), -5, 5}, std::string(" word")},
{{std::string("hello word"), 1, 12}, std::string("hello
word")},
{{std::string("HELLO,!^%"), 4, 2}, std::string("LO")},
-{{std::string(""), 5, 4}, Null()},
+{{std::string(""), 5, 4}, std::string("")},
+{{std::string(""), -1, 4}, std::string("")},
+{{std::string("12"), 3, 4}, std::string("")},
+{{std::string(""), 0, 4}, std::string("")},
+{{std::string("123"), 0, 4}, std::string("")},
+{{std::string("123"), 1, 0}, std::string("")},
{{Null(), 5, 4}, Null()}};
check_function(func_name, input_types, data_set);
@@ -53,7 +58,11 @@ TEST(function_string_test, function_string_substr_test) {
{{std::string("hello word"), -5}, std::string(" word")},
{{std::string("hello word"), 1}, std::string("hello word")},
{{std::string("HELLO,!^%"), 4}, std::string("LO,!^%")},
-{{std::string(""), 5, 4}, Null()},
+{{std::string(""), 5}, std::string("")},
+{{std::string(""), -1}, std::string("")},
+{{std::string("12"), 3}, std::string("")},
+{{std::string(""), 0}, std::string("")},
+{{std::string("123"), 0}, std::string("")},
{{Null(), 5, 4}, Null()}};
check_function(func_name, input_types, data_set);
@@ -82,7 +91,10 @@ TEST(function_string_test, function_string_strleft_test) {
{{std::string("hel lo "), 5}, std::string("hel ")},
{{std::string("hello word"), 20}, std::string("hello
word")},
{{std::string("HELLO,!^%"), 7},
std::string("HELLO,!")},
-{{std::string(""), 2}, Null()},
+{{std::string(""), 2}, std::string("")},
+{{std::string(""), -2}, std::string("")},
+{{std::string(""), 0}, std::string("")},
+{{std::string("123"), 0}, std::string("")},
{{Null(), 3}, Null()}};
check_function(func_name, input_types, data_set);
diff --git
a/regression-test/data/query/sql_functions/string_functions/test_string_function.out
b/regression-test/data/query/sql_functions/string_functions/test_string_function.out
index d7c33c667e..70c7d33202 100644
---
a/regression-test/data/query/sql_functions/string_functions/test_string_function.out
+++
b/regression-test/data/query/sql_functions/string_functions/test_string_function.out
@@ -207,7 +207,7 @@ bc1
c1
-- !sql --
-\N
+
-- !sql --
bc
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10643: [BUG] Fix invalid return type for `left` and `right` function
yiguolei merged PR #10643: URL: https://github.com/apache/doris/pull/10643 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10639: [docs] Fixed description about networks in Quick Start
yiguolei merged PR #10639: URL: https://github.com/apache/doris/pull/10639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10622: [fix] substr('', 1, 5) return empty string instead of null
yiguolei merged PR #10622: URL: https://github.com/apache/doris/pull/10622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] k-i-d-d opened a new pull request, #10653: [Enhancement] [vectorized] Runtime Filter supports the slot equivalent of outer join
k-i-d-d opened a new pull request, #10653: URL: https://github.com/apache/doris/pull/10653 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] k-i-d-d closed pull request #10653: [Enhancement] [vectorized] Runtime Filter support equivalent slot of outer join
k-i-d-d closed pull request #10653: [Enhancement] [vectorized] Runtime Filter support equivalent slot of outer join URL: https://github.com/apache/doris/pull/10653 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] k-i-d-d opened a new pull request, #10654: [Enhancement] [vectorized] Runtime Filter support equivalent slot of outer join
k-i-d-d opened a new pull request, #10654: URL: https://github.com/apache/doris/pull/10654 # Proposed changes Issue Number: close #xxx ## Problem Summary: > v1.0.1 In TPCH-q7, the runtime filter `RF002[in_or_bloom] <- n1.n_nationkey` generated by the outer join of `equal join conjunct: s_nationkey = n1.n_nationkey` cannot be pushed down to `1:VOlapScanNode TABLE: supplier`. TPCH-q2 has the same problem. > trunk ``` | 6:VHASH JOIN | | | join op: INNER JOIN(BROADCAST)[The src data has been redistributed] | | | equal join conjunct: `s_nationkey` = `n1`.`n_nationkey` | | | runtime filters: RF002[in_or_bloom] <- `n1`.`n_nationkey` | | | cardinality=12002430 | | | output slot ids: 2 3 4 10 0 | | | hash output slot ids: 2 3 4 10 0 | | | tuple ids: 0 1 2 3 | 1:VOlapScanNode | | TABLE: supplier(supplier), PREAGGREGATION: ON | | runtime filters: RF002[in_or_bloom] -> `s_nationkey` | | partitions=1/1, tablets=32/32, tabletList=14148,14150,14152 ... | | cardinality=2, avgRowSize=253.33774, numNodes=1 | | tuple ids: 1 ``` ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen opened a new pull request, #10655: [feature-wip](unique-key-merge-on-write) add interface for segment key bounds, DSIP-018[3/2]
zhannngchen opened a new pull request, #10655: URL: https://github.com/apache/doris/pull/10655 Add interfaces for segment key bounds, key bounds will be used to speed up point lookup on the primary key index of each segment. For the detail, see DSIP-018:https://cwiki.apache.org/confluence/display/DORIS/DSIP-018%3A+Support+Merge-On-Write+implementation+for+UNIQUE+KEY+data+model KeyBounds will be updated by BetaRowsetWriter, will be used to construct a RowsetTree(based on IntervalTree, will be added through next patch) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] carlvinhust2012 opened a new pull request, #10656: [unit-test] add some case for array type in block_test
carlvinhust2012 opened a new pull request, #10656: URL: https://github.com/apache/doris/pull/10656 # Proposed changes 1. this pr is used for adding some case for array type in block_test. Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (No) 3. Has unit tests been added: (No) 4. Has document been added or modified: (No) 5. Does it need to update dependencies: (No) 6. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #10657: [feature](Nereids): enforcer job.
jackwener opened a new pull request, #10657: URL: https://github.com/apache/doris/pull/10657 # Proposed changes Issue Number: close #9640 ## Problem Summary: Describe the overview of changes. Add enforcer job for cascades. ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #10658: [fix](Nereids): fix ut.
jackwener opened a new pull request, #10658: URL: https://github.com/apache/doris/pull/10658 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. fix ut. ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10658: [fix](Nereids): fix ut.
github-actions[bot] commented on PR #10658: URL: https://github.com/apache/doris/pull/10658#issuecomment-1176443151 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new pull request, #10659: [enhancement](nereids) make aggregate works
morrySnow opened a new pull request, #10659: URL: https://github.com/apache/doris/pull/10659 # Proposed changes enhancement - refactor compute output expression on root fragment in nereids planner - refactor aggregate plan translator - refactor aggregate disassemble rule - add exchange node on the top of plan node tree - add contains interface on TreeNode fix - slotDescriptor should not reuse between TupleDescriptors - expression's nullable now works fine known issues - aggregate function must be the top expression in output expression (need project in ExecNode in BE) - first phase aggregate could not convert to stream mode. - OlapScanNode do not set data partition ## Checklist(Required) 1. Does it affect the original behavior: No 2. Has unit tests been added: Yes 3. Has document been added or modified: No Need 4. Does it need to update dependencies: No 5. Are there any changes that cannot be rolled back: No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary commented on issue #10642: [Bug]failed to send brpc batch,error=Host is down
stalary commented on issue #10642: URL: https://github.com/apache/doris/issues/10642#issuecomment-1176852053 You can upgrade, this version is too old. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10635: [Fix][multi-catalog]Fix coredump when reading the parquet file for multi-thread
github-actions[bot] commented on PR #10635: URL: https://github.com/apache/doris/pull/10635#issuecomment-1176899349 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10635: [Fix][multi-catalog]Fix coredump when reading the parquet file for multi-thread
github-actions[bot] commented on PR #10635: URL: https://github.com/apache/doris/pull/10635#issuecomment-1176899380 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
