[GitHub] [doris] yiguolei merged pull request #12040: [fix](plan)add changeSlotToNullableOfOuterJoinedTuples method
yiguolei merged PR #12040: URL: https://github.com/apache/doris/pull/12040 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch dev-1.1.2 updated: add changeSlotToNullableOfOuterJoinedTuples method (#12040)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/dev-1.1.2 by this push:
new 170356ce11 add changeSlotToNullableOfOuterJoinedTuples method (#12040)
170356ce11 is described below
commit 170356ce11be4d4ace4f038eb0ced130fe4e7f07
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Wed Aug 24 15:00:38 2022 +0800
add changeSlotToNullableOfOuterJoinedTuples method (#12040)
---
.../java/org/apache/doris/analysis/Analyzer.java | 26 ++
1 file changed, 26 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
index 3c62fa6112..488d234e7b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
@@ -2266,4 +2266,30 @@ public class Analyzer {
}
}
}
+
+/**
+ * Change all outer joined slots to nullable
+ * Returns the slots actually be changed from not nullable to nullable
+ */
+public List changeSlotToNullableOfOuterJoinedTuples() {
+List result = new ArrayList<>();
+for (TupleId id : globalState.outerJoinedTupleIds.keySet()) {
+TupleDescriptor tupleDescriptor =
globalState.descTbl.getTupleDesc(id);
+if (tupleDescriptor != null) {
+for (SlotDescriptor desc : tupleDescriptor.getSlots()) {
+if (!desc.getIsNullable()) {
+desc.setIsNullable(true);
+result.add(desc);
+}
+}
+}
+}
+return result;
+}
+
+public void changeSlotsToNotNullable(List slots) {
+for (SlotDescriptor slot : slots) {
+slot.setIsNullable(false);
+}
+}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #11975: [tools](ssb and tpch)optimize tools
hello-stephen commented on PR #11975: URL: https://github.com/apache/doris/pull/11975#issuecomment-1225284282 > Nice job! > > However, I think you should also modify `.github/workflows/shellcheck.yml`. The `ShellCheck` action ignores the files under `tools/ssb-tools` and `tools/tpch-tools` currently. sorry for replying late, I made a new request for compensation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11861: [fix](grouping)fix grouping function bug
yiguolei merged PR #11861: URL: https://github.com/apache/doris/pull/11861 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12013: [feature](Nereids)add normalize aggregate rule
Kikyou1997 commented on code in PR #12013:
URL: https://github.com/apache/doris/pull/12013#discussion_r953425094
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/NormalizeAggregate.java:
##
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionReplacer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * normalize aggregate's group keys to SlotReference and generate a
LogicalProject top on LogicalAggregate
+ * to hold to order of aggregate output, since aggregate output's order could
change when we do translate.
+ *
+ * Apply this rule could simplify the processing of enforce and translate.
+ *
+ * Original Plan:
+ * Aggregate(
+ * keys:[k1#1, K2#2 + 1],
+ * outputs:[k1#1, Alias(K2# + 1)#4, Alias(k1#1 + 1)#5, Alias(SUM(v1#3))#6,
+ *Alias(SUM(v1#3 + 1))#7, Alias(SUM(v1#3) + 1)#8])
+ *
+ * After rule:
+ * Project(k1#1, Alias(SR#9)#4, Alias(k1#1 + 1)#5, Alias(SR#10))#6,
Alias(SR#11))#7, Alias(SR#10 + 1)#8)
+ * +-- Aggregate(keys:[k1#1, SR#9], outputs:[k1#1, SR#9, Alias(SUM(v1#3))#10,
Alias(SUM(v1#3 + 1))#11])
+ * +-- Project(k1#1, Alias(K2#2 + 1)#9, v1#3)
+ *
+ * More example could get from UT {@link NormalizeAggregateTest}
+ */
+public class NormalizeAggregate extends OneRewriteRuleFactory {
+@Override
+public Rule build() {
+return logicalAggregate().when(aggregate ->
!aggregate.isNormalized()).then(aggregate -> {
+// substitution map used to substitute expression in aggregate's
output to use it as top projections
+Map substitutionMap = Maps.newHashMap();
+List keys = aggregate.getGroupByExpressions();
+List newOutputs = Lists.newArrayList();
Review Comment:
rename to `outputsWithNormalizedOrder`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (f875684345 -> 8b4f693ad5)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from f875684345 [fix](agg) Crashing caused by serialization in streaming aggregation (#12027) add 8b4f693ad5 [fix](grouping)fix grouping function bug (#11861) No new revisions were added by this update. Summary of changes: .../org/apache/doris/analysis/GroupByClause.java | 10 ++- .../org/apache/doris/analysis/GroupingInfo.java| 15 .../aggregate/aggregate_grouping_function.out | 43 ++ .../aggregate/aggregate_grouping_function.groovy | 93 ++ 4 files changed, 142 insertions(+), 19 deletions(-) create mode 100644 regression-test/data/query/aggregate/aggregate_grouping_function.out create mode 100644 regression-test/suites/query/aggregate/aggregate_grouping_function.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12013: [feature](Nereids)add normalize aggregate rule
Kikyou1997 commented on code in PR #12013:
URL: https://github.com/apache/doris/pull/12013#discussion_r953389481
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/NormalizeAggregate.java:
##
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionReplacer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * normalize aggregate's group keys to SlotReference and generate a
LogicalProject top on LogicalAggregate
+ * to hold to order of aggregate output, since aggregate output's order could
change when we do translate.
+ *
+ * Apply this rule could simplify the processing of enforce and translate.
+ *
+ * Original Plan:
+ * Aggregate(
+ * keys:[k1#1, K2#2 + 1],
+ * outputs:[k1#1, Alias(K2# + 1)#4, Alias(k1#1 + 1)#5, Alias(SUM(v1#3))#6,
+ *Alias(SUM(v1#3 + 1))#7, Alias(SUM(v1#3) + 1)#8])
+ *
+ * After rule:
+ * Project(k1#1, Alias(SR#9)#4, Alias(k1#1 + 1)#5, Alias(SR#10))#6,
Alias(SR#11))#7, Alias(SR#10 + 1)#8)
+ * +-- Aggregate(keys:[k1#1, SR#9], outputs:[k1#1, SR#9, Alias(SUM(v1#3))#10,
Alias(SUM(v1#3 + 1))#11])
+ * +-- Project(k1#1, Alias(K2#2 + 1)#9, v1#3)
+ *
+ * More example could get from UT {@link NormalizeAggregateTest}
+ */
+public class NormalizeAggregate extends OneRewriteRuleFactory {
+@Override
+public Rule build() {
+return logicalAggregate().when(aggregate ->
!aggregate.isNormalized()).then(aggregate -> {
+// substitution map used to substitute expression in aggregate's
output to use it as top projections
+Map substitutionMap = Maps.newHashMap();
Review Comment:
I think you could use Visitor to do the same things
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-spark-connector] chovy-3012 opened a new pull request, #48: [bug] fix stream dataframe writing to doris json parse exception
chovy-3012 opened a new pull request, #48: URL: https://github.com/apache/doris-spark-connector/pull/48 # Proposed changes Issue Number: close #47 ## Problem Summary: stream dataframe can write to doris ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) No 3. Has unit tests been added: (Yes/No/No Need) No Need 5. Has document been added or modified: (Yes/No/No Need) No Need 7. Does it need to update dependencies: (Yes/No) No 9. Are there any changes that cannot be rolled back: (Yes/No) No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch dev-1.1.2 updated: [fix](grouping)fix grouping function bug (#11861)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/dev-1.1.2 by this push:
new 076dd67347 [fix](grouping)fix grouping function bug (#11861)
076dd67347 is described below
commit 076dd67347c79d57a0604c6905ae5f0b09218936
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Wed Aug 24 15:05:25 2022 +0800
[fix](grouping)fix grouping function bug (#11861)
---
.../org/apache/doris/analysis/GroupByClause.java | 10 ++-
.../org/apache/doris/analysis/GroupingInfo.java| 15
.../aggregate/aggregate_grouping_function.out | 43 ++
.../aggregate/aggregate_grouping_function.groovy | 93 ++
4 files changed, 142 insertions(+), 19 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/GroupByClause.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/GroupByClause.java
index 14d1804e71..faf0fd2626 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/GroupByClause.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/GroupByClause.java
@@ -30,9 +30,7 @@ import org.apache.logging.log4j.Logger;
import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
-import java.util.Optional;
import java.util.Set;
-import java.util.stream.Collectors;
/**
* Wraps all information of group by clause. support normal GROUP BY clause
and extended GROUP BY clause like
@@ -286,8 +284,12 @@ public class GroupByClause implements ParseNode {
Analyzer analyzer) {
groupingExprs = Expr.substituteList(groupingExprs, smap, analyzer,
true);
for (VirtualSlotRef vs : groupingSlots) {
-
vs.setRealSlots(Optional.ofNullable(Expr.substituteList(vs.getRealSlots(),
smap, analyzer, true)).orElse(
-new ArrayList<>()).stream().map(e -> (SlotRef)
e).collect(Collectors.toList()));
+ArrayList exprs = Expr.substituteList(vs.getRealSlots(),
smap, analyzer, true);
+if (exprs != null) {
+vs.setRealSlots(exprs);
+} else {
+vs.setRealSlots(new ArrayList());
+}
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/GroupingInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/GroupingInfo.java
index e153c5f929..b7ba987cc6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/GroupingInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/GroupingInfo.java
@@ -17,7 +17,6 @@
package org.apache.doris.analysis;
-import org.apache.doris.catalog.Table;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
@@ -270,20 +269,6 @@ public class GroupingInfo {
public void substituteGroupingFn(Expr expr, Analyzer analyzer) throws
AnalysisException {
if (expr instanceof GroupingFunctionCallExpr) {
-// TODO(yangzhengguo) support expression in grouping functions
-for (Expr child: expr.getChildren()) {
-if (!(child instanceof SlotRef)) {
-throw new AnalysisException("grouping functions only
support column in current version.");
-// expr from inline view
-} else if (((SlotRef)
child).getDesc().getParent().getTable().getType()
-== Table.TableType.INLINE_VIEW) {
-InlineViewRef ref = (InlineViewRef) ((SlotRef)
child).getDesc().getParent().getRef();
-int colIndex = ref.getColLabels().indexOf(((SlotRef)
child).getColumnName());
-if (colIndex != -1 &&
!(ref.getViewStmt().getResultExprs().get(colIndex) instanceof SlotRef)) {
-throw new AnalysisException("grouping functions only
support column in current version.");
-}
-}
-}
// if is substituted skip
if (expr.getChildren().size() == 1 && expr.getChild(0) instanceof
VirtualSlotRef) {
return;
diff --git
a/regression-test/data/query/aggregate/aggregate_grouping_function.out
b/regression-test/data/query/aggregate/aggregate_grouping_function.out
new file mode 100644
index 00..292966bd15
--- /dev/null
+++ b/regression-test/data/query/aggregate/aggregate_grouping_function.out
@@ -0,0 +1,43 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !select --
+1 2022-08-01 \N 6
+0 2022-08-01 aaa 2
+0 2022-08-01 bbb 2
+0 2022-08-01 ccc 2
+1 2022-08-02 \N 6
+0 2022-08-02 aaa 2
+0 2022-08-02 bbb 2
+0 2022-08-02 ccc 2
+1 2022-08-03 \N 6
+0 2022-08-03 a
[GitHub] [doris] github-actions[bot] commented on pull request #11649: [Fix] fix `cast(array as array<>)` causes be core dump
github-actions[bot] commented on PR #11649: URL: https://github.com/apache/doris/pull/11649#issuecomment-1225291167 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11649: [Fix] fix `cast(array as array<>)` causes be core dump
github-actions[bot] commented on PR #11649: URL: https://github.com/apache/doris/pull/11649#issuecomment-1225291199 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch asf-site updated: add
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 435cfa54478 add
435cfa54478 is described below
commit 435cfa5447883ad5910604cf26db035d53686f11
Author: jiafeng.zhang
AuthorDate: Wed Aug 24 15:10:42 2022 +0800
add
---
.asf.yaml | 18 ++
remove-non-reserved-dir.sh | 39 +++
2 files changed, 57 insertions(+)
diff --git a/.asf.yaml b/.asf.yaml
new file mode 100644
index 000..bdb953b5ffc
--- /dev/null
+++ b/.asf.yaml
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+publish:
+ whoami: asf-site
diff --git a/remove-non-reserved-dir.sh b/remove-non-reserved-dir.sh
new file mode 100644
index 000..32eeab8a5cf
--- /dev/null
+++ b/remove-non-reserved-dir.sh
@@ -0,0 +1,39 @@
+#!/bin/bash
+
+set -eo pipefail
+
+ROOT=`dirname "$0"`
+ROOT=`cd "$ROOT"; pwd`
+
+reserved=(
+build
+remove-non-reserved-dir.sh
+versions.json
+.asf.yaml
+.nojekyll
+.git
+.
+..
+)
+
+function contains() {
+local n=$#
+local value=${!n}
+for ((i=1;i < $#;i++)) {
+if [ "${!i}" == "${value}" ]; then
+echo "y"
+return 0
+fi
+}
+echo "n"
+return 1
+}
+
+for file in `ls -a $ROOT`
+do
+echo "${reserved[@]}"
+if [ $(contains "${reserved[@]}" "$file") == "n" ]; then
+echo "delete $ROOT/$file"
+rm -rf $ROOT/$file
+fi
+done
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11984: [Fix](remote) Fix thread safety issue in cache
github-actions[bot] commented on PR #11984: URL: https://github.com/apache/doris/pull/11984#issuecomment-1225292007 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
924060929 commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r953359770
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##
@@ -92,9 +92,10 @@ public List buildRules() {
RuleType.BINDING_JOIN_SLOT.build(
logicalJoin().thenApply(ctx -> {
LogicalJoin join = ctx.root;
-Optional cond = join.getCondition()
+Optional cond = join.getOtherJoinCondition()
Review Comment:
> but bind hash equal condition will bring more robustness
+1
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/FindHashConditionForJoin.java:
##
@@ -0,0 +1,69 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.JoinUtils;
+
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * this rule aims to find a conjunct list from on clause expression, which
could
+ * be used to build hash-table.
+ *
+ * For example:
+ * A join B on A.x=B.x and A.y>1 and A.x+1=B.x+B.y and A.z=B.z+A.x and
(A.z=B.z or A.x=B.x)
+ * {A.x=B.x, A.x+1=B.x+B.y} could be used to build hash table,
+ * but {A.y>1, A.z=B.z+A.z, (A.z=B.z or A.x=B.x)} are not.
+ *
+ * CAUTION:
+ * This rule must be applied after BindSlotReference
+ */
+public class FindHashConditionForJoin extends OneRewriteRuleFactory {
+@Override
+public Rule build() {
+return logicalJoin().then(join -> {
+Pair, List> pair =
JoinUtils.extractExpressionForHashTable(join);
+List hashJoinConjuncts = pair.first;
+hashJoinConjuncts.addAll(join.getHashJoinConjuncts());
+List otherJoinPredicates = pair.second;
+
+if (!hashJoinConjuncts.isEmpty()) {
+Optional condition = Optional.empty();
+if (!otherJoinPredicates.isEmpty()) {
+condition =
Optional.of(ExpressionUtils.and(otherJoinPredicates));
+}
+return new LogicalJoin(join.getJoinType(),
+hashJoinConjuncts,
+condition,
+Optional.empty(),
+Optional.empty(),
+join.left(), join.right());
+} else {
+return join;
+}
Review Comment:
```java
Pair, List> pair =
JoinUtils.extractExpressionForHashTable(join);
List extractedHashJoinConjuncts = pair.first;
List remaindNonHashJoinConjuncts = pair.second;
if (!extractedHashJoinConjuncts.isEmpty()) {
List combinedHashJoinConjuncts = ImmutableList.builder()
.addAll(join.getHashJoinConjuncts())
.addAll(extractedHashJoinConjuncts)
.build();
return new LogicalJoin(join.getJoinType(),
combinedHashJoinConjuncts,
remaindNonHashJoinConjuncts,
Optional.empty(),
Optional.empty(),
join.left(), join.right());
} else {
return join;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #12037: [Fix](Planner) Show rollup info in explain result.
morrySnow commented on PR #12037: URL: https://github.com/apache/doris/pull/12037#issuecomment-1225296204 it seems like we print rollup name after table name in a pair of parentheses currently. Just like ``` lineorder(lineorder) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: fix
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new f0be964f77b fix
f0be964f77b is described below
commit f0be964f77b2ecc7e80ef483c18426f2308e0c73
Author: jiafeng.zhang
AuthorDate: Wed Aug 24 15:17:41 2022 +0800
fix
---
.github/workflows/cron-deploy-website.yml | 22 ++
.github/workflows/manual-deploy.yml | 10 ++
2 files changed, 32 insertions(+)
diff --git a/.github/workflows/cron-deploy-website.yml
b/.github/workflows/cron-deploy-website.yml
index 17ee1e915a6..084fe2f32f3 100644
--- a/.github/workflows/cron-deploy-website.yml
+++ b/.github/workflows/cron-deploy-website.yml
@@ -34,6 +34,16 @@ jobs:
touch build/.dummy
ls build
export DORIS_COMMIT=`git rev-parse HEAD`
+- name: Upload files to OSS
+ uses: ./.github/actions/aliyun-oss-website-action
+ with:
+ accessKeyId: ${{ secrets.ALIYUN_ACCESS_KEY_ID }}
+ accessKeySecret: ${{ secrets.ALIYUN_ACCESS_KEY_SECRET }}
+ bucket: ${{ secrets.ALIYUN_OSS_BUCKET }}
+ # use your own endpoint
+ endpoint: ${{ secrets.ALIYUN_OSS_ENDPOINT }}
+ folder: build
+
- name: Deploy website
if: ${{ github.event.inputs.branch == 'master' }}
run: |
@@ -55,3 +65,15 @@ jobs:
git add .
git commit -m "Automated deployment with doris master"
git push --verbose "https://${{ secrets.GITHUB_TOKEN
}}@github.com/apache/doris-website.git" asf-site:asf-site
+
+- name: Deploy Branch
+ if: ${{ github.event.inputs.branch != 'master' }}
+ uses: peaceiris/actions-gh-pages@v3
+ with:
+github_token: ${{ secrets.GITHUB_TOKEN }}
+publish_branch: asf-site
+publish_dir: ./build
+destination_dir: ${{ github.event.inputs.branch }}
+user_name: 'github-actions[bot]'
+user_email: 'github-actions[bot]@users.noreply.github.com'
+commit_message: 'Automated deployment with doris branch ${{
github.event.inputs.branch }}@${{ env.DORIS_COMMIT }}'
diff --git a/.github/workflows/manual-deploy.yml
b/.github/workflows/manual-deploy.yml
index 9cd93d454bf..c1bb95e6697 100644
--- a/.github/workflows/manual-deploy.yml
+++ b/.github/workflows/manual-deploy.yml
@@ -37,6 +37,16 @@ jobs:
touch build/.dummy
ls build
export DORIS_COMMIT=`git rev-parse HEAD`
+- name: Upload files to OSS
+ uses: ./.github/actions/aliyun-oss-website-action
+ with:
+ accessKeyId: ${{ secrets.ALIYUN_ACCESS_KEY_ID }}
+ accessKeySecret: ${{ secrets.ALIYUN_ACCESS_KEY_SECRET }}
+ bucket: ${{ secrets.ALIYUN_OSS_BUCKET }}
+ # use your own endpoint
+ endpoint: ${{ secrets.ALIYUN_OSS_ENDPOINT }}
+ folder: build
+
- name: Deploy website
if: ${{ github.event.inputs.branch == 'master' }}
run: |
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #12013: [feature](Nereids)add normalize aggregate rule
morrySnow commented on code in PR #12013:
URL: https://github.com/apache/doris/pull/12013#discussion_r953438977
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/NormalizeAggregate.java:
##
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionReplacer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * normalize aggregate's group keys to SlotReference and generate a
LogicalProject top on LogicalAggregate
+ * to hold to order of aggregate output, since aggregate output's order could
change when we do translate.
+ *
+ * Apply this rule could simplify the processing of enforce and translate.
+ *
+ * Original Plan:
+ * Aggregate(
+ * keys:[k1#1, K2#2 + 1],
+ * outputs:[k1#1, Alias(K2# + 1)#4, Alias(k1#1 + 1)#5, Alias(SUM(v1#3))#6,
+ *Alias(SUM(v1#3 + 1))#7, Alias(SUM(v1#3) + 1)#8])
+ *
+ * After rule:
+ * Project(k1#1, Alias(SR#9)#4, Alias(k1#1 + 1)#5, Alias(SR#10))#6,
Alias(SR#11))#7, Alias(SR#10 + 1)#8)
+ * +-- Aggregate(keys:[k1#1, SR#9], outputs:[k1#1, SR#9, Alias(SUM(v1#3))#10,
Alias(SUM(v1#3 + 1))#11])
+ * +-- Project(k1#1, Alias(K2#2 + 1)#9, v1#3)
+ *
+ * More example could get from UT {@link NormalizeAggregateTest}
+ */
+public class NormalizeAggregate extends OneRewriteRuleFactory {
+@Override
+public Rule build() {
+return logicalAggregate().when(aggregate ->
!aggregate.isNormalized()).then(aggregate -> {
+// substitution map used to substitute expression in aggregate's
output to use it as top projections
+Map substitutionMap = Maps.newHashMap();
+List keys = aggregate.getGroupByExpressions();
+List newOutputs = Lists.newArrayList();
+
+// keys
+Map> partitionedKeys = keys.stream()
+
.collect(Collectors.groupingBy(SlotReference.class::isInstance));
+List newKeys = Lists.newArrayList();
+List bottomProjections = Lists.newArrayList();
+if (partitionedKeys.containsKey(false)) {
+// process non-SlotReference keys
+newKeys.addAll(partitionedKeys.get(false).stream()
+.map(e -> new Alias(e, e.toSql()))
+.peek(a -> substitutionMap.put(a.child(), a.toSlot()))
+.peek(bottomProjections::add)
+.map(Alias::toSlot)
+.collect(Collectors.toList()));
+}
+if (partitionedKeys.containsKey(true)) {
+// process SlotReference keys
+partitionedKeys.get(true).stream()
+.map(SlotReference.class::cast)
+.peek(s -> substitutionMap.put(s, s))
Review Comment:
merge into one need use lambda statement instead of lambda expression. It is
look urgly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
--
[GitHub] [doris] morrySnow commented on a diff in pull request #12013: [feature](Nereids)add normalize aggregate rule
morrySnow commented on code in PR #12013:
URL: https://github.com/apache/doris/pull/12013#discussion_r953438977
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/NormalizeAggregate.java:
##
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionReplacer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * normalize aggregate's group keys to SlotReference and generate a
LogicalProject top on LogicalAggregate
+ * to hold to order of aggregate output, since aggregate output's order could
change when we do translate.
+ *
+ * Apply this rule could simplify the processing of enforce and translate.
+ *
+ * Original Plan:
+ * Aggregate(
+ * keys:[k1#1, K2#2 + 1],
+ * outputs:[k1#1, Alias(K2# + 1)#4, Alias(k1#1 + 1)#5, Alias(SUM(v1#3))#6,
+ *Alias(SUM(v1#3 + 1))#7, Alias(SUM(v1#3) + 1)#8])
+ *
+ * After rule:
+ * Project(k1#1, Alias(SR#9)#4, Alias(k1#1 + 1)#5, Alias(SR#10))#6,
Alias(SR#11))#7, Alias(SR#10 + 1)#8)
+ * +-- Aggregate(keys:[k1#1, SR#9], outputs:[k1#1, SR#9, Alias(SUM(v1#3))#10,
Alias(SUM(v1#3 + 1))#11])
+ * +-- Project(k1#1, Alias(K2#2 + 1)#9, v1#3)
+ *
+ * More example could get from UT {@link NormalizeAggregateTest}
+ */
+public class NormalizeAggregate extends OneRewriteRuleFactory {
+@Override
+public Rule build() {
+return logicalAggregate().when(aggregate ->
!aggregate.isNormalized()).then(aggregate -> {
+// substitution map used to substitute expression in aggregate's
output to use it as top projections
+Map substitutionMap = Maps.newHashMap();
+List keys = aggregate.getGroupByExpressions();
+List newOutputs = Lists.newArrayList();
+
+// keys
+Map> partitionedKeys = keys.stream()
+
.collect(Collectors.groupingBy(SlotReference.class::isInstance));
+List newKeys = Lists.newArrayList();
+List bottomProjections = Lists.newArrayList();
+if (partitionedKeys.containsKey(false)) {
+// process non-SlotReference keys
+newKeys.addAll(partitionedKeys.get(false).stream()
+.map(e -> new Alias(e, e.toSql()))
+.peek(a -> substitutionMap.put(a.child(), a.toSlot()))
+.peek(bottomProjections::add)
+.map(Alias::toSlot)
+.collect(Collectors.toList()));
+}
+if (partitionedKeys.containsKey(true)) {
+// process SlotReference keys
+partitionedKeys.get(true).stream()
+.map(SlotReference.class::cast)
+.peek(s -> substitutionMap.put(s, s))
Review Comment:
merge into one need use lambda statement instead of lambda expression. It is
look ugly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---
[GitHub] [doris] morrySnow commented on a diff in pull request #12013: [feature](Nereids)add normalize aggregate rule
morrySnow commented on code in PR #12013:
URL: https://github.com/apache/doris/pull/12013#discussion_r953440653
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/NormalizeAggregate.java:
##
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionReplacer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * normalize aggregate's group keys to SlotReference and generate a
LogicalProject top on LogicalAggregate
+ * to hold to order of aggregate output, since aggregate output's order could
change when we do translate.
+ *
+ * Apply this rule could simplify the processing of enforce and translate.
+ *
+ * Original Plan:
+ * Aggregate(
+ * keys:[k1#1, K2#2 + 1],
+ * outputs:[k1#1, Alias(K2# + 1)#4, Alias(k1#1 + 1)#5, Alias(SUM(v1#3))#6,
+ *Alias(SUM(v1#3 + 1))#7, Alias(SUM(v1#3) + 1)#8])
+ *
+ * After rule:
+ * Project(k1#1, Alias(SR#9)#4, Alias(k1#1 + 1)#5, Alias(SR#10))#6,
Alias(SR#11))#7, Alias(SR#10 + 1)#8)
+ * +-- Aggregate(keys:[k1#1, SR#9], outputs:[k1#1, SR#9, Alias(SUM(v1#3))#10,
Alias(SUM(v1#3 + 1))#11])
+ * +-- Project(k1#1, Alias(K2#2 + 1)#9, v1#3)
+ *
+ * More example could get from UT {@link NormalizeAggregateTest}
+ */
+public class NormalizeAggregate extends OneRewriteRuleFactory {
+@Override
+public Rule build() {
+return logicalAggregate().when(aggregate ->
!aggregate.isNormalized()).then(aggregate -> {
+// substitution map used to substitute expression in aggregate's
output to use it as top projections
+Map substitutionMap = Maps.newHashMap();
+List keys = aggregate.getGroupByExpressions();
+List newOutputs = Lists.newArrayList();
Review Comment:
the new Output's purpose is not to normalize order of output, but to pump up
complex expression to top project. And the top project is used to maintain the
output order of aggregate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 commented on a diff in pull request #11933: [fix](union)the result exprs of union node should substitute by child node's smap
starocean999 commented on code in PR #11933:
URL: https://github.com/apache/doris/pull/11933#discussion_r953445341
##
fe/fe-core/src/main/java/org/apache/doris/planner/SetOperationNode.java:
##
@@ -144,6 +144,13 @@ public List> getMaterializedConstExprLists() {
@Override
public void finalize(Analyzer analyzer) throws UserException {
super.finalize(analyzer);
+// the resultExprLists should be substituted by child's output smap
Review Comment:
modified, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg commented on pull request #12006: [Bug](function) fix aggFnParams set not correct
yangzhg commented on PR #12006: URL: https://github.com/apache/doris/pull/12006#issuecomment-1225308573 This bug will occur when enable_vec_exec, and your modification seems not fix this bug -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 commented on pull request #12012: [bug](vectorized) fix bug of tuple is null null side do not set
starocean999 commented on PR #12012: URL: https://github.com/apache/doris/pull/12012#issuecomment-1225308407 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: fix
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 1e79aed76d2 fix 1e79aed76d2 is described below commit 1e79aed76d2e4cb9e4df702a6fed5f73ba555a33 Author: jiafeng.zhang AuthorDate: Wed Aug 24 15:34:47 2022 +0800 fix --- sidebars.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sidebars.json b/sidebars.json index 8216b16940e..c2c9381f24e 100644 --- a/sidebars.json +++ b/sidebars.json @@ -43,8 +43,8 @@ "type": "category", "label": "Index", "items": [ -"data-table/index/bloomfilter", "data-table/index/prefix-index", +"data-table/index/bloomfilter", "data-table/index/bitmap-index" ] } - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #12037: [Fix](Planner) Show rollup info in explain result.
cambyzju commented on code in PR #12037:
URL: https://github.com/apache/doris/pull/12037#discussion_r953479209
##
fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java:
##
@@ -846,6 +846,8 @@ public String getNodeExplainString(String prefix,
TExplainLevel detailLevel) {
output.append(prefix).append("runtime filters: ");
output.append(getRuntimeFilterExplainString(false));
}
+
+output.append(prefix).append("rollup:
").append(olapTable.getIndexNameById(selectedIndexId)).append("\n");
Review Comment:
getIndexNameById may return null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] jeffreys-cat opened a new pull request, #65: [fix] cp .asf.yaml and version to build folder
jeffreys-cat opened a new pull request, #65: URL: https://github.com/apache/doris-website/pull/65 - [[fix] cp .asf.yaml and version to build folder](https://github.com/apache/doris-website/commit/ef4343bcb7a38e84e13d896fda024ce89c25e4ce) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] morningman merged pull request #65: [fix] cp .asf.yaml and version to build folder
morningman merged PR #65: URL: https://github.com/apache/doris-website/pull/65 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: [fix] cp .asf.yaml and version to build folder (#65)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 6b74085b132 [fix] cp .asf.yaml and version to build folder (#65) 6b74085b132 is described below commit 6b74085b132af00d6cc276b110ad4c9b4a18d592 Author: Jeffrey AuthorDate: Wed Aug 24 16:05:32 2022 +0800 [fix] cp .asf.yaml and version to build folder (#65) --- .asf-site.yaml | 19 +++ .github/workflows/manual-deploy-website.yml | 2 ++ 2 files changed, 21 insertions(+) diff --git a/.asf-site.yaml b/.asf-site.yaml new file mode 100644 index 000..cb5ff7187f3 --- /dev/null +++ b/.asf-site.yaml @@ -0,0 +1,19 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +publish: + whoami: asf-site \ No newline at end of file diff --git a/.github/workflows/manual-deploy-website.yml b/.github/workflows/manual-deploy-website.yml index 46a33f9d0c1..acc93f443ed 100644 --- a/.github/workflows/manual-deploy-website.yml +++ b/.github/workflows/manual-deploy-website.yml @@ -39,6 +39,8 @@ jobs: yarn cache clean yarn && yarn build touch build/.dummy +cp .asf-site.yaml ./build/.asf.yaml +cp versions.json ./build/ ls build export DORIS_COMMIT=`git rev-parse HEAD` - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch asf-site updated: add
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 4318690a2a6 add
4318690a2a6 is described below
commit 4318690a2a696231a8603e5e84bc2f7475c7e2e2
Author: jiafeng.zhang
AuthorDate: Wed Aug 24 16:13:13 2022 +0800
add
---
.asf.yaml | 18 ++
remove-non-reserved-dir.sh | 39 +++
2 files changed, 57 insertions(+)
diff --git a/.asf.yaml b/.asf.yaml
new file mode 100644
index 000..bdb953b5ffc
--- /dev/null
+++ b/.asf.yaml
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+publish:
+ whoami: asf-site
diff --git a/remove-non-reserved-dir.sh b/remove-non-reserved-dir.sh
new file mode 100644
index 000..32eeab8a5cf
--- /dev/null
+++ b/remove-non-reserved-dir.sh
@@ -0,0 +1,39 @@
+#!/bin/bash
+
+set -eo pipefail
+
+ROOT=`dirname "$0"`
+ROOT=`cd "$ROOT"; pwd`
+
+reserved=(
+build
+remove-non-reserved-dir.sh
+versions.json
+.asf.yaml
+.nojekyll
+.git
+.
+..
+)
+
+function contains() {
+local n=$#
+local value=${!n}
+for ((i=1;i < $#;i++)) {
+if [ "${!i}" == "${value}" ]; then
+echo "y"
+return 0
+fi
+}
+echo "n"
+return 1
+}
+
+for file in `ls -a $ROOT`
+do
+echo "${reserved[@]}"
+if [ $(contains "${reserved[@]}" "$file") == "n" ]; then
+echo "delete $ROOT/$file"
+rm -rf $ROOT/$file
+fi
+done
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (8b4f693ad5 -> d87ab69ead)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 8b4f693ad5 [fix](grouping)fix grouping function bug (#11861) add d87ab69ead [bug](vectorized) fix bug of tuple is null null side do not set (#12012) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/analysis/Analyzer.java | 23 .../org/apache/doris/analysis/InlineViewRef.java | 7 ++- .../java/org/apache/doris/analysis/TableRef.java | 2 + .../doris/analysis/TupleIsNullPredicate.java | 62 +++--- .../org/apache/doris/planner/HashJoinNode.java | 6 +-- .../apache/doris/planner/SingleNodePlanner.java| 13 +++-- .../doris/analysis/TupleIsNullPredicateTest.java | 2 +- .../data/query/keyword/test_keyword.out| 7 +++ .../suites/query/keyword/test_keyword.groovy | 4 +- 9 files changed, 63 insertions(+), 63 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12012: [bug](vectorized) fix bug of tuple is null null side do not set
yiguolei merged PR #12012: URL: https://github.com/apache/doris/pull/12012 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #11787: [Bug] BE crash with query `select * from (select 1 as a) b left join (select 2 as a) c using(a);`
yiguolei closed issue #11787: [Bug] BE crash with query `select * from (select 1 as a) b left join (select 2 as a) c using(a);` URL: https://github.com/apache/doris/issues/11787 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz opened a new pull request, #12041: Parquet temp test
wsjz opened a new pull request, #12041: URL: https://github.com/apache/doris/pull/12041 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #12039: [ci](shellcheck)shellcheck include ssb-tools and tpch-tools
adonis0147 commented on PR #12039: URL: https://github.com/apache/doris/pull/12039#issuecomment-1225365389 Hi @hello-stephen , `ShellCheck` action failed. You can run the script `build-support/shell-check.sh` on your machine to check these issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 commented on pull request #12033: [fix](ut) Fix fe ut NullPointer exception for DeleteHandlerTest
Gabriel39 commented on PR #12033: URL: https://github.com/apache/doris/pull/12033#issuecomment-1225379586 Hi @caiconghui I have fix this failure test and you need rebase master and push again to resolve the P0 regression failure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11948: [enhancement](ldap) optimize LDAP authentication.
github-actions[bot] commented on PR #11948: URL: https://github.com/apache/doris/pull/11948#issuecomment-1225381068 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11948: [enhancement](ldap) optimize LDAP authentication.
github-actions[bot] commented on PR #11948: URL: https://github.com/apache/doris/pull/11948#issuecomment-1225381016 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 commented on pull request #11970: [feature-wip](parquet-reader) parquert scanner can read data
Gabriel39 commented on PR #11970: URL: https://github.com/apache/doris/pull/11970#issuecomment-1225381396 Hi @wsjz I have fixed a failure regression case and you need rebase master and push again to resolve this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #12042: [feature-wip](scan) add profile for new olap scan node
morningman opened a new pull request, #12042: URL: https://github.com/apache/doris/pull/12042 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #12039: [ci](shellcheck)shellcheck include ssb-tools and tpch-tools
hello-stephen commented on PR #12039: URL: https://github.com/apache/doris/pull/12039#issuecomment-1225414640 > Hi @hello-stephen , `ShellCheck` action failed. You can run the script `build-support/shell-check.sh` on your machine to check these issues. OK, thanks. I use VSCode with ShellCheck extension and the check passed yesterday, but today not. :( maybe I should use build-support/shell-check.sh instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz closed pull request #12041: Parquet temp test
wsjz closed pull request #12041: Parquet temp test URL: https://github.com/apache/doris/pull/12041 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on pull request #12041: Parquet temp test
wsjz commented on PR #12041: URL: https://github.com/apache/doris/pull/12041#issuecomment-1225425838 abort -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui commented on pull request #12033: [fix](ut) Fix fe ut NullPointer exception for DeleteHandlerTest
caiconghui commented on PR #12033: URL: https://github.com/apache/doris/pull/12033#issuecomment-1225427184 > Hi @caiconghui I have fix this failure test and you need rebase master and push again to resolve this. ok -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on pull request #11970: [feature-wip](parquet-reader) parquert scanner can read data
wsjz commented on PR #11970: URL: https://github.com/apache/doris/pull/11970#issuecomment-1225434536 > Hi @wsjz I have fixed a failure regression case and you need rebase master and push again to resolve this. OK -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #12039: [ci](shellcheck)shellcheck include ssb-tools and tpch-tools
adonis0147 commented on PR #12039: URL: https://github.com/apache/doris/pull/12039#issuecomment-1225440612 I'm not sure whether the VSCode extension you mentioned supports the configuration `.shellcheckrc`. If it did, I think you could finish the job more easily. Besides, the version of `shellcheck` should be greater than `0.8`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11948: [enhancement](ldap) optimize LDAP authentication.
yiguolei merged PR #11948: URL: https://github.com/apache/doris/pull/11948 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](ldap) optimize LDAP authentication. (#11948)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b619bb2000 [enhancement](ldap) optimize LDAP authentication. (#11948)
b619bb2000 is described below
commit b619bb20007c671ea53ad8064a5fa969fd240940
Author: luozenglin <37725793+luozeng...@users.noreply.github.com>
AuthorDate: Wed Aug 24 17:08:14 2022 +0800
[enhancement](ldap) optimize LDAP authentication. (#11948)
* [enhancement](ldap) optimize LDAP authentication.
1. Support caching LDAP user information.
2. HTTP authentication supports LDAP.
3. LDAP temporary users support default user property.
4. LDAP configuration supports the `admin show config` and `admin set
config` commands.
---
build.sh | 1 +
{fe/conf => conf}/ldap.conf| 20 +-
docs/en/docs/admin-manual/privilege-ldap/ldap.md | 2 -
.../zh-CN/docs/admin-manual/privilege-ldap/ldap.md | 2 -
.../java/org/apache/doris/common/ConfigBase.java | 26 ++-
.../java/org/apache/doris/common/LdapConfig.java | 32 +--
.../org/apache/doris/ldap/LdapAuthenticate.java| 71 +--
.../java/org/apache/doris/ldap/LdapClient.java | 34 ++--
.../java/org/apache/doris/ldap/LdapManager.java| 216 +
.../org/apache/doris/ldap/LdapPrivsChecker.java| 38 ++--
.../java/org/apache/doris/ldap/LdapUserInfo.java | 84
.../java/org/apache/doris/mysql/MysqlProto.java| 10 +-
.../org/apache/doris/mysql/privilege/PaloAuth.java | 12 ++
.../doris/mysql/privilege/UserPropertyMgr.java | 46 +++--
.../java/org/apache/doris/qe/ConnectContext.java | 11 --
.../java/org/apache/doris/qe/ConnectScheduler.java | 9 +-
.../apache/doris/ldap/LdapAuthenticateTest.java| 59 ++
.../java/org/apache/doris/ldap/LdapClientTest.java | 16 +-
.../org/apache/doris/ldap/LdapManagerTest.java | 88 +
.../apache/doris/ldap/LdapPrivsCheckerTest.java| 45 +++--
.../org/apache/doris/mysql/MysqlProtoTest.java | 13 +-
21 files changed, 593 insertions(+), 242 deletions(-)
diff --git a/build.sh b/build.sh
index 53cdf4ecd0..661cac059d 100755
--- a/build.sh
+++ b/build.sh
@@ -432,6 +432,7 @@ if [[ "${BUILD_FE}" -eq 1 ]]; then
cp -r -p "${DORIS_HOME}/bin"/*_fe.sh "${DORIS_OUTPUT}/fe/bin"/
cp -r -p "${DORIS_HOME}/conf/fe.conf" "${DORIS_OUTPUT}/fe/conf"/
+cp -r -p "${DORIS_HOME}/conf/ldap.conf" "${DORIS_OUTPUT}/fe/conf"/
cp -r -p "${DORIS_HOME}/conf"/*.xml "${DORIS_OUTPUT}/fe/conf"/
rm -rf "${DORIS_OUTPUT}/fe/lib"/*
cp -r -p "${DORIS_HOME}/fe/fe-core/target/lib"/* "${DORIS_OUTPUT}/fe/lib"/
diff --git a/fe/conf/ldap.conf b/conf/ldap.conf
similarity index 87%
rename from fe/conf/ldap.conf
rename to conf/ldap.conf
index 7c1aabf068..f783c53ea9 100644
--- a/fe/conf/ldap.conf
+++ b/conf/ldap.conf
@@ -41,14 +41,16 @@ ldap_user_basedn = ou=people,dc=domain,dc=com
ldap_user_filter = (&(uid={login}))
ldap_group_basedn = ou=group,dc=domain,dc=com
+# ldap_cache_time_out_s = 12 * 60 * 60;
+
# LDAP pool configuration
#
https://docs.spring.io/spring-ldap/docs/2.3.3.RELEASE/reference/#pool-configuration
-#max_active = 8
-#max_total = -1
-#max_idle = 8
-#min_idle = 0
-#max_wait = -1
-#when_exhausted = 1
-#test_on_borrow = false
-#test_on_return = false
-#test_while_idle = false
+# ldap_pool_max_active = 8
+# ldap_pool_max_total = -1
+# ldap_pool_max_idle = 8
+# ldap_pool_min_idle = 0
+# ldap_pool_max_wait = -1
+# ldap_pool_when_exhausted = 1
+# ldap_pool_test_on_borrow = false
+# ldap_pool_test_on_return = false
+# ldap_pool_test_while_idle = false
diff --git a/docs/en/docs/admin-manual/privilege-ldap/ldap.md
b/docs/en/docs/admin-manual/privilege-ldap/ldap.md
index 01a139fb25..3e9c2a41b3 100644
--- a/docs/en/docs/admin-manual/privilege-ldap/ldap.md
+++ b/docs/en/docs/admin-manual/privilege-ldap/ldap.md
@@ -171,5 +171,3 @@ If jack also belongs to the LDAP groups doris_qa, doris_pm;
Doris exists roles:
## Limitations of LDAP authentication
* The current LDAP feature of Doris only supports plaintext password
authentication, that is, when a user logs in, the password is transmitted in
plaintext between client and fe and between fe and LDAP service.
-* The current LDAP authentication only supports password authentication under
mysql protocol. If you use the Http interface, you cannot use LDAP users for
authentication.
-* Temporary users do not have user properties.
\ No newline at end of file
diff --git a/docs/zh-CN/docs/admin-manual/privilege-ldap/ldap.md
b/docs/zh-CN/docs/admin-manual/privilege-ldap/ldap.md
index b1459f8a8d..feba8f37a4 100644
--- a/docs/zh-CN/docs/admin-manual/privilege-ldap/ldap.md
+++ b/docs/zh-CN/docs/admin-manual/privilege-ldap/ldap.md
@@ -188,5 +188,3 @@ member: uid=jack,ou=aidp,dc=domain,dc=com
## LDA
[GitHub] [doris] yiguolei commented on a diff in pull request #11940: [refactor](status) Refactor status handling in agent task
yiguolei commented on code in PR #11940:
URL: https://github.com/apache/doris/pull/11940#discussion_r953554584
##
be/src/olap/task/engine_storage_migration_task.cpp:
##
@@ -341,19 +290,10 @@ void EngineStorageMigrationTask::_generate_new_header(
Status EngineStorageMigrationTask::_copy_index_and_data_files(
const string& full_path, const std::vector&
consistent_rowsets) const {
-Status status = Status::OK();
for (const auto& rs : consistent_rowsets) {
-status = rs->copy_files_to(full_path, rs->rowset_id());
-if (!status.ok()) {
-Status ret = FileUtils::remove_all(full_path);
Review Comment:
This line is removed, then there exists garbage in this folder?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #12043: [fix](bdbje) add reserved disk config to avoid too many reserved bdbje files
morningman opened a new pull request, #12043: URL: https://github.com/apache/doris/pull/12043 # Proposed changes Issue Number: close #xxx ## Problem summary Thanks to @dh-cloud and see #7273 for details ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 opened a new pull request, #12044: [feature-wip](unique-key-merge-on-write) fix that version is awlays 0 when update delete bitmap
liaoxin01 opened a new pull request, #12044: URL: https://github.com/apache/doris/pull/12044 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. introduced by pr #11953 ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11940: [refactor](status) Refactor status handling in agent task
yiguolei commented on code in PR #11940:
URL: https://github.com/apache/doris/pull/11940#discussion_r953586724
##
be/src/agent/task_worker_pool.cpp:
##
@@ -1259,11 +1217,6 @@ void
TaskWorkerPool::_report_tablet_worker_thread_callback() {
DorisMetrics::instance()->report_all_tablets_requests_skip->increment(1);
continue;
}
-if (!build_all_report_tablets_info_status.ok()) {
-LOG(WARNING) << "build all report tablets info failed. status: "
- << build_all_report_tablets_info_status;
Review Comment:
If build failed, some logic in handle report maybe wrong, it think this
logic should not be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
924060929 commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r953353208
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalHashJoin.java:
##
@@ -38,9 +40,10 @@
RIGHT_CHILD_TYPE extends Plan>
extends AbstractPhysicalJoin {
-public PhysicalHashJoin(JoinType joinType, Optional condition,
LogicalProperties logicalProperties,
+public PhysicalHashJoin(JoinType joinType, List
hashJoinConjuncts,
+Optional condition, LogicalProperties
logicalProperties,
Review Comment:
change `Optional condition` to `List
nonHashJoinConjuncts`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
924060929 commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r953329100
##
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/RewriteJob.java:
##
@@ -40,13 +41,14 @@ public class RewriteJob extends BatchRulesJob {
public RewriteJob(CascadesContext cascadesContext) {
super(cascadesContext);
ImmutableList jobs = new ImmutableList.Builder()
-.add(bottomUpBatch(ImmutableList.of(new
MergeConsecutiveProjects(
-.add(topDownBatch(ImmutableList.of(new
ExpressionNormalization(
-.add(topDownBatch(ImmutableList.of(new ReorderJoin(
-.add(topDownBatch(ImmutableList.of(new
PushPredicateThroughJoin(
+// add rules in reverse order due to jobPool is stack.
.add(topDownBatch(ImmutableList.of(new
AggregateDisassemble(
-.build().reverse(); // reverse due to jobPool is stack.
Review Comment:
we should keep the order as the present makes it easier to view. the reverse
can hide in the base job class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
englefly commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r953593125
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java:
##
@@ -43,7 +45,8 @@
extends LogicalBinary implements
Join {
private final JoinType joinType;
-private final Optional condition;
+private final Optional otherJoinCondition;
Review Comment:
shall we do it in the next pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11940: [refactor](status) Refactor status handling in agent task
yiguolei commented on code in PR #11940:
URL: https://github.com/apache/doris/pull/11940#discussion_r953594770
##
be/src/olap/task/engine_batch_load_task.cpp:
##
@@ -344,35 +307,26 @@ Status EngineBatchLoadTask::_delete_data(const TPushReq&
request,
Status res = Status::OK();
if (tablet_info_vec == nullptr) {
-LOG(WARNING) << "invalid tablet info parameter which is nullptr
pointer.";
-return Status::OLAPInternalError(OLAP_ERR_CE_CMD_PARAMS_ERROR);
+return Status::InvalidArgument("invalid tablet_info_vec which is
nullptr");
}
// 1. Get all tablets with same tablet_id
TabletSharedPtr tablet =
StorageEngine::instance()->tablet_manager()->get_tablet(request.tablet_id);
if (tablet == nullptr) {
-LOG(WARNING) << "can't find tablet. tablet=" << request.tablet_id;
-return Status::OLAPInternalError(OLAP_ERR_TABLE_NOT_FOUND);
+return Status::InternalError("could not find tablet {}",
request.tablet_id);
}
// 2. Process delete data by push interface
PushHandler push_handler;
-if (request.__isset.transaction_id) {
-res = push_handler.process_streaming_ingestion(tablet, request,
PUSH_FOR_DELETE,
- tablet_info_vec);
-} else {
-res = Status::OLAPInternalError(OLAP_ERR_PUSH_BATCH_PROCESS_REMOVED);
+if (!request.__isset.transaction_id) {
+return Status::InvalidArgument("transaction_id is not set");
}
-
+res = push_handler.process_streaming_ingestion(tablet, request,
PUSH_FOR_DELETE,
+ tablet_info_vec);
if (!res.ok()) {
-LOG(WARNING) << "fail to push empty version for delete data. "
- << "res=" << res << "tablet=" << tablet->full_name();
DorisMetrics::instance()->delete_requests_failed->increment(1);
-return res;
}
-
-LOG(INFO) << "finish to process delete data. res=" << res;
return res;
Review Comment:
All these tasks have a same problem. If the task is success, there is no
log. So if a table lost some data or data is wrong, we could not find what
happened on this tablet. I think we should print some success log to help us
debug. Maybe could add a log in EngineTask.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] gavinchou commented on pull request #12044: [feature-wip](unique-key-merge-on-write) fix that version is awlays 0 when update delete bitmap
gavinchou commented on PR #12044: URL: https://github.com/apache/doris/pull/12044#issuecomment-1225495579 Nice catch! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
924060929 commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r953598315
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java:
##
@@ -43,7 +45,8 @@
extends LogicalBinary implements
Join {
private final JoinType joinType;
-private final Optional condition;
+private final Optional otherJoinCondition;
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 commented on pull request #12037: [Fix](Planner) Show rollup info in explain result.
wangshuo128 commented on PR #12037: URL: https://github.com/apache/doris/pull/12037#issuecomment-1225499928 > it seems like we print rollup name after table name in a pair of parentheses currently. Just like > > ``` > lineorder(lineorder) > ``` Yes, rollup info kept but showing in different place. :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 closed pull request #12037: [Fix](Planner) Show rollup info in explain result.
wangshuo128 closed pull request #12037: [Fix](Planner) Show rollup info in explain result. URL: https://github.com/apache/doris/pull/12037 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] carlvinhust2012 opened a new pull request, #12045: [fix](array-type) fix the be core dump when use collect_list result t…
carlvinhust2012 opened a new pull request, #12045:
URL: https://github.com/apache/doris/pull/12045
# Proposed changes
1. this pr is used to fix the be core dump when use collect_list result to
insert.
2. the core dump stack is as below:
*** Current BE git commitID: 55fdb555b ***
*** SIGSEGV unkown detail explain (@0x0) received by PID 63892 (TID
0x7f1004d7c700) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/common/signal_handler.h:420
1# 0x7F10583D4920 in /lib64/libc.so.6
2# __dynamic_cast in /home/disk1/hugo_work/doris_env/output/be/lib/palo_be
3# doris::vectorized::ColumnVector::insert_range_from(doris::vectorized::IColumn const&, unsigned long,
unsigned long) at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/vec/columns/column_vector.cpp:255
4#
doris::vectorized::ColumnNullable::insert_range_from(doris::vectorized::IColumn
const&, unsigned long, unsigned long) at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/vec/columns/column_nullable.cpp:179
5# doris::vectorized::ColumnArray::insert_from(doris::vectorized::IColumn
const&, unsigned long) at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/vec/columns/column_array.cpp:239
6#
doris::vectorized::ColumnNullable::insert_from(doris::vectorized::IColumn
const&, unsigned long) at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/vec/columns/column_nullable.cpp:204
7# doris::vectorized::MutableBlock::add_row(doris::vectorized::Block
const*, int) at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/vec/core/block.cpp:966
8#
doris::stream_load::VNodeChannel::add_row(std::pair const&, long) at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/vec/sink/vtablet_sink.cpp:188
9# void
doris::stream_load::IndexChannel::add_row >(std::pair const&, long) at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/exec/tablet_sink.h:403
10# doris::stream_load::VOlapTableSink::send(doris::RuntimeState*,
doris::vectorized::Block*) at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/vec/sink/vtablet_sink.cpp:522
11# doris::PlanFragmentExecutor::open_vectorized_internal() at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/runtime/plan_fragment_executor.cpp:308
12# doris::PlanFragmentExecutor::open() at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/runtime/plan_fragment_executor.cpp:249
13# doris::FragmentExecState::execute() at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/runtime/fragment_mgr.cpp:246
14#
doris::FragmentMgr::_exec_actual(std::shared_ptr,
std::function) at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/runtime/fragment_mgr.cpp:500
15# std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data
const&) at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/palo-toolchain/ldb_toolchain/include/c++/11/bits/std_function.h:291
16# doris::ThreadPool::dispatch_thread() at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/util/threadpool.cpp:548
17# doris::Thread::supervise_thread(void*) at
/home/disk1/hugo_work/doris_dev/baidu/bdg/doris/core/be/src/util/thread.cpp:426
18# start_thread in /lib64/libpthread.so.0
19# clone in /lib64/libc.so.6
Issue Number: #7570
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
4. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
5. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
6. Does it need to update dependencies:
- [ ] Yes
- [ ] No
7. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11940: [refactor](status) Refactor status handling in agent task
yiguolei commented on code in PR #11940:
URL: https://github.com/apache/doris/pull/11940#discussion_r953603686
##
be/src/olap/task/engine_clone_task.cpp:
##
@@ -96,176 +93,108 @@ Status EngineCloneTask::_do_clone() {
// if missed version size is 0, then it is useless to clone from
remote be, it means local data is
// completed. Or remote be will just return header not the rowset
files. clone will failed.
-if (missed_versions.size() == 0) {
-LOG(INFO) << "missed version size = 0, skip clone and return
success. tablet id="
+if (missed_versions.empty()) {
+LOG(INFO) << "missed version size = 0, skip clone and return
success. tablet_id="
<< _clone_req.tablet_id;
-_set_tablet_info(Status::OK(), is_new_tablet);
+_set_tablet_info(is_new_tablet);
return Status::OK();
}
+LOG(INFO) << "clone to existed tablet. missed_versions_size=" <<
missed_versions.size()
+ << ", allow_incremental_clone=" << allow_incremental_clone
+ << ", signature=" << _signature << ", tablet_id=" <<
_clone_req.tablet_id
+ << ", committed_version=" << _clone_req.committed_version;
+
// try to download missing version from src backend.
// if tablet on src backend does not contains missing version, it will
download all versions,
// and set allow_incremental_clone to false
-status = _make_and_download_snapshots(*(tablet->data_dir()),
local_data_path, &src_host,
- &src_file_path, _error_msgs,
&missed_versions,
- &allow_incremental_clone);
-
-LOG(INFO) << "tablet exist with number of missing version: " <<
missed_versions.size()
- << ", try to incremental clone succeed: " <<
allow_incremental_clone
- << ", signature: " << _signature << ", tablet id: " <<
_clone_req.tablet_id
- << ", schema hash: " << _clone_req.schema_hash
- << ", clone version: " << _clone_req.committed_version
- << ", download snapshot: " << status;
-
-if (status.ok()) {
-Status olap_status =
-_finish_clone(tablet.get(), local_data_path,
_clone_req.committed_version,
- allow_incremental_clone);
-if (!olap_status.ok()) {
-LOG(WARNING) << "failed to finish clone. [table=" <<
tablet->full_name()
- << " res=" << olap_status << "]";
-_error_msgs->push_back("clone error.");
-status = Status::InternalError("Failed to finish clone");
-}
-}
+RETURN_IF_ERROR(_make_and_download_snapshots(*(tablet->data_dir()),
local_data_path,
+ &src_host,
&src_file_path, &missed_versions,
+
&allow_incremental_clone));
+RETURN_IF_ERROR(_finish_clone(tablet.get(), local_data_path,
_clone_req.committed_version,
+ allow_incremental_clone));
} else {
LOG(INFO) << "clone tablet not exist, begin clone a new tablet from
remote be. "
- << "signature:" << _signature << ", tablet_id:" <<
_clone_req.tablet_id
- << ", schema_hash:" << _clone_req.schema_hash
- << ", committed_version:" << _clone_req.committed_version;
+ << "signature=" << _signature << ", tablet_id=" <<
_clone_req.tablet_id
+ << ", committed_version=" << _clone_req.committed_version;
// create a new tablet in this be
// Get local disk from olap
string local_shard_root_path;
DataDir* store = nullptr;
-Status olap_status = StorageEngine::instance()->obtain_shard_path(
-_clone_req.storage_medium, &local_shard_root_path, &store);
-if (!olap_status.ok()) {
-LOG(WARNING) << "clone get local root path failed. signature: " <<
_signature;
-_error_msgs->push_back("clone get local root path failed.");
-status = Status::InternalError("Clone to get local root path
failed");
-}
-std::stringstream tablet_dir_stream;
-tablet_dir_stream << local_shard_root_path << "/" <<
_clone_req.tablet_id << "/"
- << _clone_req.schema_hash;
+RETURN_IF_ERROR(StorageEngine::instance()->obtain_shard_path(
+_clone_req.storage_medium, &local_shard_root_path, &store));
+auto tablet_dir = fmt::format("{}/{}/{}", local_shard_root_path,
_clone_req.tablet_id,
+ _clone_req.schema_hash);
+
+Defer remove_useless_dir {[&] {
+if (status.ok()) {
+return;
+}
+
[GitHub] [doris] yiguolei commented on a diff in pull request #11940: [refactor](status) Refactor status handling in agent task
yiguolei commented on code in PR #11940:
URL: https://github.com/apache/doris/pull/11940#discussion_r953606295
##
be/src/olap/task/engine_clone_task.cpp:
##
@@ -324,36 +251,24 @@ Status
EngineCloneTask::_make_and_download_snapshots(DataDir& data_dir,
remote_url_prefix = ss.str();
}
-st = _download_files(&data_dir, remote_url_prefix, local_data_path);
-if (!st.ok()) {
-LOG(WARNING) << "fail to download and convert tablet, remote=" <<
remote_url_prefix
- << ", error=" << st.to_string();
-status = Status::InternalError("Fail to download and convert
tablet");
-// when there is an error, keep this program executing to release
snapshot
-}
+status = _download_files(&data_dir, remote_url_prefix,
local_data_path);
+// when there is an error, keep this program executing to release
snapshot
Review Comment:
Losing log here, should print log when download files failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui merged pull request #12033: [fix](ut) Fix fe ut NullPointer exception for DeleteHandlerTest
caiconghui merged PR #12033: URL: https://github.com/apache/doris/pull/12033 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui closed issue #12032: [Bug] (ut) run DeleteHandlerTest encounter NPE
caiconghui closed issue #12032: [Bug] (ut) run DeleteHandlerTest encounter NPE URL: https://github.com/apache/doris/issues/12032 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](ut) Fix fe ut npe for DeleteHandlerTest (#12033)
This is an automated email from the ASF dual-hosted git repository.
caiconghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2057edbea0 [fix](ut) Fix fe ut npe for DeleteHandlerTest (#12033)
2057edbea0 is described below
commit 2057edbea059b4f618dc8c432c5bac816be3ff17
Author: caiconghui <55968745+caicong...@users.noreply.github.com>
AuthorDate: Wed Aug 24 18:09:02 2022 +0800
[fix](ut) Fix fe ut npe for DeleteHandlerTest (#12033)
---
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java| 2 +-
.../test/java/org/apache/doris/load/DeleteHandlerTest.java| 11 +++
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
index 060a044074..3cfa5d8502 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
@@ -453,7 +453,7 @@ public class Env {
return journalObservable;
}
-private SystemInfoService getClusterInfo() {
+public SystemInfoService getClusterInfo() {
return this.systemInfo;
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/load/DeleteHandlerTest.java
b/fe/fe-core/src/test/java/org/apache/doris/load/DeleteHandlerTest.java
index 01311f0754..5c2c005972 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/load/DeleteHandlerTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/load/DeleteHandlerTest.java
@@ -43,6 +43,7 @@ import org.apache.doris.mysql.privilege.PaloAuth;
import org.apache.doris.persist.EditLog;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.QueryStateException;
+import org.apache.doris.system.SystemInfoService;
import org.apache.doris.task.AgentBatchTask;
import org.apache.doris.task.AgentTask;
import org.apache.doris.task.AgentTaskExecutor;
@@ -93,6 +94,8 @@ public class DeleteHandlerTest {
private AgentTaskQueue agentTaskQueue;
@Mocked
private AgentTaskExecutor executor;
+@Mocked
+private SystemInfoService systemInfoService;
private Database db;
private PaloAuth auth;
@@ -167,6 +170,14 @@ public class DeleteHandlerTest {
env.getEditLog();
minTimes = 0;
result = editLog;
+
+env.getClusterInfo();
+minTimes = 0;
+result = systemInfoService;
+
+systemInfoService.getBackendIds(false);
+minTimes = 0;
+result = Lists.newArrayList(1L);
}
};
globalTransactionMgr.addDatabaseTransactionMgr(db.getId());
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11984: [Fix](remote) Fix thread safety issue in cache
yiguolei merged PR #11984: URL: https://github.com/apache/doris/pull/11984 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (2057edbea0 -> 54fc038dc5)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 2057edbea0 [fix](ut) Fix fe ut npe for DeleteHandlerTest (#12033) add 54fc038dc5 [Fix](remote) Fix thread safety issue in cache (#11984) No new revisions were added by this update. Summary of changes: be/src/common/config.h | 4 + be/src/io/cache/sub_file_cache.cpp | 114 +--- be/src/io/cache/sub_file_cache.h | 1 + be/src/io/cache/whole_file_cache.cpp | 132 +++ be/src/io/cache/whole_file_cache.h | 1 + be/src/runtime/exec_env.h| 14 +++ be/src/runtime/exec_env_init.cpp | 17 +++ be/src/util/doris_metrics.h | 2 + docs/en/docs/admin-manual/config/be-config.md| 12 +++ docs/zh-CN/docs/admin-manual/config/be-config.md | 12 +++ 10 files changed, 224 insertions(+), 85 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #11992: [Bug] Thread safety issue in cache
yiguolei closed issue #11992: [Bug] Thread safety issue in cache URL: https://github.com/apache/doris/issues/11992 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] YangShaw opened a new pull request, #12046: [enhancement](nereids)support one expression-rewrite rule: inPredicateToEqualTo
YangShaw opened a new pull request, #12046: URL: https://github.com/apache/doris/pull/12046 # Proposed changes Issue Number: close #xxx ## Problem summary Add one expression rewrite rule: rewrite InPredicate to an EqualTo Expression, if there exists exactly one element in InPredicate Options. Examples: * where A in (x) ==> where A = x * where A not in (x) ==> where not A = x Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12046: [enhancement](nereids)support one expression-rewrite rule: inPredicateToEqualTo
924060929 commented on code in PR #12046:
URL: https://github.com/apache/doris/pull/12046#discussion_r953620783
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/InPredicateToEqualToRule.java:
##
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+package org.apache.doris.nereids.rules.expression.rewrite.rules;
+
+import
org.apache.doris.nereids.rules.expression.rewrite.AbstractExpressionRewriteRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.InPredicate;
+
+/**
+ * Rewrites InPredicate to an EqualTo Expression, if there exists exactly one
element in InPredicate.Options.
+ * Examples:
+ * where A in (x) ==> where A = x
+ * where A not in (x) ==> where not A = x (After ExpressionTranslator, "not A
= x" will be translated to "A != x")
+ */
+public class InPredicateToEqualToRule extends AbstractExpressionRewriteRule {
+
+public static InPredicateToEqualToRule INSTANCE = new
InPredicateToEqualToRule();
+
+@Override
+public Expression visitInPredicate(InPredicate inPredicate,
ExpressionRewriteContext context) {
+if (inPredicate.getOptions().size() > 1) {
Review Comment:
```suggestion
if (inPredicate.getOptions().size() != 1) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12013: [feature](Nereids)add normalize aggregate rule
github-actions[bot] commented on PR #12013: URL: https://github.com/apache/doris/pull/12013#issuecomment-1225529003 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12013: [feature](Nereids)add normalize aggregate rule
github-actions[bot] commented on PR #12013: URL: https://github.com/apache/doris/pull/12013#issuecomment-1225529046 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12046: [enhancement](nereids)support one expression-rewrite rule: inPredicateToEqualTo
924060929 commented on code in PR #12046:
URL: https://github.com/apache/doris/pull/12046#discussion_r953625412
##
regression-test/suites/nereids_syntax_p0/inpredicate.groovy:
##
@@ -55,5 +55,13 @@ suite("inpredicate") {
order_qt_in_predicate_8 """
SELECT * FROM supplier WHERE s_nation not in ('PERU', 'ETHIOPIA');
"""
+
+order_qt_in_predicate_9 """
+SELECT * FROM supplier WHERE s_suppkey in (1);
+"""
+
+order_qt_in_predicate_10 """
+SELECT * FROM supplier WHERE s_suppkey not in (1);
Review Comment:
```suggestion
SELECT * FROM supplier WHERE s_suppkey not in (15);
```
##
regression-test/suites/nereids_syntax_p0/inpredicate.groovy:
##
@@ -55,5 +55,13 @@ suite("inpredicate") {
order_qt_in_predicate_8 """
SELECT * FROM supplier WHERE s_nation not in ('PERU', 'ETHIOPIA');
"""
+
+order_qt_in_predicate_9 """
+SELECT * FROM supplier WHERE s_suppkey in (1);
Review Comment:
```suggestion
SELECT * FROM supplier WHERE s_suppkey in (15);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12013: [feature](Nereids)add normalize aggregate rule
yiguolei merged PR #12013: URL: https://github.com/apache/doris/pull/12013 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature](Nereids)add normalize aggregate rule (#12013)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b32aac9195 [feature](Nereids)add normalize aggregate rule (#12013)
b32aac9195 is described below
commit b32aac919528f47a58247b4d05aa973a8643f885
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Wed Aug 24 18:30:18 2022 +0800
[feature](Nereids)add normalize aggregate rule (#12013)
---
.../org/apache/doris/nereids/rules/RuleType.java | 1 +
.../expression/rewrite/ExpressionRewrite.java | 4 +-
.../rules/rewrite/AggregateDisassemble.java| 3 +-
.../rules/rewrite/logical/NormalizeAggregate.java | 138 +
.../trees/plans/logical/LogicalAggregate.java | 27 +++-
.../trees/plans/logical/LogicalOlapScan.java | 2 +-
.../rewrite/logical/NormalizeAggregateTest.java| 168 +
7 files changed, 331 insertions(+), 12 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index 7f4c22ba71..9dd73d04b4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -45,6 +45,7 @@ public enum RuleType {
CHECK_ANALYSIS(RuleTypeClass.CHECK),
// rewrite rules
+NORMALIZE_AGGREGATE(RuleTypeClass.REWRITE),
AGGREGATE_DISASSEMBLE(RuleTypeClass.REWRITE),
COLUMN_PRUNE_PROJECTION(RuleTypeClass.REWRITE),
ELIMINATE_ALIAS_NODE(RuleTypeClass.REWRITE),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/ExpressionRewrite.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/ExpressionRewrite.java
index b285eaa2fa..96e61abc10 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/ExpressionRewrite.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/ExpressionRewrite.java
@@ -96,8 +96,8 @@ public class ExpressionRewrite implements RewriteRuleFactory {
if (outputExpressions.containsAll(newOutputExpressions)) {
return agg;
}
-return new LogicalAggregate<>(newGroupByExprs,
newOutputExpressions, agg.isDisassembled(),
-agg.getAggPhase(), agg.child());
+return new LogicalAggregate<>(newGroupByExprs,
newOutputExpressions,
+agg.isDisassembled(), agg.isNormalized(),
agg.getAggPhase(), agg.child());
}).toRule(RuleType.REWRITE_AGG_EXPRESSION);
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java
index 8a0363e103..1e8da9a14f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java
@@ -51,7 +51,6 @@ import java.util.stream.Collectors;
* TODO:
* 1. use different class represent different phase aggregate
* 2. if instance count is 1, shouldn't disassemble the agg plan
- * 3. we need another rule to removing duplicated expressions in group by
expression list
*/
public class AggregateDisassemble extends OneRewriteRuleFactory {
@@ -123,6 +122,7 @@ public class AggregateDisassemble extends
OneRewriteRuleFactory {
localGroupByExprs,
localOutputExprs,
true,
+aggregate.isNormalized(),
AggPhase.LOCAL,
aggregate.child()
);
@@ -130,6 +130,7 @@ public class AggregateDisassemble extends
OneRewriteRuleFactory {
globalGroupByExprs,
globalOutputExprs,
true,
+aggregate.isNormalized(),
AggPhase.GLOBAL,
localAggregate
);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/NormalizeAggregate.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/NormalizeAggregate.java
new file mode 100644
index 00..5aa70a0af3
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/NormalizeAggregate.java
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (t
[GitHub] [doris] zhannngchen commented on a diff in pull request #12044: [feature-wip](unique-key-merge-on-write) fix that version is awlays 0 when update delete bitmap
zhannngchen commented on code in PR #12044:
URL: https://github.com/apache/doris/pull/12044#discussion_r953629112
##
be/src/olap/txn_manager.cpp:
##
@@ -309,6 +309,9 @@ Status TxnManager::publish_txn(OlapMeta* meta, TPartitionId
partition_id,
// save meta need access disk, it maybe very slow, so that it is not
in global txn lock
// it is under a single txn lock
if (rowset_ptr != nullptr) {
+// TODO(ygl): rowset is already set version here, memory is
changed, if save failed
Review Comment:
The comments should be removed? L336 returned an OLAPInternalError
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zxealous commented on a diff in pull request #11959: [feature](remote)Add cache files cleaner for remote olap files
zxealous commented on code in PR #11959: URL: https://github.com/apache/doris/pull/11959#discussion_r953629420 ## docs/zh-CN/docs/admin-manual/config/be-config.md: ## @@ -1600,3 +1600,22 @@ webserver默认工作线程数 * 描述: 最少进行合并的版本数，当选中的小数据量的rowset个数，大于这个值是才会进行真正的合并 * 默认值: 10 +### `generate_cache_cleaner_task_interval_sec` +* 类型：int64 +* 描述：缓存文件的清理间隔，单位：秒 +* 默认值：43200（12小时） + +### `file_cache_type` +* 类型：string +* 描述：缓存文件的类型。whole_file_cache：将segment文件整个下载，sub_file_cache：将segment文件按大小切分成多个文件。 +* 默认值：104857600（100MB） Review Comment: default value ## docs/en/docs/admin-manual/config/be-config.md: ## @@ -1576,3 +1576,22 @@ Translated with www.DeepL.com/Translator (free version) * Description: at least the number of versions to be compaction, and the number of rowsets with a small amount of data in the selection. If it is greater than this value, the real compaction will be carried out * Default: 10 +### `generate_cache_cleaner_task_interval_sec` +* Type：int64 +* Description：Cleaning interval of cache files, in seconds +* Default：43200（12 hours） + +### `file_cache_type` +* Type：string +* Description：Type of cache file. whole_ file_ Cache: download the entire segment file, sub_ file_ Cache: the segment file is divided into multiple files by size. +* Default：104857600（100MB） Review Comment: the default value is an empty string -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed pull request #11892: [bugfix](fix) fix dcheck failure of outer join with select const in i…
yiguolei closed pull request #11892: [bugfix](fix) fix dcheck failure of outer join with select const in i… URL: https://github.com/apache/doris/pull/11892 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on pull request #11892: [bugfix](fix) fix dcheck failure of outer join with select const in i…
yiguolei commented on PR #11892: URL: https://github.com/apache/doris/pull/11892#issuecomment-1225536056 duplicate with https://github.com/apache/doris/pull/12012 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11778: [typo](doc) Change wrong words
yiguolei merged PR #11778: URL: https://github.com/apache/doris/pull/11778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (b32aac9195 -> 87dd9c6b8b)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from b32aac9195 [feature](Nereids)add normalize aggregate rule (#12013) add 87dd9c6b8b [typo](doc) Change wrong words #11778 No new revisions were added by this update. Summary of changes: .../sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zxealous opened a new pull request, #12047: [fix](docs) Fix configs default value in docs
zxealous opened a new pull request, #12047: URL: https://github.com/apache/doris/pull/12047 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [x] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12044: [feature-wip](unique-key-merge-on-write) fix that version is awlays 0 when update delete bitmap
github-actions[bot] commented on PR #12044: URL: https://github.com/apache/doris/pull/12044#issuecomment-1225551458 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Jibing-Li opened a new pull request, #12048: New file scan node
Jibing-Li opened a new pull request, #12048: URL: https://github.com/apache/doris/pull/12048 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #12049: [Bug](runtime filter) fix bug for late-arrival runtime filters
Gabriel39 opened a new pull request, #12049: URL: https://github.com/apache/doris/pull/12049 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #12045: [fix](array-type) fix the be core dump when use collect_list result t…
cambyzju commented on code in PR #12045:
URL: https://github.com/apache/doris/pull/12045#discussion_r953668457
##
be/src/vec/columns/column_array.cpp:
##
@@ -235,7 +235,13 @@ void ColumnArray::insert_from(const IColumn& src_, size_t
n) {
size_t size = src.size_at(n);
size_t offset = src.offset_at(n);
-get_data().insert_range_from(src.get_data(), offset, size);
+// Note: here we should process the case of
'Nullable(Array(NotNullable(nest)))'
Review Comment:
```suggestion
// Note: here we should process the case of 'Array(NotNullable(nest))'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #12045: [fix](array-type) fix the be core dump when use collect_list result t…
cambyzju commented on code in PR #12045:
URL: https://github.com/apache/doris/pull/12045#discussion_r953668845
##
be/src/vec/columns/column_array.cpp:
##
@@ -235,7 +235,13 @@ void ColumnArray::insert_from(const IColumn& src_, size_t
n) {
size_t size = src.size_at(n);
size_t offset = src.offset_at(n);
-get_data().insert_range_from(src.get_data(), offset, size);
+// Note: here we should process the case of
'Nullable(Array(NotNullable(nest)))'
+if (get_data().is_nullable() && !src.get_data().is_nullable()) {
+reinterpret_cast(&get_data())
+->insert_range_from_not_nullable(src.get_data(), offset, size);
+} else {
Review Comment:
add DCHEC for `!get_data().is_nullable() && src.get_data().is_nullable()` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary opened a new issue, #12050: [Bug] (ctas) Ctas due to be crash when select contains default timestamp
stalary opened a new issue, #12050:
URL: https://github.com/apache/doris/issues/12050
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Version
master
### What's Wrong?
Ctas due to be crash when select contains default timestamp
### What You Expected?
normal
### How to Reproduce?
1.
```
CREATE TABLE `test_time2` (
`timekey` date NULL,
`dt` datetime NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE=OLAP
DUPLICATE KEY(`timekey`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`timekey`) BUCKETS 5
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2",
"disable_auto_compaction" = "false"
)
```
2.
```
insert into test_time2(timekey) values('2022-08-24');
```
3.
```
create table test1 PROPERTIES ( "replication_allocation" =
"tag.location.default: 1", "in_memory" = "false", "storage_format" = "V2" ) as
select * from test_time2;
```
### Anything Else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] carlvinhust2012 commented on a diff in pull request #12045: [fix](array-type) fix the be core dump when use collect_list result t…
carlvinhust2012 commented on code in PR #12045:
URL: https://github.com/apache/doris/pull/12045#discussion_r953674794
##
be/src/vec/columns/column_array.cpp:
##
@@ -235,7 +235,13 @@ void ColumnArray::insert_from(const IColumn& src_, size_t
n) {
size_t size = src.size_at(n);
size_t offset = src.offset_at(n);
-get_data().insert_range_from(src.get_data(), offset, size);
+// Note: here we should process the case of
'Nullable(Array(NotNullable(nest)))'
+if (get_data().is_nullable() && !src.get_data().is_nullable()) {
+reinterpret_cast(&get_data())
+->insert_range_from_not_nullable(src.get_data(), offset, size);
+} else {
Review Comment:
if add DCHEC to replace, the insert_from() will just support the only one
case. It is unreasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] carlvinhust2012 commented on a diff in pull request #12045: [fix](array-type) fix the be core dump when use collect_list result t…
carlvinhust2012 commented on code in PR #12045:
URL: https://github.com/apache/doris/pull/12045#discussion_r953675854
##
be/src/vec/columns/column_array.cpp:
##
@@ -235,7 +235,13 @@ void ColumnArray::insert_from(const IColumn& src_, size_t
n) {
size_t size = src.size_at(n);
size_t offset = src.offset_at(n);
-get_data().insert_range_from(src.get_data(), offset, size);
+// Note: here we should process the case of
'Nullable(Array(NotNullable(nest)))'
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] MRYOG closed issue #11988: [Bug] 分区过多时，delete from table超时
MRYOG closed issue #11988: [Bug] 分区过多时，delete from table超时 URL: https://github.com/apache/doris/issues/11988 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] MRYOG commented on issue #11988: [Bug] 分区过多时，delete from table超时
MRYOG commented on issue #11988: URL: https://github.com/apache/doris/issues/11988#issuecomment-1225592014 已解决 1 执行时设置 SET delete_without_partition=TRUE 2 执行语句：DELETE FROM test_table PARTITION p20220820 WHERE day_id=20220820 3 执行删除时 添加指定分区名称(PARTITION p20220820)即可 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] MRYOG commented on issue #11987: [Bug] Delete 语句Where 条件不能跟加 1=1
MRYOG commented on issue #11987: URL: https://github.com/apache/doris/issues/11987#issuecomment-1225594041 Understand, thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] MRYOG closed issue #11987: [Bug] Delete 语句Where 条件不能跟加 1=1
MRYOG closed issue #11987: [Bug] Delete 语句Where 条件不能跟加 1=1 URL: https://github.com/apache/doris/issues/11987 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] jeffreys-cat opened a new pull request, #66: [fix] update workflows
jeffreys-cat opened a new pull request, #66: URL: https://github.com/apache/doris-website/pull/66 Update ```cron-deploy-website.yml``` and ```manual-deploy.yml]``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: [fix] update workflows (#66)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 57ce626d6b1 [fix] update workflows (#66)
57ce626d6b1 is described below
commit 57ce626d6b111ea2ae2d663cf041aba44aea1cdf
Author: Jeffrey
AuthorDate: Wed Aug 24 19:37:03 2022 +0800
[fix] update workflows (#66)
update workflows
---
.github/workflows/cron-deploy-website.yml | 33 +++-
.github/workflows/manual-deploy.yml | 36 +--
2 files changed, 28 insertions(+), 41 deletions(-)
diff --git a/.github/workflows/cron-deploy-website.yml
b/.github/workflows/cron-deploy-website.yml
index 084fe2f32f3..c6e56a9b4b1 100644
--- a/.github/workflows/cron-deploy-website.yml
+++ b/.github/workflows/cron-deploy-website.yml
@@ -32,8 +32,11 @@ jobs:
yarn cache clean
yarn && yarn build
touch build/.dummy
+cp .asf-site.yaml ./build/.asf.yaml
+cp versions.json ./build/
ls build
export DORIS_COMMIT=`git rev-parse HEAD`
+
- name: Upload files to OSS
uses: ./.github/actions/aliyun-oss-website-action
with:
@@ -44,27 +47,17 @@ jobs:
endpoint: ${{ secrets.ALIYUN_OSS_ENDPOINT }}
folder: build
-- name: Deploy website
+- name: Deploy Website
if: ${{ github.event.inputs.branch == 'master' }}
- run: |
-git config --global http.postBuffer 524288000
-git fetch
-git checkout -b asf-site remotes/origin/asf-site
-/bin/bash remove-non-reserved-dir.sh
-cp -r build/* ./
-rm -rf build/
-rm -rf .docusaurus
-rm -rf node_modules
-rm -rf doris
-rm -rf yarn.lock
-rm -rf versioned_docs/
-rm -rf versioned_sidebars
-rm -rf i18n/
-git config user.name "github-actions[bot]"
-git config user.email "github-actions[bot]@users.noreply.github.com"
-git add .
-git commit -m "Automated deployment with doris master"
-git push --verbose "https://${{ secrets.GITHUB_TOKEN
}}@github.com/apache/doris-website.git" asf-site:asf-site
+ uses: peaceiris/actions-gh-pages@v3
+ with:
+github_token: ${{ github.token }}
+publish_branch: asf-site
+publish_dir: ./build
+destination_dir: ./
+user_name: 'github-actions[bot]'
+user_email: 'github-actions[bot]@users.noreply.github.com'
+commit_message: 'Automated deployment with doris branch ${{
github.event.inputs.branch }}@${{ env.DORIS_COMMIT }}'
- name: Deploy Branch
if: ${{ github.event.inputs.branch != 'master' }}
diff --git a/.github/workflows/manual-deploy.yml
b/.github/workflows/manual-deploy.yml
index c1bb95e6697..941ef545a11 100644
--- a/.github/workflows/manual-deploy.yml
+++ b/.github/workflows/manual-deploy.yml
@@ -35,8 +35,11 @@ jobs:
yarn cache clean
yarn && yarn build
touch build/.dummy
+cp .asf-site.yaml ./build/.asf.yaml
+cp versions.json ./build/
ls build
export DORIS_COMMIT=`git rev-parse HEAD`
+
- name: Upload files to OSS
uses: ./.github/actions/aliyun-oss-website-action
with:
@@ -46,28 +49,19 @@ jobs:
# use your own endpoint
endpoint: ${{ secrets.ALIYUN_OSS_ENDPOINT }}
folder: build
-
-- name: Deploy website
+
+- name: Deploy Website
if: ${{ github.event.inputs.branch == 'master' }}
- run: |
-git config --global http.postBuffer 524288000
-git fetch
-git checkout -b asf-site remotes/origin/asf-site
-/bin/bash remove-non-reserved-dir.sh
-cp -r build/* ./
-rm -rf build/
-rm -rf .docusaurus
-rm -rf node_modules
-rm -rf doris
-rm -rf yarn.lock
-rm -rf versioned_docs/
-rm -rf versioned_sidebars
-rm -rf i18n/
-git config user.name "github-actions[bot]"
-git config user.email "github-actions[bot]@users.noreply.github.com"
-git add .
-git commit -m "Automated deployment with doris master"
-git push --verbose "https://${{ secrets.GITHUB_TOKEN
}}@github.com/apache/doris-website.git" asf-site:asf-site
+ uses: peaceiris/actions-gh-pages@v3
+ with:
+github_token: ${{ github.token }}
+publish_branch: asf-site
+publish_dir: ./build
+destination_dir: ./
+user_name: 'github-actions[bot]'
+user_email: 'github-actions[bot]@users.noreply.github.com'
+commit_message: 'Automated deployment with doris branch ${{
github.event.inputs.branch }}@${{ env.DORIS_COMMIT }}'
+
- name: Deploy Branch
if: ${{ github.event.inputs.branch != 'master' }}
uses: peaceiris/actions-gh-pages@v3
[GitHub] [doris-website] hf200012 merged pull request #66: [fix] update workflows
hf200012 merged PR #66: URL: https://github.com/apache/doris-website/pull/66 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11933: [fix](union)the result exprs of union node should substitute by child node's smap
yiguolei merged PR #11933: URL: https://github.com/apache/doris/pull/11933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](union)the result exprs of union node should substitute by child node's smap (#11933)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5219d2aab0 [fix](union)the result exprs of union node should
substitute by child node's smap (#11933)
5219d2aab0 is described below
commit 5219d2aab062d4faf60ba5f516c2528bc488fcc4
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Wed Aug 24 19:43:40 2022 +0800
[fix](union)the result exprs of union node should substitute by child
node's smap (#11933)
union node's result exprs should be substitued by child node's smap first,
then the following "computePassthrough" method would have correct information
to do its job.
---
.../org/apache/doris/planner/SetOperationNode.java | 12 +-
.../data/correctness/test_union_with_subquery.out | 4 ++
.../correctness/test_union_with_subquery.groovy| 49 ++
3 files changed, 63 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/SetOperationNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/SetOperationNode.java
index 6e56f6ffd2..bd130a8957 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/SetOperationNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/SetOperationNode.java
@@ -144,6 +144,15 @@ public abstract class SetOperationNode extends PlanNode {
@Override
public void finalize(Analyzer analyzer) throws UserException {
super.finalize(analyzer);
+// the resultExprLists should be substituted by child's output smap
+// because the result exprs are column A, B, but the child output
exprs are column B, A
+// after substituted, the next computePassthrough method will get
correct info to do its job
+List> substitutedResultExprLists = Lists.newArrayList();
+for (int i = 0; i < resultExprLists.size(); ++i) {
+substitutedResultExprLists.add(Expr.substituteList(
+resultExprLists.get(i), children.get(i).getOutputSmap(),
analyzer, true));
+}
+resultExprLists = substitutedResultExprLists;
// In Doris-6380, moved computePassthrough() and the materialized
position of resultExprs/constExprs
// from this.init() to this.finalize(), and will not call
SetOperationNode::init() again at the end
// of createSetOperationNodeFragment().
@@ -179,8 +188,7 @@ public abstract class SetOperationNode extends PlanNode {
newExprList.add(exprList.get(j));
}
}
-materializedResultExprLists.add(
-Expr.substituteList(newExprList,
getChild(i).getOutputSmap(), analyzer, true));
+materializedResultExprLists.add(newExprList);
}
Preconditions.checkState(
materializedResultExprLists.size() == getChildren().size());
diff --git a/regression-test/data/correctness/test_union_with_subquery.out
b/regression-test/data/correctness/test_union_with_subquery.out
new file mode 100644
index 00..a4cca003b9
--- /dev/null
+++ b/regression-test/data/correctness/test_union_with_subquery.out
@@ -0,0 +1,4 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !sql --
+1 1
+
diff --git a/regression-test/suites/correctness/test_union_with_subquery.groovy
b/regression-test/suites/correctness/test_union_with_subquery.groovy
new file mode 100644
index 00..5362231fda
--- /dev/null
+++ b/regression-test/suites/correctness/test_union_with_subquery.groovy
@@ -0,0 +1,49 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_union_with_subquery") {
+sql """ DROP TABLE IF EXISTS A_union; """
+sql """
+create table A_union ( a int not null, b varchar(10) null )ENGINE=OLAP
+DISTRIBUTED BY HASH(a) BUCKETS 1
+PROPERTIES (
+"replication_allocation" = "tag.location.default: 1",
+"in_memory" = "false",
+"storage_format" = "V2"
+);
+"""
+
+sql """ insert into A_union values(
[GitHub] [doris] starocean999 opened a new pull request, #12051: [fix](union)the result exprs of union node should substitute by child node's smap
starocean999 opened a new pull request, #12051: URL: https://github.com/apache/doris/pull/12051 # Proposed changes Issue Number: close #xxx ## Problem summary see https://github.com/apache/doris/pull/11933 ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12006: [Bug](function) fix aggFnParams set not correct
github-actions[bot] commented on PR #12006: URL: https://github.com/apache/doris/pull/12006#issuecomment-1225627818 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
