[GitHub] [doris] caiconghui closed pull request #12526: [fix](log) fix wrong logger for ExprSubstitutionMap
caiconghui closed pull request #12526: [fix](log) fix wrong logger for ExprSubstitutionMap URL: https://github.com/apache/doris/pull/12526 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui opened a new pull request, #12546: [fix](log) fix wrong logger for ExprSubstitutionMap
caiconghui opened a new pull request, #12546: URL: https://github.com/apache/doris/pull/12546 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12546: [fix](log) fix wrong logger for ExprSubstitutionMap
github-actions[bot] commented on PR #12546: URL: https://github.com/apache/doris/pull/12546#issuecomment-1244991523 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12546: [fix](log) fix wrong logger for ExprSubstitutionMap
github-actions[bot] commented on PR #12546: URL: https://github.com/apache/doris/pull/12546#issuecomment-1244991586 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #11805: [feature](Nereids)Push down not slot references expression of on clause
Kikyou1997 commented on code in PR #11805:
URL: https://github.com/apache/doris/pull/11805#discussion_r969234202
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/SingleSidePredicate.java:
##
@@ -0,0 +1,103 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * push down expression which is not slot reference
+ */
+public class SingleSidePredicate extends OneRewriteRuleFactory {
+/*
+ * rewrite example:
+ * join(t1.a + 1 = t2.b + 2)join(c = d)
+ * / \ /\
+ */ \ / \
+ * / \ > /\
+ * / \ / \
+ * olapScan(t1) olapScan(t2)project(t1.a + 1 as c)
project(t2.b + 2 as d)
+ *|
|
+ *|
|
+ *|
|
+ *|
|
+ * olapScan(t1)
olapScan(t2)
+ *TODO: now t1.a + t2.a = t1.b is not in hashJoinConjuncts. The rule will
not handle it.
+ */
+@Override
+public Rule build() {
+return logicalJoin()
+.when(join ->
join.getHashJoinConjuncts().stream().anyMatch(expr ->
+expr.children().stream().anyMatch(expr1 -> !(expr1
instanceof Slot
+.then(join -> {
+List> exprsOfJoinRelation =
+Lists.newArrayList(Lists.newArrayList(),
Lists.newArrayList());
+Map exprMap = Maps.newHashMap();
+Plan left = join.left();
+join.getHashJoinConjuncts().forEach(conjunct -> {
+Preconditions.checkArgument(conjunct instanceof
EqualTo);
+Expression[] exprs = conjunct.children().toArray(new
Expression[0]);
+int tag = checkIfSwap(exprs[0], left);
+exprsOfJoinRelation.get(0).add(exprs[tag]);
+exprsOfJoinRelation.get(1).add(exprs[tag ^ 1]);
Review Comment:
interesting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #11805: [feature](Nereids)Push down not slot references expression of on clause
Kikyou1997 commented on code in PR #11805:
URL: https://github.com/apache/doris/pull/11805#discussion_r969241594
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/SingleSidePredicate.java:
##
@@ -0,0 +1,103 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * push down expression which is not slot reference
+ */
+public class SingleSidePredicate extends OneRewriteRuleFactory {
+/*
+ * rewrite example:
+ * join(t1.a + 1 = t2.b + 2)join(c = d)
+ * / \ /\
+ */ \ / \
+ * / \ > /\
+ * / \ / \
+ * olapScan(t1) olapScan(t2)project(t1.a + 1 as c)
project(t2.b + 2 as d)
+ *|
|
+ *|
|
+ *|
|
+ *|
|
+ * olapScan(t1)
olapScan(t2)
+ *TODO: now t1.a + t2.a = t1.b is not in hashJoinConjuncts. The rule will
not handle it.
+ */
+@Override
+public Rule build() {
+return logicalJoin()
+.when(join ->
join.getHashJoinConjuncts().stream().anyMatch(expr ->
+expr.children().stream().anyMatch(expr1 -> !(expr1
instanceof Slot
+.then(join -> {
+List> exprsOfJoinRelation =
+Lists.newArrayList(Lists.newArrayList(),
Lists.newArrayList());
+Map exprMap = Maps.newHashMap();
+Plan left = join.left();
+join.getHashJoinConjuncts().forEach(conjunct -> {
+Preconditions.checkArgument(conjunct instanceof
EqualTo);
+Expression[] exprs = conjunct.children().toArray(new
Expression[0]);
+int tag = checkIfSwap(exprs[0], left);
Review Comment:
Better add some comment here to explain why this swap operation is
neccessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui merged pull request #12546: [fix](log) fix wrong logger for ExprSubstitutionMap
caiconghui merged PR #12546: URL: https://github.com/apache/doris/pull/12546 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [fix](log) fix wrong logger for ExprSubstitutionMap (#12546)
This is an automated email from the ASF dual-hosted git repository.
caiconghui pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new 82061ad3a0 [fix](log) fix wrong logger for ExprSubstitutionMap (#12546)
82061ad3a0 is described below
commit 82061ad3a05c70ff3087b8b55da13775a45748c4
Author: caiconghui <55968745+caicong...@users.noreply.github.com>
AuthorDate: Tue Sep 13 15:22:47 2022 +0800
[fix](log) fix wrong logger for ExprSubstitutionMap (#12546)
Co-authored-by: caiconghui1
---
.../org/apache/doris/analysis/ExprSubstitutionMap.java| 15 ++-
1 file changed, 6 insertions(+), 9 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ExprSubstitutionMap.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ExprSubstitutionMap.java
index 2afe97523f..9726543f36 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ExprSubstitutionMap.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ExprSubstitutionMap.java
@@ -19,18 +19,15 @@ package org.apache.doris.analysis;
import org.apache.doris.common.AnalysisException;
-import java.util.List;
-import java.util.Objects;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
import com.google.common.base.Joiner;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
-import com.google.common.base.Joiner;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.Objects;
/**
* Map of expression substitutions: lhs[i] gets substituted with rhs[i].
@@ -41,7 +38,7 @@ import com.google.common.collect.Lists;
* See Expr.substitute() and related functions for details on the actual
substitution.
*/
public final class ExprSubstitutionMap {
-private final static Logger LOG =
LoggerFactory.getLogger(ExprSubstitutionMap.class);
+private final static Logger LOG =
LogManager.getLogger(ExprSubstitutionMap.class);
private boolean checkAnalyzed_ = true;
private List lhs_; // left-hand side
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zenoyang commented on a diff in pull request #10084: [feature](vectorized) Support block aggregate in scanner
zenoyang commented on code in PR #10084:
URL: https://github.com/apache/doris/pull/10084#discussion_r969180648
##
be/src/util/simd/bits.h:
##
@@ -89,16 +92,31 @@ inline size_t count_zero_num(const int8_t* __restrict data,
size_t size) {
return num;
}
-inline size_t count_zero_num(const int8_t* __restrict data, const uint8_t*
__restrict null_map,
Review Comment:
Because `__restrict` is redundant for `const` pointers. I see this function
is called in `VRuntimeFilterWrapper::execute` and cannot be remove.
##
be/src/util/simd/bits.h:
##
@@ -89,16 +92,31 @@ inline size_t count_zero_num(const int8_t* __restrict data,
size_t size) {
return num;
}
-inline size_t count_zero_num(const int8_t* __restrict data, const uint8_t*
__restrict null_map,
Review Comment:
Because `__restrict` is redundant for `const` pointers. I see this function
is called in `VRuntimeFilterWrapper::execute` and cannot be remove.
##
be/src/vec/columns/column_decimal.cpp:
##
@@ -240,13 +240,13 @@ ColumnPtr ColumnDecimal::filter(const IColumn::Filter&
filt, ssize_t result_s
* completely pass or do not pass the filter.
* Therefore, we will optimistically check the parts of `SIMD_BYTES`
values.
*/
-static constexpr size_t SIMD_BYTES = 32;
+static constexpr size_t SIMD_BYTES = 64;
const UInt8* filt_end_sse = filt_pos + size / SIMD_BYTES * SIMD_BYTES;
while (filt_pos < filt_end_sse) {
-uint32_t mask = simd::bytes32_mask_to_bits32_mask(filt_pos);
+auto mask = simd::bytes64_mask_to_bits64_mask(filt_pos);
-if (0x == mask) {
+if (0x == mask) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zenoyang commented on a diff in pull request #10084: [feature](vectorized) Support block aggregate in scanner
zenoyang commented on code in PR #10084:
URL: https://github.com/apache/doris/pull/10084#discussion_r969245544
##
be/src/vec/olap/block_reader.cpp:
##
@@ -168,6 +177,49 @@ Status BlockReader::init(const ReaderParams& read_params) {
return Status::OK();
}
+Status BlockReader::next_block_with_aggregation(Block* block, MemPool*
mem_pool,
+ObjectPool* agg_pool, bool*
eof) {
+return _block_aggregator ? _agg_next_block(block, mem_pool, agg_pool, eof)
Review Comment:
Great suggestion, I've changed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangbo closed pull request #12404: [fix](planner) fix projection may return error results
wangbo closed pull request #12404: [fix](planner) fix projection may return error results URL: https://github.com/apache/doris/pull/12404 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangbo opened a new pull request, #12551: [fix](planner) fix projection may return error results
wangbo opened a new pull request, #12551: URL: https://github.com/apache/doris/pull/12551 2 fix tuple is null offset incorrect # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #11805: [feature](Nereids)Push down not slot references expression of on clause
Kikyou1997 commented on code in PR #11805:
URL: https://github.com/apache/doris/pull/11805#discussion_r969261760
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/SingleSidePredicate.java:
##
@@ -0,0 +1,103 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * push down expression which is not slot reference
+ */
+public class SingleSidePredicate extends OneRewriteRuleFactory {
+/*
+ * rewrite example:
+ * join(t1.a + 1 = t2.b + 2)join(c = d)
+ * / \ /\
+ */ \ / \
+ * / \ > /\
+ * / \ / \
+ * olapScan(t1) olapScan(t2)project(t1.a + 1 as c)
project(t2.b + 2 as d)
+ *|
|
+ *|
|
+ *|
|
+ *|
|
+ * olapScan(t1)
olapScan(t2)
+ *TODO: now t1.a + t2.a = t1.b is not in hashJoinConjuncts. The rule will
not handle it.
+ */
+@Override
+public Rule build() {
+return logicalJoin()
+.when(join ->
join.getHashJoinConjuncts().stream().anyMatch(expr ->
+expr.children().stream().anyMatch(expr1 -> !(expr1
instanceof Slot
+.then(join -> {
+List> exprsOfJoinRelation =
+Lists.newArrayList(Lists.newArrayList(),
Lists.newArrayList());
+Map exprMap = Maps.newHashMap();
+Plan left = join.left();
+join.getHashJoinConjuncts().forEach(conjunct -> {
+Preconditions.checkArgument(conjunct instanceof
EqualTo);
+Expression[] exprs = conjunct.children().toArray(new
Expression[0]);
+int tag = checkIfSwap(exprs[0], left);
+exprsOfJoinRelation.get(0).add(exprs[tag]);
+exprsOfJoinRelation.get(1).add(exprs[tag ^ 1]);
+Arrays.stream(exprs).sequential().forEach(expr ->
+exprMap.put(expr, new Alias(expr, "expr_" +
expr.hashCode(;
+});
+LogicalJoin join1 =
join.withHashJoinConjuncts(join.getHashJoinConjuncts()
+.stream().map(equalTo ->
equalTo.withChildren(equalTo.children()
+.stream().map(expr ->
exprMap.get(expr).toSlot())
+.collect(Collectors.toList(
+.collect(Collectors.toList()));
+Iterator
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #11805: [feature](Nereids)Push down not slot references expression of on clause
Kikyou1997 commented on code in PR #11805:
URL: https://github.com/apache/doris/pull/11805#discussion_r969263233
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java:
##
@@ -250,4 +250,9 @@ public LEFT_CHILD_TYPE left() {
public RIGHT_CHILD_TYPE right() {
return (RIGHT_CHILD_TYPE) child(1);
}
+
+public LogicalJoin withHashJoinConjuncts(List
hashJoinConjuncts) {
Review Comment:
I think you can add a method named with `withHashJoinConjunctsAndChidren`,
so that you could finish the creation of new JoinNode in one call.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java:
##
@@ -250,4 +250,9 @@ public LEFT_CHILD_TYPE left() {
public RIGHT_CHILD_TYPE right() {
return (RIGHT_CHILD_TYPE) child(1);
}
+
+public LogicalJoin withHashJoinConjuncts(List
hashJoinConjuncts) {
Review Comment:
I think you could add a method named with `withHashJoinConjunctsAndChidren`,
so that you could finish the creation of new JoinNode in one call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12461: [Feature](Nereids)add relation id
Kikyou1997 commented on code in PR #12461:
URL: https://github.com/apache/doris/pull/12461#discussion_r969266049
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalRelation.java:
##
@@ -125,4 +132,7 @@ public List getSelectedPartitionIds() {
return selectedPartitionIds;
}
+public RelationId getId() {
Review Comment:
If this method is common for both logical and physical scan, you'd better to
an declaration in the `Scan` interface
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalRelation.java:
##
@@ -125,4 +132,7 @@ public List getSelectedPartitionIds() {
return selectedPartitionIds;
}
+public RelationId getId() {
Review Comment:
If this method is common for both logical and physical scan, you'd better to
add an declaration in the `Scan` interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #12552: [Bug](scan node) try to fix invalid call to nullptr slot
BiteThet opened a new pull request, #12552: URL: https://github.com/apache/doris/pull/12552 # Proposed changes testing ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chovy-3012 commented on issue #12098: Release Note 1.1.2
chovy-3012 commented on issue #12098: URL: https://github.com/apache/doris/issues/12098#issuecomment-1245056178 👏🤙 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] JNSimba opened a new pull request, #61: [Feature] Support flink table lookup join
JNSimba opened a new pull request, #61: URL: https://github.com/apache/doris-flink-connector/pull/61 # Proposed changes Issue Number: close #xxx ## Problem Summary: At present, the data of the flink stream join doris table is to read the doris table into the memory at one time and join it, such as kafka join doris. This makes it impossible to read the newly added data of the doris table. Therefore, lookup join is supported on the flink doris connector. The principle is to search in doris in real time according to the data in the fact stream. tips: For query performance, the cache setting is supported, which is disabled by default. `lookup.cache.max-rows` //The maximum row of the cache `lookup.cache.ttl`.//cache's ttl `lookup.max-retries`. //The maximum number of retries to read doris example: ```sql CREATE TABLE fact_table ( `id` BIGINT, `name` STRING, `city` STRING, `process_time` as proctime() ) WITH ( 'connector' = 'kafka', 'topic' = 'test_lookup', 'properties.bootstrap.servers' = '127.0.0.1:9092', 'properties.group.id' = 'testGroup', 'scan.startup.mode' = 'earliest-offset', 'format' = 'csv' ); create table dim_city( `city` STRING, `level` INT , `province` STRING, `country` STRING ) WITH ( 'connector' = 'doris', 'fenodes' = '127.0.0.1:8030', 'table.identifier' = 'test.dim_city', 'username' = 'root', 'password' = '', 'lookup.cache.max-rows' = '1000', 'lookup.cache.ttl' = '6' ); SELECT a.id, a.name, a.city, c.province, c.country,c.level FROM fact_table a left join dim_city FOR SYSTEM_TIME AS OF a.process_time AS c ON a.city = c.city ``` ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] wangyf0555 opened a new pull request, #94: [feature]add build versions plugin
wangyf0555 opened a new pull request, #94: URL: https://github.com/apache/doris-website/pull/94 What this plugin does： - The version number of the markDown content is displayed - Package the specified version of the document -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12316: [Enhancement](compaction) reduce VMergeIterator copy block
yiguolei merged PR #12316: URL: https://github.com/apache/doris/pull/12316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Enhancement](compaction) reduce VMergeIterator copy block (#12316)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2306e46658 [Enhancement](compaction) reduce VMergeIterator copy block
(#12316)
2306e46658 is described below
commit 2306e46658681a1d294975d810c82c1e209d016b
Author: Pxl
AuthorDate: Tue Sep 13 16:19:34 2022 +0800
[Enhancement](compaction) reduce VMergeIterator copy block (#12316)
This pr change make VMergeIterator support return row reference to instead
copy a full block.
---
be/src/olap/compaction.cpp| 4 +-
be/src/olap/iterators.h | 10 +-
be/src/olap/rowset/beta_rowset_reader.cpp | 26 ++-
be/src/olap/rowset/beta_rowset_reader.h | 2 +
be/src/olap/rowset/rowset_reader.h| 5 +-
be/src/vec/core/block.h | 19 +++
be/src/vec/olap/block_reader.h| 2 +-
be/src/vec/olap/vcollect_iterator.cpp | 90 ++-
be/src/vec/olap/vcollect_iterator.h | 64 ++--
be/src/vec/olap/vgeneric_iterators.cpp| 259 +-
10 files changed, 306 insertions(+), 175 deletions(-)
diff --git a/be/src/olap/compaction.cpp b/be/src/olap/compaction.cpp
index 8f9e8910d9..2e9b4a6b55 100644
--- a/be/src/olap/compaction.cpp
+++ b/be/src/olap/compaction.cpp
@@ -236,7 +236,7 @@ Status Compaction::do_compaction_impl(int64_t permits) {
<< ". elapsed time=" << watch.get_elapse_second()
<< "s. cumulative_compaction_policy="
<< _tablet->cumulative_compaction_policy()->name()
- << ", compact_row_per_second=" << _input_row_num /
watch.get_elapse_second();
+ << ", compact_row_per_second=" << int(_input_row_num /
watch.get_elapse_second());
return Status::OK();
}
@@ -336,7 +336,7 @@ Status Compaction::check_correctness(const
Merger::Statistics& stats) {
// 1. check row number
if (_input_row_num != _output_rowset->num_rows() + stats.merged_rows +
stats.filtered_rows) {
LOG(WARNING) << "row_num does not match between cumulative input and
output! "
- << "input_row_num=" << _input_row_num
+ << "tablet=" << _tablet->full_name() << ",
input_row_num=" << _input_row_num
<< ", merged_row_num=" << stats.merged_rows
<< ", filtered_row_num=" << stats.filtered_rows
<< ", output_row_num=" << _output_rowset->num_rows();
diff --git a/be/src/olap/iterators.h b/be/src/olap/iterators.h
index 3d9690f70b..22f081d0eb 100644
--- a/be/src/olap/iterators.h
+++ b/be/src/olap/iterators.h
@@ -94,8 +94,8 @@ public:
// Used to read data in RowBlockV2 one by one
class RowwiseIterator {
public:
-RowwiseIterator() {}
-virtual ~RowwiseIterator() {}
+RowwiseIterator() = default;
+virtual ~RowwiseIterator() = default;
// Initialize this iterator and make it ready to read with
// input options.
@@ -116,6 +116,12 @@ public:
return Status::NotSupported("to be implemented");
}
+virtual Status next_block_view(vectorized::BlockView* block_view) {
+return Status::NotSupported("to be implemented");
+}
+
+virtual bool support_return_data_by_ref() { return false; }
+
virtual Status current_block_row_locations(std::vector*
block_row_locations) {
return Status::NotSupported("to be implemented");
}
diff --git a/be/src/olap/rowset/beta_rowset_reader.cpp
b/be/src/olap/rowset/beta_rowset_reader.cpp
index 17223c9eb6..df15b72f62 100644
--- a/be/src/olap/rowset/beta_rowset_reader.cpp
+++ b/be/src/olap/rowset/beta_rowset_reader.cpp
@@ -24,7 +24,6 @@
#include "olap/row_block.h"
#include "olap/row_block2.h"
#include "olap/row_cursor.h"
-#include "olap/rowset/segment_v2/segment_iterator.h"
#include "olap/schema.h"
#include "olap/tablet_meta.h"
#include "vec/core/block.h"
@@ -119,7 +118,9 @@ Status BetaRowsetReader::init(RowsetReaderContext*
read_context) {
for (uint32_t seg_id = 0; seg_id < rowset()->num_segments(); ++seg_id)
{
auto d = read_context->delete_bitmap->get_agg(
{rowset_id, seg_id, read_context->version.second});
-if (d->isEmpty()) continue; // Empty delete bitmap for the segment
+if (d->isEmpty()) {
+continue; // Empty delete bitmap for the segment
+}
VLOG_TRACE << "Get the delete bitmap for rowset: " <<
rowset_id.to_string()
<< ", segment id:" << seg_id << ", size:" <<
d->cardinality();
read_options.delete_bitmap.emplace(seg_id, std::move(d));
@@ -323,6 +324,27 @@ Status BetaRowsetReader::next_block(vectorized::Block*
block) {
return Status::OK();
}
+Status BetaRowsetReader::next_block_view(vectorized::BlockView* block_view) {
+SCOPED_RAW_TIMER(
[GitHub] [doris] eldenmoon commented on pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
eldenmoon commented on PR #12293: URL: https://github.com/apache/doris/pull/12293#issuecomment-1245063171 @adonis0147 plz review again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon closed pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
eldenmoon closed pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum… URL: https://github.com/apache/doris/pull/12293 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12470: [fix](frontend) fix notify update storage policy agent task null exception
yiguolei merged PR #12470: URL: https://github.com/apache/doris/pull/12470 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](frontend) fix notify update storage policy agent task null exception #12470
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b98a3ed86c [fix](frontend) fix notify update storage policy agent task
null exception #12470
b98a3ed86c is described below
commit b98a3ed86ca753d430047be287c8056f920c8fc2
Author: deardeng <565620...@qq.com>
AuthorDate: Tue Sep 13 16:20:11 2022 +0800
[fix](frontend) fix notify update storage policy agent task null exception
#12470
---
.../src/main/java/org/apache/doris/policy/StoragePolicy.java | 7 ++-
.../src/main/java/org/apache/doris/task/AgentBatchTask.java| 10 ++
2 files changed, 12 insertions(+), 5 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/policy/StoragePolicy.java
b/fe/fe-core/src/main/java/org/apache/doris/policy/StoragePolicy.java
index 1200745c52..653052cd59 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/policy/StoragePolicy.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/policy/StoragePolicy.java
@@ -181,11 +181,8 @@ public class StoragePolicy extends Policy {
}
if (props.containsKey(COOLDOWN_TTL)) {
hasCooldownTtl = true;
-if (Integer.parseInt(props.get(COOLDOWN_TTL)) < 0) {
-throw new AnalysisException("cooldown_ttl must >= 0.");
-}
-this.cooldownTtl = props.get(COOLDOWN_TTL);
-this.cooldownTtlMs = getMsByCooldownTtl(this.cooldownTtl);
+// second
+this.cooldownTtl =
String.valueOf(getMsByCooldownTtl(props.get(COOLDOWN_TTL)) / 1000);
}
if (hasCooldownDatetime && hasCooldownTtl) {
throw new AnalysisException(COOLDOWN_DATETIME + " and " +
COOLDOWN_TTL + " can't be set together.");
diff --git a/fe/fe-core/src/main/java/org/apache/doris/task/AgentBatchTask.java
b/fe/fe-core/src/main/java/org/apache/doris/task/AgentBatchTask.java
index 20e5c3b5df..2319bf8886 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/task/AgentBatchTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/task/AgentBatchTask.java
@@ -33,6 +33,7 @@ import org.apache.doris.thrift.TCompactionReq;
import org.apache.doris.thrift.TCreateTabletReq;
import org.apache.doris.thrift.TDownloadReq;
import org.apache.doris.thrift.TDropTabletReq;
+import org.apache.doris.thrift.TGetStoragePolicy;
import org.apache.doris.thrift.TMoveDirReq;
import org.apache.doris.thrift.TNetworkAddress;
import org.apache.doris.thrift.TPublishVersionRequest;
@@ -349,6 +350,15 @@ public class AgentBatchTask implements Runnable {
tAgentTaskRequest.setCompactionReq(request);
return tAgentTaskRequest;
}
+case NOTIFY_UPDATE_STORAGE_POLICY: {
+NotifyUpdateStoragePolicyTask notifyUpdateStoragePolicyTask =
(NotifyUpdateStoragePolicyTask) task;
+TGetStoragePolicy request =
notifyUpdateStoragePolicyTask.toThrift();
+if (LOG.isDebugEnabled()) {
+LOG.debug(request.toString());
+}
+tAgentTaskRequest.setUpdatePolicy(request);
+return tAgentTaskRequest;
+}
default:
LOG.debug("could not find task type for task [{}]", task);
return null;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] hf200012 merged pull request #94: [feature]add build versions plugin
hf200012 merged PR #94: URL: https://github.com/apache/doris-website/pull/94 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: [feature]add build versions plugin (#94)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new e99fba72a75 [feature]add build versions plugin (#94)
e99fba72a75 is described below
commit e99fba72a7583cb702f20e1fa6407dcfe4d5fa72
Author: wangyongfeng <943155...@qq.com>
AuthorDate: Tue Sep 13 16:21:49 2022 +0800
[feature]add build versions plugin (#94)
* add build versions plugin
---
buildVersions.json | 1 +
config/versions-plugin.js | 20 +
docusaurus.config.js | 2 ++
package.json | 3 ++-
src/components/VersionsDoc/README.md | 21 ++
src/components/VersionsDoc/index.tsx | 39 ++
src/components/VersionsDoc/styles.scss | 23
src/theme/MDXComponents.js | 9
8 files changed, 117 insertions(+), 1 deletion(-)
diff --git a/buildVersions.json b/buildVersions.json
new file mode 100644
index 000..0637a088a01
--- /dev/null
+++ b/buildVersions.json
@@ -0,0 +1 @@
+[]
\ No newline at end of file
diff --git a/config/versions-plugin.js b/config/versions-plugin.js
new file mode 100644
index 000..401dce76d5c
--- /dev/null
+++ b/config/versions-plugin.js
@@ -0,0 +1,20 @@
+const fs = require('fs-extra')
+
+const filePath = './buildVersions.json'
+
+async function readVersionsFile () {
+const content = await fs.readFile(filePath, 'utf-8')
+const versionData = JSON.parse(content)
+return versionData
+}
+
+module.exports = function (context, options) {
+return {
+name: 'versions-plugin',
+async contentLoaded({ content, actions }) {
+const versions = await readVersionsFile()
+const { setGlobalData } = actions;
+setGlobalData({ versions });
+},
+};
+};
\ No newline at end of file
diff --git a/docusaurus.config.js b/docusaurus.config.js
index 383ca8d7ce6..47fee481130 100644
--- a/docusaurus.config.js
+++ b/docusaurus.config.js
@@ -6,6 +6,7 @@ const lightCodeTheme =
require('prism-react-renderer/themes/github');
const showAllVersions = true;
const { ssrTemplate } = require('./config/ssrTemplate');
const customDocusaurusPlugin = require('./config/custom-docusaurus-plugin');
+const versionsPlugin = require('./config/versions-plugin')
console.log(process.env);
@@ -40,6 +41,7 @@ const config = {
projectName: 'doris', // Usually your repo name.
plugins: [
'docusaurus-plugin-sass',
+versionsPlugin,
[
'content-docs',
/** @type {import('@docusaurus/plugin-content-docs').Options} */
diff --git a/package.json b/package.json
index 96868b0556f..8222f5262f7 100644
--- a/package.json
+++ b/package.json
@@ -6,7 +6,8 @@
"docusaurus": "docusaurus",
"start": "docusaurus start",
"start:zh-CN": "docusaurus start --locale zh-CN",
-"build": "PWA_SERVICE_WORKER_URL=https://doris.apache.org docusaurus
build",
+"build": "PWA_SERVICE_WORKER_URL=https://doris.apache.org/sw.js
docusaurus build",
+"build:version": "docusaurus set-versions && docusaurus build",
"swizzle": "docusaurus swizzle",
"deploy": "docusaurus deploy",
"clear": "docusaurus clear",
diff --git a/src/components/VersionsDoc/README.md
b/src/components/VersionsDoc/README.md
new file mode 100644
index 000..3579274b760
--- /dev/null
+++ b/src/components/VersionsDoc/README.md
@@ -0,0 +1,21 @@
+# How to use ``
+
+You can wrap a piece of Markdown content using the , and set the
version number, like this:
+
+```html
+
+This is markdown content...
+
+```
+> Tips: the value must be a string that split by commas.
+
+Then in our website, the MarkDown content will show the version number to
which it belongs.
+
+If you want to build a single version docs, you can update buildVersions.json:
+
+```json
+["1.1", "1.2"]
+```
+Then run the build:versions command.
+
+
diff --git a/src/components/VersionsDoc/index.tsx
b/src/components/VersionsDoc/index.tsx
new file mode 100644
index 000..97e11a60a4d
--- /dev/null
+++ b/src/components/VersionsDoc/index.tsx
@@ -0,0 +1,39 @@
+import React from 'react';
+import clsx from 'clsx';
+import Translate from '@docusaurus/Translate';
+import './styles.scss';
+import { usePluginData } from '@docusaurus/useGlobalData';
+
+
+export default function VersionsDoc(props): JSX.Element {
+const { children, value = '' } = props;
+const versionsPluginData: any = usePluginData('versions-plugin');
+const getBuildVersions = (versionsPluginData): string[] => {
+if (versionsPluginData) {
+const versionsData = versionsPluginData.versions;
+if (Array.isArray(versionsData) && versionsData.length > 0) {
+
[GitHub] [doris] Wilson-BT opened a new issue, #12556: [Bug] databalance not work when add new node to doris cluster
Wilson-BT opened a new issue, #12556: URL: https://github.com/apache/doris/issues/12556 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris: 1.1.1 and 1.1.2 table : 3 replica ### What's Wrong? When i add new nodes to doris cluster, the data-balance not work.  On the picture, i have add 3 nodes to doris cluster，but tabletNum have not change. ### What You Expected? tablet will balance between new nodes and old node. ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Wilson-BT closed issue #12556: [Bug] databalance not work when add new node to doris cluster
Wilson-BT closed issue #12556: [Bug] databalance not work when add new node to doris cluster URL: https://github.com/apache/doris/issues/12556 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
adonis0147 commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969306888
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
Review Comment:
```suggestion
Status
ArrayFileColumnIterator::_calculate_offsets(vectorized::ColumnArray::ColumnOffsets&
offsets, ssize_t start)
```
1. Typo: `_caculate_offsets` -> `_calculate_offsets`
2. For function arguments, I think it is better to put the mutable arguments
before the immutable ones.
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
Review Comment:
We would better do single job in one function. According to the name of this
function, we should calculate the new offsets only and put the assignment of
`num_items` **outside** the function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] JNSimba closed issue #57: [Enhancement] Added *.DS_Store into .gitignore for Mac OS
JNSimba closed issue #57: [Enhancement] Added *.DS_Store into .gitignore for Mac OS URL: https://github.com/apache/doris-flink-connector/issues/57 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
adonis0147 commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969341725
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
+// caculate real offsets
+ordinal_t first_offset = offsets_data[start - 1]; // -1 is valid
+ordinal_t first_ord = offsets_data[start];
+for (ssize_t i = start; i + 1 < offsets_data.size(); ++i) {
+offsets_data[i] = first_offset + (offsets_data[i + 1] - first_ord);
+}
Review Comment:
Why should we re-calculate the offsets here? Some questions here:
1. Does `vectorized::MutableColumnPtr& dst` in
`ArrayFileColumnIterator::next_batch` share some data with others?
2. Does the `offsets_data` read from different pages cause the
re-calculation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #12557: [fix](column) ColumnNullable::replace_column_data should DCHECK size …
dataroaring opened a new pull request, #12557: URL: https://github.com/apache/doris/pull/12557 …> self_row # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #12558: (column) ColumnNullable::replace_column_data should DCHECK size > sel…
dataroaring opened a new pull request, #12558: URL: https://github.com/apache/doris/pull/12558 …f_row # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
adonis0147 commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969360351
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
+// caculate real offsets
Review Comment:
typo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 1018ji commented on issue #12098: Release Note 1.1.2
1018ji commented on issue #12098: URL: https://github.com/apache/doris/issues/12098#issuecomment-1245128394 🤙 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
eldenmoon commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969369777
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
+// caculate real offsets
+ordinal_t first_offset = offsets_data[start - 1]; // -1 is valid
+ordinal_t first_ord = offsets_data[start];
+for (ssize_t i = start; i + 1 < offsets_data.size(); ++i) {
+offsets_data[i] = first_offset + (offsets_data[i + 1] - first_ord);
+}
Review Comment:
for eg. an array full offset is (0, 1, 3, 4, 7) but we only need row_id(0,
2, 4)， so we need to recaculate offset value we need read from page
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
eldenmoon commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969373305
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
Review Comment:
2. But I saw many doris code which put immutable before mutable arguments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
adonis0147 commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969380915
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
+// caculate real offsets
+ordinal_t first_offset = offsets_data[start - 1]; // -1 is valid
+ordinal_t first_ord = offsets_data[start];
+for (ssize_t i = start; i + 1 < offsets_data.size(); ++i) {
+offsets_data[i] = first_offset + (offsets_data[i + 1] - first_ord);
+}
Review Comment:
Ok, that is the scenario `ArrayFileColumnIterator::read_by_rowids`. What
about the scenario in read data sequently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
adonis0147 commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969380915
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
+// caculate real offsets
+ordinal_t first_offset = offsets_data[start - 1]; // -1 is valid
+ordinal_t first_ord = offsets_data[start];
+for (ssize_t i = start; i + 1 < offsets_data.size(); ++i) {
+offsets_data[i] = first_offset + (offsets_data[i + 1] - first_ord);
+}
Review Comment:
Ok, that is the scenario `ArrayFileColumnIterator::read_by_rowids`. What
about the scenario that we read data sequently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
eldenmoon commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969382694
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
+// caculate real offsets
+ordinal_t first_offset = offsets_data[start - 1]; // -1 is valid
+ordinal_t first_ord = offsets_data[start];
+for (ssize_t i = start; i + 1 < offsets_data.size(); ++i) {
+offsets_data[i] = first_offset + (offsets_data[i + 1] - first_ord);
+}
Review Comment:
1. 2 both answer is yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #12558: (column) ColumnNullable::replace_column_data should DCHECK size > sel…
BiteThet commented on PR #12558: URL: https://github.com/apache/doris/pull/12558#issuecomment-1245151089 need update pr title -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] geniusjoe opened a new pull request, #62: [docs] update readme.md flink version
geniusjoe opened a new pull request, #62: URL: https://github.com/apache/doris-flink-connector/pull/62 ## Problem Summary: Since flink client changed groupId format, we may not build current project using 1.14 and below version. I think it's better to change README.md to the 1.15 version.   ## Checklist(Required) 1. Does it affect the original behavior: No 2. Has unit tests been added: No Need 3. Has document been added or modified: Yes 4. Does it need to update dependencies: No 5. Are there any changes that cannot be rolled back: No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
adonis0147 commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969389961
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
+// caculate real offsets
+ordinal_t first_offset = offsets_data[start - 1]; // -1 is valid
+ordinal_t first_ord = offsets_data[start];
+for (ssize_t i = start; i + 1 < offsets_data.size(); ++i) {
+offsets_data[i] = first_offset + (offsets_data[i + 1] - first_ord);
+}
Review Comment:
Maybe there is a potential optimization point if we don't read by rowids. It
seems that the item_writer doesn't reset the next_rowid in the same segment. So
I don't think these answer are yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
adonis0147 commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969389961
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
+// caculate real offsets
+ordinal_t first_offset = offsets_data[start - 1]; // -1 is valid
+ordinal_t first_ord = offsets_data[start];
+for (ssize_t i = start; i + 1 < offsets_data.size(); ++i) {
+offsets_data[i] = first_offset + (offsets_data[i + 1] - first_ord);
+}
Review Comment:
Maybe there is a potential optimization point if we don't read by rowids. It
seems that the item_writer doesn't reset the next_rowid in the same segment. So
I don't think these answers are yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
eldenmoon commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969395056
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
+// caculate real offsets
+ordinal_t first_offset = offsets_data[start - 1]; // -1 is valid
+ordinal_t first_ord = offsets_data[start];
+for (ssize_t i = start; i + 1 < offsets_data.size(); ++i) {
+offsets_data[i] = first_offset + (offsets_data[i + 1] - first_ord);
+}
Review Comment:
sequenctly also should be recaculate, eg. (0, 1, 3, 100, 103, 109, 114)
, assume the offset=100 row_id = 20, we only need row_id from [20, 23), so
after recaculate the offset became( 0, 3, 9)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #12560: [improvement](runtime) log status via to_string
dataroaring opened a new pull request, #12560: URL: https://github.com/apache/doris/pull/12560 Status is treated as bool when we log it via operator << due to Status::bool() method. # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
adonis0147 commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969401143
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
+// caculate real offsets
+ordinal_t first_offset = offsets_data[start - 1]; // -1 is valid
+ordinal_t first_ord = offsets_data[start];
+for (ssize_t i = start; i + 1 < offsets_data.size(); ++i) {
+offsets_data[i] = first_offset + (offsets_data[i + 1] - first_ord);
+}
Review Comment:
Ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] hf200012 merged pull request #95: [docs]1.1.2 Release Note
hf200012 merged PR #95: URL: https://github.com/apache/doris-website/pull/95 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: [docs]1.1.2 Release Note (#95)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 01b46abfc5b [docs]1.1.2 Release Note (#95)
01b46abfc5b is described below
commit 01b46abfc5b9b8fd0f215d3abb07d29399fb7f39
Author: Luzhijing <82810928+luzhij...@users.noreply.github.com>
AuthorDate: Tue Sep 13 17:53:58 2022 +0800
[docs]1.1.2 Release Note (#95)
---
docs/releasenotes/release-1.1.2.md | 84
.../current/releasenotes/release-1.1.2.md | 91 ++
sidebars.json | 3 +-
3 files changed, 177 insertions(+), 1 deletion(-)
diff --git a/docs/releasenotes/release-1.1.2.md
b/docs/releasenotes/release-1.1.2.md
new file mode 100644
index 000..223b65fda06
--- /dev/null
+++ b/docs/releasenotes/release-1.1.2.md
@@ -0,0 +1,84 @@
+---
+{
+"title": "Release 1.1.2",
+"language": "en"
+}
+---
+
+
+
+
+In this release, Doris Team has fixed more than 170 issues or performance
improvement since 1.1.1. This release is a bugfix release on 1.1 and all users
are encouraged to upgrade to this release.
+
+# Features
+
+### New MemTracker
+
+Introduced new MemTracker for both vectorized engine and non-vectorized engine
which is more accurate.
+
+### Add API for showing current queries and kill query
+
+### Support read/write emoji of UTF16 via ODBC Table
+
+# Improvements
+
+### Data Lake related improvements
+
+- Improved HDFS ORC File scan performance about 300%.
[#11501](https://github.com/apache/doris/pull/11501)
+
+- Support HDFS HA mode when query Iceberg table.
+
+- Support query Hive data created by [Apache Tez](https://tez.apache.org/)
+
+- Add Ali OSS as Hive external support.
+
+### Add support for string and text type in Spark Load
+
+
+### Add reuse block in non-vectorized engine and have 50% performance
improvement in some cases. [#11392](https://github.com/apache/doris/pull/11392)
+
+### Improve like or regex performance
+
+### Disable tcmalloc's aggressive_memory_decommit
+
+It will have 40% performance gains in load or query.
+
+Currently it is a config, you can change it by set config
`tc_enable_aggressive_memory_decommit`.
+
+# Bug Fix
+
+### Some issues about FE that will cause FE failure or data corrupt.
+
+- Add reserved disk config to avoid too many reserved BDB-JE
files.**(Serious)** In an HA environment, BDB JE will retains as many
reserved files. The BDB-je log doesn't delete until approaching a disk limit.
+
+- Fix fatal bug in BDB-JE which will cause FE replica could not start
correctly or data corrupted.** (Serious)**
+
+### Fe will hang on waitFor_rpc during query and BE will hang in high
concurrent scenarios.
+
+[#12459](https://github.com/apache/doris/pull/12459)
[#12458](https://github.com/apache/doris/pull/12458)
[#12392](https://github.com/apache/doris/pull/12392)
+
+### A fatal issue in vectorized storage engine which will cause wrong result.
**(Serious)**
+
+[#11754](https://github.com/apache/doris/pull/11754)
[#11694](https://github.com/apache/doris/pull/11694)
+
+### Lots of planner related issues that will cause BE core or in abnormal
state.
+
+[#12080](https://github.com/apache/doris/pull/12080)
[#12075](https://github.com/apache/doris/pull/12075)
[#12040](https://github.com/apache/doris/pull/12040)
[#12003](https://github.com/apache/doris/pull/12003)
[#12007](https://github.com/apache/doris/pull/12007)
[#11971](https://github.com/apache/doris/pull/11971)
[#11933](https://github.com/apache/doris/pull/11933)
[#11861](https://github.com/apache/doris/pull/11861)
[#11859](https://github.com/apache/doris/pull/11859) [#11855](https: [...]
+
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/releasenotes/release-1.1.2.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/releasenotes/release-1.1.2.md
new file mode 100644
index 000..c0c5e582ab1
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/releasenotes/release-1.1.2.md
@@ -0,0 +1,91 @@
+---
+{
+"title": "Release 1.1.2",
+"language": "zh-CN"
+}
+---
+
+
+
+
+在 Apache Doris 1.1.2 版本中，我们引入了新的 Memtracker、极大程度上避免 OOM
类问题的发生，提升了向量化执行引擎在多数查询场景的性能表现，修复了诸多导致 BE 和 FE
发生异常的问题，优化了在湖仓联邦查询场景的部分体验问题并提升访问外部数据的性能。
+
+相较于 1.1.1 版本，在 1.1.2 版本中有超过 170 个 Issue 和性能优化项被合入，系统稳定性和性能都得到进一步加强。与此同时，1.1.2
版本还将作为 Apache Doris 首个 LTS （Long-term Support）长周期支持版本，后续长期维护和支持，推荐所有用户下载和升级。
+
+# 新增功能
+
+### MemTracker
+
+MemTracker 是一个用于分析内存使用情况的统计工具，在 1.1.1 版本中我们引入了简易版 Memtracker 用以控制 BE 侧内存。在
1.1.2 版本中，我们引入了新的 MemTracker，在向量化执行引擎和非向量化执行引擎中都更为准确。
+
+### 增加展示和取消正在执行 Query 的 API
+
+`GET /rest/v2/manager/query/current_queries`
+
+`GET /rest/v2/manager/query/kill/{query_id}`
+
+具体使用参考文档 [Query Profile
Action](https://doris.apache.org/zh-CN/docs/dev/admin-manual/http-actions/fe/manager/query-profile-action?_highligh
[GitHub] [doris] eldenmoon commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
eldenmoon commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969415507
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
Review Comment:
I think both would be ok， since if we put the assignment of `num_items`
outside the function, we need to do one more cast
`static_cast(*offsets)`, this seems a
little duplicated code, num_items as a output of `_calculate_offsets` may be
straightforward?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
eldenmoon commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969420064
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
+// caculate real offsets
Review Comment:
done
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] alex1528 opened a new issue, #12561: [Bug]be can't start
alex1528 opened a new issue, #12561: URL: https://github.com/apache/doris/issues/12561 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1.2 ### What's Wrong? Sep 13 18:05:37 be01 kernel: traps: doris_be[15696] trap invalid opcode ip:20f0685 sp:7ffd6baf1160 error:0 in doris_be[40+6fc1000] Sep 13 18:05:38 be01 systemd-coredump[15698]: Process 15696 (doris_be) of user 0 dumped core. Stack trace of thread 15696: #0 0x020f0685 _Z27check_required_instructionsv (doris_be) #1 0x0601751d __libc_csu_init (doris_be) #2 0x14a504f5973e __libc_start_main (libc.so.6) #3 0x020ee8aa _start (doris_be) -- Subject: Process 15696 (doris_be) dumped core ### What You Expected? can started ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] lsy3993 opened a new pull request, #12562: [test](window-function) add big table in regression of window function
lsy3993 opened a new pull request, #12562: URL: https://github.com/apache/doris/pull/12562 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [x] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #12293: [Enhancement](array-type) record offsets info to speed up array colum…
adonis0147 commented on code in PR #12293:
URL: https://github.com/apache/doris/pull/12293#discussion_r969430640
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -466,6 +486,42 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
return Status::OK();
}
+Status ArrayFileColumnIterator::_seek_by_offsets(ordinal_t ord) {
+// using offsets info
+ordinal_t offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&offset));
+RETURN_IF_ERROR(_item_iterator->seek_to_ordinal(offset));
+return Status::OK();
+}
+
+Status ArrayFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+RETURN_IF_ERROR(_offset_iterator->seek_to_ordinal(ord));
+if (_array_reader->is_nullable()) {
+RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+}
+return _seek_by_offsets(ord);
+}
+
+Status ArrayFileColumnIterator::_caculate_offsets(ssize_t start,
+
vectorized::MutableColumnPtr& offsets,
+ size_t* num_items) {
+auto& column_offsets =
static_cast(*offsets);
+ordinal_t last_offset = 0;
+RETURN_IF_ERROR(_peek_one_offset(&last_offset));
+
+auto& offsets_data = column_offsets.get_data();
+*num_items = last_offset - offsets_data[start];
Review Comment:
We could just do the cast outside and use the explicit type inside the
function. No duplicate casts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] lihuigang opened a new pull request, #12563: [fix](planner) fix orthogonal_bitmap_union_count planner : wrong PREAGGREGATION
lihuigang opened a new pull request, #12563: URL: https://github.com/apache/doris/pull/12563 # Proposed changes Issue Number: close #xxx ## Problem summary Execution plan display when using orthogonal_bitmap_union_count function: PREAGGREGATION: OFF Reason: Invalid Aggregate Operator: orthogonal_bitmap_union_count The correct plan is: PREAGGREGATION: ON ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12552: [fix](new-scan) try to fix invalid call to nullptr slot
github-actions[bot] commented on PR #12552: URL: https://github.com/apache/doris/pull/12552#issuecomment-1245227620 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12552: [fix](new-scan) try to fix invalid call to nullptr slot
github-actions[bot] commented on PR #12552: URL: https://github.com/apache/doris/pull/12552#issuecomment-1245227679 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] catpineapple opened a new pull request, #12564: [fix](doc) fix upgrade doc description
catpineapple opened a new pull request, #12564: URL: https://github.com/apache/doris/pull/12564 # Proposed changes Issue Number: close #xxx ## Problem summary Complements upgrade documentation description and documentation jump link ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] catpineapple opened a new pull request, #96: [fix](doc) fix upgrade doc description
catpineapple opened a new pull request, #96: URL: https://github.com/apache/doris-website/pull/96 Complements upgrade documentation description and documentation jump link -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12160: [feature](light-schema-change) fix light-schema-change and add more cases
github-actions[bot] commented on PR #12160: URL: https://github.com/apache/doris/pull/12160#issuecomment-1245240108 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12552: [fix](new-scan) try to fix invalid call to nullptr slot
yiguolei merged PR #12552: URL: https://github.com/apache/doris/pull/12552 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](new-scan) try to fix invalid call to nullptr slot (#12552)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 9e49f68663 [fix](new-scan) try to fix invalid call to nullptr slot
(#12552)
9e49f68663 is described below
commit 9e49f68663c95776ef89da68a984932a91e335a5
Author: Pxl
AuthorDate: Tue Sep 13 18:54:29 2022 +0800
[fix](new-scan) try to fix invalid call to nullptr slot (#12552)
---
be/src/vec/exec/scan/vscan_node.cpp | 13 +++--
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/be/src/vec/exec/scan/vscan_node.cpp
b/be/src/vec/exec/scan/vscan_node.cpp
index 23661f7c5b..0ded3e80e3 100644
--- a/be/src/vec/exec/scan/vscan_node.cpp
+++ b/be/src/vec/exec/scan/vscan_node.cpp
@@ -272,11 +272,11 @@ Status
VScanNode::_append_rf_into_conjuncts(std::vector& vexprs) {
RETURN_IF_ERROR(new_vconjunct_ctx_ptr->prepare(_state, _row_descriptor));
RETURN_IF_ERROR(new_vconjunct_ctx_ptr->open(_state));
if (_vconjunct_ctx_ptr) {
-(*(_vconjunct_ctx_ptr.get()))->mark_as_stale();
+(*_vconjunct_ctx_ptr)->mark_as_stale();
_stale_vexpr_ctxs.push_back(std::move(_vconjunct_ctx_ptr));
}
_vconjunct_ctx_ptr.reset(new doris::vectorized::VExprContext*);
-*(_vconjunct_ctx_ptr.get()) = new_vconjunct_ctx_ptr;
+*_vconjunct_ctx_ptr = new_vconjunct_ctx_ptr;
return Status::OK();
}
@@ -353,7 +353,7 @@ Status VScanNode::_normalize_conjuncts() {
if (new_root) {
(*_vconjunct_ctx_ptr)->set_root(new_root);
} else {
-(*(_vconjunct_ctx_ptr.get()))->mark_as_stale();
+(*_vconjunct_ctx_ptr)->mark_as_stale();
_stale_vexpr_ctxs.push_back(std::move(_vconjunct_ctx_ptr));
_vconjunct_ctx_ptr.reset(nullptr);
}
@@ -405,10 +405,10 @@ VExpr* VScanNode::_normalize_predicate(VExpr*
conjunct_expr_root) {
auto impl = conjunct_expr_root->get_impl();
// If impl is not null, which means this a conjuncts from runtime
filter.
VExpr* cur_expr = impl ? const_cast(impl) :
conjunct_expr_root;
-SlotDescriptor* slot;
+SlotDescriptor* slot = nullptr;
ColumnValueRangeType* range = nullptr;
PushDownType pdt = PushDownType::UNACCEPTABLE;
-_eval_const_conjuncts(cur_expr, *(_vconjunct_ctx_ptr.get()), &pdt);
+_eval_const_conjuncts(cur_expr, *_vconjunct_ctx_ptr, &pdt);
if (pdt == PushDownType::UNACCEPTABLE &&
(_is_predicate_acting_on_slot(cur_expr, in_predicate_checker,
&slot, &range) ||
_is_predicate_acting_on_slot(cur_expr, eq_predicate_checker,
&slot, &range))) {
@@ -437,7 +437,8 @@ VExpr* VScanNode::_normalize_predicate(VExpr*
conjunct_expr_root) {
},
*range);
}
-if (pdt == PushDownType::ACCEPTABLE &&
_is_key_column(slot->col_name())) {
+if (pdt == PushDownType::ACCEPTABLE && slot != nullptr &&
+_is_key_column(slot->col_name())) {
return nullptr;
} else {
// for PARTIAL_ACCEPTABLE and UNACCEPTABLE, do not remove expr
from the tree
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12514: [fix](comments) modify comments of setting global variables
yiguolei merged PR #12514: URL: https://github.com/apache/doris/pull/12514 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](comments) modify comments of setting global variables #12514
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6a3385437b [fix](comments) modify comments of setting global variables
#12514
6a3385437b is described below
commit 6a3385437b4658c1cd1bd1a8bac209ce58d2b159
Author: Henry2SS <45096548+henry...@users.noreply.github.com>
AuthorDate: Tue Sep 13 19:13:57 2022 +0800
[fix](comments) modify comments of setting global variables #12514
Co-authored-by: wuhangze
---
fe/fe-core/src/main/java/org/apache/doris/qe/VariableMgr.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/VariableMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/VariableMgr.java
index 7efbf0c7a1..d6c224e68c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/VariableMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/VariableMgr.java
@@ -269,8 +269,7 @@ public class VariableMgr {
}
private static void setGlobalVarAndWriteEditLog(VarContext ctx, String
name, String value) throws DdlException {
-// set global variable should not affect variables of current session.
-// global variable will only make effect when connecting in.
+// global variable will make effect when is set immediately.
wlock.lock();
try {
setValue(ctx.getObj(), ctx.getField(), value);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] leafgodgood commented on issue #12525: [Feature] Support the use of group_concat function and the use of separator
leafgodgood commented on issue #12525: URL: https://github.com/apache/doris/issues/12525#issuecomment-1245263316 > master branch，navicat is 15.0.27 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model
github-actions[bot] commented on PR #11952: URL: https://github.com/apache/doris/pull/11952#issuecomment-1245265868 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model
yiguolei merged PR #11952: URL: https://github.com/apache/doris/pull/11952 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](storage) For debugging problems: add session variable `skip_storage_engine_merge` to treat agg and unique data model as dup model (#11952)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6bf5fc6db5 [improvement](storage) For debugging problems: add session
variable `skip_storage_engine_merge` to treat agg and unique data model as dup
model (#11952)
6bf5fc6db5 is described below
commit 6bf5fc6db50f1c543b9c5b50bef00ceb4b60ff3e
Author: TengJianPing <18241664+jackte...@users.noreply.github.com>
AuthorDate: Tue Sep 13 19:18:56 2022 +0800
[improvement](storage) For debugging problems: add session variable
`skip_storage_engine_merge` to treat agg and unique data model as dup model
(#11952)
For debug purpose:
Add session variable skip_storage_engine_merge, when set to true, tables of
aggregate key model and unique key model will be read as duplicate key model.
Add session variable skip_delete_predicate, when set to true, rows deleted
with delete statement will be selected.
---
be/src/runtime/runtime_state.h | 9
be/src/vec/exec/volap_scanner.cpp | 16 ++-
docs/en/docs/advanced/variables.md | 6 ++
docs/zh-CN/docs/advanced/variables.md | 6 ++
.../java/org/apache/doris/common/util/Util.java| 4 +++-
.../java/org/apache/doris/qe/SessionVariable.java | 24 ++
gensrc/thrift/PaloInternalService.thrift | 6 ++
7 files changed, 65 insertions(+), 6 deletions(-)
diff --git a/be/src/runtime/runtime_state.h b/be/src/runtime/runtime_state.h
index 8b49f0a161..1b7042815a 100644
--- a/be/src/runtime/runtime_state.h
+++ b/be/src/runtime/runtime_state.h
@@ -349,6 +349,15 @@ public:
return segment_v2::CompressionTypePB::SNAPPY;
}
+bool skip_storage_engine_merge() const {
+return _query_options.__isset.skip_storage_engine_merge &&
+ _query_options.skip_storage_engine_merge;
+}
+
+bool skip_delete_predicate() const {
+return _query_options.__isset.skip_delete_predicate &&
_query_options.skip_delete_predicate;
+}
+
const std::vector& tablet_commit_infos() const {
return _tablet_commit_infos;
}
diff --git a/be/src/vec/exec/volap_scanner.cpp
b/be/src/vec/exec/volap_scanner.cpp
index efa80b36ac..4b820aa1ce 100644
--- a/be/src/vec/exec/volap_scanner.cpp
+++ b/be/src/vec/exec/volap_scanner.cpp
@@ -171,7 +171,11 @@ Status VOlapScanner::_init_tablet_reader_params(
->rowset_meta()
->is_segments_overlapping());
-_tablet_reader_params.direct_mode = _aggregation || single_version;
+if (_runtime_state->skip_storage_engine_merge()) {
+_tablet_reader_params.direct_mode = true;
+} else {
+_tablet_reader_params.direct_mode = _aggregation || single_version;
+}
RETURN_IF_ERROR(_init_return_columns(!_tablet_reader_params.direct_mode));
@@ -192,10 +196,12 @@ Status VOlapScanner::_init_tablet_reader_params(
std::copy(function_filters.cbegin(), function_filters.cend(),
std::inserter(_tablet_reader_params.function_filters,
_tablet_reader_params.function_filters.begin()));
-auto& delete_preds = _tablet->delete_predicates();
-std::copy(delete_preds.cbegin(), delete_preds.cend(),
- std::inserter(_tablet_reader_params.delete_predicates,
-_tablet_reader_params.delete_predicates.begin()));
+if (!_runtime_state->skip_delete_predicate()) {
+auto& delete_preds = _tablet->delete_predicates();
+std::copy(delete_preds.cbegin(), delete_preds.cend(),
+ std::inserter(_tablet_reader_params.delete_predicates,
+
_tablet_reader_params.delete_predicates.begin()));
+}
// Merge the columns in delete predicate that not in latest schema in to
current tablet schema
for (auto& del_pred_rs : _tablet_reader_params.delete_predicates) {
diff --git a/docs/en/docs/advanced/variables.md
b/docs/en/docs/advanced/variables.md
index 5b388c8a05..537a271e73 100644
--- a/docs/en/docs/advanced/variables.md
+++ b/docs/en/docs/advanced/variables.md
@@ -526,3 +526,9 @@ Translated with www.DeepL.com/Translator (free version)
* `trim_tailing_spaces_for_external_table_query`
Used to control whether trim the tailing spaces while quering Hive external
tables. The default is false.
+
+* `skip_storage_engine_merge`
+ For debugging purpose. In vectorized execution engine, in case of problems
of reading data of Aggregate Key model and Unique Key model, setting value to
`true` will read data as Duplicate Key model.
+
+* `skip_delete_predicate`
+ For debugging purpose. In vectorized execution engine, in case of problems
of reading data, setting value to `true` will also read deleted data.
diff --git a/docs/zh-CN/docs/a
[GitHub] [doris] github-actions[bot] commented on pull request #12551: [fix](planner) fix projection may return error results
github-actions[bot] commented on PR #12551: URL: https://github.com/apache/doris/pull/12551#issuecomment-1245275892 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated (82061ad3a0 -> d8bfab1a2a)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch branch-1.1-lts in repository https://gitbox.apache.org/repos/asf/doris.git from 82061ad3a0 [fix](log) fix wrong logger for ExprSubstitutionMap (#12546) add d8bfab1a2a [bugfix](odbc) escape identifiers for sqlserver and postgresql (#12487) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/doris/catalog/OdbcTable.java | 14 ++ 1 file changed, 14 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12551: [fix](planner) fix projection may return error results
github-actions[bot] commented on PR #12551: URL: https://github.com/apache/doris/pull/12551#issuecomment-1245275944 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [Bug](doe) fix closing scanner twice (#12408)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new 9db16fa4ee [Bug](doe) fix closing scanner twice (#12408)
9db16fa4ee is described below
commit 9db16fa4ee413393144508cec66672df277aee1f
Author: Gabriel
AuthorDate: Wed Sep 7 22:45:30 2022 +0800
[Bug](doe) fix closing scanner twice (#12408)
---
be/src/vec/exec/ves_http_scanner.cpp | 4
be/src/vec/exec/ves_http_scanner.h | 1 -
2 files changed, 5 deletions(-)
diff --git a/be/src/vec/exec/ves_http_scanner.cpp
b/be/src/vec/exec/ves_http_scanner.cpp
index c06da0617b..11e3e87f94 100644
--- a/be/src/vec/exec/ves_http_scanner.cpp
+++ b/be/src/vec/exec/ves_http_scanner.cpp
@@ -19,10 +19,6 @@
namespace doris::vectorized {
-VEsHttpScanner::~VEsHttpScanner() {
-close();
-}
-
Status VEsHttpScanner::get_next(std::vector&
columns,
MemPool* tuple_pool, bool* eof,
const std::map&
docvalue_context) {
diff --git a/be/src/vec/exec/ves_http_scanner.h
b/be/src/vec/exec/ves_http_scanner.h
index 5fa83db6fd..a315e3e776 100644
--- a/be/src/vec/exec/ves_http_scanner.h
+++ b/be/src/vec/exec/ves_http_scanner.h
@@ -30,7 +30,6 @@ public:
bool doc_value_mode)
: EsHttpScanner(state, profile, tuple_id, properties,
conjunct_ctxs, counter,
doc_value_mode) {};
-~VEsHttpScanner();
Status get_next(std::vector& columns,
MemPool* tuple_pool,
bool* eof, const std::map&
docvalue_context);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] hf200012 merged pull request #96: [fix](doc) fix upgrade doc description
hf200012 merged PR #96: URL: https://github.com/apache/doris-website/pull/96 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: doc_upgrade (#96)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new a71c0f62712 doc_upgrade (#96) a71c0f62712 is described below commit a71c0f62712db13fd99c9b711b18835153fe18f0 Author: catpineapple <42031973+catpineap...@users.noreply.github.com> AuthorDate: Tue Sep 13 19:36:27 2022 +0800 doc_upgrade (#96) --- docs/admin-manual/cluster-management/upgrade.md| 28 +++-- .../maint-monitor/metadata-operation.md| 2 +- .../admin-manual/cluster-management/upgrade.md | 29 +++--- .../maint-monitor/metadata-operation.md| 2 +- 4 files changed, 54 insertions(+), 7 deletions(-) diff --git a/docs/admin-manual/cluster-management/upgrade.md b/docs/admin-manual/cluster-management/upgrade.md index 450867bcca9..36919592982 100644 --- a/docs/admin-manual/cluster-management/upgrade.md +++ b/docs/admin-manual/cluster-management/upgrade.md @@ -56,6 +56,24 @@ Doris can upgrade smoothly by rolling upgrades. The following steps are recommen 2. **important! ! Metadata needs to be backed up before upgrading(The entire directory needs to be backed up)! !** +Let's assume that the path of `meta_dir` specified in fe.conf is `path/to/doris-meta`. In a normal Doris cluster, the directory structure of metadata should be as follows: + +``` +/path/to/doris-meta/ +|-- bdb/ +| |-- .jdb +| |-- je.config.csv +| |-- je.info.0 +| |-- je.info.0.lck +| |-- je.lck +| `-- je.stat.csv +`-- image/ +|-- ROLE +|-- VERSION +`-- image. +``` +The metadata directory to be backed up is `doris-meta/` + ## Test the correctness of BE upgrade 1. Arbitrarily select a BE node and deploy the latest doris_be binary file. @@ -79,13 +97,13 @@ Doris can upgrade smoothly by rolling upgrades. The following steps are recommen ## Upgrade preparation 1. After data validation, the new version of BE and FE binary files are distributed to their respective directories. -2. Usually small version upgrade, BE only needs to upgrade doris_be; FE only needs to upgrade palo-fe.jar. If it is a large version upgrade, you may need to upgrade other files (including but not limited to bin / lib / etc.) If you are not sure whether you need to replace other files, it is recommended to replace all of them. +2. Usually small version upgrade, BE only needs to upgrade doris_be; FE only needs to upgrade doris-fe.jar(which was doris-fe.jar in previous versions). If it is a large version upgrade, you may need to upgrade other files (including but not limited to bin / , lib / etc.) If you are not sure whether you need to replace other files, it is recommended to replace all of them. ## rolling upgrade 1. Confirm that the new version of the file is deployed. Restart FE and BE instances one by one. 2. It is suggested that BE be restarted one by one and FE be restarted one by one. Because Doris usually guarantees backward compatibility between FE and BE, that is, the old version of FE can access the new version of BE. However, the old version of BE may not be supported to access the new version of FE. -3. It is recommended to restart the next instance after confirming the previous instance started successfully. Refer to the Installation Deployment Document for the identification of successful instance startup. +3. It is recommended to restart the next instance after confirming the previous instance started successfully. Refer to the [Installation Deployment Document](../../install/install-deploy.md) for the identification of successful instance startup. ## About version rollback Because the database is a stateful service, Doris cannot support version rollback (version downgrade) in most cases. In some cases, the rollback of the 3-bit or 4-bit version can be supported, but the rollback of the 2-bit version will not be supported. @@ -93,3 +111,9 @@ Because the database is a stateful service, Doris cannot support version rollbac Therefore, it is recommended to upgrade some nodes and observe the business operation (gray upgrade) to reduce the upgrade risk. **Illegal rollback operation may cause data loss and damage.** + +## Documentation +1. [Doris metadata design document](../../../community/design/metadata-design.md) +2. [Metadata Operations and Maintenance](../../admin-manual/maint-monitor/metadata-operation.md) +3. [Data replica management](../../admin-manual/maint-monitor/tablet-repair-and-balance.md) +4. [Installation Deployment Document](../../install/install-deploy.md) diff --git a/docs/admin-manual/maint-monitor/metadata-operation.md b/docs/
[GitHub] [doris] yiguolei merged pull request #12519: [enhance](information_schema) show hll type and bitmap type instead of unknown
yiguolei merged PR #12519: URL: https://github.com/apache/doris/pull/12519 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhance](information_schema) show hll type and bitmap type instead of unknown (#12519)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 58508aea13 [enhance](information_schema) show hll type and bitmap type
instead of unknown (#12519)
58508aea13 is described below
commit 58508aea13341af659bddf7701d126241573e60b
Author: AlexYue
AuthorDate: Tue Sep 13 19:43:42 2022 +0800
[enhance](information_schema) show hll type and bitmap type instead of
unknown (#12519)
Before this pr, when querying data type of hll/bitmap column, 'unknown'
would be returned instead of the correct data type of queried column.
---
.../exec/schema_scanner/schema_columns_scanner.cpp | 12 +
.../data/query/system/test_query_sys_data_type.out | 10 +++
.../query/system/test_query_sys_data_type.groovy | 31 ++
3 files changed, 53 insertions(+)
diff --git a/be/src/exec/schema_scanner/schema_columns_scanner.cpp
b/be/src/exec/schema_scanner/schema_columns_scanner.cpp
index 6102e128af..35d6f28518 100644
--- a/be/src/exec/schema_scanner/schema_columns_scanner.cpp
+++ b/be/src/exec/schema_scanner/schema_columns_scanner.cpp
@@ -122,6 +122,12 @@ std::string
SchemaColumnsScanner::to_mysql_data_type_string(TColumnDesc& desc) {
case TPrimitiveType::DECIMALV2: {
return "decimal";
}
+case TPrimitiveType::HLL: {
+return "hll";
+}
+case TPrimitiveType::OBJECT: {
+return "bitmap";
+}
default:
return "unknown";
}
@@ -180,6 +186,12 @@ std::string
SchemaColumnsScanner::type_to_string(TColumnDesc& desc) {
stream << ")";
return stream.str();
}
+case TPrimitiveType::HLL: {
+return "hll";
+}
+case TPrimitiveType::OBJECT: {
+return "bitmap";
+}
default:
return "unknown";
}
diff --git a/regression-test/data/query/system/test_query_sys_data_type.out
b/regression-test/data/query/system/test_query_sys_data_type.out
new file mode 100644
index 00..c5273d2c5c
--- /dev/null
+++ b/regression-test/data/query/system/test_query_sys_data_type.out
@@ -0,0 +1,10 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !sql --
+dt date
+id int
+name char
+province char
+os char
+set1 hll
+set2 bitmap
+
diff --git
a/regression-test/suites/query/system/test_query_sys_data_type.groovy
b/regression-test/suites/query/system/test_query_sys_data_type.groovy
new file mode 100644
index 00..81ae6f5571
--- /dev/null
+++ b/regression-test/suites/query/system/test_query_sys_data_type.groovy
@@ -0,0 +1,31 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_query_sys_data_type", 'query,p0') {
+def tbName = "test_data_type"
+def dbName = "test_query_db"
+sql "CREATE DATABASE IF NOT EXISTS ${dbName}"
+sql "USE ${dbName}"
+
+sql """ DROP TABLE IF EXISTS ${tbName} """
+sql """
+create table ${tbName} (dt date, id int, name char(10), province
char(10), os char(1), set1 hll hll_union, set2 bitmap bitmap_union)
+distributed by hash(id) buckets 1 properties("replication_num"="1");
+"""
+
+qt_sql "select column_name, data_type from information_schema.columns
where table_schema = '${dbName}' and table_name = '${tbName}'"
+}
\ No newline at end of file
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 commented on a diff in pull request #10084: [feature](vectorized) Support block aggregate in scanner
Gabriel39 commented on code in PR #10084:
URL: https://github.com/apache/doris/pull/10084#discussion_r969515435
##
be/src/vec/olap/block_aggregator.cpp:
##
@@ -0,0 +1,234 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/olap/block_aggregator.h"
+
+#include "util/simd/bits.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/columns_number.h"
+#include "vec/core/types.h"
+#include "vec/data_types/data_type_factory.hpp"
+
+namespace doris::vectorized {
+
+template
+inline void compare_previous(const IColumn& col, uint8_t* flags) {
+if (col.is_nullable()) {
+auto col_ptr = reinterpret_cast(&col);
+for (int i = 1; i < col_ptr->size(); ++i) {
+flags[i] &= (col_ptr->compare_at(i, i - 1, col, -1) == 0);
+}
+} else {
+auto col_ptr = reinterpret_cast(&col);
+for (int i = 1; i < col_ptr->size(); ++i) {
+flags[i] &= (col_ptr->compare_at(i, i - 1, col, -1) == 0);
+}
+}
+}
+
+BlockAggregator::BlockAggregator(const TabletSchema* tablet_schema,
+ const std::vector& output_columns,
+ const std::vector& return_columns,
+ const std::unordered_set* null_set,
const int batch_size)
+: _tablet_schema(tablet_schema),
+ _output_columns(output_columns),
+ _sorted_output_columns(output_columns),
+ _return_columns(return_columns),
+ _null_set(null_set),
+ _batch_size(batch_size) {
+_has_agg_data = false;
+_do_aggregate = true;
+_agg_ratio = config::block_aggregate_ratio;
+_num_columns = _output_columns.size();
+_num_key_columns = 0;
+
+// todo(zeno) Make sure column prune done
+std::sort(_sorted_output_columns.begin(), _sorted_output_columns.end());
+_output_columns_loc.resize(_num_columns);
+for (int i = 0; i < _num_columns; ++i) {
+for (int j = 0; j < _num_columns; ++j) {
+if (_sorted_output_columns[i] == output_columns[j]) {
+_output_columns_loc[i] = j;
+break;
+}
+}
+}
+
+for (auto index : _sorted_output_columns) {
+if (_tablet_schema->column(index).is_key()) {
+_num_key_columns++;
+}
+}
+
+_current_row = 0;
+_source_size = 0;
+_eq_previous.reserve(_batch_size);
+_agg_index.reserve(_batch_size);
+_agg_range.reserve(_batch_size);
+
+// init _key_comparator
+for (int i = 0; i < _num_key_columns; ++i) {
+_key_comparator.emplace_back(
+
_get_comparator(_tablet_schema->column(_sorted_output_columns[i])));
+}
+
+// init _column_aggregator
+for (int i = 0; i < _num_key_columns; ++i) {
+
_column_aggregator.emplace_back(std::make_unique());
+}
+
+for (int i = _num_key_columns; i < _num_columns; ++i) {
+bool is_nullable = (_null_set != nullptr &&
+_null_set->find(_sorted_output_columns[i]) !=
_null_set->end());
+auto col = _tablet_schema->column(_sorted_output_columns[i]);
+auto data_type =
vectorized::DataTypeFactory::instance().create_data_type(col, is_nullable);
+
_column_aggregator.emplace_back(std::make_unique(col,
data_type));
+}
+
+aggregate_reset();
+}
+
+CompareFunc BlockAggregator::_get_comparator(const TabletColumn& col) {
+switch (col.type()) {
+case OLAP_FIELD_TYPE_TINYINT:
+return &compare_previous;
+case OLAP_FIELD_TYPE_SMALLINT:
+return &compare_previous;
+case OLAP_FIELD_TYPE_INT:
+return &compare_previous;
+case OLAP_FIELD_TYPE_BIGINT:
+return &compare_previous;
+case OLAP_FIELD_TYPE_LARGEINT:
+return &compare_previous;
+case OLAP_FIELD_TYPE_BOOL:
+return &compare_previous;
+case OLAP_FIELD_TYPE_FLOAT:
+return &compare_previous;
+case OLAP_FIELD_TYPE_DOUBLE:
+return &compare_previous;
+case OLAP_FIELD_TYPE_DECIMAL:
+return &compare_previous>;
+case OLAP_FIELD_TYPE_CHAR:
+case OLAP_FIELD_TYPE_VARCHAR:
+case OLAP_FIELD_TYPE_STRING:
+return &compare_previous;
+
[GitHub] [doris] dependabot[bot] commented on pull request #12520: Bump snakeyaml from 1.15 to 1.26 in /fe
dependabot[bot] commented on PR #12520: URL: https://github.com/apache/doris/pull/12520#issuecomment-1245299343 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting `@dependabot ignore this major version` or `@dependabot ignore this minor version`. If you change your mind, just re-open this PR and I'll resolve any conflicts on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed pull request #12520: Bump snakeyaml from 1.15 to 1.26 in /fe
yiguolei closed pull request #12520: Bump snakeyaml from 1.15 to 1.26 in /fe URL: https://github.com/apache/doris/pull/12520 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] lihuigang closed pull request #12563: [fix](planner) fix orthogonal_bitmap_union_count planner : wrong PREAGGREGATION
lihuigang closed pull request #12563: [fix](planner) fix orthogonal_bitmap_union_count planner : wrong PREAGGREGATION URL: https://github.com/apache/doris/pull/12563 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] deardeng opened a new pull request, #12565: [feature](regression) Enhancement regression frame, support http post…
deardeng opened a new pull request, #12565: URL: https://github.com/apache/doris/pull/12565 … client # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... Enhancement regression frame, support http post client -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12551: [fix](planner) fix projection may return error results
yiguolei merged PR #12551: URL: https://github.com/apache/doris/pull/12551 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [fix](planner) fix projection may return error results (#12551)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new b16f2f94f6 [fix](planner) fix projection may return error results
(#12551)
b16f2f94f6 is described below
commit b16f2f94f6e28aab116aa07ab6d020aad518691b
Author: wangbo
AuthorDate: Tue Sep 13 20:03:21 2022 +0800
[fix](planner) fix projection may return error results (#12551)
---
be/src/vec/columns/column_vector.cpp | 14 +++
be/src/vec/columns/column_vector.h | 2 ++
be/src/vec/exec/join/vhash_join_node.cpp | 41 +---
be/src/vec/exec/join/vhash_join_node.h | 1 +
4 files changed, 55 insertions(+), 3 deletions(-)
diff --git a/be/src/vec/columns/column_vector.cpp
b/be/src/vec/columns/column_vector.cpp
index dfe1bce3b3..5b371b64aa 100644
--- a/be/src/vec/columns/column_vector.cpp
+++ b/be/src/vec/columns/column_vector.cpp
@@ -239,6 +239,20 @@ void ColumnVector::insert_indices_from(const IColumn&
src, const int* indices
}
}
+template
+void ColumnVector::insert_join_nullmap(const int* indices_begin, const int*
indices_end) {
+auto origin_size = size();
+auto new_size = indices_end - indices_begin;
+data.resize(origin_size + new_size);
+
+for (int i = 0; i < new_size; ++i) {
+int offset = indices_begin[i];
+if constexpr (std::is_same_v) {
+data[origin_size + i] = (offset == -1) ? T{JOIN_NULL_HINT} : 0;
+}
+}
+}
+
template
ColumnPtr ColumnVector::filter(const IColumn::Filter& filt, ssize_t
result_size_hint) const {
size_t size = data.size();
diff --git a/be/src/vec/columns/column_vector.h
b/be/src/vec/columns/column_vector.h
index e57ffe4a9c..4d4ceb14e2 100644
--- a/be/src/vec/columns/column_vector.h
+++ b/be/src/vec/columns/column_vector.h
@@ -275,6 +275,8 @@ public:
void insert_indices_from(const IColumn& src, const int* indices_begin,
const int* indices_end) override;
+void insert_join_nullmap(const int* indices_begin, const int* indices_end);
+
void fill(const value_type& element, size_t num) {
auto old_size = data.size();
auto new_size = old_size + num;
diff --git a/be/src/vec/exec/join/vhash_join_node.cpp
b/be/src/vec/exec/join/vhash_join_node.cpp
index 88f8fa5d95..23036f150c 100644
--- a/be/src/vec/exec/join/vhash_join_node.cpp
+++ b/be/src/vec/exec/join/vhash_join_node.cpp
@@ -179,7 +179,8 @@ struct ProcessHashTableProbe {
// output build side result column
template
void build_side_output_column(MutableColumns& mcol, int column_offset, int
column_length,
- const std::vector& output_slot_flags,
int size) {
+ const std::vector& output_slot_flags,
+ const std::vector& right_check_flags,
int size) {
constexpr auto is_semi_anti_join = JoinOpType::value ==
TJoinOp::RIGHT_ANTI_JOIN ||
JoinOpType::value ==
TJoinOp::RIGHT_SEMI_JOIN ||
JoinOpType::value ==
TJoinOp::LEFT_ANTI_JOIN ||
@@ -195,6 +196,12 @@ struct ProcessHashTableProbe {
auto& column =
_build_blocks[0].get_by_position(i).column;
mcol[i + column_offset]->insert_indices_from(
*column, _build_block_rows.data(),
_build_block_rows.data() + size);
+} else if (right_check_flags[i] &&
+ mcol[i + column_offset]->is_nullable()) {
+reinterpret_cast(mcol[i +
column_offset].get())
+->get_null_map_column()
+.insert_join_nullmap(_build_block_rows.data(),
+ _build_block_rows.data()
+ size);
} else {
mcol[i + column_offset]->resize(size);
}
@@ -261,6 +268,12 @@ struct ProcessHashTableProbe {
}
}
}
+} else if (right_check_flags[i] &&
+ mcol[i + column_offset]->is_nullable()) {
+reinterpret_cast(mcol[i +
column_offset].get())
+->get_null_map_column()
+.insert_join_nullmap(_build_block_rows.data(),
+ _build_block_rows.data()
+ size);
} else {
mcol[i + column_offset]->resize(size);
}
@@ -396,7 +409,8 @@ struct ProcessHashTableProbe {
{
SCOPED_TIMER(_build_side_output_timer);
build_side
[GitHub] [doris] HappenLee opened a new pull request, #12566: [thirdparty](lib) Add xxhash lib
HappenLee opened a new pull request, #12566: URL: https://github.com/apache/doris/pull/12566 # Proposed changes Add xxhash lib for BE, which is the faster hash method by test. ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12511: [feature](Nereids): Eliminate redundant filter and limit.
github-actions[bot] commented on PR #12511: URL: https://github.com/apache/doris/pull/12511#issuecomment-1245319503 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12511: [feature](Nereids): Eliminate redundant filter and limit.
github-actions[bot] commented on PR #12511: URL: https://github.com/apache/doris/pull/12511#issuecomment-1245319556 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener merged pull request #12511: [feature](Nereids): Eliminate redundant filter and limit.
jackwener merged PR #12511: URL: https://github.com/apache/doris/pull/12511 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (58508aea13 -> 9a5be4bab5)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 58508aea13 [enhance](information_schema) show hll type and bitmap type
instead of unknown (#12519)
add 9a5be4bab5 [feature](Nereids): Eliminate redundant filter and limit.
(#12511)
No new revisions were added by this update.
Summary of changes:
.../doris/nereids/jobs/batch/RewriteJob.java | 6 ++--
.../doris/nereids/parser/LogicalPlanBuilder.java | 2 +-
.../org/apache/doris/nereids/rules/RuleType.java | 17 +++
...icalEmptyRelation.java => EliminateFilter.java} | 14 -
...gicalEmptyRelation.java => EliminateLimit.java} | 7 ++---
.../trees/expressions/literal/BooleanLiteral.java | 14 -
.../nereids/trees/expressions/literal/Literal.java | 2 +-
.../apache/doris/nereids/util/ExpressionUtils.java | 2 +-
.../nereids/rules/analysis/CheckAnalysisTest.java | 2 +-
.../rules/expression/rewrite/TypeCoercionTest.java | 8 ++---
...iveLimitsTest.java => EliminateFilterTest.java} | 34 --
...tiveLimitsTest.java => EliminateLimitTest.java} | 31 ++--
.../rules/rewrite/logical/JoinReorderTest.java | 2 +-
.../logical/MergeConsecutiveFilterTest.java| 8 ++---
.../logical/MergeConsecutiveLimitsTest.java| 6 ++--
.../logical/MergeConsecutiveProjectsTest.java | 14 -
.../trees/expressions/ExpectedInputTypesTest.java | 30 +--
17 files changed, 110 insertions(+), 89 deletions(-)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/{LogicalLimitZeroToLogicalEmptyRelation.java
=> EliminateFilter.java} (73%)
rename
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/{LogicalLimitZeroToLogicalEmptyRelation.java
=> EliminateLimit.java} (84%)
copy
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/{MergeConsecutiveLimitsTest.java
=> EliminateFilterTest.java} (53%)
copy
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/{MergeConsecutiveLimitsTest.java
=> EliminateLimitTest.java} (57%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 merged pull request #12407: [Opt](vectorized) Speed up bucket shuffle join hash compute
Gabriel39 merged PR #12407: URL: https://github.com/apache/doris/pull/12407 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Opt](vectorized) Speed up bucket shuffle join hash compute (#12407)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d913ca5731 [Opt](vectorized) Speed up bucket shuffle join hash compute
(#12407)
d913ca5731 is described below
commit d913ca573116139b4cb95f6f649ce2a4b3870788
Author: HappenLee
AuthorDate: Tue Sep 13 20:19:22 2022 +0800
[Opt](vectorized) Speed up bucket shuffle join hash compute (#12407)
* [Opt](vectorized) Speed up bucket shuffle join hash compute
---
be/src/exec/tablet_info.cpp| 12 ++---
be/src/runtime/data_stream_sender.cpp | 5 +-
be/src/runtime/define_primitive_type.h | 58 ++
be/src/runtime/primitive_type.h| 38 +-
be/src/runtime/raw_value.h | 27 ++
be/src/util/hash_util.hpp | 7 +++
be/src/vec/columns/column.h| 21
be/src/vec/columns/column_const.cpp| 18 +++
be/src/vec/columns/column_const.h | 3 ++
be/src/vec/columns/column_decimal.cpp | 33
be/src/vec/columns/column_decimal.h| 2 +
be/src/vec/columns/column_nullable.cpp | 19 +++
be/src/vec/columns/column_nullable.h | 2 +
be/src/vec/columns/column_string.cpp | 20
be/src/vec/columns/column_string.h | 3 ++
be/src/vec/columns/column_vector.cpp | 33 +++-
be/src/vec/columns/column_vector.h | 3 ++
be/src/vec/sink/vdata_stream_sender.cpp| 18 ++-
.../org/apache/doris/analysis/DateLiteral.java | 28 ---
.../org/apache/doris/analysis/DecimalLiteral.java | 16 --
.../org/apache/doris/analysis/LargeIntLiteral.java | 6 +++
21 files changed, 274 insertions(+), 98 deletions(-)
diff --git a/be/src/exec/tablet_info.cpp b/be/src/exec/tablet_info.cpp
index c89e03f0be..08e77d9565 100644
--- a/be/src/exec/tablet_info.cpp
+++ b/be/src/exec/tablet_info.cpp
@@ -220,10 +220,7 @@ Status OlapTablePartitionParam::init() {
if (slot != nullptr) {
hash_val = RawValue::zlib_crc32(slot, slot_desc->type(),
hash_val);
} else {
-//nullptr is treat as 0 when hash
-static const int INT_VALUE = 0;
-static const TypeDescriptor INT_TYPE(TYPE_INT);
-hash_val = RawValue::zlib_crc32(&INT_VALUE, INT_TYPE,
hash_val);
+hash_val = HashUtil::zlib_crc_hash_null(hash_val);
}
}
return hash_val % num_buckets;
@@ -492,16 +489,13 @@ Status VOlapTablePartitionParam::init() {
uint32_t hash_val = 0;
for (int i = 0; i < _distributed_slot_locs.size(); ++i) {
auto slot_desc = _slots[_distributed_slot_locs[i]];
-auto column =
key->first->get_by_position(_distributed_slot_locs[i]).column;
+auto& column =
key->first->get_by_position(_distributed_slot_locs[i]).column;
auto val = column->get_data_at(key->second);
if (val.data != nullptr) {
hash_val = RawValue::zlib_crc32(val.data, val.size,
slot_desc->type().type,
hash_val);
} else {
-// NULL is treat as 0 when hash
-static const int INT_VALUE = 0;
-static const TypeDescriptor INT_TYPE(TYPE_INT);
-hash_val = RawValue::zlib_crc32(&INT_VALUE, INT_TYPE,
hash_val);
+hash_val = HashUtil::zlib_crc_hash_null(hash_val);
}
}
return hash_val % num_buckets;
diff --git a/be/src/runtime/data_stream_sender.cpp
b/be/src/runtime/data_stream_sender.cpp
index 13b2308d7b..38d5aa1d51 100644
--- a/be/src/runtime/data_stream_sender.cpp
+++ b/be/src/runtime/data_stream_sender.cpp
@@ -626,10 +626,7 @@ Status DataStreamSender::process_distribute(RuntimeState*
state, TupleRow* row,
if (partition_val != nullptr) {
hash_val = RawValue::zlib_crc32(partition_val,
ctx->root()->type(), hash_val);
} else {
-//nullptr is treat as 0 when hash
-static const int INT_VALUE = 0;
-static const TypeDescriptor INT_TYPE(TYPE_INT);
-hash_val = RawValue::zlib_crc32(&INT_VALUE, INT_TYPE, hash_val);
+hash_val = HashUtil::zlib_crc_hash_null(hash_val);
}
}
hash_val %= part->distributed_bucket();
diff --git a/be/src/runtime/define_primitive_type.h
b/be/src/runtime/define_primitive_type.h
new file mode 100644
index 00..aa5e140a6e
--- /dev/null
+++ b/be/src/runtime/defin
[GitHub] [doris] github-actions[bot] commented on pull request #12558: [Bug](column) ColumnNullable::replace_column_data should DCHECK size > sel…
github-actions[bot] commented on PR #12558: URL: https://github.com/apache/doris/pull/12558#issuecomment-1245352608 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12558: [Bug](column) ColumnNullable::replace_column_data should DCHECK size > sel…
github-actions[bot] commented on PR #12558: URL: https://github.com/apache/doris/pull/12558#issuecomment-1245352670 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liwenqi1996 opened a new pull request, #12567: [sample](flink-connector) add function
liwenqi1996 opened a new pull request, #12567: URL: https://github.com/apache/doris/pull/12567 # Proposed changes ## Problem summary Describe your changes. add doris date delete function ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dujl opened a new pull request, #12568: fix ut errors when cherry-pick pr "add tablet erros when close_wait return error"
dujl opened a new pull request, #12568: URL: https://github.com/apache/doris/pull/12568 # Proposed changes Issue Number: close #xxx when cherry-pick https://github.com/apache/doris/commit/75b3707a28a19c20d3b79604dfa80e5803fd59a0 to branch dev-1.1.2, It has some merge conflict This pr fix it. ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [N ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ N] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ N] No Need 4. Does it need to update dependencies: - [ ] Yes - [ N] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ N] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12564: [fix](doc) fix upgrade doc description
github-actions[bot] commented on PR #12564: URL: https://github.com/apache/doris/pull/12564#issuecomment-1245366185 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataalive commented on issue #12561: [Bug]be can't start
dataalive commented on issue #12561: URL: https://github.com/apache/doris/issues/12561#issuecomment-1245370225 kernel version? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dujl commented on pull request #12566: [thirdparty](lib) Add xxhash lib
dujl commented on PR #12566: URL: https://github.com/apache/doris/pull/12566#issuecomment-1245383007 Is there any test data for performance improvement? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] JNSimba commented on a diff in pull request #12567: [sample](flink-connector) add function
JNSimba commented on code in PR #12567: URL: https://github.com/apache/doris/pull/12567#discussion_r969621841 ## samples/doris-demo/flink-demo-v1.1/src/main/java/org/apache/doris/demo/flink/dbsync/DatabaseFullDelSync.java: ## @@ -0,0 +1,238 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +package org.apache.doris.demo.flink.dbsync; + +import com.alibaba.fastjson.JSON; +import com.alibaba.fastjson.JSONObject; +import com.ververica.cdc.connectors.mysql.source.MySqlSource; +import com.ververica.cdc.debezium.JsonDebeziumDeserializationSchema; +import org.apache.doris.flink.cfg.DorisExecutionOptions; +import org.apache.doris.flink.cfg.DorisOptions; +import org.apache.doris.flink.cfg.DorisReadOptions; +import org.apache.doris.flink.sink.DorisSink; +import org.apache.doris.flink.sink.writer.SimpleStringSerializer; +import org.apache.flink.api.common.eventtime.WatermarkStrategy; +import org.apache.flink.api.common.functions.FilterFunction; +import org.apache.flink.api.common.functions.FlatMapFunction; +import org.apache.flink.streaming.api.datastream.DataStreamSource; +import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator; +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.apache.flink.util.Collector; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.*; Review Comment: not import * -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zenoyang commented on a diff in pull request #10084: [feature](vectorized) Support block aggregate in scanner
zenoyang commented on code in PR #10084:
URL: https://github.com/apache/doris/pull/10084#discussion_r969622110
##
be/src/vec/olap/block_aggregator.cpp:
##
@@ -0,0 +1,234 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/olap/block_aggregator.h"
+
+#include "util/simd/bits.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/columns_number.h"
+#include "vec/core/types.h"
+#include "vec/data_types/data_type_factory.hpp"
+
+namespace doris::vectorized {
+
+template
+inline void compare_previous(const IColumn& col, uint8_t* flags) {
+if (col.is_nullable()) {
+auto col_ptr = reinterpret_cast(&col);
+for (int i = 1; i < col_ptr->size(); ++i) {
+flags[i] &= (col_ptr->compare_at(i, i - 1, col, -1) == 0);
+}
+} else {
+auto col_ptr = reinterpret_cast(&col);
+for (int i = 1; i < col_ptr->size(); ++i) {
+flags[i] &= (col_ptr->compare_at(i, i - 1, col, -1) == 0);
+}
+}
+}
+
+BlockAggregator::BlockAggregator(const TabletSchema* tablet_schema,
+ const std::vector& output_columns,
+ const std::vector& return_columns,
+ const std::unordered_set* null_set,
const int batch_size)
+: _tablet_schema(tablet_schema),
+ _output_columns(output_columns),
+ _sorted_output_columns(output_columns),
+ _return_columns(return_columns),
+ _null_set(null_set),
+ _batch_size(batch_size) {
+_has_agg_data = false;
+_do_aggregate = true;
+_agg_ratio = config::block_aggregate_ratio;
+_num_columns = _output_columns.size();
+_num_key_columns = 0;
+
+// todo(zeno) Make sure column prune done
+std::sort(_sorted_output_columns.begin(), _sorted_output_columns.end());
+_output_columns_loc.resize(_num_columns);
+for (int i = 0; i < _num_columns; ++i) {
+for (int j = 0; j < _num_columns; ++j) {
+if (_sorted_output_columns[i] == output_columns[j]) {
+_output_columns_loc[i] = j;
+break;
+}
+}
+}
+
+for (auto index : _sorted_output_columns) {
+if (_tablet_schema->column(index).is_key()) {
+_num_key_columns++;
+}
+}
+
+_current_row = 0;
+_source_size = 0;
+_eq_previous.reserve(_batch_size);
+_agg_index.reserve(_batch_size);
+_agg_range.reserve(_batch_size);
+
+// init _key_comparator
+for (int i = 0; i < _num_key_columns; ++i) {
+_key_comparator.emplace_back(
+
_get_comparator(_tablet_schema->column(_sorted_output_columns[i])));
+}
+
+// init _column_aggregator
+for (int i = 0; i < _num_key_columns; ++i) {
+
_column_aggregator.emplace_back(std::make_unique());
+}
+
+for (int i = _num_key_columns; i < _num_columns; ++i) {
+bool is_nullable = (_null_set != nullptr &&
+_null_set->find(_sorted_output_columns[i]) !=
_null_set->end());
+auto col = _tablet_schema->column(_sorted_output_columns[i]);
+auto data_type =
vectorized::DataTypeFactory::instance().create_data_type(col, is_nullable);
+
_column_aggregator.emplace_back(std::make_unique(col,
data_type));
+}
+
+aggregate_reset();
+}
+
+CompareFunc BlockAggregator::_get_comparator(const TabletColumn& col) {
+switch (col.type()) {
+case OLAP_FIELD_TYPE_TINYINT:
+return &compare_previous;
+case OLAP_FIELD_TYPE_SMALLINT:
+return &compare_previous;
+case OLAP_FIELD_TYPE_INT:
+return &compare_previous;
+case OLAP_FIELD_TYPE_BIGINT:
+return &compare_previous;
+case OLAP_FIELD_TYPE_LARGEINT:
+return &compare_previous;
+case OLAP_FIELD_TYPE_BOOL:
+return &compare_previous;
+case OLAP_FIELD_TYPE_FLOAT:
+return &compare_previous;
+case OLAP_FIELD_TYPE_DOUBLE:
+return &compare_previous;
+case OLAP_FIELD_TYPE_DECIMAL:
+return &compare_previous>;
+case OLAP_FIELD_TYPE_CHAR:
+case OLAP_FIELD_TYPE_VARCHAR:
+case OLAP_FIELD_TYPE_STRING:
+return &compare_previous;
+
