[GitHub] [doris] luozenglin opened a new pull request, #11948: [enhancement](ldap) optimize LDAP authentication.
luozenglin opened a new pull request, #11948: URL: https://github.com/apache/doris/pull/11948 1. Support caching LDAP user information. 2. HTTP authentication supports LDAP. 3. LDAP temporary users support default user property. 4. LDAP configuration supports the `admin show config` and `admin set config` commands. # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [x] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [x] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #11950: [refactor](Nereids): merge rewrite job
jackwener opened a new pull request, #11950: URL: https://github.com/apache/doris/pull/11950 # Proposed changes Issue Number: close #xxx ## Problem summary merge rewrite job ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary commented on a diff in pull request #11945: [improve](fe): use `Pair.of` to replace `new Pair<>()`
stalary commented on code in PR #11945:
URL: https://github.com/apache/doris/pull/11945#discussion_r950812804
##
fe/fe-core/src/main/java/org/apache/doris/common/Pair.java:
##
@@ -37,41 +38,31 @@
@SerializedName(value = "second")
public S second;
-public Pair(F first, S second) {
+private Pair(F first, S second) {
this.first = first;
this.second = second;
}
-public static Pair create(F first, S second) {
-return new Pair(first, second);
+public static Pair of(F first, S second) {
+return Pair.of(first, second);
Review Comment:
endless loop？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #11945: [improve](fe): use `Pair.of` to replace `new Pair<>()`
jackwener commented on code in PR #11945:
URL: https://github.com/apache/doris/pull/11945#discussion_r950814351
##
fe/fe-core/src/main/java/org/apache/doris/common/Pair.java:
##
@@ -37,41 +38,31 @@
@SerializedName(value = "second")
public S second;
-public Pair(F first, S second) {
+private Pair(F first, S second) {
this.first = first;
this.second = second;
}
-public static Pair create(F first, S second) {
-return new Pair(first, second);
+public static Pair of(F first, S second) {
+return Pair.of(first, second);
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary commented on pull request #11945: [improve](fe): use `Pair.of` to replace `new Pair<>()`
stalary commented on PR #11945: URL: https://github.com/apache/doris/pull/11945#issuecomment-1221507173 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary commented on issue #11932: [Bug] doris的group_concat函数不支持distinct
stalary commented on issue #11932: URL: https://github.com/apache/doris/issues/11932#issuecomment-1221508234 Wait for 1.1.2 publish, you can use it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary commented on a diff in pull request #11942: [Enhancement](restore) restore table or partition with the same replication num as before the backup
stalary commented on code in PR #11942:
URL: https://github.com/apache/doris/pull/11942#discussion_r950816440
##
fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java:
##
@@ -162,14 +166,17 @@ public RestoreJob() {
}
public RestoreJob(String label, String backupTs, long dbId, String dbName,
BackupJobInfo jobInfo, boolean allowLoad,
-ReplicaAllocation replicaAlloc, long timeoutMs, int metaVersion,
Env env, long repoId) {
+ReplicaAllocation replicaAlloc, long timeoutMs, int metaVersion,
boolean reserveReplica, Env env,
+long repoId) {
super(JobType.RESTORE, label, dbId, dbName, timeoutMs, env, repoId);
this.backupTimestamp = backupTs;
this.jobInfo = jobInfo;
this.allowLoad = allowLoad;
this.replicaAlloc = replicaAlloc;
this.state = RestoreJobState.PENDING;
this.metaVersion = metaVersion;
+this.reserveReplica = reserveReplica;
+properties.put(PROP_RESERVE_REPLICA, String.valueOf(reserveReplica));
Review Comment:
Why assign a value here, which looks different from the rest of the
parameters?
##
docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/RESTORE.md:
##
@@ -56,6 +56,7 @@ PROPERTIES ("key"="value", ...);
- PROPERTIES 目前支持以下属性：
- "backup_timestamp" = "2018-05-04-16-45-08"：指定了恢复对应备份的哪个时间版本，必填。该信息可以通过
`SHOW SNAPSHOT ON repo;` 语句获得。
- "replication_num" =
"3"：指定恢复的表或分区的副本数。默认为3。若恢复已存在的表或分区，则副本数必须和已存在表或分区的副本数相同。同时，必须有足够的 host 容纳多个副本。
+ - "reserve_replica" =
"true":默认为false。当该属性为true时，会忽略'replication_num'属性，恢复的表或分区的副本数将与备份之前一样。支持多个表或表内多个分区有不同的副本数。
Review Comment:
```suggestion
- "reserve_replica" = "true"：默认为 false。当该属性为 true 时，会忽略 replication_num
属性，恢复的表或分区的副本数将与备份之前一样。支持多个表或表内多个分区有不同的副本数。
```
##
fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java:
##
@@ -1858,6 +1893,7 @@ public void readFields(DataInput in) throws IOException {
String value = Text.readString(in);
properties.put(key, value);
}
+reserveReplica =
Boolean.parseBoolean(properties.get(PROP_RESERVE_REPLICA));
Review Comment:
What does this do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary commented on issue #11905: [Bug] Failed to execute CREATE TABLE LIKE or CREATE TABLE AS SELECT When the SQL statement contains DEFAULT CURRENT_TIMESTAMP
stalary commented on issue #11905: URL: https://github.com/apache/doris/issues/11905#issuecomment-1221516327 Let me have a look, this may need support, can you post your sql? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11927: [feature](spark-load)Spark load supports string type data import
github-actions[bot] commented on PR #11927: URL: https://github.com/apache/doris/pull/11927#issuecomment-1221527876 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11801: [improvement](agg) Serialize the fixed-length aggregation results with corresponding columns instead of ColumnString
github-actions[bot] commented on PR #11801: URL: https://github.com/apache/doris/pull/11801#issuecomment-1221541815 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11945: [improve](fe): use `Pair.of` to replace `new Pair<>()`
yiguolei merged PR #11945: URL: https://github.com/apache/doris/pull/11945 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improve](fe): use `Pair.of` to replace `new Pair<>()` (#11945)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new adfef85c0c [improve](fe): use `Pair.of` to replace `new Pair<>()` (#11945) adfef85c0c is described below commit adfef85c0ca0093927c9bf447a2f42e7fbabd08e Author: jakevin AuthorDate: Mon Aug 22 08:27:40 2022 +0800 [improve](fe): use `Pair.of` to replace `new Pair<>()` (#11945) --- .../apache/doris/alter/SchemaChangeHandler.java| 2 +- .../org/apache/doris/analysis/BinaryPredicate.java | 2 +- .../java/org/apache/doris/analysis/CastExpr.java | 8 ++-- .../org/apache/doris/analysis/DataDescription.java | 2 +- .../java/org/apache/doris/analysis/InsertStmt.java | 4 +- .../org/apache/doris/analysis/LoadColumnsInfo.java | 2 +- .../org/apache/doris/analysis/PartitionName.java | 4 +- .../analysis/RoutineLoadDataSourceProperties.java | 2 +- .../java/org/apache/doris/analysis/SelectStmt.java | 2 +- .../apache/doris/analysis/SetUserPropertyStmt.java | 2 +- .../java/org/apache/doris/analysis/TableRef.java | 2 +- .../org/apache/doris/backup/BackupHandler.java | 4 +- .../org/apache/doris/backup/BrokerStorage.java | 2 +- .../java/org/apache/doris/backup/Repository.java | 2 +- .../java/org/apache/doris/backup/RestoreJob.java | 4 +- .../java/org/apache/doris/catalog/Database.java| 2 +- .../main/java/org/apache/doris/catalog/Env.java| 14 +++--- .../org/apache/doris/catalog/InfoSchemaDb.java | 2 +- .../org/apache/doris/catalog/SparkResource.java| 2 +- .../main/java/org/apache/doris/catalog/Tablet.java | 36 +++ .../main/java/org/apache/doris/catalog/Type.java | 2 +- .../doris/clone/DynamicPartitionScheduler.java | 6 +-- .../apache/doris/clone/PartitionRebalancer.java| 2 +- .../java/org/apache/doris/clone/TabletChecker.java | 4 +- .../org/apache/doris/clone/TabletScheduler.java| 2 +- .../clone/TwoDimensionalGreedyRebalanceAlgo.java | 2 +- .../main/java/org/apache/doris/common/Pair.java| 27 --- .../doris/common/proc/FrontendsProcNode.java | 2 +- .../apache/doris/common/profile/CounterNode.java | 2 +- .../org/apache/doris/common/util/BrokerUtil.java | 2 +- .../org/apache/doris/common/util/DebugUtil.java| 4 +- .../org/apache/doris/common/util/KafkaUtil.java| 4 +- .../apache/doris/common/util/RuntimeProfile.java | 6 +-- .../org/apache/doris/deploy/DeployManager.java | 12 ++--- .../apache/doris/deploy/impl/K8sDeployManager.java | 2 +- .../apache/doris/httpv2/rest/MetaInfoAction.java | 4 +- .../doris/httpv2/rest/manager/HttpUtils.java | 2 +- .../doris/httpv2/rest/manager/NodeAction.java | 6 +-- .../doris/httpv2/restv2/MetaInfoActionV2.java | 4 +- .../src/main/java/org/apache/doris/load/Load.java | 4 +- .../main/java/org/apache/doris/load/Source.java| 2 +- .../org/apache/doris/load/loadv2/SparkLoadJob.java | 4 +- .../doris/load/routineload/KafkaProgress.java | 8 ++-- .../load/routineload/KafkaRoutineLoadJob.java | 2 +- .../apache/doris/mysql/privilege/UserProperty.java | 2 +- .../doris/mysql/privilege/UserPropertyInfo.java| 2 +- .../java/org/apache/doris/nereids/memo/Group.java | 6 +-- .../apache/doris/nereids/memo/GroupExpression.java | 4 +- .../java/org/apache/doris/nereids/memo/Memo.java | 8 ++-- .../rules/exploration/join/JoinLAsscomHelper.java | 2 +- .../org/apache/doris/nereids/util/JoinUtils.java | 2 +- .../apache/doris/persist/BackendTabletsInfo.java | 2 +- .../apache/doris/planner/DistributedPlanner.java | 2 +- .../org/apache/doris/planner/HashJoinNode.java | 2 +- .../apache/doris/planner/PredicatePushDown.java| 4 +- .../apache/doris/planner/SingleNodePlanner.java| 6 +-- .../java/org/apache/doris/qe/ConnectProcessor.java | 4 +- .../main/java/org/apache/doris/qe/Coordinator.java | 10 ++-- .../java/org/apache/doris/qe/SimpleScheduler.java | 2 +- .../org/apache/doris/rewrite/InferFiltersRule.java | 24 +- .../doris/statistics/OlapScanStatsDerive.java | 4 +- .../org/apache/doris/system/SystemInfoService.java | 2 +- .../org/apache/doris/task/ExportPendingTask.java | 2 +- .../org/apache/doris/task/HadoopLoadEtlTask.java | 2 +- .../doris/transaction/DatabaseTransactionMgr.java | 2 +- .../org/apache/doris/backup/BrokerStorageTest.java | 2 +- .../org/apache/doris/catalog/AdminStmtTest.java| 2 +- .../java/org/apache/doris/catalog/TabletTest.java | 8 ++-- .../org/apache/doris/catalog/UserPropertyTest.java | 28 +-- .../java/org/apache/doris/clone/RebalanceTest.java | 8 ++-- .../TwoDimensionalGreedyRebalanceAlgoTest.java | 2 +- .../apache/doris/load/loadv2/SparkLoadJobTest.java | 2 +- .../load/routineload/KafkaRoutineLoadJobTest.java | 2 +- .../exploration
[GitHub] [doris] yiguolei merged pull request #11946: [improve](nereids): remove FakeJoin.java
yiguolei merged PR #11946: URL: https://github.com/apache/doris/pull/11946 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improve](nereids): remove FakeJoin.java (#11946)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 19496ef9a0 [improve](nereids): remove FakeJoin.java (#11946)
19496ef9a0 is described below
commit 19496ef9a017922b7a950de1377a634d28d7c980
Author: jakevin
AuthorDate: Mon Aug 22 08:28:20 2022 +0800
[improve](nereids): remove FakeJoin.java (#11946)
---
.../doris/nereids/stats/StatsCalculatorTest.java | 14 ---
.../apache/doris/nereids/trees/plans/FakeJoin.java | 43 --
2 files changed, 8 insertions(+), 49 deletions(-)
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/stats/StatsCalculatorTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/stats/StatsCalculatorTest.java
index 290cbd7d32..67045678aa 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/stats/StatsCalculatorTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/stats/StatsCalculatorTest.java
@@ -32,11 +32,11 @@ import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.expressions.functions.AggregateFunction;
import org.apache.doris.nereids.trees.expressions.functions.Sum;
-import org.apache.doris.nereids.trees.plans.FakeJoin;
import org.apache.doris.nereids.trees.plans.GroupPlan;
import org.apache.doris.nereids.trees.plans.JoinType;
import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.trees.plans.logical.LogicalLimit;
import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
@@ -165,9 +165,7 @@ public class StatsCalculatorTest {
@Test
public void testHashJoin() {
-List qualifier = new ArrayList<>();
-qualifier.add("test");
-qualifier.add("t");
+List qualifier = ImmutableList.of("test", "t");
SlotReference slot1 = new SlotReference("c1", IntegerType.INSTANCE,
true, qualifier);
SlotReference slot2 = new SlotReference("c2", IntegerType.INSTANCE,
true, qualifier);
ColumnStats columnStats1 = new ColumnStats();
@@ -190,8 +188,12 @@ public class StatsCalculatorTest {
EqualTo equalTo = new EqualTo(slot1, slot2);
-FakeJoin fakeSemiJoin = new FakeJoin(JoinType.LEFT_SEMI_JOIN,
Optional.of(equalTo));
-FakeJoin fakeInnerJoin = new FakeJoin(JoinType.INNER_JOIN,
Optional.of(equalTo));
+LogicalOlapScan scan1 = PlanConstructor.newLogicalOlapScan(0, "t", 0);
+LogicalOlapScan scan2 = PlanConstructor.newLogicalOlapScan(0, "t", 0);
+LogicalJoin fakeSemiJoin = new
LogicalJoin<>(
+JoinType.LEFT_SEMI_JOIN, Optional.of(equalTo), scan1, scan2);
+LogicalJoin fakeInnerJoin = new
LogicalJoin<>(
+JoinType.INNER_JOIN, Optional.of(equalTo), scan1, scan2);
StatsDeriveResult semiJoinStats = JoinEstimation.estimate(leftStats,
rightStats, fakeSemiJoin);
Assertions.assertEquals(leftRowCount, semiJoinStats.getRowCount());
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/FakeJoin.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/FakeJoin.java
deleted file mode 100644
index 431abccae1..00
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/FakeJoin.java
+++ /dev/null
@@ -1,43 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.nereids.trees.plans;
-
-import org.apache.doris.nereids.trees.expressions.Expression;
-import org.apache.doris.nereids.trees.plans.algebra.Join;
-
-import java.util.Optional;
-
-public class FakeJoin implements Join {
-private final JoinType joinType;
-private final Optional condition;
-
-public FakeJoin(JoinType joinType, Optional condition) {
-this.joinType = joinType;
-this.condition = condition;
[GitHub] [doris] yiguolei merged pull request #11939: [fix](memtracker) Remove useless memory exceed check
yiguolei merged PR #11939: URL: https://github.com/apache/doris/pull/11939 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](memtracker) Remove useless memory exceed check #11939
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5eb576 [fix](memtracker) Remove useless memory exceed check #11939
5eb576 is described below
commit 5eb576a32cf172aa462e4ab7755510868656
Author: Xinyi Zou
AuthorDate: Mon Aug 22 08:40:19 2022 +0800
[fix](memtracker) Remove useless memory exceed check #11939
---
be/src/exec/broker_scan_node.cpp | 6 +-
be/src/exec/tablet_sink.cpp | 3 +--
be/src/vec/exec/file_scan_node.cpp| 6 +-
be/src/vec/exec/vbroker_scan_node.cpp | 6 +-
be/src/vec/sink/vtablet_sink.cpp | 8 ++--
5 files changed, 6 insertions(+), 23 deletions(-)
diff --git a/be/src/exec/broker_scan_node.cpp b/be/src/exec/broker_scan_node.cpp
index 32baad6fcd..9cff2da4ff 100644
--- a/be/src/exec/broker_scan_node.cpp
+++ b/be/src/exec/broker_scan_node.cpp
@@ -348,11 +348,7 @@ Status BrokerScanNode::scanner_scan(const
TBrokerScanRange& scan_range,
// stop pushing more batch if
// 1. too many batches in queue, or
// 2. at least one batch in queue and memory exceed limit.
- (_batch_queue.size() >= _max_buffered_batches ||
-(thread_context()
- ->_thread_mem_tracker_mgr->limiter_mem_tracker()
- ->any_limit_exceeded() &&
- !_batch_queue.empty( {
+ (_batch_queue.size() >= _max_buffered_batches ||
!_batch_queue.empty())) {
_queue_writer_cond.wait_for(l, std::chrono::seconds(1));
}
// Process already set failed, so we just return OK
diff --git a/be/src/exec/tablet_sink.cpp b/be/src/exec/tablet_sink.cpp
index 281f94aee2..cdb503551b 100644
--- a/be/src/exec/tablet_sink.cpp
+++ b/be/src/exec/tablet_sink.cpp
@@ -303,8 +303,7 @@ Status NodeChannel::add_row(Tuple* input_tuple, int64_t
tablet_id) {
// _cancelled may be set by rpc callback, and it's possible that
_cancelled might be set in any of the steps below.
// It's fine to do a fake add_row() and return OK, because we will check
_cancelled in next add_row() or mark_close().
while (!_cancelled && _pending_batches_num > 0 &&
- (_pending_batches_bytes > _max_pending_batches_bytes ||
-
_parent->_mem_tracker->limit_exceeded(_max_pending_batches_bytes))) {
+ _pending_batches_bytes > _max_pending_batches_bytes) {
SCOPED_ATOMIC_TIMER(&_mem_exceeded_block_ns);
std::this_thread::sleep_for(std::chrono::milliseconds(10));
}
diff --git a/be/src/vec/exec/file_scan_node.cpp
b/be/src/vec/exec/file_scan_node.cpp
index ec8a5165ce..e84c0e9371 100644
--- a/be/src/vec/exec/file_scan_node.cpp
+++ b/be/src/vec/exec/file_scan_node.cpp
@@ -403,11 +403,7 @@ Status FileScanNode::scanner_scan(const TFileScanRange&
scan_range, ScannerCount
// stop pushing more batch if
// 1. too many batches in queue, or
// 2. at least one batch in queue and memory exceed limit.
- (_block_queue.size() >= _max_buffered_batches ||
-(thread_context()
- ->_thread_mem_tracker_mgr->limiter_mem_tracker()
- ->any_limit_exceeded() &&
- !_block_queue.empty( {
+ (_block_queue.size() >= _max_buffered_batches ||
!_block_queue.empty())) {
_queue_writer_cond.wait_for(l, std::chrono::seconds(1));
}
// Process already set failed, so we just return OK
diff --git a/be/src/vec/exec/vbroker_scan_node.cpp
b/be/src/vec/exec/vbroker_scan_node.cpp
index 98c16112c1..2f0593b3f6 100644
--- a/be/src/vec/exec/vbroker_scan_node.cpp
+++ b/be/src/vec/exec/vbroker_scan_node.cpp
@@ -246,11 +246,7 @@ Status VBrokerScanNode::scanner_scan(const
TBrokerScanRange& scan_range, Scanner
// stop pushing more batch if
// 1. too many batches in queue, or
// 2. at least one batch in queue and memory exceed limit.
- (_block_queue.size() >= _max_buffered_batches ||
-(thread_context()
- ->_thread_mem_tracker_mgr->limiter_mem_tracker()
- ->any_limit_exceeded() &&
- !_block_queue.empty( {
+ (_block_queue.size() >= _max_buffered_batches ||
!_block_queue.empty())) {
_queue_writer_cond.wait_for(l, std::chrono::seconds(1));
}
// Process already set failed, so we just return OK
diff --git a/be/src/vec/sink/vtablet_sink.cpp b/be/src/vec/sink/vtablet_sink.cpp
index 4965562c3c..7a0faee0d6 100644
--- a/be/src/vec/sink/vtablet_sink.cpp
+++ b/be/src/vec/sink/vtablet_sink.cpp
@@ -178,12 +178,8 @@ Status VNodeChannel::add_row(con
[GitHub] [doris] yiguolei merged pull request #11944: [dev-1.1.2](memory) Default `STRICT_MEMORY_USE` reduce hash table memory
yiguolei merged PR #11944: URL: https://github.com/apache/doris/pull/11944 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch dev-1.1.2 updated: [dev-1.1.2](memory) Default STRICT_MEMORY_USE reduce hash table memory #11944
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/dev-1.1.2 by this push:
new 642abd91b3 [dev-1.1.2](memory) Default STRICT_MEMORY_USE reduce hash
table memory #11944
642abd91b3 is described below
commit 642abd91b362089d066299dd37329e8b94209031
Author: Xinyi Zou
AuthorDate: Mon Aug 22 08:45:48 2022 +0800
[dev-1.1.2](memory) Default STRICT_MEMORY_USE reduce hash table memory
#11944
---
be/src/vec/common/hash_table/hash_table.h | 2 +-
build.sh | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/be/src/vec/common/hash_table/hash_table.h
b/be/src/vec/common/hash_table/hash_table.h
index a60e71e42b..ed59476083 100644
--- a/be/src/vec/common/hash_table/hash_table.h
+++ b/be/src/vec/common/hash_table/hash_table.h
@@ -244,7 +244,7 @@ template
struct HashTableGrower {
/// The state of this structure is enough to get the buffer size of the
hash table.
doris::vectorized::UInt8 size_degree = initial_size_degree;
-doris::vectorized::Int64 double_grow_degree = 31; // 2GB
+doris::vectorized::Int64 double_grow_degree = 26; // 64M
/// The size of the hash table in the cells.
size_t buf_size() const { return 1ULL << size_degree; }
diff --git a/build.sh b/build.sh
index c58dfc61bc..ad0cf680b0 100755
--- a/build.sh
+++ b/build.sh
@@ -223,7 +223,7 @@ if [[ -z ${USE_MEM_TRACKER} ]]; then
USE_MEM_TRACKER=ON
fi
if [[ -z ${STRICT_MEMORY_USE} ]]; then
-STRICT_MEMORY_USE=OFF
+STRICT_MEMORY_USE=ON
fi
echo "Get params:
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11943: [dev-1.1.2](cherry-pick) Optimize readability of mem exceed limit error message
yiguolei merged PR #11943: URL: https://github.com/apache/doris/pull/11943 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch dev-1.1.2 updated: [dev-1.1.2](cherry-pick) Optimize readability of mem exceed limit error message #11943
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/dev-1.1.2 by this push:
new 3e2ae923f7 [dev-1.1.2](cherry-pick) Optimize readability of mem exceed
limit error message #11943
3e2ae923f7 is described below
commit 3e2ae923f7aa5f4d3f3e4dc9c64a9982a15c764d
Author: Xinyi Zou
AuthorDate: Mon Aug 22 08:46:34 2022 +0800
[dev-1.1.2](cherry-pick) Optimize readability of mem exceed limit error
message #11943
---
be/src/runtime/memory/mem_tracker.cpp| 39 ++---
be/src/runtime/memory/mem_tracker.h | 8 +--
be/src/runtime/memory/mem_tracker_limiter.cpp| 70 +++-
be/src/runtime/memory/mem_tracker_limiter.h | 29 +++---
be/src/runtime/memory/mem_tracker_task_pool.cpp | 4 +-
be/src/runtime/memory/thread_mem_tracker_mgr.cpp | 16 ++
be/src/runtime/memory/thread_mem_tracker_mgr.h | 4 +-
be/src/runtime/runtime_state.cpp | 4 +-
be/src/runtime/thread_context.h | 9 +--
be/src/service/doris_main.cpp| 1 +
10 files changed, 116 insertions(+), 68 deletions(-)
diff --git a/be/src/runtime/memory/mem_tracker.cpp
b/be/src/runtime/memory/mem_tracker.cpp
index dda2c43102..c4801d6611 100644
--- a/be/src/runtime/memory/mem_tracker.cpp
+++ b/be/src/runtime/memory/mem_tracker.cpp
@@ -42,7 +42,7 @@ struct TrackerGroup {
// Multiple groups are used to reduce the impact of locks.
static std::vector mem_tracker_pool(1000);
-NewMemTracker::NewMemTracker(const std::string& label, RuntimeProfile*
profile, bool is_limiter) {
+NewMemTracker::NewMemTracker(const std::string& label, RuntimeProfile*
profile) {
if (profile == nullptr) {
_consumption =
std::make_shared(TUnit::BYTES);
} else {
@@ -58,30 +58,24 @@ NewMemTracker::NewMemTracker(const std::string& label,
RuntimeProfile* profile,
_consumption = profile->AddSharedHighWaterMarkCounter(COUNTER_NAME,
TUnit::BYTES);
}
-_is_limiter = is_limiter;
-if (!_is_limiter) {
-if (thread_context()->_thread_mem_tracker_mgr->limiter_mem_tracker()) {
-_label = fmt::format(
-"{} | {}", label,
-
thread_context()->_thread_mem_tracker_mgr->limiter_mem_tracker()->label());
-} else {
-_label = label + " | ";
-}
-
-_bind_group_num =
-
thread_context()->_thread_mem_tracker_mgr->limiter_mem_tracker()->group_num();
-{
-std::lock_guard
l(mem_tracker_pool[_bind_group_num].group_lock);
-_tracker_group_it =
mem_tracker_pool[_bind_group_num].trackers.insert(
-mem_tracker_pool[_bind_group_num].trackers.end(), this);
-}
+if (thread_context()->_thread_mem_tracker_mgr->limiter_mem_tracker()) {
+_label = fmt::format(
+"{} | {}", label,
+
thread_context()->_thread_mem_tracker_mgr->limiter_mem_tracker()->label());
} else {
-_label = label;
+_label = label + " | ";
+}
+
+_bind_group_num =
thread_context()->_thread_mem_tracker_mgr->limiter_mem_tracker()->group_num();
+{
+std::lock_guard
l(mem_tracker_pool[_bind_group_num].group_lock);
+_tracker_group_it = mem_tracker_pool[_bind_group_num].trackers.insert(
+mem_tracker_pool[_bind_group_num].trackers.end(), this);
}
}
NewMemTracker::~NewMemTracker() {
-if (!_is_limiter) {
+if (_bind_group_num != -1) {
std::lock_guard
l(mem_tracker_pool[_bind_group_num].group_lock);
if (_tracker_group_it !=
mem_tracker_pool[_bind_group_num].trackers.end()) {
mem_tracker_pool[_bind_group_num].trackers.erase(_tracker_group_it);
@@ -102,8 +96,9 @@ NewMemTracker::Snapshot NewMemTracker::make_snapshot(size_t
level) const {
return snapshot;
}
-void NewMemTracker::make_group_snapshot(std::vector*
snapshots, size_t level,
- int64_t group_num, std::string
related_label) {
+void NewMemTracker::make_group_snapshot(std::vector*
snapshots,
+size_t level, int64_t group_num,
+std::string related_label) {
std::lock_guard l(mem_tracker_pool[group_num].group_lock);
for (auto tracker : mem_tracker_pool[group_num].trackers) {
if (split(tracker->label(), " | ")[1] == related_label) {
diff --git a/be/src/runtime/memory/mem_tracker.h
b/be/src/runtime/memory/mem_tracker.h
index 41d2fe4d1c..258f244aae 100644
--- a/be/src/runtime/memory/mem_tracker.h
+++ b/be/src/runtime/memory/mem_tracker.h
@@ -44,8 +44,9 @@ public:
};
// Creates and adds the tracker to the mem_tracker_pool.
-NewMemTracker(const std::string& label = std::string(), RuntimeProfile*
profile = nullptr,
[GitHub] [doris] yiguolei merged pull request #11921: [refractor](bitmap) bitmap serialize and deserialize refractor
yiguolei merged PR #11921: URL: https://github.com/apache/doris/pull/11921 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [refractor](bitmap) bitmap serialize and deserialize refractor (#11921)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 83ea4ea984 [refractor](bitmap) bitmap serialize and deserialize
refractor (#11921)
83ea4ea984 is described below
commit 83ea4ea984a462a2c7e1f8915c1a5ad36f21cd30
Author: camby <104178...@qq.com>
AuthorDate: Mon Aug 22 08:52:20 2022 +0800
[refractor](bitmap) bitmap serialize and deserialize refractor (#11921)
Co-authored-by: cambyzju
---
be/src/vec/data_types/data_type_bitmap.cpp | 40 ++---
be/test/vec/core/column_complex_test.cpp | 56 ++
2 files changed, 74 insertions(+), 22 deletions(-)
diff --git a/be/src/vec/data_types/data_type_bitmap.cpp
b/be/src/vec/data_types/data_type_bitmap.cpp
index cb1a2f2e45..5c49f9c65c 100644
--- a/be/src/vec/data_types/data_type_bitmap.cpp
+++ b/be/src/vec/data_types/data_type_bitmap.cpp
@@ -45,24 +45,22 @@ char* DataTypeBitMap::serialize(const IColumn& column,
char* buf) const {
auto& data_column = assert_cast(*ptr);
// serialize the bitmap size array, row num saves at index 0
-const auto row_num = column.size();
-size_t bitmap_size_array[row_num + 1];
-bitmap_size_array[0] = row_num;
-for (size_t i = 0; i < row_num; ++i) {
+size_t* meta_ptr = (size_t*)buf;
+meta_ptr[0] = column.size();
+for (size_t i = 0; i < meta_ptr[0]; ++i) {
auto& bitmap = const_cast(data_column.get_element(i));
-bitmap_size_array[i + 1] = bitmap.getSizeInBytes();
+meta_ptr[i + 1] = bitmap.getSizeInBytes();
}
-auto allocate_len_size = sizeof(size_t) * (row_num + 1);
-memcpy(buf, bitmap_size_array, allocate_len_size);
-buf += allocate_len_size;
+
// serialize each bitmap
-for (size_t i = 0; i < row_num; ++i) {
+char* data_ptr = buf + sizeof(size_t) * (meta_ptr[0] + 1);
+for (size_t i = 0; i < meta_ptr[0]; ++i) {
auto& bitmap = const_cast(data_column.get_element(i));
-bitmap.write(buf);
-buf += bitmap_size_array[i + 1];
+bitmap.write(data_ptr);
+data_ptr += meta_ptr[i + 1];
}
-return buf;
+return data_ptr;
}
const char* DataTypeBitMap::deserialize(const char* buf, IColumn* column)
const {
@@ -70,19 +68,17 @@ const char* DataTypeBitMap::deserialize(const char* buf,
IColumn* column) const
auto& data = data_column.get_data();
// deserialize the bitmap size array
-size_t row_num = *reinterpret_cast(buf);
-buf += sizeof(size_t);
-size_t bitmap_size_array[row_num];
-memcpy(bitmap_size_array, buf, sizeof(size_t) * row_num);
-buf += sizeof(size_t) * row_num;
+const size_t* meta_ptr = reinterpret_cast(buf);
+
// deserialize each bitmap
-data.resize(row_num);
-for (int i = 0; i < row_num; ++i) {
-data[i].deserialize(buf);
-buf += bitmap_size_array[i];
+data.resize(meta_ptr[0]);
+const char* data_ptr = buf + sizeof(size_t) * (meta_ptr[0] + 1);
+for (size_t i = 0; i < meta_ptr[0]; ++i) {
+data[i].deserialize(data_ptr);
+data_ptr += meta_ptr[i + 1];
}
-return buf;
+return data_ptr;
}
MutableColumnPtr DataTypeBitMap::create_column() const {
diff --git a/be/test/vec/core/column_complex_test.cpp
b/be/test/vec/core/column_complex_test.cpp
index 1eb8cd906d..ce9e4d60f3 100644
--- a/be/test/vec/core/column_complex_test.cpp
+++ b/be/test/vec/core/column_complex_test.cpp
@@ -44,4 +44,60 @@ TEST(ColumnComplexTest, BasicTest) {
TEST(ColumnComplexType, DataTypeBitmapTest) {
std::make_shared();
}
+
+class ColumnBitmapTest : public testing::Test {
+public:
+virtual void SetUp() override {}
+virtual void TearDown() override {}
+
+void check_bitmap_column(const IColumn& l, const IColumn& r) {
+ASSERT_EQ(l.size(), r.size());
+const auto& l_col = assert_cast(l);
+const auto& r_col = assert_cast(r);
+for (size_t i = 0; i < l_col.size(); ++i) {
+auto& l_bitmap = const_cast(l_col.get_element(i));
+auto& r_bitmap = const_cast(r_col.get_element(i));
+ASSERT_EQ(l_bitmap.xor_cardinality(r_bitmap), 0);
+}
+}
+
+void check_serialize_and_deserialize(MutableColumnPtr& col) {
+auto column = assert_cast(col.get());
+auto size = _bitmap_type.get_uncompressed_serialized_bytes(*column);
+std::unique_ptr buf = std::make_unique(size);
+auto result = _bitmap_type.serialize(*column, buf.get());
+ASSERT_EQ(result, buf.get() + size);
+
+auto column2 = _bitmap_type.create_column();
+_bitmap_type.deserialize(buf.get(), column2.get());
+check_bitmap_column(*column, *column2.get());
+}
+
+private:
+DataTypeBitMap _bitmap_type;
+};
+
+TEST_F(ColumnBitmapTest, SerializeAndDeserialize) {
+auto column
[doris-website] branch master updated: sidebar sort
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 435683c86f7 sidebar sort
435683c86f7 is described below
commit 435683c86f7eb93b96739cf277e38b1ffb1c0b0c
Author: jiafeng.zhang
AuthorDate: Mon Aug 22 08:55:40 2022 +0800
sidebar sort
---
.../current/data-table/index/bloomfilter.md | 4 ++--
sidebars.json | 8
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-table/index/bloomfilter.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-table/index/bloomfilter.md
index 4382feccaaa..49fcde10dc2 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-table/index/bloomfilter.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-table/index/bloomfilter.md
@@ -42,9 +42,9 @@ BloomFilter是由Bloom在1970年提出的一种多哈希函数映射的快速查
## Doris BloomFilter索引及使用使用场景
-举个例子：如果要查找一个占用100字节存储空间大小的短行，一个64KB的HFile数据块应该包含(64 * 1024)/100 = 655.53 =
~700行，如果仅能在整个数据块的起始行键上建立索引，那么它是无法给你提供细粒度的索引信息的。因为要查找的行数据可能会落在该数据块的行区间上，也可能行数据没在该数据块上，也可能是表中根本就不存在该行数据，也或者是行数据在另一个HFile里，甚至在MemStore里。以上这几种情况，都会导致从磁盘读取数据块时带来额外的IO开销，也会滥用数据块的缓存，当面对一个巨大的数据集且处于高并发读时，会严重影响性能。
+举个例子：如果要查找一个占用100字节存储空间大小的短行，一个64KB的数据块应该包含(64 * 1024)/100 = 655.53 =
~700行，如果仅能在整个数据块的起始行键上建立索引，那么它是无法给你提供细粒度的索引信息的。因为要查找的行数据可能会落在该数据块的行区间上，也可能行数据没在该数据块上，也可能是表中根本就不存在该行数据，也或者是行数据在另一个数据块里，甚至在内存里。以上这几种情况，都会导致从磁盘读取数据块时带来额外的IO开销，也会滥用数据块的缓存，当面对一个巨大的数据集且处于高并发读时，会严重影响性能。
-因此，HBase提供了布隆过滤器，它允许你对存储在每个数据块的数据做一个反向测试。当某行被请求时，通过布隆过滤器先检查该行是否不在这个数据块，布隆过滤器要么确定回答该行不在，要么回答它不知道。这就是为什么我们称它是反向测试。布隆过滤器同样也可以应用到行里的单元上，当访问某列标识符时可以先使用同样的反向测试。
+因此，Doris
提供了布隆过滤器，它允许你对存储在每个数据块的数据做一个反向测试。当某行被请求时，通过布隆过滤器先检查该行是否不在这个数据块，布隆过滤器要么确定回答该行不在，要么回答它不知道。这就是为什么我们称它是反向测试。布隆过滤器同样也可以应用到行里的单元上，当访问某列标识符时可以先使用同样的反向测试。
但布隆过滤器也不是没有代价。存储这个额外的索引层次会占用额外的空间。布隆过滤器随着它们的索引对象数据增长而增长，所以行级布隆过滤器比列标识符级布隆过滤器占用空间要少。当空间不是问题时，它们可以帮助你榨干系统的性能潜力。
Doris的BloomFilter索引可以通过建表的时候指定，或者通过表的ALTER操作来完成。Bloom
Filter本质上是一种位图结构，用于快速的判断一个给定的值是否在一个集合中。这种判断会产生小概率的误判。即如果返回false，则一定不在这个集合内。而如果范围true，则有可能在这个集合内。
diff --git a/sidebars.json b/sidebars.json
index 1999efd7e4c..26d7a10c637 100644
--- a/sidebars.json
+++ b/sidebars.json
@@ -2,16 +2,16 @@
"docs": [
{
"type": "category",
-"label": "Getting Started",
+"label": "Doris Introduction",
"items": [
-"get-starting/get-starting"
+"summary/basic-summary"
]
},
{
"type": "category",
-"label": "Doris Introduction",
+"label": "Getting Started",
"items": [
-"summary/basic-summary"
+"get-starting/get-starting"
]
},
{
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11947: [fix](memtracker) Improve memory tracking accuracy for exec nodes
yiguolei merged PR #11947: URL: https://github.com/apache/doris/pull/11947 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](memtracker) Improve memory tracking accuracy for exec nodes (#11947)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b1fd701493 [fix](memtracker) Improve memory tracking accuracy for exec
nodes (#11947)
b1fd701493 is described below
commit b1fd701493919e89936662f84f151b3071cc4578
Author: Xinyi Zou
AuthorDate: Mon Aug 22 08:56:05 2022 +0800
[fix](memtracker) Improve memory tracking accuracy for exec nodes (#11947)
---
be/src/exec/blocking_join_node.cpp| 1 +
be/src/exec/hash_join_node.cpp| 1 +
be/src/exec/olap_scan_node.cpp| 2 ++
be/src/exec/tablet_sink.cpp | 4 +++-
be/src/runtime/memory/mem_tracker_limiter.cpp | 8
be/src/service/internal_service.cpp | 3 +++
be/src/vec/exec/join/vhash_join_node.cpp | 1 +
be/src/vec/exec/vblocking_join_node.cpp | 1 +
be/src/vec/exec/volap_scan_node.cpp | 2 ++
be/src/vec/sink/vtablet_sink.cpp | 1 +
10 files changed, 19 insertions(+), 5 deletions(-)
diff --git a/be/src/exec/blocking_join_node.cpp
b/be/src/exec/blocking_join_node.cpp
index 10419a3499..5e5aee3714 100644
--- a/be/src/exec/blocking_join_node.cpp
+++ b/be/src/exec/blocking_join_node.cpp
@@ -93,6 +93,7 @@ Status BlockingJoinNode::close(RuntimeState* state) {
void BlockingJoinNode::build_side_thread(RuntimeState* state,
std::promise* status) {
SCOPED_ATTACH_TASK(state);
+SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
status->set_value(construct_build_side(state));
}
diff --git a/be/src/exec/hash_join_node.cpp b/be/src/exec/hash_join_node.cpp
index cbba3c607c..f5a7fec132 100644
--- a/be/src/exec/hash_join_node.cpp
+++ b/be/src/exec/hash_join_node.cpp
@@ -179,6 +179,7 @@ Status HashJoinNode::close(RuntimeState* state) {
void HashJoinNode::build_side_thread(RuntimeState* state,
std::promise* status) {
SCOPED_ATTACH_TASK(state);
+SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
status->set_value(construct_hash_table(state));
}
diff --git a/be/src/exec/olap_scan_node.cpp b/be/src/exec/olap_scan_node.cpp
index 048a922bae..99caa8afb9 100644
--- a/be/src/exec/olap_scan_node.cpp
+++ b/be/src/exec/olap_scan_node.cpp
@@ -1480,6 +1480,7 @@ Status
OlapScanNode::normalize_bloom_filter_predicate(SlotDescriptor* slot) {
void OlapScanNode::transfer_thread(RuntimeState* state) {
// scanner open pushdown to scanThread
SCOPED_ATTACH_TASK(state);
+SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
Status status = Status::OK();
for (auto scanner : _olap_scanners) {
status = Expr::clone_if_not_exists(_conjunct_ctxs, state,
scanner->conjunct_ctxs());
@@ -1663,6 +1664,7 @@ void OlapScanNode::transfer_thread(RuntimeState* state) {
void OlapScanNode::scanner_thread(OlapScanner* scanner) {
SCOPED_ATTACH_TASK(_runtime_state);
+SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
Thread::set_self_name("olap_scanner");
if (UNLIKELY(_transfer_done)) {
_scanner_done = true;
diff --git a/be/src/exec/tablet_sink.cpp b/be/src/exec/tablet_sink.cpp
index cdb503551b..a1213d348d 100644
--- a/be/src/exec/tablet_sink.cpp
+++ b/be/src/exec/tablet_sink.cpp
@@ -468,8 +468,9 @@ int NodeChannel::try_send_and_fetch_status(RuntimeState*
state,
}
void NodeChannel::try_send_batch(RuntimeState* state) {
-SCOPED_ATTACH_TASK(state);
SCOPED_ATOMIC_TIMER(&_actual_consume_ns);
+SCOPED_ATTACH_TASK(state);
+SCOPED_CONSUME_MEM_TRACKER(_node_channel_tracker.get());
AddBatchReq send_batch;
{
debug::ScopedTSANIgnoreReadsAndWrites ignore_tsan;
@@ -1321,6 +1322,7 @@ Status OlapTableSink::_validate_data(RuntimeState* state,
RowBatch* batch, Bitma
void OlapTableSink::_send_batch_process(RuntimeState* state) {
SCOPED_TIMER(_non_blocking_send_timer);
SCOPED_ATTACH_TASK(state);
+SCOPED_CONSUME_MEM_TRACKER(_mem_tracker.get());
do {
int running_channels_num = 0;
for (auto index_channel : _channels) {
diff --git a/be/src/runtime/memory/mem_tracker_limiter.cpp
b/be/src/runtime/memory/mem_tracker_limiter.cpp
index 50eb011028..84209888ef 100644
--- a/be/src/runtime/memory/mem_tracker_limiter.cpp
+++ b/be/src/runtime/memory/mem_tracker_limiter.cpp
@@ -226,11 +226,11 @@ std::string MemTrackerLimiter::log_usage(int
max_recursive_depth,
Status MemTrackerLimiter::mem_limit_exceeded_log(const std::string& msg) {
DCHECK(_limit != -1);
std::string detail = fmt::format(
-"{}, backend={} free memory left={}. If is query, can change the
limit by `set "
-"exec_mem_limit=xxx`, details mem usage see be.INFO.",
+"{}, backend={} memory used={}, free memory left={}. If is query,
can change the limit "
+"by `set exec_mem_limit=xxx`, details mem usage see be.INFO.",
msg, BackendOptions::get_localhost(),
-
[GitHub] [doris-website] hf200012 merged pull request #56: [fix] fix extra space and backslash
hf200012 merged PR #56: URL: https://github.com/apache/doris-website/pull/56 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: fix extra space and backslash (#56)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new ccb4d4e22bf fix extra space and backslash (#56)
ccb4d4e22bf is described below
commit ccb4d4e22bfc14d3c1bf73badf14ca1f35f81874
Author: timey
AuthorDate: Mon Aug 22 08:56:32 2022 +0800
fix extra space and backslash (#56)
fix extra space and backslash
---
.../current/developer-guide/fe-eclipse-dev.md | 2 +-
src/pages/download/index.tsx| 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/developer-guide/fe-eclipse-dev.md
b/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/developer-guide/fe-eclipse-dev.md
index dc2b155982b..b3179e6d084 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/developer-guide/fe-eclipse-dev.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/developer-guide/fe-eclipse-dev.md
@@ -38,7 +38,7 @@ FE 模块需要部分生成代码，如 Thrift、Protobuf, jflex, cup 等框架
windows下开发获取生成代码步骤
-1. 在 Linux 下， 进入 `fe 目录下执行以下命令：
+1. 在 Linux 下， 进入 fe 目录下执行以下命令：
```
mvn generate-sources
diff --git a/src/pages/download/index.tsx b/src/pages/download/index.tsx
index e5d58bf0ec9..7ce8392dda0 100644
--- a/src/pages/download/index.tsx
+++ b/src/pages/download/index.tsx
@@ -169,7 +169,7 @@ export default function Download(): JSX.Element {
If the CPU does not support the avx2
instruction set, select the no avx2 version. You
can check whether it is supported by
-cat /proc/cpu info
+cat /proc/cpuinfo
. The avx2 instruction will improve the
computational efficiency of data structures such
as bloom filter.
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #11926: [Bug] Spark load does not support string type data import
yiguolei closed issue #11926: [Bug] Spark load does not support string type data import URL: https://github.com/apache/doris/issues/11926 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11927: [feature](spark-load)Spark load supports string type data import
yiguolei merged PR #11927: URL: https://github.com/apache/doris/pull/11927 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature](spark-load)Spark load supports string type data import (#11927)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 915d8989c5 [feature](spark-load)Spark load supports string type data
import (#11927)
915d8989c5 is described below
commit 915d8989c5774638190d487d77c2079a422aea27
Author: jiafeng.zhang
AuthorDate: Mon Aug 22 08:56:59 2022 +0800
[feature](spark-load)Spark load supports string type data import (#11927)
---
be/src/olap/push_handler.cpp | 1 +
.../apache/doris/load/loadv2/dpp/ColumnParser.java | 23 ++
.../org/apache/doris/load/loadv2/dpp/DppUtils.java | 6 ++
.../org/apache/doris/load/loadv2/dpp/SparkDpp.java | 12 +++
4 files changed, 42 insertions(+)
diff --git a/be/src/olap/push_handler.cpp b/be/src/olap/push_handler.cpp
index 7cae605904..e6b858015a 100644
--- a/be/src/olap/push_handler.cpp
+++ b/be/src/olap/push_handler.cpp
@@ -905,6 +905,7 @@ Status PushBrokerReader::fill_field_row(RowCursorCell* dst,
const char* src, boo
case OLAP_FIELD_TYPE_DOUBLE:
case OLAP_FIELD_TYPE_CHAR:
case OLAP_FIELD_TYPE_VARCHAR:
+case OLAP_FIELD_TYPE_STRING:
case OLAP_FIELD_TYPE_HLL:
case OLAP_FIELD_TYPE_OBJECT: {
dst->set_is_null(src_null);
diff --git
a/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/ColumnParser.java
b/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/ColumnParser.java
index e72d471a9f..1425dd3bef 100644
---
a/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/ColumnParser.java
+++
b/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/ColumnParser.java
@@ -29,6 +29,7 @@ import java.math.BigInteger;
import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeFormatterBuilder;
+
// Parser to validate value for different type
public abstract class ColumnParser implements Serializable {
@@ -62,6 +63,9 @@ public abstract class ColumnParser implements Serializable {
return new DateParser();
} else if (columnType.equalsIgnoreCase("DATETIME")) {
return new DatetimeParser();
+} else if (columnType.equalsIgnoreCase("STRING")
+|| columnType.equalsIgnoreCase("TEXT")) {
+return new StringTypeParser(etlColumn);
} else if (columnType.equalsIgnoreCase("VARCHAR")
|| columnType.equalsIgnoreCase("CHAR")
|| columnType.equalsIgnoreCase("BITMAP")
@@ -208,6 +212,25 @@ class StringParser extends ColumnParser {
}
}
+class StringTypeParser extends ColumnParser {
+
+private EtlJobConfig.EtlColumn etlColumn;
+
+public StringTypeParser(EtlJobConfig.EtlColumn etlColumn) {
+this.etlColumn = etlColumn;
+}
+
+@Override
+public boolean parse(String value) {
+try {
+return value.getBytes("UTF-8").length <=
DppUtils.STRING_LENGTH_LIMIT;
+} catch (Exception e) {
+throw new RuntimeException("string check failed ", e);
+}
+}
+}
+
+
class DecimalParser extends ColumnParser {
public static int PRECISION = 27;
diff --git
a/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/DppUtils.java
b/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/DppUtils.java
index c53e1ae087..9a252b38b7 100644
--- a/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/DppUtils.java
+++ b/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/DppUtils.java
@@ -41,6 +41,8 @@ import java.util.zip.CRC32;
public class DppUtils {
public static final String BUCKET_ID = "__bucketId__";
+public static final int STRING_LENGTH_LIMIT = 1048576;
+
public static Class getClassFromDataType(DataType dataType) {
if (dataType == null) {
return null;
@@ -94,6 +96,8 @@ public class DppUtils {
case "HLL":
case "CHAR":
case "VARCHAR":
+case "STRING":
+case "TEXT":
case "BITMAP":
case "OBJECT":
return String.class;
@@ -142,6 +146,8 @@ public class DppUtils {
break;
case "CHAR":
case "VARCHAR":
+case "STRING":
+case "TEXT":
case "OBJECT":
dataType = DataTypes.StringType;
break;
diff --git
a/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
b/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
index ab7e791a1c..5d951ad70b 100644
--- a/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
+++ b/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
@@ -405,6 +405,18 @@ public final class SparkDpp implements
java.io.Serializable {
return false;
}
brea
[GitHub] [doris-website] hf200012 merged pull request #55: [doc](fix) add STRING conversion notice
hf200012 merged PR #55: URL: https://github.com/apache/doris-website/pull/55 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: [doc](fix) add STRING conversion notice (#55)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 5ddc065c138 [doc](fix) add STRING conversion notice (#55) 5ddc065c138 is described below commit 5ddc065c138fd5a48c2c5b12796ef019acea40eb Author: a19920714liou <562951...@qq.com> AuthorDate: Mon Aug 22 08:57:08 2022 +0800 [doc](fix) add STRING conversion notice (#55) add STRING conversion notice --- .../sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN.md | 1 + .../sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN.md | 1 + 2 files changed, 2 insertions(+) diff --git a/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN.md b/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN.md index 27a77319d84..697ef2a912b 100644 --- a/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN.md +++ b/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN.md @@ -116,6 +116,7 @@ Notice: - DATE is converted to DATETIME (hours, minutes and seconds are automatically filled with zeros, for example: `2019-12-09` <--> `2019-12-09 00:00:00`) - Convert FLOAT to DOUBLE - INT is converted to DATE (if the INT type data is illegal, the conversion fails, and the original data remains unchanged) + - All can be converted to STRING except DATE and DATETIME, but STRING cannot be converted to any other type 5. Reorder the column at the specified index diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN.md index 743d29b8d42..287ca8a3f3b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN.md @@ -116,6 +116,7 @@ MODIFY COLUMN column_name column_type [KEY | agg_type] [NULL | NOT NULL] [DEFAUL - DATE 转换成 DATETIME(时分秒自动补零， 例如: `2019-12-09` <--> `2019-12-09 00:00:00`) - FLOAT 转换成 DOUBLE - INT 转换成 DATE (如果INT类型数据不合法则转换失败，原始数据不变) + - 除DATE与DATETIME以外都可以转换成STRING，但是STRING不能转换任何其他类型 5. 对指定 index 的列进行重新排序 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11866: [improvement](config)Enable insert strict
github-actions[bot] commented on PR #11866: URL: https://github.com/apache/doris/pull/11866#issuecomment-1221672263 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #11917: [Bugfix](datetime) fix DateLiteral range check is no longer valid
morningman commented on code in PR #11917:
URL: https://github.com/apache/doris/pull/11917#discussion_r950935452
##
fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java:
##
@@ -435,6 +435,9 @@ private void init(String s, Type type) throws
AnalysisException {
this.roundFloor(((ScalarType) type).getScalarScale());
}
this.type = type;
+if (checkRange() || checkDate()) {
Review Comment:
Should it be:
```
if (!checkRange() || !checkDate())
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #11951: [FixAssist](compaction) add DCHECK in BlockReader::_unique_key_next_block to reason problem
dataroaring opened a new pull request, #11951: URL: https://github.com/apache/doris/pull/11951 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11801: [improvement](agg) Serialize the fixed-length aggregation results with corresponding columns instead of ColumnString
yiguolei merged PR #11801: URL: https://github.com/apache/doris/pull/11801 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](agg) Serialize the fixed-length aggregation results with corresponding columns instead of ColumnString (#11801)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new dc8f64b3e3 [improvement](agg) Serialize the fixed-length aggregation
results with corresponding columns instead of ColumnString (#11801)
dc8f64b3e3 is described below
commit dc8f64b3e3563c41857e30485dbd4d68ea9da7ab
Author: Jerry Hu
AuthorDate: Mon Aug 22 10:12:06 2022 +0800
[improvement](agg) Serialize the fixed-length aggregation results with
corresponding columns instead of ColumnString (#11801)
---
be/src/vec/CMakeLists.txt | 1 +
.../vec/aggregate_functions/aggregate_function.h | 84 +++-
.../aggregate_functions/aggregate_function_avg.h | 72 ++-
.../aggregate_functions/aggregate_function_count.h | 96 +
.../aggregate_function_min_max.h | 121 ++-
.../aggregate_function_nothing.h | 3 +
.../aggregate_functions/aggregate_function_null.h | 10 +
.../aggregate_functions/aggregate_function_sum.h | 53 +
be/src/vec/columns/column_fixed_length_object.h| 197 ++
be/src/vec/core/field.h| 12 ++
be/src/vec/core/types.h| 3 +
be/src/vec/data_types/data_type.cpp| 2 +
be/src/vec/data_types/data_type_factory.cpp| 3 +
be/src/vec/data_types/data_type_factory.hpp| 1 +
.../data_types/data_type_fixed_length_object.cpp | 67 +++
.../vec/data_types/data_type_fixed_length_object.h | 59 ++
be/src/vec/exec/vaggregation_node.cpp | 223 +
be/src/vec/exec/vaggregation_node.h| 36 +++-
be/src/vec/exprs/vectorized_agg_fn.cpp | 14 ++
be/src/vec/exprs/vectorized_agg_fn.h | 6 +
.../org/apache/doris/planner/AggregationNode.java | 1 +
gensrc/proto/types.proto | 1 +
gensrc/thrift/PlanNodes.thrift | 3 +-
23 files changed, 969 insertions(+), 99 deletions(-)
diff --git a/be/src/vec/CMakeLists.txt b/be/src/vec/CMakeLists.txt
index bc37c49310..8743013590 100644
--- a/be/src/vec/CMakeLists.txt
+++ b/be/src/vec/CMakeLists.txt
@@ -66,6 +66,7 @@ set(VEC_FILES
data_types/data_type_array.cpp
data_types/data_type_bitmap.cpp
data_types/data_type_factory.cpp
+ data_types/data_type_fixed_length_object.cpp
data_types/data_type_hll.cpp
data_types/data_type_nothing.cpp
data_types/data_type_nothing.cpp
diff --git a/be/src/vec/aggregate_functions/aggregate_function.h
b/be/src/vec/aggregate_functions/aggregate_function.h
index ae9a3667e5..6309b0c994 100644
--- a/be/src/vec/aggregate_functions/aggregate_function.h
+++ b/be/src/vec/aggregate_functions/aggregate_function.h
@@ -28,6 +28,7 @@
#include "vec/core/column_numbers.h"
#include "vec/core/field.h"
#include "vec/core/types.h"
+#include "vec/data_types/data_type_string.h"
namespace doris::vectorized {
@@ -76,6 +77,9 @@ public:
/// Delete data for aggregation.
virtual void destroy(AggregateDataPtr __restrict place) const noexcept = 0;
+virtual void destroy_vec(AggregateDataPtr __restrict place,
+ const size_t num_rows) const noexcept = 0;
+
/// Reset aggregation state
virtual void reset(AggregateDataPtr place) const = 0;
@@ -117,17 +121,29 @@ public:
virtual void serialize_vec(const std::vector& places,
size_t offset,
BufferWritable& buf, const size_t num_rows)
const = 0;
+virtual void serialize_to_column(const std::vector&
places, size_t offset,
+ MutableColumnPtr& dst, const size_t
num_rows) const = 0;
+
+virtual void serialize_without_key_to_column(ConstAggregateDataPtr
__restrict place,
+ MutableColumnPtr& dst) const
= 0;
+
/// Deserializes state. This function is called only for empty (just
created) states.
virtual void deserialize(AggregateDataPtr __restrict place,
BufferReadable& buf,
Arena* arena) const = 0;
-virtual void deserialize_vec(AggregateDataPtr places, ColumnString*
column, Arena* arena,
+virtual void deserialize_vec(AggregateDataPtr places, const ColumnString*
column, Arena* arena,
size_t num_rows) const = 0;
+virtual void deserialize_from_column(AggregateDataPtr places, const
IColumn& column,
+ Arena* arena, size_t num_rows) const
= 0;
+
/// Deserializes state and merge it with current aggregation function.
virtual void deserialize_and_merge(AggregateDataPtr __restrict place,
BufferReadable& buf,
Arena* arena) const = 0;
+virtual void deserialize_and_mer
[GitHub] [doris] morningman merged pull request #11845: [feature-wip](parquet-reader) decode parquet time & datetime & decimal
morningman merged PR #11845: URL: https://github.com/apache/doris/pull/11845 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature-wip](parquet-reader) decode parquet time & datetime & decimal (#11845)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6d925054de [feature-wip](parquet-reader) decode parquet time &
datetime & decimal (#11845)
6d925054de is described below
commit 6d925054de3be67d27893a47939e88a30e60e735
Author: Ashin Gau
AuthorDate: Mon Aug 22 10:15:35 2022 +0800
[feature-wip](parquet-reader) decode parquet time & datetime & decimal
(#11845)
1. Spark can set the timestamp precision by the following configuration:
spark.sql.parquet.outputTimestampType = INT96(NANOS), TIMESTAMP_MICROS,
TIMESTAMP_MILLIS
DATETIME V1 only keeps the second precision, DATETIME V2 keeps the
microsecond precision.
2. If using DECIMAL V2, the BE saves the value as decimal128, and keeps the
precision of decimal as (precision=27, scale=9). DECIMAL V3 can maintain the
right precision of decimal
---
be/src/exec/schema_scanner.cpp | 7 +-
be/src/exec/schema_scanner.h | 3 +
be/src/vec/exec/file_hdfs_scanner.cpp | 6 +-
be/src/vec/exec/format/parquet/parquet_common.cpp | 166 --
be/src/vec/exec/format/parquet/parquet_common.h| 248 +
.../parquet/vparquet_column_chunk_reader.cpp | 36 ++-
.../format/parquet/vparquet_column_chunk_reader.h | 11 +-
.../exec/format/parquet/vparquet_column_reader.cpp | 6 +-
.../exec/format/parquet/vparquet_column_reader.h | 9 +-
.../exec/format/parquet/vparquet_group_reader.cpp | 8 +-
.../exec/format/parquet/vparquet_group_reader.h| 3 +-
.../exec/format/parquet/vparquet_page_reader.cpp | 2 +-
be/src/vec/exec/format/parquet/vparquet_reader.cpp | 8 +-
be/src/vec/exec/format/parquet/vparquet_reader.h | 3 +-
be/src/vec/runtime/vdatetime_value.cpp | 10 +-
be/src/vec/runtime/vdatetime_value.h | 2 +
be/src/vec/utils/arrow_column_to_doris_column.cpp | 3 +-
.../test_data/parquet_scanner/type-decoder.parquet | Bin 3405 -> 4391 bytes
be/test/vec/exec/parquet/parquet_thrift_test.cpp | 146 ++--
19 files changed, 609 insertions(+), 68 deletions(-)
diff --git a/be/src/exec/schema_scanner.cpp b/be/src/exec/schema_scanner.cpp
index da5edec413..9071293ff1 100644
--- a/be/src/exec/schema_scanner.cpp
+++ b/be/src/exec/schema_scanner.cpp
@@ -138,7 +138,12 @@ Status SchemaScanner::create_tuple_desc(ObjectPool* pool) {
if (_columns[i].type == TYPE_DECIMALV2) {
t_slot_desc.__set_slotType(TypeDescriptor::create_decimalv2_type(27,
9).to_thrift());
} else {
-
t_slot_desc.__set_slotType(TypeDescriptor(_columns[i].type).to_thrift());
+TypeDescriptor descriptor(_columns[i].type);
+if (_columns[i].precision >= 0 && _columns[i].scale >= 0) {
+descriptor.precision = _columns[i].precision;
+descriptor.scale = _columns[i].scale;
+}
+t_slot_desc.__set_slotType(descriptor.to_thrift());
}
t_slot_desc.__set_colName(_columns[i].name);
t_slot_desc.__set_columnPos(i);
diff --git a/be/src/exec/schema_scanner.h b/be/src/exec/schema_scanner.h
index ea2bc28748..2450da24d4 100644
--- a/be/src/exec/schema_scanner.h
+++ b/be/src/exec/schema_scanner.h
@@ -63,6 +63,9 @@ public:
PrimitiveType type;
int size;
bool is_null;
+/// Only set if type == TYPE_DECIMAL or DATETIMEV2
+int precision = -1;
+int scale = -1;
};
SchemaScanner(ColumnDesc* columns, int column_num);
virtual ~SchemaScanner();
diff --git a/be/src/vec/exec/file_hdfs_scanner.cpp
b/be/src/vec/exec/file_hdfs_scanner.cpp
index ab6401de8b..6459da6fff 100644
--- a/be/src/vec/exec/file_hdfs_scanner.cpp
+++ b/be/src/vec/exec/file_hdfs_scanner.cpp
@@ -59,9 +59,9 @@ Status ParquetFileHdfsScanner::_get_next_reader(int
_next_range) {
std::unique_ptr file_reader;
RETURN_IF_ERROR(FileFactory::create_file_reader(_state->exec_env(),
_profile, _params, range,
file_reader));
-_reader.reset(new ParquetReader(file_reader.release(),
_file_slot_descs.size(),
-_state->query_options().batch_size,
range.start_offset,
-range.size));
+_reader.reset(new ParquetReader(
+file_reader.release(), _file_slot_descs.size(),
_state->query_options().batch_size,
+range.start_offset, range.size,
const_cast(&_state->timezone_obj(;
auto tuple_desc = _state->desc_tbl().get_tuple_descriptor(_tupleId);
Status status =
_reader->init_reader(tuple_desc, _file_slot_descs, _conjunct_ctxs,
_state->timezone());
diff --git a/be/src/vec/exec/format/parquet/parquet_common.cpp
b/be/src/vec/exec/format/parque
[GitHub] [doris] jacktengg opened a new pull request, #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model
jacktengg opened a new pull request, #11952:
URL: https://github.com/apache/doris/pull/11952
# Proposed changes
Issue Number: close #xxx
## Problem summary
For debug purpose: add session variable agg_as_duplicate, when set to true,
tables of aggregate key model and unique key model will be read as duplicate
key model:
```
mysql> create table t1(a int, b int) engine = olap unique key (a)
distributed by hash(a) buckets 16 properties("replication_allocation" =
"tag.location.default: 1");
Query OK, 0 rows affected (0.28 sec)
mysql> insert into t1 values (1,1),(2,1);
Query OK, 2 rows affected (0.25 sec)
{'label':'insert_e46de5e8df5246d4_ac3d12726252db7b', 'status':'VISIBLE',
'txnId':'2'}
mysql> select * from t1;
+--+--+
| a| b|
+--+--+
|2 |1 |
|1 |1 |
+--+--+
2 rows in set (0.47 sec)
mysql> insert into t1 values (1,11),(2,11);
Query OK, 2 rows affected (0.17 sec)
{'label':'insert_7c44d5acd7b54bc7_82618edd8eef815b', 'status':'VISIBLE',
'txnId':'3'}
mysql> select * from t1;
+--+--+
| a| b|
+--+--+
|1 | 11 |
|2 | 11 |
+--+--+
2 rows in set (0.36 sec)
mysql> show variables like "%agg%";
+---+---+
| Variable_name | Value |
+---+---+
| agg_as_duplicate | false |
| disable_streaming_preaggregations | false |
+---+---+
2 rows in set (0.00 sec)
mysql> set agg_as_duplicate = true;
Query OK, 0 rows affected (0.01 sec)
mysql> select * from t1;
+--+--+
| a| b|
+--+--+
|2 |1 |
|2 | 11 |
|1 |1 |
|1 | 11 |
+--+--+
4 rows in set (0.36 sec)
```
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11951: [FixAssist](compaction) add DCHECK in BlockReader::_unique_key_next_block to reason problem
github-actions[bot] commented on PR #11951: URL: https://github.com/apache/doris/pull/11951#issuecomment-1221717954 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11951: [FixAssist](compaction) add DCHECK in BlockReader::_unique_key_next_block to reason problem
github-actions[bot] commented on PR #11951: URL: https://github.com/apache/doris/pull/11951#issuecomment-1221717976 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] yuanyuan8983 opened a new pull request, #57: Change error port
yuanyuan8983 opened a new pull request, #57: URL: https://github.com/apache/doris-website/pull/57 Change the wrong port in JDBC demo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on a diff in pull request #11942: [Enhancement](restore) restore table or partition with the same replication num as before the backup
nextdreamblue commented on code in PR #11942:
URL: https://github.com/apache/doris/pull/11942#discussion_r950964011
##
fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java:
##
@@ -1858,6 +1893,7 @@ public void readFields(DataInput in) throws IOException {
String value = Text.readString(in);
properties.put(key, value);
}
+reserveReplica =
Boolean.parseBoolean(properties.get(PROP_RESERVE_REPLICA));
Review Comment:
主要是为了兼容老的元数据，这个新的属性通过properties这个已有的变量去保存，可以很好地兼容之前的恢复任务元数据。如果不保存，会有丢失这个属性信息，所以折中了下，使用properties这个目前看来其实没有使用的变量。
##
fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java:
##
@@ -162,14 +166,17 @@ public RestoreJob() {
}
public RestoreJob(String label, String backupTs, long dbId, String dbName,
BackupJobInfo jobInfo, boolean allowLoad,
-ReplicaAllocation replicaAlloc, long timeoutMs, int metaVersion,
Env env, long repoId) {
+ReplicaAllocation replicaAlloc, long timeoutMs, int metaVersion,
boolean reserveReplica, Env env,
+long repoId) {
super(JobType.RESTORE, label, dbId, dbName, timeoutMs, env, repoId);
this.backupTimestamp = backupTs;
this.jobInfo = jobInfo;
this.allowLoad = allowLoad;
this.replicaAlloc = replicaAlloc;
this.state = RestoreJobState.PENDING;
this.metaVersion = metaVersion;
+this.reserveReplica = reserveReplica;
+properties.put(PROP_RESERVE_REPLICA, String.valueOf(reserveReplica));
Review Comment:
理由同下
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yixiutt opened a new pull request, #11953: [feature-wip](unique-key-merge-on-write) opt lock and only save valid…
yixiutt opened a new pull request, #11953: URL: https://github.com/apache/doris/pull/11953 … delete_bitmap 1. use rlock in most logic instead of wrlock 2. filter stale rowset's delete bitmap in save meta 3. add a delete_bitmap lock to handle compaction and publish_txn confict # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on a diff in pull request #11942: [Enhancement](restore) restore table or partition with the same replication num as before the backup
nextdreamblue commented on code in PR #11942:
URL: https://github.com/apache/doris/pull/11942#discussion_r950965400
##
fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java:
##
@@ -1858,6 +1893,7 @@ public void readFields(DataInput in) throws IOException {
String value = Text.readString(in);
properties.put(key, value);
}
+reserveReplica =
Boolean.parseBoolean(properties.get(PROP_RESERVE_REPLICA));
Review Comment:
如果不这样保存，而是使用单独一个值去write和read，那么老的没有该值的恢复任务在加载的时候，直接会报错，元数据读取失败。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model
yiguolei commented on code in PR #11952:
URL: https://github.com/apache/doris/pull/11952#discussion_r950966213
##
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##
@@ -519,6 +521,12 @@ public class SessionVariable implements Serializable,
Writable {
@VariableMgr.VarAttr(name = ENABLE_FUNCTION_PUSHDOWN)
public boolean enableFunctionPushdown;
+/**
+ * For debugg purpose, dont' merge unique key and agg key when reading
data.
+ */
+@VariableMgr.VarAttr(name = AGG_AS_DUPLICATE)
+private boolean aggAsDuplicate = false;
Review Comment:
the variable name： skip_storage_engine_merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model
yiguolei commented on code in PR #11952:
URL: https://github.com/apache/doris/pull/11952#discussion_r950967191
##
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##
@@ -519,6 +521,12 @@ public class SessionVariable implements Serializable,
Writable {
@VariableMgr.VarAttr(name = ENABLE_FUNCTION_PUSHDOWN)
public boolean enableFunctionPushdown;
+/**
+ * For debugg purpose, dont' merge unique key and agg key when reading
data.
+ */
+@VariableMgr.VarAttr(name = AGG_AS_DUPLICATE)
+private boolean aggAsDuplicate = false;
+
public String getBlockEncryptionMode() {
Review Comment:
also add another variable: skip_delete_predicate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model
yiguolei commented on code in PR #11952: URL: https://github.com/apache/doris/pull/11952#discussion_r950967537 ## be/src/vec/exec/volap_scanner.cpp: ## @@ -172,6 +172,11 @@ Status VOlapScanner::_init_tablet_reader_params( ->is_segments_overlapping()); _tablet_reader_params.direct_mode = _aggregation || single_version; Review Comment: This line is useless? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on a diff in pull request #11942: [Enhancement](restore) restore table or partition with the same replication num as before the backup
nextdreamblue commented on code in PR #11942:
URL: https://github.com/apache/doris/pull/11942#discussion_r950968930
##
fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java:
##
@@ -162,14 +166,17 @@ public RestoreJob() {
}
public RestoreJob(String label, String backupTs, long dbId, String dbName,
BackupJobInfo jobInfo, boolean allowLoad,
-ReplicaAllocation replicaAlloc, long timeoutMs, int metaVersion,
Env env, long repoId) {
+ReplicaAllocation replicaAlloc, long timeoutMs, int metaVersion,
boolean reserveReplica, Env env,
+long repoId) {
super(JobType.RESTORE, label, dbId, dbName, timeoutMs, env, repoId);
this.backupTimestamp = backupTs;
this.jobInfo = jobInfo;
this.allowLoad = allowLoad;
this.replicaAlloc = replicaAlloc;
this.state = RestoreJobState.PENDING;
this.metaVersion = metaVersion;
+this.reserveReplica = reserveReplica;
+properties.put(PROP_RESERVE_REPLICA, String.valueOf(reserveReplica));
Review Comment:
把该值放到properties，只是为了通过properties来持久化到元数据，否则元数据会有前后兼容问题
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on a diff in pull request #11942: [Enhancement](restore) restore table or partition with the same replication num as before the backup
nextdreamblue commented on code in PR #11942:
URL: https://github.com/apache/doris/pull/11942#discussion_r950964182
##
fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java:
##
@@ -162,14 +166,17 @@ public RestoreJob() {
}
public RestoreJob(String label, String backupTs, long dbId, String dbName,
BackupJobInfo jobInfo, boolean allowLoad,
-ReplicaAllocation replicaAlloc, long timeoutMs, int metaVersion,
Env env, long repoId) {
+ReplicaAllocation replicaAlloc, long timeoutMs, int metaVersion,
boolean reserveReplica, Env env,
+long repoId) {
super(JobType.RESTORE, label, dbId, dbName, timeoutMs, env, repoId);
this.backupTimestamp = backupTs;
this.jobInfo = jobInfo;
this.allowLoad = allowLoad;
this.replicaAlloc = replicaAlloc;
this.state = RestoreJobState.PENDING;
this.metaVersion = metaVersion;
+this.reserveReplica = reserveReplica;
+properties.put(PROP_RESERVE_REPLICA, String.valueOf(reserveReplica));
Review Comment:
理由同下
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] icedrugs89 commented on issue #11888: [Bug] 数据列内容不一致
icedrugs89 commented on issue #11888:
URL: https://github.com/apache/doris/issues/11888#issuecomment-1221737338
与社区的技术人员沟通后，采用如下的步骤验证：
1.be.conf增加enable_storage_vectorization = false重启
2.执行select * from fin_dw.dwd_trade_draw_pdi a where date(a.time_loan)
>='2022-01-01' and a.loan_state = 'ARJ' and a.loan_no >'0' and a.loan_no not
in('NULL','null','') limit 5;
返回结果数据为空，符合预期，向量化存储对某些类型的值方面可能存在一些问题，关闭后恢复
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #11954: [Chore](script) support custom python version on build script
BiteThet opened a new pull request, #11954: URL: https://github.com/apache/doris/pull/11954 # Proposed changes Sometime, we want to specify a python version for build script. we can set `export DORIS_BUILD_PYTHON_VERSION=python3` now. ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11802: [feature](Nereids)add type coercion rule for nereids
morrySnow commented on code in PR #11802:
URL: https://github.com/apache/doris/pull/11802#discussion_r950979587
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ComparisonPredicate.java:
##
@@ -39,45 +37,21 @@ public abstract class ComparisonPredicate extends
Expression implements BinaryEx
* @param rightright child of comparison predicate
*/
public ComparisonPredicate(Expression left, Expression right, String
symbol) {
-super(left, right);
-this.symbol = symbol;
+super(left, right, symbol);
}
@Override
public DataType getDataType() throws UnboundException {
return BooleanType.INSTANCE;
}
-@Override
-public boolean nullable() throws UnboundException {
-return left().nullable() || right().nullable();
-}
-
-@Override
-public String toSql() {
-return "(" + left().toSql() + ' ' + symbol + ' ' + right().toSql() +
")";
-}
-
public R accept(ExpressionVisitor visitor, C context) {
return visitor.visitComparisonPredicate(this, context);
}
@Override
-public int hashCode() {
-return Objects.hash(symbol, left(), right());
-}
-
-@Override
-public boolean equals(Object o) {
-if (this == o) {
-return true;
-}
-if (o == null || getClass() != o.getClass()) {
-return false;
-}
-ComparisonPredicate other = (ComparisonPredicate) o;
-return Objects.equals(left(), other.left())
-&& Objects.equals(right(), other.right());
+public AbstractDataType inputType() {
+return AnyDataType.INSTANCE;
Review Comment:
this is comparison, such as equalTo, so need BooleanType.INSTANCE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangstar333 commented on a diff in pull request #11936: [Bug] (bitmap) intersect_count function use in string cause ASAN error
zhangstar333 commented on code in PR #11936:
URL: https://github.com/apache/doris/pull/11936#discussion_r950981024
##
be/src/util/bitmap_intersect.h:
##
@@ -242,4 +266,91 @@ struct BitmapIntersect {
std::map _bitmaps;
};
+template <>
+struct BitmapIntersect {
+public:
+BitmapIntersect() = default;
+
+explicit BitmapIntersect(const char* src) { deserialize(src); }
+
+void add_key(const std::string_view key) {
+BitmapValue empty_bitmap;
+_bitmaps[key] = empty_bitmap;
+}
+
+void update(const std::string_view& key, const BitmapValue& bitmap) {
+if (_bitmaps.find(key) != _bitmaps.end()) {
+_bitmaps[key] |= bitmap;
+}
+}
+
+void merge(const BitmapIntersect& other) {
+for (auto& kv : other._bitmaps) {
+if (_bitmaps.find(kv.first) != _bitmaps.end()) {
+_bitmaps[kv.first] |= kv.second;
+} else {
+_bitmaps[kv.first] = kv.second;
+}
+}
+}
+
+// intersection
+BitmapValue intersect() const {
+BitmapValue result;
+auto it = _bitmaps.begin();
+result |= it->second;
+it++;
+for (; it != _bitmaps.end(); it++) {
+result &= it->second;
+}
+return result;
+}
+
+// calculate the intersection for _bitmaps's bitmap values
+int64_t intersect_count() const {
+if (_bitmaps.empty()) {
+return 0;
+}
+return intersect().cardinality();
+}
+
+// the serialize size
+size_t size() {
+size_t size = 4;
+for (auto& kv : _bitmaps) {
+size += detail::Helper::serialize_size(kv.first);
+size += kv.second.getSizeInBytes();
+}
+return size;
+}
+
+//must call size() first
+void serialize(char* dest) {
+char* writer = dest;
+*(int32_t*)writer = _bitmaps.size();
+writer += 4;
+for (auto& kv : _bitmaps) {
+writer = detail::Helper::write_to(kv.first, writer);
+kv.second.write(writer);
+writer += kv.second.getSizeInBytes();
+}
+}
+
+void deserialize(const char* src) {
+const char* reader = src;
+int32_t bitmaps_size = *(int32_t*)reader;
+reader += 4;
+for (int32_t i = 0; i < bitmaps_size; i++) {
+std::string key;
+detail::Helper::read_from(&reader, &key);
+BitmapValue bitmap(reader);
+reader += bitmap.getSizeInBytes();
+_bitmaps[key] = bitmap;
+}
+}
+
+private:
+phmap::flat_hash_map _bitmaps;
Review Comment:
Is the ASAN error caused by the stringValue stored here as Key?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11936: [Bug] (bitmap) intersect_count function use in string cause ASAN error
github-actions[bot] commented on PR #11936: URL: https://github.com/apache/doris/pull/11936#issuecomment-1221751854 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11936: [Bug] (bitmap) intersect_count function use in string cause ASAN error
github-actions[bot] commented on PR #11936: URL: https://github.com/apache/doris/pull/11936#issuecomment-1221751882 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11802: [feature](Nereids)add type coercion rule for nereids
morrySnow commented on code in PR #11802:
URL: https://github.com/apache/doris/pull/11802#discussion_r950983547
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckAnalysis.java:
##
@@ -0,0 +1,48 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.analysis;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.typecoercion.TypeCheckResult;
+import org.apache.doris.nereids.trees.plans.Plan;
+
+import java.util.Optional;
+
+/**
+ * Check analysis rule to check semantic correct after analysis by Nereids.
+ */
+public class CheckAnalysis extends OneAnalysisRuleFactory {
+@Override
+public Rule build() {
+return
any().then(this::checkExpressionInputTypes).toRule(RuleType.CHECK_ANALYSIS);
+}
+
+private Plan checkExpressionInputTypes(Plan plan) {
+final Optional firstFailed =
plan.getExpressions().stream()
+.map(Expression::checkInputDataTypes)
+.filter(TypeCheckResult::failed)
+.findFirst();
+
+if (firstFailed.isPresent()) {
+throw new RuntimeException(firstFailed.get().getMessage());
Review Comment:
ok, btw exception will be refactor totally after we could generate best
state-of-the-art plan on TPC-H benchmark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11802: [feature](Nereids)add type coercion rule for nereids
morrySnow commented on code in PR #11802:
URL: https://github.com/apache/doris/pull/11802#discussion_r950983767
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/TypeCoercion.java:
##
@@ -0,0 +1,180 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rewrite.rules;
+
+import org.apache.doris.nereids.annotation.Developing;
+import
org.apache.doris.nereids.rules.expression.rewrite.AbstractExpressionRewriteRule;
+import
org.apache.doris.nereids.rules.expression.rewrite.ExpressionRewriteContext;
+import org.apache.doris.nereids.trees.expressions.BinaryOperator;
+import org.apache.doris.nereids.trees.expressions.CaseWhen;
+import org.apache.doris.nereids.trees.expressions.Cast;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.InPredicate;
+import
org.apache.doris.nereids.trees.expressions.typecoercion.ImplicitCastInputTypes;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.coercion.AbstractDataType;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * a rule to add implicit cast for expressions.
+ * This class is inspired by spark's TypeCoercion.
+ */
+@Developing
+public class TypeCoercion extends AbstractExpressionRewriteRule {
+
+// TODO:
+// 1. DecimalPrecision Process
+// 2. Divide process
+// 3. String promote with numeric in binary arithmetic
+// 4. Date and DateTime process
+
+public static final TypeCoercion INSTANCE = new TypeCoercion();
+
+@Override
+public Expression rewrite(Expression expr, ExpressionRewriteContext ctx) {
+if (expr instanceof ImplicitCastInputTypes) {
+return visitImplicitCastInputTypes(expr, ctx);
+} else {
+return super.rewrite(expr, ctx);
+}
+}
+
+// TODO: add other expression visitor function to do type coercion if
necessary.
+
+@Override
+public Expression visitBinaryOperator(BinaryOperator binaryOperator,
ExpressionRewriteContext context) {
+Expression left = rewrite(binaryOperator.left(), context);
+Expression right = rewrite(binaryOperator.right(), context);
+
+return
Optional.of(TypeCoercionUtils.canHandleTypeCoercion(left.getDataType(),
right.getDataType()))
+.filter(Boolean::booleanValue)
+.map(b ->
TypeCoercionUtils.findTightestCommonType(left.getDataType(),
right.getDataType()))
+.filter(Optional::isPresent)
+.map(Optional::get)
+.filter(ct -> binaryOperator.inputType().acceptsType(ct))
+.filter(ct -> !left.getDataType().equals(ct) ||
!right.getDataType().equals(ct))
+.map(commonType -> {
+Expression newLeft =
TypeCoercionUtils.castIfNotSameType(left, commonType);
+Expression newRight =
TypeCoercionUtils.castIfNotSameType(right, commonType);
+return binaryOperator.withChildren(newLeft, newRight);
+})
+.orElse(binaryOperator.withChildren(left, right));
+}
+
+@Override
+public Expression visitCaseWhen(CaseWhen caseWhen,
ExpressionRewriteContext context) {
+List rewrittenChildren = caseWhen.children().stream()
+.map(e -> rewrite(e, context)).collect(Collectors.toList());
+CaseWhen newCaseWhen = caseWhen.withChildren(rewrittenChildren);
+List dataTypesForCoercion =
newCaseWhen.dataTypesForCoercion();
+if (dataTypesForCoercion.size() <= 1) {
+return newCaseWhen;
+}
+DataType first = dataTypesForCoercion.get(0);
+if (dataTypesForCoercion.stream().allMatch(dataType ->
dataType.equals(first))) {
+return newCaseWhen;
+}
+Optional optionalCommonType =
TypeCoercionUtils.findWiderCommonType(dataTypesForCoercion);
+return optionalCommonType
+.
[GitHub] [doris] mrhhsg opened a new pull request, #11955: [improvement](compress) Support compress/decompress block with lz4
mrhhsg opened a new pull request, #11955: URL: https://github.com/apache/doris/pull/11955 # Proposed changes Issue Number: close #xxx ## Problem summary LZ4 is more faster than snappy. https://user-images.githubusercontent.com/1179834/185831817-181766a1-703b-4e28-8b7a-13c49ef578de.png";> via: https://blog.cloudflare.com/squeezing-the-firehose/ ## Checklist(Required) 1. Does it affect the original behavior: - [x] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] jeffreys-cat closed pull request #33: [CDN] Upload files to OSS
jeffreys-cat closed pull request #33: [CDN] Upload files to OSS URL: https://github.com/apache/doris-website/pull/33 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #11865: [Bug] Turn on insert into strict mode
yiguolei closed issue #11865: [Bug] Turn on insert into strict mode URL: https://github.com/apache/doris/issues/11865 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11866: [improvement](config)Enable insert strict
yiguolei merged PR #11866: URL: https://github.com/apache/doris/pull/11866 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](config)Enable insert strict (#11866)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 26deebccb8 [improvement](config)Enable insert strict (#11866)
26deebccb8 is described below
commit 26deebccb810b7c54b602ee203bcd48aa347ea1e
Author: jiafeng.zhang
AuthorDate: Mon Aug 22 11:32:17 2022 +0800
[improvement](config)Enable insert strict (#11866)
---
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 40feef115f..7f4b2a5846 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -359,7 +359,7 @@ public class SessionVariable implements Serializable,
Writable {
public int parallelExecInstanceNum = 1;
@VariableMgr.VarAttr(name = ENABLE_INSERT_STRICT, needForward = true)
-public boolean enableInsertStrict = false;
+public boolean enableInsertStrict = true;
@VariableMgr.VarAttr(name = ENABLE_ODBC_TRANSCATION)
public boolean enableOdbcTransaction = false;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11816: [enhancement](memory) Reduce virtual memory used by PaddedPODArray
yiguolei merged PR #11816: URL: https://github.com/apache/doris/pull/11816 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](memory) Reduce virtual memory used by PaddedPODArray (#11816)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 92cef580f3 [enhancement](memory) Reduce virtual memory used by
PaddedPODArray (#11816)
92cef580f3 is described below
commit 92cef580f32d5b8085032f16eb001f1e8512d0ef
Author: Xinyi Zou
AuthorDate: Mon Aug 22 11:33:07 2022 +0800
[enhancement](memory) Reduce virtual memory used by PaddedPODArray (#11816)
---
be/src/exec/hash_join_node.cpp| 2 +-
be/src/exec/set_operation_node.cpp| 3 ++-
be/src/vec/olap/vgeneric_iterators.cpp| 3 ++-
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java | 3 ++-
4 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/be/src/exec/hash_join_node.cpp b/be/src/exec/hash_join_node.cpp
index f5a7fec132..53efed1113 100644
--- a/be/src/exec/hash_join_node.cpp
+++ b/be/src/exec/hash_join_node.cpp
@@ -148,7 +148,7 @@ Status HashJoinNode::prepare(RuntimeState* state) {
_is_null_safe_eq_join.end());
_hash_tbl.reset(new HashTable(_build_expr_ctxs, _probe_expr_ctxs,
_build_tuple_size,
stores_nulls, _is_null_safe_eq_join, id(),
- state->batch_size() * 2));
+
BitUtil::RoundUpToPowerOfTwo(state->batch_size(;
_probe_batch.reset(new RowBatch(child(0)->row_desc(),
state->batch_size()));
diff --git a/be/src/exec/set_operation_node.cpp
b/be/src/exec/set_operation_node.cpp
index 7e00e69fbe..882019b199 100644
--- a/be/src/exec/set_operation_node.cpp
+++ b/be/src/exec/set_operation_node.cpp
@@ -146,7 +146,8 @@ Status SetOperationNode::open(RuntimeState* state) {
}
// initial build hash table used for remove duplicated
_hash_tbl.reset(new HashTable(_child_expr_lists[0], _child_expr_lists[1],
_build_tuple_size,
- true, _find_nulls, id(), state->batch_size()
* 2));
+ true, _find_nulls, id(),
+
BitUtil::RoundUpToPowerOfTwo(state->batch_size(;
RowBatch build_batch(child(0)->row_desc(), state->batch_size());
RETURN_IF_ERROR(child(0)->open(state));
diff --git a/be/src/vec/olap/vgeneric_iterators.cpp
b/be/src/vec/olap/vgeneric_iterators.cpp
index 557ef8e057..83fdcbb105 100644
--- a/be/src/vec/olap/vgeneric_iterators.cpp
+++ b/be/src/vec/olap/vgeneric_iterators.cpp
@@ -239,7 +239,8 @@ private:
bool _valid = false;
mutable bool _skip = false;
size_t _index_in_block = -1;
-int _block_row_max = 4096;
+// 4096 minus 16 + 16 bytes padding that in padding pod array
+int _block_row_max = 4064;
int _num_columns;
int _num_key_columns;
std::vector* _compare_columns;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 7f4b2a5846..ccc7a8764d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -336,8 +336,9 @@ public class SessionVariable implements Serializable,
Writable {
@VariableMgr.VarAttr(name = CODEGEN_LEVEL)
public int codegenLevel = 0;
+// 1024 minus 16 + 16 bytes padding that in padding pod array
@VariableMgr.VarAttr(name = BATCH_SIZE)
-public int batchSize = 1024;
+public int batchSize = 992;
@VariableMgr.VarAttr(name = DISABLE_STREAMING_PREAGGREGATIONS)
public boolean disableStreamPreaggregations = false;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] jeffreys-cat opened a new pull request, #58: [CDN] Use aliyun-oss-website-action in our repo
jeffreys-cat opened a new pull request, #58: URL: https://github.com/apache/doris-website/pull/58 Because ```aliyun-oss-website-action``` is not created by Github or verified creators, and it does not comply with Apache rules, We use ```git submodule``` use it in our repo. ```actions/checkout``` need set to submodules: recursive makes it work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #11473: [Bug][Vectorized] Fix nullptr deref in data sink
morningman merged PR #11473: URL: https://github.com/apache/doris/pull/11473 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: [CDN] Use aliyun-oss-website-action in our repo (#58)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new b602157c183 [CDN] Use aliyun-oss-website-action in our repo (#58)
b602157c183 is described below
commit b602157c183c76537f7ed0a97e3cb37b9cf96877
Author: Jeffrey
AuthorDate: Mon Aug 22 11:45:00 2022 +0800
[CDN] Use aliyun-oss-website-action in our repo (#58)
Use aliyun-oss-website-action in our repo
---
.github/workflows/actions/aliyun-oss-website-action | 1 +
.github/workflows/manual-deploy-website.yml | 18 +-
.gitmodules | 3 +++
3 files changed, 21 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/actions/aliyun-oss-website-action
b/.github/workflows/actions/aliyun-oss-website-action
new file mode 16
index 000..eefa35a2bb1
--- /dev/null
+++ b/.github/workflows/actions/aliyun-oss-website-action
@@ -0,0 +1 @@
+Subproject commit eefa35a2bb18c61ff710bb603a253c4a0ca582a9
diff --git a/.github/workflows/manual-deploy-website.yml
b/.github/workflows/manual-deploy-website.yml
index 9cd93d454bf..5fffc2e242c 100644
--- a/.github/workflows/manual-deploy-website.yml
+++ b/.github/workflows/manual-deploy-website.yml
@@ -1,4 +1,4 @@
-name: Manual Deploy
+name: Manual Deploy And Upload To OSS
on:
workflow_dispatch:
@@ -11,9 +11,13 @@ on:
jobs:
build-and-deploy:
runs-on: ubuntu-latest
+environment: Production
steps:
- name: Checkout
uses: actions/checkout@master
+ with:
+ persist-credentials: false
+ submodules: recursive
- name: Use Node.js
uses: actions/setup-node@v1
@@ -37,6 +41,17 @@ jobs:
touch build/.dummy
ls build
export DORIS_COMMIT=`git rev-parse HEAD`
+
+- name: Upload files to OSS
+ uses: ./.github/actions/aliyun-oss-website-action
+ with:
+ accessKeyId: ${{ secrets.ALIYUN_ACCESS_KEY_ID }}
+ accessKeySecret: ${{ secrets.ALIYUN_ACCESS_KEY_SECRET }}
+ bucket: ${{ secrets.ALIYUN_OSS_BUCKET }}
+ # use your own endpoint
+ endpoint: ${{ secrets.ALIYUN_OSS_ENDPOINT }}
+ folder: build
+
- name: Deploy website
if: ${{ github.event.inputs.branch == 'master' }}
run: |
@@ -58,6 +73,7 @@ jobs:
git add .
git commit -m "Automated deployment with doris master"
git push --verbose "https://${{ secrets.GITHUB_TOKEN
}}@github.com/apache/doris-website.git" asf-site:asf-site
+
- name: Deploy Branch
if: ${{ github.event.inputs.branch != 'master' }}
uses: peaceiris/actions-gh-pages@v3
diff --git a/.gitmodules b/.gitmodules
new file mode 100644
index 000..40d7771241e
--- /dev/null
+++ b/.gitmodules
@@ -0,0 +1,3 @@
+[submodule ".github/workflows/actions/aliyun-oss-website-action"]
+ path = .github/workflows/actions/aliyun-oss-website-action
+ url = https://github.com/fangbinwei/aliyun-oss-website-action.git
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] hf200012 merged pull request #57: Change error port
hf200012 merged PR #57: URL: https://github.com/apache/doris-website/pull/57 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] hf200012 merged pull request #58: [CDN] Use aliyun-oss-website-action in our repo
hf200012 merged PR #58: URL: https://github.com/apache/doris-website/pull/58 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix][Vectorized] Fix nullptr deref in data sink (#11473)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0b33824eef [fix][Vectorized] Fix nullptr deref in data sink (#11473)
0b33824eef is described below
commit 0b33824eef912d8ec6f1366d29a1f62c1c1e6bb0
Author: Amos Bird
AuthorDate: Mon Aug 22 11:44:55 2022 +0800
[fix][Vectorized] Fix nullptr deref in data sink (#11473)
brpc cache may return nullptr.
---
be/src/vec/sink/vdata_stream_sender.cpp | 7 +++
1 file changed, 7 insertions(+)
diff --git a/be/src/vec/sink/vdata_stream_sender.cpp
b/be/src/vec/sink/vdata_stream_sender.cpp
index e137d3ca18..e5b6a5c721 100644
--- a/be/src/vec/sink/vdata_stream_sender.cpp
+++ b/be/src/vec/sink/vdata_stream_sender.cpp
@@ -69,6 +69,13 @@ Status VDataStreamSender::Channel::init(RuntimeState* state)
{
_brpc_stub =
state->exec_env()->brpc_internal_client_cache()->get_client(_brpc_dest_addr);
}
+if (!_brpc_stub) {
+std::string msg = fmt::format("Get rpc stub failed, dest_addr={}:{}",
+ _brpc_dest_addr.hostname,
_brpc_dest_addr.port);
+LOG(WARNING) << msg;
+return Status::InternalError(msg);
+}
+
// In bucket shuffle join will set fragment_instance_id (-1, -1)
// to build a camouflaged empty channel. the ip and port is '0.0.0.0:0"
// so the empty channel not need call function close_internal()
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: Change error port (#57)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new ac077a1fd3b Change error port (#57)
ac077a1fd3b is described below
commit ac077a1fd3b88ce9d50c1d2c18c44664d9dcc432
Author: yuanyuan8983 <99315889+yuanyuan8...@users.noreply.github.com>
AuthorDate: Mon Aug 22 11:45:18 2022 +0800
Change error port (#57)
* Update jdbc-load.md
---
docs/data-operate/import/import-scenes/jdbc-load.md | 2 +-
.../current/data-operate/import/import-scenes/jdbc-load.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/data-operate/import/import-scenes/jdbc-load.md
b/docs/data-operate/import/import-scenes/jdbc-load.md
index f2314077c0e..ea3d35e3bdf 100644
--- a/docs/data-operate/import/import-scenes/jdbc-load.md
+++ b/docs/data-operate/import/import-scenes/jdbc-load.md
@@ -79,7 +79,7 @@ public class DorisJDBCDemo {
private static final String JDBC_DRIVER = "com.mysql.jdbc.Driver";
private static final String DB_URL_PATTERN =
"jdbc:mysql://%s:%d/%s?rewriteBatchedStatements=true";
private static final String HOST = "127.0.0.1"; // Leader Node host
-private static final int PORT = 8030; // http port of Leader Node
+private static final int PORT = 9030; // query port of Leader Node
private static final String DB = "demo";
private static final String TBL = "test_1";
private static final String USER = "admin";
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-scenes/jdbc-load.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-scenes/jdbc-load.md
index 0dd586e0306..1ab77c49c1b 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-scenes/jdbc-load.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-scenes/jdbc-load.md
@@ -79,7 +79,7 @@ public class DorisJDBCDemo {
private static final String JDBC_DRIVER = "com.mysql.jdbc.Driver";
private static final String DB_URL_PATTERN =
"jdbc:mysql://%s:%d/%s?rewriteBatchedStatements=true";
private static final String HOST = "127.0.0.1"; // Leader Node host
-private static final int PORT = 8030; // http port of Leader Node
+private static final int PORT = 9030; // query port of Leader Node
private static final String DB = "demo";
private static final String TBL = "test_1";
private static final String USER = "admin";
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11955: [improvement](compress) Support compress/decompress block with lz4
github-actions[bot] commented on PR #11955: URL: https://github.com/apache/doris/pull/11955#issuecomment-1221769974 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11955: [improvement](compress) Support compress/decompress block with lz4
github-actions[bot] commented on PR #11955: URL: https://github.com/apache/doris/pull/11955#issuecomment-1221770003 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #11802: [feature](Nereids)add type coercion rule for nereids
morrySnow commented on PR #11802: URL: https://github.com/apache/doris/pull/11802#issuecomment-1221770129 > inpredicate.groovy done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #11802: [feature](Nereids)add type coercion rule for nereids
morrySnow commented on PR #11802: URL: https://github.com/apache/doris/pull/11802#issuecomment-1221770279 > you can uncomment the `order_qt_in_predicate_5` and `order_qt_in_predicate_6` in `regression-test/suites/nereids_syntax_p0/inpredicate.groovy` for regression testing done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11795: [enhancement](Nereids)support max function
github-actions[bot] commented on PR #11795: URL: https://github.com/apache/doris/pull/11795#issuecomment-1221776576 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jacktengg commented on a diff in pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model
jacktengg commented on code in PR #11952:
URL: https://github.com/apache/doris/pull/11952#discussion_r950996455
##
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##
@@ -519,6 +521,12 @@ public class SessionVariable implements Serializable,
Writable {
@VariableMgr.VarAttr(name = ENABLE_FUNCTION_PUSHDOWN)
public boolean enableFunctionPushdown;
+/**
+ * For debugg purpose, dont' merge unique key and agg key when reading
data.
+ */
+@VariableMgr.VarAttr(name = AGG_AS_DUPLICATE)
+private boolean aggAsDuplicate = false;
Review Comment:
done
##
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##
@@ -519,6 +521,12 @@ public class SessionVariable implements Serializable,
Writable {
@VariableMgr.VarAttr(name = ENABLE_FUNCTION_PUSHDOWN)
public boolean enableFunctionPushdown;
+/**
+ * For debugg purpose, dont' merge unique key and agg key when reading
data.
+ */
+@VariableMgr.VarAttr(name = AGG_AS_DUPLICATE)
+private boolean aggAsDuplicate = false;
+
public String getBlockEncryptionMode() {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] jeffreys-cat opened a new pull request, #59: [CDN] Use aliyun-oss-website-action in our repo
jeffreys-cat opened a new pull request, #59: URL: https://github.com/apache/doris-website/pull/59 Update submodule path -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg commented on a diff in pull request #11917: [Bugfix](datetime) fix DateLiteral range check is no longer valid
yangzhg commented on code in PR #11917:
URL: https://github.com/apache/doris/pull/11917#discussion_r950997358
##
fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java:
##
@@ -435,6 +435,9 @@ private void init(String s, Type type) throws
AnalysisException {
this.roundFloor(((ScalarType) type).getScalarScale());
}
this.type = type;
+if (checkRange() || checkDate()) {
Review Comment:
`checkRange` and `checkDate` return true is error 😂
https://github.com/yangzhg/doris/blob/881670566c0aa577dd83af69c7b3d9f7a3986ab2/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java#L1298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg commented on a diff in pull request #11917: [Bugfix](datetime) fix DateLiteral range check is no longer valid
yangzhg commented on code in PR #11917:
URL: https://github.com/apache/doris/pull/11917#discussion_r950997358
##
fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java:
##
@@ -435,6 +435,9 @@ private void init(String s, Type type) throws
AnalysisException {
this.roundFloor(((ScalarType) type).getScalarScale());
}
this.type = type;
+if (checkRange() || checkDate()) {
Review Comment:
`checkRange` and `checkDate` return true is error 😂
[](https://github.com/yangzhg/doris/blob/881670566c0aa577dd83af69c7b3d9f7a3986ab2/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java#L1298)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg commented on a diff in pull request #11917: [Bugfix](datetime) fix DateLiteral range check is no longer valid
yangzhg commented on code in PR #11917:
URL: https://github.com/apache/doris/pull/11917#discussion_r950997358
##
fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java:
##
@@ -435,6 +435,9 @@ private void init(String s, Type type) throws
AnalysisException {
this.roundFloor(((ScalarType) type).getScalarScale());
}
this.type = type;
+if (checkRange() || checkDate()) {
Review Comment:
`checkRange` and `checkDate` return true is error 😂
https://github.com/apache/doris/blob/881670566c0aa577dd83af69c7b3d9f7a3986ab2/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java#L1298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] hf200012 merged pull request #59: [CDN] Use aliyun-oss-website-action in our repo
hf200012 merged PR #59: URL: https://github.com/apache/doris-website/pull/59 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: [CDN] Use aliyun-oss-website-action in our repo (#59)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new e3eee449bfb [CDN] Use aliyun-oss-website-action in our repo (#59)
e3eee449bfb is described below
commit e3eee449bfbbd12c1c919ecfa16e0d7ea1ae997b
Author: Jeffrey
AuthorDate: Mon Aug 22 12:14:45 2022 +0800
[CDN] Use aliyun-oss-website-action in our repo (#59)
* [CDN] Use aliyun-oss-website-action in our repo
---
.github/{workflows => }/actions/aliyun-oss-website-action | 0
.gitmodules | 4 ++--
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/actions/aliyun-oss-website-action
b/.github/actions/aliyun-oss-website-action
similarity index 100%
rename from .github/workflows/actions/aliyun-oss-website-action
rename to .github/actions/aliyun-oss-website-action
diff --git a/.gitmodules b/.gitmodules
index 40d7771241e..60e0f4401f0 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,3 +1,3 @@
-[submodule ".github/workflows/actions/aliyun-oss-website-action"]
- path = .github/workflows/actions/aliyun-oss-website-action
+[submodule ".github/actions/aliyun-oss-website-action"]
+ path = .github/actions/aliyun-oss-website-action
url = https://github.com/fangbinwei/aliyun-oss-website-action.git
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #11861: [fix](grouping)fix grouping function bug
morrySnow commented on PR #11861: URL: https://github.com/apache/doris/pull/11861#issuecomment-1221786886 @yangzhg PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11859: [fix](agg)repeat node shouldn't change slot's nullable property of agg node
github-actions[bot] commented on PR #11859: URL: https://github.com/apache/doris/pull/11859#issuecomment-1221787431 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hf200012 opened a new pull request, #11956: [Typo](doc) Rewrite the Quick Start document
hf200012 opened a new pull request, #11956: URL: https://github.com/apache/doris/pull/11956 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
wangshuo128 commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r951000690
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/MultiJoin.java:
##
@@ -74,25 +91,30 @@ public Optional reorderJoinsAccordingToConditions() {
*/
private Plan reorderJoinsAccordingToConditions(List joinInputs,
List conjuncts) {
if (joinInputs.size() == 2) {
-Set joinOutput = getJoinOutput(joinInputs.get(0),
joinInputs.get(1));
-Map> split = splitConjuncts(conjuncts,
joinOutput);
-List joinConditions = split.get(true);
-List nonJoinConditions = split.get(false);
-
+//Set joinOutput = getJoinOutput(joinInputs.get(0),
joinInputs.get(1));
+//Map> split = splitConjuncts(conjuncts,
joinOutput);
+//List joinConditions = split.get(true);
+//List nonJoinConditions = split.get(false);
+Pair, List> pair =
JoinUtils.extractExpressionForHashTable(
+
joinInputs.get(0).getOutput().stream().map(SlotReference.class::cast).collect(Collectors.toList()),
Review Comment:
`getExprId` is defined in the base class `NamedExpression`, you could get
exprId from Slot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
englefly commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r951010422
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java:
##
@@ -145,12 +183,16 @@ public boolean equals(Object o) {
return false;
}
LogicalJoin that = (LogicalJoin) o;
-return joinType == that.joinType && Objects.equals(condition,
that.condition);
+
+return joinType == that.joinType
+&& that.getHashJoinPredicates().containsAll(hashJoinPredicates)
+&& hashJoinPredicates.containsAll(that.getHashJoinPredicates())
+&& Objects.equals(otherJoinCondition, that.otherJoinCondition);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
englefly commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r951011122
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/MultiJoin.java:
##
@@ -74,25 +93,30 @@ public Plan reorderJoinsAccordingToConditions() {
*/
private Plan reorderJoinsAccordingToConditions(List joinInputs,
List conjuncts) {
if (joinInputs.size() == 2) {
-Set joinOutput = getJoinOutput(joinInputs.get(0),
joinInputs.get(1));
-Map> split = splitConjuncts(conjuncts,
joinOutput);
-List joinConditions = split.get(true);
-List nonJoinConditions = split.get(false);
-
+//Set joinOutput = getJoinOutput(joinInputs.get(0),
joinInputs.get(1));
+//Map> split = splitConjuncts(conjuncts,
joinOutput);
+//List joinConditions = split.get(true);
+//List nonJoinConditions = split.get(false);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sangetang opened a new issue, #11957: [Bug] NPE LoadLabelCleaner
sangetang opened a new issue, #11957: URL: https://github.com/apache/doris/issues/11957 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1.1-rc03 ### What's Wrong? ERROR (LoadLabelCleaner|60) [Daemon.run():118] daemon thread got exception. name: LoadLabelCleaner java.lang.NullPointerException: null at org.apache.doris.load.loadv2.SparkLoadJob.clearSparkLauncherLog(SparkLoadJob.java:737) ~[doris-fe.jar:1.0-SNAPSHOT] at org.apache.doris.load.loadv2.LoadManager.removeOldLoadJob(LoadManager.java:426) ~[doris-fe.jar:1.0-SNAPSHOT] at org.apache.doris.catalog.Catalog$1.runAfterCatalogReady(Catalog.java:2220) ~[doris-fe.jar:1.0-SNAPSHOT] at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.0-SNAPSHOT] ### What You Expected? no error ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
morrySnow commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r951033596
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/ExpressionRewrite.java:
##
@@ -106,15 +107,25 @@ private class JoinExpressionRewrite extends
OneRewriteRuleFactory {
@Override
public Rule build() {
return logicalJoin().then(join -> {
-Optional condition = join.getCondition();
-if (!condition.isPresent()) {
+List hashJoinPredicates =
join.getHashJoinPredicates();
+Optional otherJoinCondition =
join.getOtherJoinCondition();
+if (!otherJoinCondition.isPresent() &&
hashJoinPredicates.isEmpty()) {
return join;
}
-Expression newCondition = rewriter.rewrite(condition.get());
-if (newCondition.equals(condition.get())) {
+List rewriteHashJoinPredicates = new ArrayList<>();
+boolean joinPredicatesChanged = false;
+for (Expression expr : hashJoinPredicates) {
+Expression newExpr = rewriter.rewrite(expr);
+joinPredicatesChanged = joinPredicatesChanged ||
newExpr.equals(expr);
+rewriteHashJoinPredicates.add(newExpr);
+}
Review Comment:
update joinPredicatesChanged is not a problem. But newExpr compare to old
one is hard to do with stream. So ignore this comment pls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
morrySnow commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r951034098
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/algebra/Join.java:
##
@@ -28,5 +29,9 @@
public interface Join {
JoinType getJoinType();
-Optional getCondition();
+List getHashJoinPredicates();
+
+Optional getOtherJoinCondition();
+
+Optional getOnClauseCondition();
Review Comment:
Nereids join node do not have filter condition at all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
morrySnow commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r951035097
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java:
##
@@ -132,7 +167,10 @@ public List computeOutput(Plan leftInput, Plan
rightInput) {
@Override
public String toString() {
StringBuilder sb = new StringBuilder("LogicalJoin (").append(joinType);
-condition.ifPresent(expression -> sb.append(", ").append(expression));
+sb.append(" [");
+hashJoinPredicates.stream().map(expr -> sb.append("
").append(expr)).collect(Collectors.toList());
+sb.append(" ]");
+otherJoinCondition.ifPresent(expression -> sb.append(",
").append(expression));
Review Comment:
what i mean is print explain into like this:
```
LogicalJoin(hashJoinConjuncts:[=, =],
nonHashJoinConjuncts=[ccc>ddd, eee < 1])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
morrySnow commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r951035623
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalJoin.java:
##
@@ -78,11 +90,27 @@ public boolean equals(Object o) {
return false;
}
AbstractPhysicalJoin that = (AbstractPhysicalJoin) o;
-return joinType == that.joinType && Objects.equals(condition,
that.condition);
+return joinType == that.joinType && Objects.equals(otherJoinCondition,
that.otherJoinCondition);
Review Comment:
in nereids, NLJ could do join with hashJoinConjuncts, currently we do not do
that just because BE could not execute it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
morrySnow commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r951036202
##
fe/fe-core/src/main/java/org/apache/doris/nereids/util/JoinUtils.java:
##
@@ -51,14 +55,14 @@ public static boolean onlyShuffle(AbstractPhysicalJoin
join) {
*/
public static List getEqualTo(AbstractPhysicalJoin
join) {
List eqConjuncts = Lists.newArrayList();
-if (!join.getCondition().isPresent()) {
+if (!join.getOtherJoinCondition().isPresent()) {
Review Comment:
yes, this is what i mean, but u check OtherJoinCondition here. Should we
need to check hashJoinConjuncts here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
morrySnow commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r951036619
##
fe/fe-core/src/main/java/org/apache/doris/nereids/util/JoinUtils.java:
##
@@ -87,6 +91,117 @@ private static boolean isEqualTo(List
leftSlots, List left;
+HashSet leftExprIds;
+HashSet right;
+HashSet rightExprIds;
+
+JoinSlotCoverageChecker(List left, List
right) {
+this.left = new HashSet<>(left);
+leftExprIds = new
HashSet<>(left.stream().map(SlotReference::getExprId).collect(Collectors.toList()));
+this.right = new HashSet<>(right);
+rightExprIds = new
HashSet<>(right.stream().map(SlotReference::getExprId).collect(Collectors.toList()));
+}
+
+boolean isCoveredByLeftSlots(List slots) {
+boolean covered = left.containsAll(slots);
Review Comment:
> I am not sure that only comparing ExprId covers all cases
It is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11812: [feature](Nereids) support non-equal predicates in Join
morrySnow commented on code in PR #11812:
URL: https://github.com/apache/doris/pull/11812#discussion_r951037106
##
fe/fe-core/src/main/java/org/apache/doris/nereids/util/JoinUtils.java:
##
@@ -98,19 +213,20 @@ public static Pair,
List> getOnClauseUsedSlot
List leftSlots =
Utils.getOutputSlotReference(join.left());
List rightSlots =
Utils.getOutputSlotReference(join.right());
-List equalToList = getEqualTo(join);
-
+List equalToList = join.getHashJoinPredicates().stream()
+.map(e -> (EqualTo) e).collect(Collectors.toList());
+JoinSlotCoverageChecker checker = new
JoinSlotCoverageChecker(leftSlots, rightSlots);
for (EqualTo equalTo : equalToList) {
List leftOnSlots =
equalTo.left().collect(SlotReference.class::isInstance);
List rightOnSlots =
equalTo.right().collect(SlotReference.class::isInstance);
-if (new HashSet<>(leftSlots).containsAll(leftOnSlots)
-&& new HashSet<>(rightSlots).containsAll(rightOnSlots)) {
+if (checker.isCoveredByLeftSlots(leftOnSlots)
+&& checker.isCoveredByRightSlots(rightOnSlots)) {
// TODO: need rethink about `.get(0)`
childSlots.first.add(leftOnSlots.get(0));
Review Comment:
i think addAll is needed, and get(0) will lead to mistake.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] pengxiangyu opened a new issue, #11958: [Feature] Add cache files cleaner for remote olap files
pengxiangyu opened a new issue, #11958: URL: https://github.com/apache/doris/issues/11958 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description Add cache files cleaner for remote files ### Use case Add cache files cleaner for remote files ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
