[GitHub] [incubator-doris] jackwener commented on pull request #9686: push predicate to subquery

2022-05-25 Thread GitBox


jackwener commented on PR #9686:
URL: https://github.com/apache/incubator-doris/pull/9686#issuecomment-1136861370

   There are some checkstyle problems.
   
   I think use `idea-checkstyle` to check style problem will be better.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] jackwener opened a new pull request, #9767: [enhancement](style): polish the `checkstyle`

2022-05-25 Thread GitBox


jackwener opened a new pull request, #9767:
URL: https://github.com/apache/incubator-doris/pull/9767

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   - cancel the auto-align in Javadoc
   
   ```java
   @ param1   desc
   @ paramExample desc
   ``` 
   
   - polish doc for checkstyle
   
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (No)
   2. Has unit tests been added: (No)
   3. Has document been added or modified: (yes)
   4. Does it need to update dependencies: (no)
   5. Are there any changes that cannot be rolled back: (No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] englefly opened a new issue, #9768: [Feature] window function: "min() over()" does not support preceding and following in range expression

2022-05-25 Thread GitBox


englefly opened a new issue, #9768:
URL: https://github.com/apache/incubator-doris/issues/9768

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Description
   
   `SELECT nationkey, min(nationkey) OVER (PARTITION BY regionkey ORDER BY 
nationkey ROWS BETWEEN 2 PRECEDING AND 1 FOLLOWING) AS min FROM 
tpch_tiny_nation`
   
   Doris returns error:
   java.sql.SQLException: errCode = 2, detailMessage = 'min(`nationkey`)' is 
only supported with an UNBOUNDED PRECEDING start bound.
   
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] englefly opened a new issue, #9769: [Feature] window function rank() does not work with Windowing clause

2022-05-25 Thread GitBox


englefly opened a new issue, #9769:
URL: https://github.com/apache/incubator-doris/issues/9769

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Description
   
   sql:
   select orderkey, discount,
   dense_rank() over (order by discount),
   rank() over (order by discount range between unbounded preceding and current 
row)
   from tpch_tiny_lineitem where partkey = 272
   
   Doris returns error
   java.sql.SQLException: errCode = 2, detailMessage = Windowing clause not 
allowed with 'rank()'
    
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] nextdreamblue commented on pull request #9694: [Bug] Fix create table like when having hidden columns

2022-05-25 Thread GitBox


nextdreamblue commented on PR #9694:
URL: https://github.com/apache/incubator-doris/pull/9694#issuecomment-1136942643

   @stalary the checkstyle problem, done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] jackwener commented on a diff in pull request #8862: [feature-wip](statistics) step5: show statistics job information

2022-05-25 Thread GitBox


jackwener commented on code in PR #8862:
URL: https://github.com/apache/incubator-doris/pull/8862#discussion_r881366908


##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowAnalyzeStmt.java:
##
@@ -0,0 +1,354 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.OrderByPair;
+import org.apache.doris.mysql.privilege.PaloAuth;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSetMetaData;
+import org.apache.doris.statistics.StatisticsJob;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * ShowAnalyzeStmt is used to show statistics job info.
+ * syntax:
+ *SHOW ANALYZE
+ *[TABLE | ID]
+ *[
+ *WHERE
+ *[STATE = 
["PENDING"|"SCHEDULING"|"RUNNING"|"FINISHED"|"FAILED"|"CANCELLED"]]
+ *]
+ *[ORDER BY ...]
+ *[LIMIT limit][OFFSET offset];
+ */
+public class ShowAnalyzeStmt extends ShowStmt {
+private static final String STATE_NAME = "state";
+private static final ImmutableList TITLE_NAMES = new 
ImmutableList.Builder()
+.add("id")
+.add("create_time")
+.add("start_time")
+.add("finish_time")
+.add("error_msg")
+.add("scope")
+.add("progress")
+.add("state")
+.build();
+
+private List jobIds;
+private TableName dbTableName;
+private Expr whereClause;
+private LimitElement limitElement;
+private List orderByElements;
+
+// after analyzed
+private long dbId;
+private final Set tblIds = Sets.newHashSet();
+
+private String stateValue;
+private ArrayList orderByPairs;
+
+public ShowAnalyzeStmt() {
+}
+
+public ShowAnalyzeStmt(List jobIds) {
+this.jobIds = jobIds;
+}
+
+public ShowAnalyzeStmt(TableName dbTableName,
+   Expr whereClause,
+   List orderByElements,
+   LimitElement limitElement) {
+this.dbTableName = dbTableName;
+this.whereClause = whereClause;
+this.orderByElements = orderByElements;
+this.limitElement = limitElement;
+}
+
+public List getJobIds() {
+return jobIds;
+}
+
+public long getDbId() {
+Preconditions.checkArgument(isAnalyzed(),
+"The dbId must be obtained after the parsing is complete");
+return dbId;
+}
+
+public Set getTblIds() {
+Preconditions.checkArgument(isAnalyzed(),
+"The dbId must be obtained after the parsing is complete");
+return tblIds;
+}
+
+public String getStateValue() {
+Preconditions.checkArgument(isAnalyzed(),
+"The tbl name must be obtained after the parsing is complete");
+return stateValue;
+}
+
+public ArrayList getOrderByPairs() {
+Preconditions.checkArgument(isAnalyzed(),
+"The tbl name must be obtained after the parsing is complete");
+return orderByPairs;
+}
+
+public long getLimit() {
+if (limitElement != null && limitElement.hasLimit()) {
+return limitElement.getLimit();
+}
+return -1L;
+}
+
+public long getOffset() {
+if (limitElement != null && limitElement.hasOffset()) {
+return limitElement.getOffset();
+}
+return -1L;
+}
+
+@Override
+public void analyze(Analyzer analyzer) throws AnalysisException, 
UserEx

[GitHub] [incubator-doris] yangzhg opened a new pull request, #9770: [fix] disabel transfer data large than 2GB

2022-05-25 Thread GitBox


yangzhg opened a new pull request, #9770:
URL: https://github.com/apache/incubator-doris/pull/9770

   # Proposed changes
   
   because of brpc and protobuf cannot transfer data large than 2GB, if large 
than 2GB will overflow, so add a check before send
   
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] HappenLee merged pull request #9688: [fix] group by with two NULL rows after left join

2022-05-25 Thread GitBox


HappenLee merged PR #9688:
URL: https://github.com/apache/incubator-doris/pull/9688


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] HappenLee closed issue #9544: [Bug] After left join, there are two rows of the same key after group by

2022-05-25 Thread GitBox


HappenLee closed issue #9544: [Bug] After left join,  there are two rows of the 
same key after group by
URL: https://github.com/apache/incubator-doris/issues/9544


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [fix] group by with two NULL rows after left join (#9688)

2022-05-25 Thread lihaopeng
This is an automated email from the ASF dual-hosted git repository.

lihaopeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 2725127421 [fix] group by with two NULL rows after left join (#9688)
2725127421 is described below

commit 272512742196c8d5d3d8225988d8f26d7851c58d
Author: camby <104178...@qq.com>
AuthorDate: Wed May 25 16:43:55 2022 +0800

[fix] group by with two NULL rows after left join (#9688)

Co-authored-by: cambyzju 
---
 be/src/vec/columns/column_nullable.cpp |  5 +-
 .../query/aggregate/aggregate_groupby_null.out |  7 +++
 .../query/aggregate/aggregate_groupby_null.groovy  | 58 ++
 3 files changed, 69 insertions(+), 1 deletion(-)

diff --git a/be/src/vec/columns/column_nullable.cpp 
b/be/src/vec/columns/column_nullable.cpp
index f7cf9e9d3e..215ced2383 100644
--- a/be/src/vec/columns/column_nullable.cpp
+++ b/be/src/vec/columns/column_nullable.cpp
@@ -102,7 +102,10 @@ StringRef 
ColumnNullable::serialize_value_into_arena(size_t n, Arena& arena,
 static constexpr auto s = sizeof(arr[0]);
 
 auto pos = arena.alloc_continue(s, begin);
-memcpy(pos, &arr[n], s);
+// Value of `NULL` may be 1 or JOIN_NULL_HINT, we serialize both to 1.
+// Because we need same key for both `NULL` values while processing `group 
by`.
+UInt8* val = reinterpret_cast(pos);
+*val = (arr[n] ? 1 : 0);
 
 if (arr[n]) return StringRef(pos, s);
 
diff --git a/regression-test/data/query/aggregate/aggregate_groupby_null.out 
b/regression-test/data/query/aggregate/aggregate_groupby_null.out
new file mode 100644
index 00..9c44613df2
--- /dev/null
+++ b/regression-test/data/query/aggregate/aggregate_groupby_null.out
@@ -0,0 +1,7 @@
+-- This file is automatically generated. You should know what you did if you 
want to edit this
+-- !groupby_null --
+\N 3
+
+-- !groupby_null --
+\N 3
+
diff --git 
a/regression-test/suites/query/aggregate/aggregate_groupby_null.groovy 
b/regression-test/suites/query/aggregate/aggregate_groupby_null.groovy
new file mode 100644
index 00..dc3742c5fe
--- /dev/null
+++ b/regression-test/suites/query/aggregate/aggregate_groupby_null.groovy
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("aggregate_groupby_null") {
+def leftTable = "agg_groupby_null_left"
+sql """ DROP TABLE IF EXISTS ${leftTable} """
+sql """
+CREATE TABLE IF NOT EXISTS ${leftTable} (
+id INT NULL,
+device_id STRING NULL
+)
+UNIQUE KEY(id)
+DISTRIBUTED BY HASH(id) BUCKETS 1
+PROPERTIES (
+  "replication_num" = "1"
+)
+"""
+sql """ INSERT INTO ${leftTable} VALUES (1,'1'),(2,'2'),(3,'3'),(4,'4') """
+
+def rightTable = "agg_groupby_null_right"
+sql """ DROP TABLE IF EXISTS ${rightTable} """
+sql """
+CREATE TABLE IF NOT EXISTS ${rightTable} (
+id INT NULL,
+device_name STRING NULL
+)
+UNIQUE KEY(id)
+DISTRIBUTED BY HASH(id) BUCKETS 1
+PROPERTIES (
+  "replication_num" = "1"
+)
+"""
+sql """ INSERT INTO ${rightTable} VALUES (1,'name'),(3,null) """
+
+sql """ set enable_vectorized_engine=false """
+qt_groupby_null """ SELECT rt.device_name, COUNT(${leftTable}.id) FROM 
${leftTable}
+LEFT JOIN ${rightTable} rt ON ${leftTable}.id = rt.id
+   WHERE rt.device_name is NULL group by rt.device_name """
+
+sql """ set enable_vectorized_engine=true """
+qt_groupby_null """ SELECT rt.device_name, COUNT(${leftTable}.id) FROM 
${leftTable}
+LEFT JOIN ${rightTable} rt ON ${leftTable}.id = rt.id
+   WHERE rt.device_name is NULL group by rt.device_name """
+}


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] HappenLee merged pull request #9700: [doc] Add manual for Array data type and functions

2022-05-25 Thread GitBox


HappenLee merged PR #9700:
URL: https://github.com/apache/incubator-doris/pull/9700


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [doc] Add manual for Array data type and functions (#9700)

2022-05-25 Thread lihaopeng
This is an automated email from the ASF dual-hosted git repository.

lihaopeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 2ad691edf7 [doc] Add manual for Array data type and functions (#9700)
2ad691edf7 is described below

commit 2ad691edf72a76d6d0967f0a878724046ebc3dad
Author: camby <104178...@qq.com>
AuthorDate: Wed May 25 16:44:20 2022 +0800

[doc] Add manual for Array data type and functions (#9700)

Co-authored-by: cambyzju 
---
 docs/.vuepress/sidebar/en.js   | 11 +++
 docs/.vuepress/sidebar/zh-CN.js| 11 +++
 .../array-functions/array_contains.md  | 65 +
 .../array-functions/array_position.md  | 65 +
 .../sql-functions/array-functions/element_at.md| 81 +
 .../sql-manual/sql-reference/Data-Types/ARRAY.md   | 84 ++
 .../array-functions/array_contains.md  | 65 +
 .../array-functions/array_position.md  | 65 +
 .../sql-functions/array-functions/element_at.md| 81 +
 .../sql-manual/sql-reference/Data-Types/ARRAY.md   | 84 ++
 10 files changed, 612 insertions(+)

diff --git a/docs/.vuepress/sidebar/en.js b/docs/.vuepress/sidebar/en.js
index cb677e32d3..a05ad55e7c 100644
--- a/docs/.vuepress/sidebar/en.js
+++ b/docs/.vuepress/sidebar/en.js
@@ -547,6 +547,16 @@ module.exports = [
   "WINDOW-FUNCTION-ROW-NUMBER",
 ],
   },
+  {
+title: "Array Functions",
+directoryPath: "array-functions/",
+initialOpenGroupIndex: -1,
+children: [
+  "array_contains",
+  "array_position",
+  "element_at",
+],
+  },
   "cast",
   "digital-masking",
 ],
@@ -812,6 +822,7 @@ module.exports = [
   "STRING",
   "TINYINT",
   "VARCHAR",
+  "ARRAY",
 ],
   },
   {
diff --git a/docs/.vuepress/sidebar/zh-CN.js b/docs/.vuepress/sidebar/zh-CN.js
index e1fad83642..65a972ca13 100644
--- a/docs/.vuepress/sidebar/zh-CN.js
+++ b/docs/.vuepress/sidebar/zh-CN.js
@@ -547,6 +547,16 @@ module.exports = [
   "WINDOW-FUNCTION-ROW-NUMBER",
 ],
   },
+  {
+title: "Array函数",
+directoryPath: "array-functions/",
+initialOpenGroupIndex: -1,
+children: [
+  "array_contains",
+  "array_position",
+  "element_at",
+],
+  },
   "cast",
   "digital-masking",
 ],
@@ -812,6 +822,7 @@ module.exports = [
   "STRING",
   "TINYINT",
   "VARCHAR",
+  "ARRAY",
 ],
   },
   {
diff --git a/docs/en/sql-manual/sql-functions/array-functions/array_contains.md 
b/docs/en/sql-manual/sql-functions/array-functions/array_contains.md
new file mode 100644
index 00..57091a57ed
--- /dev/null
+++ b/docs/en/sql-manual/sql-functions/array-functions/array_contains.md
@@ -0,0 +1,65 @@
+---
+{
+"title": "array_contains",
+"language": "en"
+}
+---
+
+
+
+## array_contains
+
+### description
+
+ Syntax
+
+`BOOLEAN array_contains(ARRAY arr, T value)`
+
+Check if a value presents in an array column. Return below values:
+
+```
+1- if value presents in an array;
+0- if value does not present in an array;
+NULL - when array is NULL;
+```
+
+### notice
+
+`Only supported in vectorized engine`
+
+### example
+
+```
+mysql> set enable_vectorized_engine=true;
+
+mysql> SELECT id,c_array,array_contains(c_array, 5) FROM `array_test`;
++--+-+--+
+| id   | c_array | array_contains(`c_array`, 5) |
++--+-+--+
+|1 | [1, 2, 3, 4, 5] |1 |
+|2 | [6, 7, 8]   |0 |
+|3 | []  |0 |
+|4 | NULL| NULL |
++--+-+--+
+```
+
+### keywords
+
+ARRAY_CONTAINS
diff --git a/docs/en/sql-manual/sql-functions/array-functions/array_position.md 
b/docs/en/sql-manual/sql-functions/array-functions/array_position.md
new file mode 100644
index 00..dd47628c27
--- /dev/null
+++ b/docs/en/sql-manual/sql-functions/array-functions/array_position.md
@@ -0,0 +1,65 @@
+---
+{
+"title": "array_position",
+"language": "en"
+}
+---
+
+
+
+## array_position
+
+### description
+
+ Syntax
+
+`BIGINT array_position(ARRAY arr, T value)`
+
+Returns a position/index of first occurrence of the `value` in the given array.
+
+```
+position - value 

[GitHub] [incubator-doris] jacktengg opened a new pull request, #9771: [improvement] Show detail status code string for BE http api

2022-05-25 Thread GitBox


jacktengg opened a new pull request, #9771:
URL: https://github.com/apache/incubator-doris/pull/9771

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Failure message for BE http api does not show detail status code:
   ```
   "status": "Fail",
   "msg": ""
   }
   ```
   Improved to:
   ```
   {
   "status": "Not found",
   "msg": "Tablet not found. tablet_id=1202"
   }
   ```
   
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] Gabriel39 opened a new pull request, #9773: [BUG] fix correctness for vectorized compaction

2022-05-25 Thread GitBox


Gabriel39 opened a new pull request, #9773:
URL: https://github.com/apache/incubator-doris/pull/9773

   # Proposed changes
   
   Issue Number: close #9766 
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9770: [fix] disabel transfer data large than 2GB

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-doris/pull/9770#issuecomment-1136976048

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] HappenLee opened a new pull request, #9772: [Opt][VecLoad] Opt the vec stream load performance

2022-05-25 Thread GitBox


HappenLee opened a new pull request, #9772:
URL: https://github.com/apache/incubator-doris/pull/9772

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Reduce unless virtual function call and switch case to improve load 
performance of stream load.
   
    1. Test table:
   ```
   CREATE TABLE `test1` (
 `id` int(11) NOT NULL COMMENT "",
 `k1` varchar(32) NOT NULL COMMENT "",
 `k2` varchar(128) NOT NULL COMMENT "",
 `m` decimal(10, 3) NULL DEFAULT "10.5" COMMENT "",
 `d` date NULL DEFAULT "2014-02-04" COMMENT "",
 `e` datetime NULL DEFAULT "2014-02-04 15:36:00" COMMENT "",
 `f` boolean NOT NULL COMMENT "",
 INDEX index_id (`id`) USING BITMAP COMMENT ''
   ) ENGINE=OLAP
   DUPLICATE KEY(`id`, `k1`)
   COMMENT "OLAP"
   DISTRIBUTED BY HASH(`id`) BUCKETS 16
   PROPERTIES (
   "replication_allocation" = "tag.location.default: 1",
   "in_memory" = "false",
   "storage_format" = "V2"
   )
   ```
    2. Test data (1kw)
   
   ```
   
1,VOtWRdM0vqgMRD4g,QuSHANt7mA0hcsE8O6ggzOMOSLWtmfDs,99.6441922075,1982-11-10,1982-11-10
 02:57:53,1
   
,82N10yffLWGu6WM9,zDm0ekBEoYyfFfGGovRgp2hySHw3r31Y,3.9179255308,1992-06-13,1992-06-13
 21:54:32,1
   
9998,To1UUHYi5uy0m0g7,HGihOdi9OyH5LawdgbcgOsGxTaH0du1H,24.4223120552,2007-04-01,2007-04-01
 05:15:26,0
   
9997,zGaXQadwfnchbvhc,XtkA3KuKgK0BuGQONGx4mkZnbm54grSP,47.5443607568,2010-04-26,2010-04-26
 03:54:28,0
   ```
   # 3. opt resutl
   |  origin   | opt  |
   |    |   |
   | 25s  | 19s |
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (No/)
   2. Has unit tests been added: (No Need)
   3. Has document been added or modified: (No Need)
   4. Does it need to update dependencies: (No)
   5. Are there any changes that cannot be rolled back: (Yes)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9770: [fix] disabel transfer data large than 2GB

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-doris/pull/9770#issuecomment-1136975953

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #9663: [Enhancement]Add prefetch in join node

2022-05-25 Thread GitBox


yiguolei commented on code in PR #9663:
URL: https://github.com/apache/incubator-doris/pull/9663#discussion_r881422957


##
be/src/vec/exec/join/vhash_join_node.cpp:
##
@@ -77,18 +81,44 @@ struct ProcessHashTableBuild {
 inserted_rows.reserve(_batch_size);
 }
 
+std::vector cached_hash_values(CACHE_NUM);
+std::vector cached_key_holders(CACHE_NUM);
+
+size_t cached_count = 0;
 for (size_t k = 0; k < _rows; ++k) {
+if (UNLIKELY(k % CACHE_NUM == 0)) {
+cached_count = std::min(CACHE_NUM, _rows - k);
+for (size_t i = k; i < _rows && i < (k + CACHE_NUM); ++i) {

Review Comment:
   for (size_t i = 0; i < cache_count; ++i) {
   if constexpr (ignore_null) {
   if ((*null_map)[i+k]) {
   if ((*null_map)[i+k]) {
   continue;
   }
   }
   
   cached_key_holders[i ] =
   key_getter.get_key_holder(i + k, 
_join_node->_arena);
   cached_hash_values[i ] = key_getter.get_hash(
   hash_table_ctx.hash_table, cached_key_holders[i 
]);
   }



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #9663: [Enhancement]Add prefetch in join node

2022-05-25 Thread GitBox


yiguolei commented on code in PR #9663:
URL: https://github.com/apache/incubator-doris/pull/9663#discussion_r881422957


##
be/src/vec/exec/join/vhash_join_node.cpp:
##
@@ -77,18 +81,44 @@ struct ProcessHashTableBuild {
 inserted_rows.reserve(_batch_size);
 }
 
+std::vector cached_hash_values(CACHE_NUM);
+std::vector cached_key_holders(CACHE_NUM);
+
+size_t cached_count = 0;
 for (size_t k = 0; k < _rows; ++k) {
+if (UNLIKELY(k % CACHE_NUM == 0)) {
+cached_count = std::min(CACHE_NUM, _rows - k);
+for (size_t i = k; i < _rows && i < (k + CACHE_NUM); ++i) {

Review Comment:
   for (size_t i = 0; i < cache_count; ++i) {
   if constexpr (ignore_null) {
   if ((*null_map)[i+k]) {
   if ((*null_map)[i+k]) {
   continue;
   }
   }
   cached_key_holders[i ] =
   key_getter.get_key_holder(i + k, 
_join_node->_arena);
   cached_hash_values[i ] = key_getter.get_hash(
   hash_table_ctx.hash_table, cached_key_holders[i 
]);
   }



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #9663: [Enhancement]Add prefetch in join node

2022-05-25 Thread GitBox


yiguolei commented on code in PR #9663:
URL: https://github.com/apache/incubator-doris/pull/9663#discussion_r881422957


##
be/src/vec/exec/join/vhash_join_node.cpp:
##
@@ -77,18 +81,44 @@ struct ProcessHashTableBuild {
 inserted_rows.reserve(_batch_size);
 }
 
+std::vector cached_hash_values(CACHE_NUM);
+std::vector cached_key_holders(CACHE_NUM);
+
+size_t cached_count = 0;
 for (size_t k = 0; k < _rows; ++k) {
+if (UNLIKELY(k % CACHE_NUM == 0)) {
+cached_count = std::min(CACHE_NUM, _rows - k);
+for (size_t i = k; i < _rows && i < (k + CACHE_NUM); ++i) {

Review Comment:
   for (size_t i = 0; i < cache_count; ++i) {
   if constexpr (ignore_null) {
   if ((*null_map)[i+k]) {
   if ((*null_map)[i+k]) {
   continue;
   }
   }
   cached_key_holders[i ] =
   key_getter.get_key_holder(i + k, 
_join_node->_arena);
   cached_hash_values[i ] = key_getter.get_hash(
   hash_table_ctx.hash_table, cached_key_holders[i 
]);
   }



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] Gabriel39 opened a new pull request, #9774: [Improvement] remove unused code in vectorized compaction

2022-05-25 Thread GitBox


Gabriel39 opened a new pull request, #9774:
URL: https://github.com/apache/incubator-doris/pull/9774

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] xiepengcheng01 commented on a diff in pull request #9772: [Opt][VecLoad] Opt the vec stream load performance

2022-05-25 Thread GitBox


xiepengcheng01 commented on code in PR #9772:
URL: https://github.com/apache/incubator-doris/pull/9772#discussion_r881432756


##
be/src/vec/exec/vbroker_scanner.cpp:
##
@@ -20,11 +20,11 @@
 #include 
 
 #include 
-#include 
 
 #include "exec/exec_node.h"
 #include "exec/plain_text_line_reader.h"
 #include "exec/text_converter.h"
+#include "exec/text_converter.hpp"

Review Comment:
   can remove `exec/text_converter.h`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9763: [security] update canal version to fix fastjson security issue

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9763:
URL: https://github.com/apache/incubator-doris/pull/9763#issuecomment-1137055444

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9763: [security] update canal version to fix fastjson security issue

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9763:
URL: https://github.com/apache/incubator-doris/pull/9763#issuecomment-1137055487

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #9763: [security] update canal version to fix fastjson security issue

2022-05-25 Thread GitBox


yangzhg merged PR #9763:
URL: https://github.com/apache/incubator-doris/pull/9763


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [security] update canal version to fix fastjson security issue (#9763)

2022-05-25 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new be026addde [security] update canal version to fix fastjson security 
issue (#9763)
be026addde is described below

commit be026adddee8ad9154bd81e34141f3f075739edc
Author: Zhengguo Yang 
AuthorDate: Wed May 25 18:22:37 2022 +0800

[security] update canal version to fix fastjson security issue (#9763)
---
 fe/fe-core/pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml
index 009cf9df90..b975bdafdd 100644
--- a/fe/fe-core/pom.xml
+++ b/fe/fe-core/pom.xml
@@ -431,7 +431,7 @@ under the License.
 
 com.alibaba.otter
 canal.client
-1.1.4
+1.1.6
 
 
 ch.qos.logback
@@ -447,7 +447,7 @@ under the License.
 
 com.alibaba.otter
 canal.protocol
-1.1.4
+1.1.6
 
 
 ch.qos.logback


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] jackwener opened a new issue, #9775: [Enhancement] Remove the redundant `this.`

2022-05-25 Thread GitBox


jackwener opened a new issue, #9775:
URL: https://github.com/apache/incubator-doris/issues/9775

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Description
   
   Remove uncessary `this.`
   
   Most scenario, it just use in `constructor`.
   
   ### Solution
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] stalary commented on pull request #9694: [Bug] Fix create table like when having hidden columns

2022-05-25 Thread GitBox


stalary commented on PR #9694:
URL: https://github.com/apache/incubator-doris/pull/9694#issuecomment-1137075942

   LGTM @yiguolei 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #9770: [fix] disabel transfer data large than 2GB

2022-05-25 Thread GitBox


yangzhg merged PR #9770:
URL: https://github.com/apache/incubator-doris/pull/9770


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [fix] disable transfer data large than 2GB by brpc (#9770)

2022-05-25 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new f5bef328fe [fix] disable transfer data large than 2GB by brpc (#9770)
f5bef328fe is described below

commit f5bef328fe4c4701bb3eccfb02f992d68c6fa5ae
Author: Zhengguo Yang 
AuthorDate: Wed May 25 18:41:13 2022 +0800

[fix] disable transfer data large than 2GB by brpc (#9770)

because of brpc and protobuf cannot transfer data large than 2GB, if large 
than 2GB will overflow, so add a check before send
---
 be/src/runtime/row_batch.cpp | 11 ++-
 be/src/vec/core/block.cpp| 35 ---
 2 files changed, 30 insertions(+), 16 deletions(-)

diff --git a/be/src/runtime/row_batch.cpp b/be/src/runtime/row_batch.cpp
index 520a5642f9..5acf7634de 100644
--- a/be/src/runtime/row_batch.cpp
+++ b/be/src/runtime/row_batch.cpp
@@ -273,10 +273,6 @@ Status RowBatch::serialize(PRowBatch* output_batch, 
size_t* uncompressed_size,
 try {
 // Allocation of extra-long contiguous memory may fail, and data 
compression cannot be used if it fails
 _compression_scratch.resize(max_compressed_size);
-} catch (const std::bad_alloc& e) {
-can_compress = false;
-LOG(WARNING) << "Try to alloc " << max_compressed_size
- << " bytes for compression scratch failed. " << 
e.what();
 } catch (...) {
 can_compress = false;
 std::exception_ptr p = std::current_exception();
@@ -309,11 +305,8 @@ Status RowBatch::serialize(PRowBatch* output_batch, 
size_t* uncompressed_size,
 *compressed_size = pb_size;
 if (pb_size > std::numeric_limits::max()) {
 // the protobuf has a hard limit of 2GB for serialized data.
-return Status::InternalError(
-fmt::format("The rowbatch is large than 2GB({}), can not 
send by Protobuf. "
-"please set BE config 
'transfer_data_by_brpc_attachment' to true "
-"and restart BE.",
-pb_size));
+return Status::InternalError(fmt::format(
+"The rowbatch is large than 2GB({}), can not send by 
Protobuf.", pb_size));
 }
 } else {
 *uncompressed_size = pb_size + tuple_byte_size;
diff --git a/be/src/vec/core/block.cpp b/be/src/vec/core/block.cpp
index aa482fcfbf..4866a1129e 100644
--- a/be/src/vec/core/block.cpp
+++ b/be/src/vec/core/block.cpp
@@ -669,7 +669,16 @@ Status Block::serialize(PBlock* pblock, size_t* 
uncompressed_bytes, size_t* comp
 
 // serialize data values
 // when data type is HLL, content_uncompressed_size maybe larger than real 
size.
-allocated_buf->resize(content_uncompressed_size);
+try {
+allocated_buf->resize(content_uncompressed_size);
+} catch (...) {
+std::exception_ptr p = std::current_exception();
+std::string msg = fmt::format("Try to alloc {} bytes for allocated_buf 
failed. reason {}",
+  content_uncompressed_size,
+  p ? p.__cxa_exception_type()->name() : 
"null");
+LOG(WARNING) << msg;
+return Status::BufferAllocFailed(msg);
+}
 char* buf = allocated_buf->data();
 for (const auto& c : *this) {
 buf = c.type->serialize(*(c.column), buf);
@@ -678,12 +687,21 @@ Status Block::serialize(PBlock* pblock, size_t* 
uncompressed_bytes, size_t* comp
 
 // compress
 if (config::compress_rowbatches && content_uncompressed_size > 0) {
-// Try compressing the content to compression_scratch,
-// swap if compressed data is smaller
+size_t max_compressed_size = 
snappy::MaxCompressedLength(content_uncompressed_size);
 std::string compression_scratch;
-uint32_t max_compressed_size = 
snappy::MaxCompressedLength(content_uncompressed_size);
-compression_scratch.resize(max_compressed_size);
-
+try {
+// Try compressing the content to compression_scratch,
+// swap if compressed data is smaller
+// Allocation of extra-long contiguous memory may fail, and data 
compression cannot be used if it fails
+compression_scratch.resize(max_compressed_size);
+} catch (...) {
+std::exception_ptr p = std::current_exception();
+std::string msg =
+fmt::format("Try to alloc {} bytes for compression scratch 
failed. reason {}",
+max_compressed_size, p ? 
p.__cxa_exception_type()->name() : "null");
+LOG(WARNING) << msg;
+return Status::BufferAllocFailed(msg);
+}
 size_t compressed_size = 0;
 char* compressed_output = compression_

[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9774: [Improvement] remove unused code in vectorized compaction

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9774:
URL: https://github.com/apache/incubator-doris/pull/9774#issuecomment-1137086006

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] LOVEGISER opened a new pull request, #9776: website document update

2022-05-25 Thread GitBox


LOVEGISER opened a new pull request, #9776:
URL: https://github.com/apache/incubator-doris/pull/9776

   # Proposed changes
   website document update 
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (No)
   2. Has unit tests been added: (No Need)
   3. Has document been added or modified: (Yes)
   4. Does it need to update dependencies: (No)
   5. Are there any changes that cannot be rolled back: (Yes)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] pengxiangyu opened a new issue, #9777: [Feature] Add cancel export operation

2022-05-25 Thread GitBox


pengxiangyu opened a new issue, #9777:
URL: https://github.com/apache/incubator-doris/issues/9777

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Description
   
   _No response_
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9773: [BUG] fix correctness for vectorized compaction

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9773:
URL: https://github.com/apache/incubator-doris/pull/9773#issuecomment-1137119322

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yiguolei merged pull request #9743: [Improvement] fix typo

2022-05-25 Thread GitBox


yiguolei merged PR #9743:
URL: https://github.com/apache/incubator-doris/pull/9743


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [Improvement] fix typo (#9743)

2022-05-25 Thread yiguolei
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 8470543144 [Improvement] fix typo (#9743)
8470543144 is described below

commit 8470543144f46ff92fe7ef4364291a9709b06fe7
Author: Gabriel 
AuthorDate: Wed May 25 19:29:01 2022 +0800

[Improvement] fix typo (#9743)
---
 be/src/vec/core/types.h|  2 +-
 .../vec/function/function_array_element_test.cpp   | 20 ++---
 be/test/vec/function/function_array_index_test.cpp | 16 ++--
 be/test/vec/function/function_ifnull_test.cpp  |  4 +-
 be/test/vec/function/function_nullif_test.cpp  |  4 +-
 be/test/vec/function/function_string_test.cpp  |  8 +-
 be/test/vec/function/function_test_util.cpp|  2 +-
 be/test/vec/function/function_test_util.h  |  2 +-
 be/test/vec/function/function_time_test.cpp| 94 +++---
 9 files changed, 76 insertions(+), 76 deletions(-)

diff --git a/be/src/vec/core/types.h b/be/src/vec/core/types.h
index 0d70204e14..3f4d4317de 100644
--- a/be/src/vec/core/types.h
+++ b/be/src/vec/core/types.h
@@ -138,7 +138,7 @@ struct TypeName {
 };
 template <>
 struct TypeName {
-static const char* get() { return "SringValue"; }
+static const char* get() { return "StringValue"; }
 };
 
 template <>
diff --git a/be/test/vec/function/function_array_element_test.cpp 
b/be/test/vec/function/function_array_element_test.cpp
index 39d9c23cab..1c1be231f6 100644
--- a/be/test/vec/function/function_array_element_test.cpp
+++ b/be/test/vec/function/function_array_element_test.cpp
@@ -90,13 +90,13 @@ TEST(function_array_element_test, element_at) {
 {
 InputTypeSet input_types = {TypeIndex::Array, TypeIndex::DateTime, 
TypeIndex::Int64};
 
-Array vec = {str_to_data_time("2022-01-02 01:00:00"), 
str_to_data_time(""),
- str_to_data_time("2022-07-08 03:00:00")};
+Array vec = {str_to_date_time("2022-01-02 01:00:00"), 
str_to_date_time(""),
+ str_to_date_time("2022-07-08 03:00:00")};
 DataSet data_set = {{{vec, Int64(0)}, Null()},
-{{vec, Int64(1)}, str_to_data_time("2022-01-02 
01:00:00")},
+{{vec, Int64(1)}, str_to_date_time("2022-01-02 
01:00:00")},
 {{vec, Int64(4)}, Null()},
-{{vec, Int64(-1)}, str_to_data_time("2022-07-08 
03:00:00")},
-{{vec, Int64(-2)}, str_to_data_time("")},
+{{vec, Int64(-1)}, str_to_date_time("2022-07-08 
03:00:00")},
+{{vec, Int64(-2)}, str_to_date_time("")},
 {{vec, Int64(-4)}, Null()},
 {{Null(), Int64(1)}, Null()},
 {{empty_arr, Int64(0)}, Null()},
@@ -109,13 +109,13 @@ TEST(function_array_element_test, element_at) {
 {
 InputTypeSet input_types = {TypeIndex::Array, TypeIndex::Date, 
TypeIndex::Int64};
 
-Array vec = {str_to_data_time("2022-01-02"), str_to_data_time(""),
- str_to_data_time("2022-07-08")};
+Array vec = {str_to_date_time("2022-01-02"), str_to_date_time(""),
+ str_to_date_time("2022-07-08")};
 DataSet data_set = {{{vec, Int64(0)}, Null()},
-{{vec, Int64(1)}, str_to_data_time("2022-01-02")},
+{{vec, Int64(1)}, str_to_date_time("2022-01-02")},
 {{vec, Int64(4)}, Null()},
-{{vec, Int64(-1)}, str_to_data_time("2022-07-08")},
-{{vec, Int64(-2)}, str_to_data_time("")},
+{{vec, Int64(-1)}, str_to_date_time("2022-07-08")},
+{{vec, Int64(-2)}, str_to_date_time("")},
 {{vec, Int64(-4)}, Null()},
 {{Null(), Int64(1)}, Null()},
 {{empty_arr, Int64(0)}, Null()},
diff --git a/be/test/vec/function/function_array_index_test.cpp 
b/be/test/vec/function/function_array_index_test.cpp
index c1a0231a7d..6584c69a75 100644
--- a/be/test/vec/function/function_array_index_test.cpp
+++ b/be/test/vec/function/function_array_index_test.cpp
@@ -113,8 +113,8 @@ TEST(function_array_index_test, array_contains) {
 {
 InputTypeSet input_types = {TypeIndex::Array, TypeIndex::Date, 
TypeIndex::Date};
 
-Array vec = {str_to_data_time("2022-01-02", false), 
str_to_data_time("", false),
- str_to_data_time("2022-07-08", false)};
+Array vec = {str_to_date_time("2022-01-02", false), 
str_to_date_time("", false),
+ str_to_date_time("2022-07-08", false)};
 DataSet data_set = {{{vec, std::string("2022-01-02")

[GitHub] [incubator-doris] Gabriel39 commented on a diff in pull request #9582: [refactor](decimalv3) optimize decimal performance and precision

2022-05-25 Thread GitBox


Gabriel39 commented on code in PR #9582:
URL: https://github.com/apache/incubator-doris/pull/9582#discussion_r881540625


##
be/src/vec/functions/function_binary_arithmetic.h:
##
@@ -449,6 +469,18 @@ struct DecimalBinaryOperation {
 private:
 /// there's implicit type convertion here
 static NativeResultType apply(NativeResultType a, NativeResultType b) {
+if (config::enable_execution_decimalv3) {
+if constexpr (OpTraits::can_overflow && check_overflow) {
+NativeResultType res;
+if (Op::template apply(a, b, res)) {
+LOG(FATAL) << "Decimal math overflow";

Review Comment:
   how about to return null when overflow?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] stalary opened a new pull request, #9778: [Bug] sql block rule npe

2022-05-25 Thread GitBox


stalary opened a new pull request, #9778:
URL: https://github.com/apache/incubator-doris/pull/9778

   # Proposed changes
   
   Issue Number: close https://github.com/apache/incubator-doris/issues/9749
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #9771: [improvement] Show detail status code string for BE http api

2022-05-25 Thread GitBox


yiguolei commented on code in PR #9771:
URL: https://github.com/apache/incubator-doris/pull/9771#discussion_r881545905


##
be/src/util/json_util.cpp:
##
@@ -32,11 +32,7 @@ std::string to_json(const Status& status) {
 writer.StartObject();
 // status
 writer.Key("status");
-if (status.ok()) {

Review Comment:
   Hi jack, I think we could move to_json(Status& status) method to  
common/status.h since there is only one method in this file and it is related 
with common/status.
   And also you  could remove json_util_test.h because we could not ensure that 
the error message always the same with the expected string.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] liaoxin01 commented on a diff in pull request #9582: [refactor](decimalv3) optimize decimal performance and precision

2022-05-25 Thread GitBox


liaoxin01 commented on code in PR #9582:
URL: https://github.com/apache/incubator-doris/pull/9582#discussion_r881554793


##
be/src/vec/functions/function_binary_arithmetic.h:
##
@@ -449,6 +469,18 @@ struct DecimalBinaryOperation {
 private:
 /// there's implicit type convertion here
 static NativeResultType apply(NativeResultType a, NativeResultType b) {
+if (config::enable_execution_decimalv3) {
+if constexpr (OpTraits::can_overflow && check_overflow) {
+NativeResultType res;
+if (Op::template apply(a, b, res)) {
+LOG(FATAL) << "Decimal math overflow";

Review Comment:
   The column may not be ColumnNullable, so we can't return null.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] pengxiangyu opened a new pull request, #9779: Add backend_rpc_timeout_second in AdminConfig

2022-05-25 Thread GitBox


pengxiangyu opened a new pull request, #9779:
URL: https://github.com/apache/incubator-doris/pull/9779

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Add backend_rpc_timeout_second in AdminConfig
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (No)
   2. Has unit tests been added: (No)
   3. Has document been added or modified: (No)
   4. Does it need to update dependencies: (No)
   5. Are there any changes that cannot be rolled back: (No)
   
   ## Further comments
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] xxiao2018 commented on pull request #8730: [Enhancement] enhancement for regression-test framework

2022-05-25 Thread GitBox


xxiao2018 commented on PR #8730:
URL: https://github.com/apache/incubator-doris/pull/8730#issuecomment-1137199570

   fix in #8930


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9753: [doc]Add Doris join optimization documentation

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9753:
URL: https://github.com/apache/incubator-doris/pull/9753#issuecomment-1137213800

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9753: [doc]Add Doris join optimization documentation

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9753:
URL: https://github.com/apache/incubator-doris/pull/9753#issuecomment-1137213841

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9761: [fix](help) fix bug of help command

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9761:
URL: https://github.com/apache/incubator-doris/pull/9761#issuecomment-1137217678

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9761: [fix](help) fix bug of help command

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9761:
URL: https://github.com/apache/incubator-doris/pull/9761#issuecomment-1137217599

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] HappenLee merged pull request #9459: [stream-load-vec]: memtable flush only if necessary after aggregated

2022-05-25 Thread GitBox


HappenLee merged PR #9459:
URL: https://github.com/apache/incubator-doris/pull/9459


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] HappenLee closed issue #8656: [Enhancement] [Vectorized] optimize pre-agg in stream load memtable

2022-05-25 Thread GitBox


HappenLee closed issue #8656: [Enhancement] [Vectorized] optimize pre-agg in 
stream load memtable
URL: https://github.com/apache/incubator-doris/issues/8656


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [stream-load-vec]: memtable flush only if necessary after aggregated (#9459)

2022-05-25 Thread lihaopeng
This is an automated email from the ASF dual-hosted git repository.

lihaopeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 73e31a2179 [stream-load-vec]: memtable flush only if necessary after 
aggregated (#9459)
73e31a2179 is described below

commit 73e31a21799c15abde903502ad5b73138b01c336
Author: spaces-x 
AuthorDate: Wed May 25 21:12:24 2022 +0800

[stream-load-vec]: memtable flush only if necessary after aggregated (#9459)


Co-authored-by: weixiang 
---
 be/src/common/config.h   |  3 +++
 be/src/olap/delta_writer.cpp |  9 ---
 be/src/olap/memtable.cpp | 56 +++-
 be/src/olap/memtable.h   | 10 +++-
 be/src/vec/core/block.cpp| 19 +++
 be/src/vec/core/block.h  |  7 ++
 6 files changed, 89 insertions(+), 15 deletions(-)

diff --git a/be/src/common/config.h b/be/src/common/config.h
index fcfc2e3f78..a73efcf24d 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -464,6 +464,9 @@ CONF_Int32(memory_max_alignment, "16");
 // write buffer size before flush
 CONF_mInt64(write_buffer_size, "209715200");
 
+// max buffer size used in memtable for the aggregated table
+CONF_mInt64(memtable_max_buffer_size, "419430400");
+
 // following 2 configs limit the memory consumption of load process on a 
Backend.
 // eg: memory limit to 80% of mem limit config but up to 100GB(default)
 // NOTICE(cmy): set these default values very large because we don't want to
diff --git a/be/src/olap/delta_writer.cpp b/be/src/olap/delta_writer.cpp
index 11ee242449..fe0ffeddbb 100644
--- a/be/src/olap/delta_writer.cpp
+++ b/be/src/olap/delta_writer.cpp
@@ -203,9 +203,12 @@ Status DeltaWriter::write(const vectorized::Block* block, 
const std::vector
 }
 }
 
-if (_mem_table->memory_usage() >= config::write_buffer_size) {
-RETURN_NOT_OK(_flush_memtable_async());
-_reset_mem_table();
+if (_mem_table->need_to_agg()) {
+_mem_table->shrink_memtable_by_agg();
+if (_mem_table->is_flush()) {
+RETURN_NOT_OK(_flush_memtable_async());
+_reset_mem_table();
+}
 }
 
 return Status::OK();
diff --git a/be/src/olap/memtable.cpp b/be/src/olap/memtable.cpp
index c7f94ead4f..cf58f7e4ba 100644
--- a/be/src/olap/memtable.cpp
+++ b/be/src/olap/memtable.cpp
@@ -126,11 +126,10 @@ void MemTable::insert(const vectorized::Block* block, 
size_t row_pos, size_t num
 }
 }
 size_t cursor_in_mutableblock = _input_mutable_block.rows();
-size_t oldsize = _input_mutable_block.allocated_bytes();
 _input_mutable_block.add_rows(block, row_pos, num_rows);
-size_t newsize = _input_mutable_block.allocated_bytes();
-_mem_usage += newsize - oldsize;
-_mem_tracker->consume(newsize - oldsize);
+size_t input_size = block->allocated_bytes() * num_rows / block->rows();
+_mem_usage += input_size;
+_mem_tracker->consume(input_size);
 
 for (int i = 0; i < num_rows; i++) {
 _row_in_blocks.emplace_back(new RowInBlock {cursor_in_mutableblock + 
i});
@@ -242,7 +241,8 @@ void MemTable::_aggregate_two_row_in_block(RowInBlock* 
new_row, RowInBlock* row_
  new_row->_row_pos, nullptr);
 }
 }
-vectorized::Block MemTable::_collect_vskiplist_results() {
+template 
+void MemTable::_collect_vskiplist_results() {
 VecTable::Iterator it(_vec_skip_list.get());
 vectorized::Block in_block = _input_mutable_block.to_block();
 // TODO: should try to insert data by column, not by row. to opt the code
@@ -251,6 +251,7 @@ vectorized::Block MemTable::_collect_vskiplist_results() {
 _output_mutable_block.add_row(&in_block, it.key()->_row_pos);
 }
 } else {
+size_t idx = 0;
 for (it.SeekToFirst(); it.Valid(); it.Next()) {
 auto& block_data = in_block.get_columns_with_type_and_name();
 // move key columns
@@ -263,11 +264,46 @@ vectorized::Block MemTable::_collect_vskiplist_results() {
 auto function = _agg_functions[i];
 function->insert_result_into(it.key()->_agg_places[i],
  
*(_output_mutable_block.get_column_by_position(i)));
-function->destroy(it.key()->_agg_places[i]);
+if constexpr (is_final) {
+function->destroy(it.key()->_agg_places[i]);
+}
 }
+if constexpr (!is_final) {
+// re-index the row_pos in VSkipList
+it.key()->_row_pos = idx;
+idx++;
+}
+}
+if constexpr (!is_final) {
+// if is not final, we collect the agg results to input_block and 
then continue to insert
+size_t shrunked_after_agg = 
_output_mutable_block.allocated_bytes();
+  

[GitHub] [incubator-doris] stalary opened a new issue, #9780: [Feature] Support Col Policy

2022-05-25 Thread GitBox


stalary opened a new issue, #9780:
URL: https://github.com/apache/incubator-doris/issues/9780

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Description
   
   Support Col Policy, similar to row policy.
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9776: [optimiz] website document update

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9776:
URL: https://github.com/apache/incubator-doris/pull/9776#issuecomment-1137277358

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9776: [optimiz] website document update

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9776:
URL: https://github.com/apache/incubator-doris/pull/9776#issuecomment-1137277475

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] morningman commented on a diff in pull request #9779: Add backend_rpc_timeout_second in AdminConfig

2022-05-25 Thread GitBox


morningman commented on code in PR #9779:
URL: https://github.com/apache/incubator-doris/pull/9779#discussion_r881765585


##
fe/fe-core/src/main/java/org/apache/doris/common/Config.java:
##
@@ -1651,4 +1651,8 @@ public class Config extends ConfigBase {
 
 @ConfField
 public static boolean enable_vectorized_load = false;
+
+@ConfField(mutable = false, masterOnly = true)
+public static int backend_rpc_timeout_ms = 6; // 1 min

Review Comment:
   Please add comment for this config and also modify the config document
   `admin-manual/config/fe-config.md`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9728: [Bug] If function adds type inference

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9728:
URL: https://github.com/apache/incubator-doris/pull/9728#issuecomment-1137433042

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] JNSimba commented on issue #9713: [Bug] doris-flink 连接器不能正常运行

2022-05-25 Thread GitBox


JNSimba commented on issue #9713:
URL: 
https://github.com/apache/incubator-doris/issues/9713#issuecomment-1137435681

   Does doris-flink-1.0-SNAPSHOT.jar also depend on flink1.13? Then confirm 
whether the scala versions on both sides are the same?
   Also, you can use the jar package of the flink doris connector in the maven 
repository


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] morningman closed issue #9465: [feature-wip][array-type] Support more sub types.

2022-05-25 Thread GitBox


morningman closed issue #9465: [feature-wip][array-type] Support more sub types.
URL: https://github.com/apache/incubator-doris/issues/9465


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] morningman merged pull request #9466: [feature-wip][array-type] Support more sub types.

2022-05-25 Thread GitBox


morningman merged PR #9466:
URL: https://github.com/apache/incubator-doris/pull/9466


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [feature-wip][array-type] Support more sub types. (#9466)

2022-05-25 Thread morningman
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 2a11a4ab99 [feature-wip][array-type] Support more sub types. (#9466)
2a11a4ab99 is described below

commit 2a11a4ab99206ebf9cf6c62c885088b8ead5bc05
Author: Adonis Ling 
AuthorDate: Thu May 26 08:41:34 2022 +0800

[feature-wip][array-type] Support more sub types. (#9466)

Please refer to #9465
---
 be/src/exprs/array_functions.cpp   |   5 +-
 be/src/exprs/literal.cpp   |  13 +-
 be/src/olap/aggregate_func.cpp |  16 +-
 be/src/olap/rowset/segment_v2/segment_writer.cpp   |   2 +-
 be/src/olap/types.h|  15 +-
 be/src/runtime/collection_value.cpp| 636 -
 be/src/runtime/collection_value.h  | 176 --
 be/src/runtime/mysql_result_writer.cpp |  22 +-
 be/src/runtime/raw_value.cpp   |   9 +-
 be/src/runtime/row_batch.cpp   |   8 +-
 be/src/runtime/tuple.h |   2 +-
 be/src/udf/udf.h   |   2 +-
 be/src/util/array_parser.h |  65 ++-
 be/src/vec/core/block.cpp  |  28 +-
 be/src/vec/olap/olap_data_convertor.cpp| 266 -
 be/src/vec/olap/olap_data_convertor.h  |  85 +--
 be/src/vec/sink/mysql_result_writer.cpp|  12 +
 be/test/exprs/array_functions_test.cpp |   2 +-
 be/test/runtime/array_test.cpp | 417 ++
 be/test/runtime/collection_value_test.cpp  |  13 +-
 be/test/testutil/array_utils.cpp   |  26 +
 be/test/testutil/array_utils.h |   2 +-
 be/test/util/array_parser_test.cpp |  54 ++
 .../java/org/apache/doris/analysis/TypeDef.java|   4 -
 .../java/org/apache/doris/catalog/ArrayType.java   |   3 +
 25 files changed, 1207 insertions(+), 676 deletions(-)

diff --git a/be/src/exprs/array_functions.cpp b/be/src/exprs/array_functions.cpp
index b9b7fcf8a4..be9f151b22 100644
--- a/be/src/exprs/array_functions.cpp
+++ b/be/src/exprs/array_functions.cpp
@@ -30,8 +30,9 @@ void ArrayFunctions::init() {}
 DCHECK_EQ(context->get_return_type().children.size(), 1);  
 \
 CollectionValue v; 
 \
 CollectionValue::init_collection(context, num_children, PRIMARY_TYPE, 
&v);  \
-for (int i = 0; i < num_children; ++i) {   
 \
-v.set(i, PRIMARY_TYPE, values + i);
 \
+auto iterator = v.iterator(PRIMARY_TYPE);  
 \
+for (int i = 0; i < num_children; ++i, iterator.next()) {  
 \
+iterator.set(values + i);  
 \
 }  
 \
 CollectionVal ret; 
 \
 v.to_collection_val(&ret); 
 \
diff --git a/be/src/exprs/literal.cpp b/be/src/exprs/literal.cpp
index 5e1bbb9846..64401b7fe2 100644
--- a/be/src/exprs/literal.cpp
+++ b/be/src/exprs/literal.cpp
@@ -183,13 +183,14 @@ Status Literal::prepare(RuntimeState* state, const 
RowDescriptor& row_desc, Expr
 if (type().type == TYPE_ARRAY) {
 DCHECK_EQ(type().children.size(), 1) << "array children type not 1";
 // init array value
-auto td = type().children.at(0).type;
-RETURN_IF_ERROR(CollectionValue::init_collection(state->obj_pool(), 
get_num_children(), td,
- &_value.array_val));
+auto child_type = type().children.at(0).type;
+RETURN_IF_ERROR(CollectionValue::init_collection(state->obj_pool(), 
get_num_children(),
+ child_type, 
&_value.array_val));
+auto iterator = _value.array_val.iterator(child_type);
 // init every item
-for (int i = 0; i < get_num_children(); ++i) {
-Expr* children = get_child(i);
-RETURN_IF_ERROR(_value.array_val.set(i, td, 
children->get_const_val(context)));
+for (int i = 0; i < get_num_children() && iterator.has_next(); ++i, 
iterator.next()) {
+Expr* child = get_child(i);
+iterator.set(child->get_const_val(context));
 }
 }
 
diff --git a/be/src/olap/aggregate_func.cpp b/be/src/olap/aggregate_func.cpp
index 4d583d476b..781ab39082 100644
--- a/be/src/olap/aggregate_func.cpp
+++ b/be/src/olap/aggregate_func.cpp
@@ -105,6 +105,8 

[GitHub] [incubator-doris] morningman merged pull request #9492: [fix](resource-tag) Consider resource tags when assigning tasks for broker & routine load

2022-05-25 Thread GitBox


morningman merged PR #9492:
URL: https://github.com/apache/incubator-doris/pull/9492


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [fix](resource-tag) Consider resource tags when assigning tasks for broker & routine load (#9492)

2022-05-25 Thread morningman
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 0c70359404 [fix](resource-tag) Consider resource tags when assigning 
tasks for broker & routine load (#9492)
0c70359404 is described below

commit 0c70359404f7b4ecfb53ec17c994d0a3a6e5c08a
Author: Mingyu Chen 
AuthorDate: Thu May 26 08:42:09 2022 +0800

[fix](resource-tag) Consider resource tags when assigning tasks for broker 
& routine load (#9492)

This CL mainly changes:
1. Broker Load
When assigning backends, use user level resource tag to find available 
backends.
If user level resource tag is not set, broker load task can be assigned 
to any BE node,
otherwise, task can only be assigned to BE node which match the user 
level tags.

2. Routine Load
The current routine load job does not have user info, so it can not get 
user level tag when assigning tasks.
So there are 2 ways:
1. For old routine load job, use tags of replica allocation info to 
select BE nodes.
2. For new routine load job, the user info will be added and persisted 
in routine load job.
---
 docs/en/admin-manual/multi-tenant.md   | 12 
 docs/zh-CN/admin-manual/multi-tenant.md| 14 +++-
 .../org/apache/doris/common/FeMetaVersion.java |  6 +-
 .../doris/load/loadv2/LoadingTaskPlanner.java  |  2 +-
 .../load/routineload/KafkaRoutineLoadJob.java  |  8 ++-
 .../doris/load/routineload/RoutineLoadJob.java | 27 +++-
 .../doris/load/routineload/RoutineLoadManager.java | 74 --
 .../load/routineload/RoutineLoadTaskScheduler.java |  3 +-
 .../mysql/privilege/UserResourceProperty.java  | 31 -
 .../org/apache/doris/planner/BrokerScanNode.java   | 45 +
 .../load/routineload/KafkaRoutineLoadJobTest.java  | 26 
 .../doris/load/routineload/RoutineLoadJobTest.java | 16 +++--
 .../load/routineload/RoutineLoadManagerTest.java   | 32 ++
 .../load/routineload/RoutineLoadSchedulerTest.java | 15 +++--
 .../transaction/GlobalTransactionMgrTest.java  | 13 ++--
 15 files changed, 223 insertions(+), 101 deletions(-)

diff --git a/docs/en/admin-manual/multi-tenant.md 
b/docs/en/admin-manual/multi-tenant.md
index fb121fa191..41663a1e4f 100644
--- a/docs/en/admin-manual/multi-tenant.md
+++ b/docs/en/admin-manual/multi-tenant.md
@@ -133,6 +133,18 @@ Node resource division refers to setting tags for BE nodes 
in a Doris cluster, a
 In this way, we have achieved physical resource isolation for different 
user queries by dividing nodes and restricting user resource usage. 
Furthermore, we can create different users for different business departments 
and restrict each user from using different resource groups. In order to avoid 
the use of resource interference between different business parts. For example, 
there is a business table in the cluster that needs to be shared by all 9 
business departments, but it is hoped [...]
 
 On the other hand, for the isolation of online and offline tasks. We can 
use resource groups to achieve this. For example, we can divide nodes into two 
resource groups, Online and Offline. The table data is still stored in 3 
copies, of which 2 copies are stored in the Online resource group, and 1 copy 
is stored in the Offline resource group. The Online resource group is mainly 
used for online data services with high concurrency and low latency. Some large 
queries or offline ETL opera [...]
+
+4. Resource group assignments for load job
+
+The resource usage of load jobs (including insert, broker load, routine 
load, stream load, etc.) can be divided into two parts:
+1. Computing resources: responsible for reading data sources, data 
transformation and distribution.
+2. Write resource: responsible for data encoding, compression and writing 
to disk.
+
+The write resource must be the node where the replica is located, and the 
computing resource can theoretically select any node to complete. Therefore, 
the allocation of resource groups for load jobs is divided into two steps:
+1. Use user-level resource tags to limit the resource groups that 
computing resources can use.
+2. Use the resource tag of the replica to limit the resource group that 
the write resource can use.
+
+So if you want all the resources used by the load operation to be limited 
to the resource group where the data is located, you only need to set the 
resource tag of the user level to the same as the resource tag of the replica.
 
 ## Single query resource limit
 
diff --git a/docs/zh-CN/admin-manual/multi-tenant.md 
b/docs/zh-CN/admin-manual/multi-tenant.md
index dadabd1a5a..acc1775c0a 100644
--- a/docs/zh-CN/admin-manual/multi-tenant.md
+++ b/docs/zh-CN/admin-manual/multi-tenant.md

[GitHub] [incubator-doris] morningman commented on pull request #9708: [refactor][be] Optimize headers

2022-05-25 Thread GitBox


morningman commented on PR #9708:
URL: https://github.com/apache/incubator-doris/pull/9708#issuecomment-1138019571

   Hi @adonis0147 please rebase to solve the conflict


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] morningman merged pull request #9761: [fix](help) fix bug of help command

2022-05-25 Thread GitBox


morningman merged PR #9761:
URL: https://github.com/apache/incubator-doris/pull/9761


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [fix](help) fix bug of help command (#9761)

2022-05-25 Thread morningman
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 32a210f426 [fix](help) fix bug of help command (#9761)
32a210f426 is described below

commit 32a210f426f9422a971a1e0157a349c6f88c9dcd
Author: Mingyu Chen 
AuthorDate: Thu May 26 08:44:00 2022 +0800

[fix](help) fix bug of help command (#9761)

This bug is introduced from #9306, that user need to execute
"help stream-load" to show the help doc.
But actually, it should be "help stream load".
---
 fe/fe-core/src/main/java/org/apache/doris/common/MarkDownParser.java | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/common/MarkDownParser.java 
b/fe/fe-core/src/main/java/org/apache/doris/common/MarkDownParser.java
index eced29b736..200c106003 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/common/MarkDownParser.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/MarkDownParser.java
@@ -166,7 +166,9 @@ public class MarkDownParser {
 }
 // Note that multiple line breaks at content's end will be merged to 
be one,
 // and other whitespace characters will be deleted.
-return Maps.immutableEntry(key.substring(headLevel).trim(),
+// Also, the header in md file is like "## STREAM-LOAD", we need to 
convert it to "STREAM LOAD",
+// so that we can execute "help stream load" to show the help doc.
+return 
Maps.immutableEntry(key.substring(headLevel).trim().replaceAll("-", " "),
 sb.toString().replaceAll("\\s+$", "\n"));
 }
 }


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] hf200012 commented on issue #9594: [Bug] PAUSE SYNC JOB 卡死

2022-05-25 Thread GitBox


hf200012 commented on issue #9594:
URL: 
https://github.com/apache/incubator-doris/issues/9594#issuecomment-1138036514

   Judging from your prompt information, it is the connection between your 
client and the server that is disconnected. You don't need to restart fe, just 
reconnect it. I made more than a dozen tables, inserted data, loaded it through 
binlog, and repeated it. Pause , resume , stop did not reproduce your problem 
@yiyulinfeng 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] mrhhsg opened a new issue, #9781: [Enhancement] Need to ignore generated files in tools by updating .gitignore

2022-05-25 Thread GitBox


mrhhsg opened a new issue, #9781:
URL: https://github.com/apache/incubator-doris/issues/9781

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Description
   
   Some generated files should be add ignored.
   ```bash
   $ git status
   On branch git_ignore
   Untracked files:
 (use "git add ..." to include in what will be committed)
   tools/ssb-tools/ssb-data/
   tools/ssb-tools/ssb-dbgen/
   tools/tpch-tools/TPC-H_Tools_v3.0.0.zip
   tools/tpch-tools/TPC-H_Tools_v3.0.0/
   tools/tpch-tools/tpc-h_v3.0.0.docx
   tools/tpch-tools/tpc-h_v3.0.0.pdf
   tools/tpch-tools/tpch-data/
   ```
   
   ### Solution
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] mrhhsg opened a new pull request, #9782: Update .gitignore to ignore generated files in tools

2022-05-25 Thread GitBox


mrhhsg opened a new pull request, #9782:
URL: https://github.com/apache/incubator-doris/pull/9782

   # Proposed changes
   
   Issue Number: close #9781
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg commented on a diff in pull request #9753: [doc]Add Doris join optimization documentation

2022-05-25 Thread GitBox


yangzhg commented on code in PR #9753:
URL: https://github.com/apache/incubator-doris/pull/9753#discussion_r882257343


##
docs/zh-CN/advanced/join-optimization/doris-join-optimization.md:
##
@@ -0,0 +1,226 @@
+---
+{
+"title": "Doris Join 优化原理",
+"language": "zh-CN"
+}
+
+
+---
+
+
+
+# Doris Join 优化原理
+
+Doris 支持两种物理算子,一类是 **Hash Join**,另一类是 **Nest Loop Join**。
+

Review Comment:
   doris doesn't really support  `Nest Loop Join`, it just uses an 
Nest-Loop-Join-like algorithm for non-equivalent joins, so I think this will 
cause some insurmountable problems for join optimization



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg commented on a diff in pull request #9753: [doc]Add Doris join optimization documentation

2022-05-25 Thread GitBox


yangzhg commented on code in PR #9753:
URL: https://github.com/apache/incubator-doris/pull/9753#discussion_r882257444


##
docs/zh-CN/advanced/join-optimization/doris-join-optimization.md:
##
@@ -0,0 +1,226 @@
+---
+{
+"title": "Doris Join 优化原理",
+"language": "zh-CN"
+}
+
+
+---
+
+
+
+# Doris Join 优化原理
+
+Doris 支持两种物理算子,一类是 **Hash Join**,另一类是 **Nest Loop Join**。
+
+- Hash Join:在右表上根据等值 Join 列建立哈希表,左表流式的利用哈希表进行 Join 计算,它的限制是只能适用于等值 Join。
+- Nest Loop Join:通过两个 for 循环,很直观。然后它适用的场景就是不等值的 
Join,例如:大于小于或者是需要求笛卡尔积的场景。它是一个通用的 Join 算子,但是性能表现差。
+
+作为分布式的 MPP 数据库, 在 Join 的过程中是需要进行数据的 Shuffle。数据需要进行拆分调度,才能保证最终的 Join 
结果是正确的。举个简单的例子,假设关系S 和 R 进行Join,N 表示参与 Join 计算的节点的数量;T 则表示关系的 Tuple 数目。
+
+
+
+## Doris Shuffle 方式
+
+Doris 支持 4 种 Shuffle 方式
+
+1. BroadCast Join

Review Comment:
   ```suggestion
   1. Broadcast Join
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg commented on a diff in pull request #9753: [doc]Add Doris join optimization documentation

2022-05-25 Thread GitBox


yangzhg commented on code in PR #9753:
URL: https://github.com/apache/incubator-doris/pull/9753#discussion_r882258297


##
docs/zh-CN/advanced/join-optimization/doris-join-optimization.md:
##
@@ -0,0 +1,226 @@
+---
+{
+"title": "Doris Join 优化原理",
+"language": "zh-CN"
+}
+
+
+---
+
+
+
+# Doris Join 优化原理
+
+Doris 支持两种物理算子,一类是 **Hash Join**,另一类是 **Nest Loop Join**。
+
+- Hash Join:在右表上根据等值 Join 列建立哈希表,左表流式的利用哈希表进行 Join 计算,它的限制是只能适用于等值 Join。
+- Nest Loop Join:通过两个 for 循环,很直观。然后它适用的场景就是不等值的 
Join,例如:大于小于或者是需要求笛卡尔积的场景。它是一个通用的 Join 算子,但是性能表现差。
+
+作为分布式的 MPP 数据库, 在 Join 的过程中是需要进行数据的 Shuffle。数据需要进行拆分调度,才能保证最终的 Join 
结果是正确的。举个简单的例子,假设关系S 和 R 进行Join,N 表示参与 Join 计算的节点的数量;T 则表示关系的 Tuple 数目。
+
+
+
+## Doris Shuffle 方式
+
+Doris 支持 4 种 Shuffle 方式
+
+1. BroadCast Join
+
+   它要求把右表全量的数据都发送到左表上,即每一个参与 Join 的节点,它都拥有右表全量的数据,也就是 T(R)。
+
+   它适用的场景是比较通用的,同时能够支持 Hash Join 和 Nest loop Join,它的网络开销 N * T(R)。
+
+   ![image-20220523152004731](/images/join/image-20220523152004731.png)
+
+   左表数据不移动,右表数据发送到左表数据的扫描节点。
+
+2. Shuffle Join
+
+   当进行 Hash Join 时候,可以通过 Join 列计算对应的 Hash 值,并进行 Hash 分桶。
+
+   它的网络开销则是:T(R) + T(N),但它只能支持 Hash Join,因为它是根据 Join 的条件也去做计算分桶的。
+
+   ![image-20220523151902368](/images/join/image-20220523151902368.png)
+
+   左右表数据根据分区,计算的记过发送到不同的分区节点上。
+
+3. Bucket Shuffle Join
+
+   Doris 的表数据本身是通过 Hash 计算分桶的,所以就可以利用表本身的分桶列的性质来进行 Join 数据的 Shuffle。假如两张表需要做 
Join,并且 Join 列是左表的分桶列,那么左表的数据其实可以不用去移动右表通过左表的数据分桶发送数据就可以完成  Join  的计算。
+
+   它的网络开销则是:T(R)相当于只 Shuffle 右表的数据就可以了。
+
+   ![image-20220523151653562](/images/join/image-20220523151653562.png)
+
+   左表数据不移动,右表数据根据分区计算的结果发送到左表扫表的节点
+
+4. Colocate
+
+   它与 Bucket Shuffle Join 相似,相当于在数据导入的时候,根据预设的 Join 列的场景已经做好了数据的 
Shuffle。那么实际查询的时候就可以直接进行 Join 计算而不需要考虑数据的 Shuffle 问题了。
+
+   ![image-20220523151619754](/images/join/image-20220523151619754.png)
+
+   数据已经预先分区,直接在本地进行 Join 计算
+
+### 四种 Shuffle 方式对比
+
+| Shuffle方式| 网络开销| 物理算子   | 适用场景   
  |
+| -- | --- | -- | 
 |
+| BroadCast  | N * T(R)| Hash Join / Nest Loop Join | 通用   
  |
+| Shuffle| T(S) + T(R) | Hash Join  | 通用   
  |
+| Bucket Shuffle | T(R)| Hash Join  | 
Join条件中存在左表的分布式列,且左表执行时为单分区   |
+| Colocate   | 0   | Hash Join  | 
Join条件中存在左表的分布式列,切左右表同属于一个Colocate Group |
+
+N : 参与 Join 计算的 Instance 个数
+
+T(关系) : 关系的 Tuple 数目
+
+上面这 4 种方式灵活度是从高到低的,它对这个数据分布的要求是越来越严格,但 Join 计算的性能也是越来越好的。
+
+## Runtime Filter  Join 优化
+
+Doris 在进行 Hash Join 计算时会在右表构建一个哈希表,左表流式的通过右表的哈希表从而得出 Join 结果。而 RuntimeFilter 
就是充分利用了右表的 Hash 表,在右表生成哈希表的时,同时生成一个基于哈希表数据的一个过滤条件,然后下推到左表的数据扫描节点。通过这样的方式,Doris 
可以在运行时进行数据过滤。
+
+假如左表是一张大表,右表是一张小表,那么利用左表生成的过滤条件就可以把绝大多数在 Join 层要过滤的数据在数据读取时就提前过滤,这样就能大幅度的提升 
Join 查询的性能。
+
+当前 Doris 支持三种类型 RuntimeFilter
+
+- 一种是 IN— IN,很好理解,将一个 hashset 下推到数据扫描节点。

Review Comment:
   ```suggestion
   - 一种是 IN,很好理解,将一个 hashset 下推到数据扫描节点。
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] carlvinhust2012 commented on a diff in pull request #9778: [Bug] sql block rule npe

2022-05-25 Thread GitBox


carlvinhust2012 commented on code in PR #9778:
URL: https://github.com/apache/incubator-doris/pull/9778#discussion_r882261551


##
regression-test/suites/sql_block_rule/test_sql_block_rule.groovy:
##
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_sql_block_rule", "sql_block_rule") {
+sql """
+USE regression_test
+  """
+
+sql """
+CREATE SQL_BLOCK_RULE test_rule_sql
+PROPERTIES("sql"="SELECT * FROM table_2", "global"= 
"true", "enable"= "true")
+  """
+
+test {
+sql "SELECT * FROM table_2"
+exception "sql match regex sql block rule: test_rule_sql"
+}
+
+sql """
+DROP SQL_BLOCK_RULE test_rule_sql
+  """
+
+sql """
+CREATE SQL_BLOCK_RULE test_rule_num
+PROPERTIES("tablet_num"="1", "global"= "true", "enable"= 
"true")
+  """
+
+test {
+sql "SELECT * FROM table_2"
+exception "sql hits sql block rule: test_rule_num, reach tablet_num : 
1"
+}
+
+qt_select """
+SHOW SQL_BLOCK_RULE
+  """
+
+sql """
+DROP SQL_BLOCK_RULE test_rule_num
+  """
+

Review Comment:
   Add a normal case maybe better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] carlvinhust2012 commented on a diff in pull request #9778: [Bug] sql block rule npe

2022-05-25 Thread GitBox


carlvinhust2012 commented on code in PR #9778:
URL: https://github.com/apache/incubator-doris/pull/9778#discussion_r882262763


##
fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRule.java:
##
@@ -33,12 +34,13 @@
 import java.util.List;
 import java.util.regex.Pattern;
 
+/**
+ * Use for block some sql by rule.
+ **/
 public class SqlBlockRule implements Writable {
 
 public static final String NAME_TYPE = "SQL BLOCK RULE NAME";
 
-public static final String DEFAULT_USER = "default";

Review Comment:
   why delete this line?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] morrySnow opened a new pull request, #9783: [enhancement] (community) update IntelliJ IDEA style file

2022-05-25 Thread GitBox


morrySnow opened a new pull request, #9783:
URL: https://github.com/apache/incubator-doris/pull/9783

   # Proposed changes
   
   update three warp rules to adapt check style.
   
   ## Problem Summary:
   
   current IntelliJ IDEA style file could not move TERNARY_OPERATION_SIGNS, 
PLACE_ASSIGNMENT_SIGN and ASSERT_STATEMENT_COLON to next line automatically.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: No
   2. Has unit tests been added: No Need
   3. Has document been added or modified: No Need
   4. Does it need to update dependencies: No
   5. Are there any changes that cannot be rolled back: No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] stalary commented on a diff in pull request #9778: [Bug] sql block rule npe

2022-05-25 Thread GitBox


stalary commented on code in PR #9778:
URL: https://github.com/apache/incubator-doris/pull/9778#discussion_r882263977


##
fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRule.java:
##
@@ -33,12 +34,13 @@
 import java.util.List;
 import java.util.regex.Pattern;
 
+/**
+ * Use for block some sql by rule.
+ **/
 public class SqlBlockRule implements Writable {
 
 public static final String NAME_TYPE = "SQL BLOCK RULE NAME";
 
-public static final String DEFAULT_USER = "default";

Review Comment:
   DEFAULT_USER is used only when the first designed, and then defined with 
global, it is no longer needed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] stalary commented on a diff in pull request #9778: [Bug] sql block rule npe

2022-05-25 Thread GitBox


stalary commented on code in PR #9778:
URL: https://github.com/apache/incubator-doris/pull/9778#discussion_r882264423


##
regression-test/suites/sql_block_rule/test_sql_block_rule.groovy:
##
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_sql_block_rule", "sql_block_rule") {
+sql """
+USE regression_test
+  """
+
+sql """
+CREATE SQL_BLOCK_RULE test_rule_sql
+PROPERTIES("sql"="SELECT * FROM table_2", "global"= 
"true", "enable"= "true")
+  """
+
+test {
+sql "SELECT * FROM table_2"
+exception "sql match regex sql block rule: test_rule_sql"
+}
+
+sql """
+DROP SQL_BLOCK_RULE test_rule_sql
+  """
+
+sql """
+CREATE SQL_BLOCK_RULE test_rule_num
+PROPERTIES("tablet_num"="1", "global"= "true", "enable"= 
"true")
+  """
+
+test {
+sql "SELECT * FROM table_2"
+exception "sql hits sql block rule: test_rule_num, reach tablet_num : 
1"
+}
+
+qt_select """
+SHOW SQL_BLOCK_RULE
+  """
+
+sql """
+DROP SQL_BLOCK_RULE test_rule_num
+  """
+

Review Comment:
   Ok, I'm going to add it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] adonis0147 opened a new issue, #9784: [Enhancement][doc] Add a note about the version of checkstyle

2022-05-25 Thread GitBox


adonis0147 opened a new issue, #9784:
URL: https://github.com/apache/incubator-doris/issues/9784

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Description
   
   
[UnusedLocalVariable](https://github.com/apache/incubator-doris/blob/32a210f426f9422a971a1e0157a349c6f88c9dcd/fe/check/checkstyle/checkstyle.xml#L167)
 check was introduced by [checkstyle 
9.3](https://checkstyle.sourceforge.io/config_coding.html#UnusedLocalVariable), 
we should remind users of the version.
   
   ### Solution
   
   Add a note about the version of checkstyle.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] adonis0147 opened a new pull request, #9785: [doc] Add a note about the version of checkstyle

2022-05-25 Thread GitBox


adonis0147 opened a new pull request, #9785:
URL: https://github.com/apache/incubator-doris/pull/9785

   # Proposed changes
   
   Issue Number: close #9784 
   
   ## Problem Summary:
   
   Please refer to #9784 
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: No
   2. Has unit tests been added: No Need
   3. Has document been added or modified: Yes
   4. Does it need to update dependencies: No
   5. Are there any changes that cannot be rolled back: No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] adonis0147 commented on pull request #9785: [doc] Add a note about the version of checkstyle

2022-05-25 Thread GitBox


adonis0147 commented on PR #9785:
URL: https://github.com/apache/incubator-doris/pull/9785#issuecomment-1138123079

   CC @morrySnow 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] morningman commented on a diff in pull request #9764: [style] (fe) wrap and whitespace rules

2022-05-25 Thread GitBox


morningman commented on code in PR #9764:
URL: https://github.com/apache/incubator-doris/pull/9764#discussion_r882264236


##
fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java:
##
@@ -134,11 +134,21 @@ public Operator converse() {
 }
 }
 
-public boolean isEquivalence() { return this == EQ || this == 
EQ_FOR_NULL; };
+public boolean isEquivalence() {
+return this == EQ || this == EQ_FOR_NULL;
+}
+
+;

Review Comment:
   remove



##
fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java:
##
@@ -134,11 +134,21 @@ public Operator converse() {
 }
 }
 
-public boolean isEquivalence() { return this == EQ || this == 
EQ_FOR_NULL; };
+public boolean isEquivalence() {
+return this == EQ || this == EQ_FOR_NULL;
+}
+
+;
+
+public boolean isUnNullSafeEquivalence() {
+return this == EQ;
+}
 
-public boolean isUnNullSafeEquivalence() { return this == EQ; };
+;

Review Comment:
   remove



##
fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java:
##
@@ -134,11 +134,21 @@ public Operator converse() {
 }
 }
 
-public boolean isEquivalence() { return this == EQ || this == 
EQ_FOR_NULL; };
+public boolean isEquivalence() {
+return this == EQ || this == EQ_FOR_NULL;
+}
+
+;

Review Comment:
   remove



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] pengxiangyu opened a new pull request, #9786: Change policy

2022-05-25 Thread GitBox


pengxiangyu opened a new pull request, #9786:
URL: https://github.com/apache/incubator-doris/pull/9786

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] adonis0147 commented on pull request #9708: [refactor][be] Optimize headers

2022-05-25 Thread GitBox


adonis0147 commented on PR #9708:
URL: https://github.com/apache/incubator-doris/pull/9708#issuecomment-1138142482

   Hi @morningman , I have resolved all conflicts. PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9734: [Bug] [Vectorized] add padding when load char type data

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9734:
URL: https://github.com/apache/incubator-doris/pull/9734#issuecomment-1138156724

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9734: [Bug] [Vectorized] add padding when load char type data

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9734:
URL: https://github.com/apache/incubator-doris/pull/9734#issuecomment-1138156751

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] hf200012 closed pull request #9697: [fix]Documentation fixes

2022-05-25 Thread GitBox


hf200012 closed pull request #9697: [fix]Documentation fixes
URL: https://github.com/apache/incubator-doris/pull/9697


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] hf200012 opened a new pull request, #9787: [fix]Documentation fixes

2022-05-25 Thread GitBox


hf200012 opened a new pull request, #9787:
URL: https://github.com/apache/incubator-doris/pull/9787

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9787: [fix]Documentation fixes

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9787:
URL: https://github.com/apache/incubator-doris/pull/9787#issuecomment-1138169415

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9787: [fix]Documentation fixes

2022-05-25 Thread GitBox


github-actions[bot] commented on PR #9787:
URL: https://github.com/apache/incubator-doris/pull/9787#issuecomment-1138169396

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] Vikasss7663 commented on issue #9775: [Enhancement] Remove the redundant `this.`

2022-05-25 Thread GitBox


Vikasss7663 commented on issue #9775:
URL: 
https://github.com/apache/incubator-doris/issues/9775#issuecomment-1138177476

   Hey @jackwener 
   I am a beginner and never contributed to any open source.
   I want to work on this, can you help me with this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] Gabriel39 commented on pull request #9582: [refactor](decimalv3) optimize decimal performance and precision

2022-05-25 Thread GitBox


Gabriel39 commented on PR #9582:
URL: https://github.com/apache/incubator-doris/pull/9582#issuecomment-1138183867

   1. Add UTs in BE
   2. rebase master to resolve conflicts


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] jackwener commented on issue #9775: [Enhancement] Remove the redundant `this.`

2022-05-25 Thread GitBox


jackwener commented on issue #9775:
URL: 
https://github.com/apache/incubator-doris/issues/9775#issuecomment-1138183981

   > Hey @jackwener I am a beginner and never contributed to any open source. I 
want to work on this, can you help me with this issue.
   
   It's ok😀, let's me explain this issue.
   
   In Java, we can use `this.` to access `class member variables` like `if 
(this.name == jackwener)`. 
   
   But, in most scenario, it's redundant. just use in constructor like:
   
   ```java
   
   class Person {
   public string name;
   
   Person(name) {
   this.name = name;
   }
   
   boolean isJack() {
this.name == "jack";  // It's redundant;
   }
   }
   ```
   
   So, we can remove the redundant like `isJack()`.
   
   example in #8862 
`fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsJobManager.java`
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] jackwener commented on issue #9775: [Enhancement] Remove the redundant `this.`

2022-05-25 Thread GitBox


jackwener commented on issue #9775:
URL: 
https://github.com/apache/incubator-doris/issues/9775#issuecomment-1138184514

   You can find code in `fe/`.
   
   If there are redundant `this.`, you can remove them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] stalary commented on pull request #9786: Change policy

2022-05-25 Thread GitBox


stalary commented on PR #9786:
URL: https://github.com/apache/incubator-doris/pull/9786#issuecomment-1138189958

   Can you submit an issue describing what you are going to do?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] BiteTheDDDDt commented on issue #9596: [Bug] [Load-Vectorized] core dump on KeyCoder::encode_ascending

2022-05-25 Thread GitBox


BiteThet commented on issue #9596:
URL: 
https://github.com/apache/incubator-doris/issues/9596#issuecomment-1138201343

   fixed on https://github.com/apache/incubator-doris/pull/9734


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] englefly opened a new issue, #9789: [Bug] variance calculate error

2022-05-25 Thread GitBox


englefly opened a new issue, #9789:
URL: https://github.com/apache/incubator-doris/issues/9789

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Version
   
   Dev-1.0.1
   
   ### What's Wrong?
   
   for column A =[34, 78, 23]
   select variance(A) 
   Expect is: 564.7
   But real is: 564.66788
   
   
   
   ### What You Expected?
   
   for column A =[34, 78, 23]
   select variance(A) 
   Expect is: 564.7
   But real is: 564.66788
   
   ### How to Reproduce?
   
   _No response_
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] hf200012 opened a new pull request, #9790: [doc][fix]Sync job doc

2022-05-25 Thread GitBox


hf200012 opened a new pull request, #9790:
URL: https://github.com/apache/incubator-doris/pull/9790

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org