[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #6219: [Feature] Support SHOW DATA SKEW stmt

2021-07-14 Thread GitBox


EmmyMiao87 commented on a change in pull request #6219:
URL: https://github.com/apache/incubator-doris/pull/6219#discussion_r669347621



##
File path: 
docs/zh-CN/sql-reference/sql-statements/Administration/ADMIN-SHOW-DATA-SKEW.md
##
@@ -0,0 +1,50 @@
+---
+{
+"title": "ADMIN SHOW DATA SKEW",
+"language": "zh-CN"
+}
+---
+
+
+
+# ADMIN SHOW DATA SKEW
+## description
+
+该语句用于查看表或某个分区的数据倾斜情况。
+
+语法:
+
+ADMIN SHOW DATA SKEW FROM [db_name.]tbl_name [PARTITION (p1)];

Review comment:
   如果 partition 是必填项,那么不用加 []

##
File path: 
docs/zh-CN/sql-reference/sql-statements/Administration/ADMIN-SHOW-DATA-SKEW.md
##
@@ -0,0 +1,50 @@
+---
+{
+"title": "ADMIN SHOW DATA SKEW",
+"language": "zh-CN"
+}
+---
+
+
+
+# ADMIN SHOW DATA SKEW
+## description
+
+该语句用于查看表或某个分区的数据倾斜情况。
+
+语法:
+
+ADMIN SHOW DATA SKEW FROM [db_name.]tbl_name [PARTITION (p1)];
+
+说明:
+
+1. 必须指定且仅指定一个分区。对于非分区表,分区名称同表名。
+   2. 结果将展示指定分区下,各个分桶的数据量,以及每个分桶数据量在总数据量中的占比。

Review comment:
   注意格式

##
File path: fe/fe-core/src/main/cup/sql_parser.cup
##
@@ -262,6 +262,7 @@ terminal String KW_ADD, KW_ADMIN, KW_AFTER, KW_AGGREGATE, 
KW_ALL, KW_ALTER, KW_A
 KW_REPAIR, KW_REPEATABLE, KW_REPOSITORY, KW_REPOSITORIES, KW_REPLACE, 
KW_REPLACE_IF_NOT_NULL, KW_REPLICA, KW_RESOURCE, KW_RESOURCES, KW_RESTORE, 
KW_RETURNS, KW_RESUME, KW_REVOKE,
 KW_RIGHT, KW_ROLE, KW_ROLES, KW_ROLLBACK, KW_ROLLUP, KW_ROUTINE, KW_ROW, 
KW_ROWS,
 KW_S3, KW_SCHEMA, KW_SCHEMAS, KW_SECOND, KW_SELECT, KW_SEMI, 
KW_SERIALIZABLE, KW_SESSION, KW_SET, KW_SETS, KW_SET_VAR, KW_SHOW, KW_SIGNED,
+   KW_SKEW,

Review comment:
   Pay attention to the format

##
File path: 
fe/fe-core/src/main/java/org/apache/doris/analysis/AdminShowDataSkewStmt.java
##
@@ -0,0 +1,99 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.cluster.ClusterNamespace;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+
+// admin show data skew from tbl [partition(p1, p2, ...)]
+public class AdminShowDataSkewStmt extends ShowStmt {
+public static final ImmutableList TITLE_NAMES = new 
ImmutableList.Builder()
+.add("BucketIdx").add("AvgDataSize")
+.add("Graph").add("Percent")
+.build();
+
+private TableRef tblRef;
+
+public AdminShowDataSkewStmt(TableRef tblRef) {
+this.tblRef = tblRef;
+}
+
+@Override
+public void analyze(Analyzer analyzer) throws UserException {
+super.analyze(analyzer);
+
+// check auth
+if 
(!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), 
PrivPredicate.ADMIN)) {
+
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, 
"ADMIN");
+}
+
+String dbName = null;
+if (Strings.isNullOrEmpty(tblRef.getName().getDb())) {
+dbName = analyzer.getDefaultDb();
+if (Strings.isNullOrEmpty(dbName)) {
+ErrorReport.reportAnalysisException(ErrorCode.ERR_NO_DB_ERROR);
+}
+} else {
+dbName = ClusterNamespace.getFullName(getClusterName(), 
tblRef.getName().getDb());
+}
+
+tblRef.getName().setDb(dbName);

Review comment:
   analyze tblRef before check partition




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---

[GitHub] [incubator-doris] xinyiZzz commented on pull request #1792: Add ChunkAllocator to accelerate chunk allocation

2021-07-14 Thread GitBox


xinyiZzz commented on pull request #1792:
URL: https://github.com/apache/incubator-doris/pull/1792#issuecomment-879684670


   ChunkAllocator and BufferAllocator have many similarities. For example, each 
core has an area, and chunks (buffer) of the same size are in a list. When 
assigning, it will be first obtained from the free list of the area of ​​the 
current core, etc.
   
   Currently ChunkAllocator has some low utilization problems, such as
   1. Memory fragmentation. For example, when a large number of small-size 
chunks are applied at the beginning, these small chunks may no longer be used 
in the future, nor will they be merged or recycled, nor will large chunks be 
split.
   - BufferAllocator will release the current free buffer after the memory 
request reaches the upper limit.
   2. local_core_alloc_count: other_core_alloc_count = 1:541, that is, there is 
a high probability of obtaining chunks from areas of other cores, which is 
locked.
   - BufferAllocator will apply first from the system when there is no buffer 
in the current core area, and will get it from other core areas only when the 
system space is insufficient, so that the effect may be better after running 
for a period of time.
   3. There is no upper and lower limit on the size of the chunk;
   4. The 2G reserved space is easy to fill up when high concurrency, although 
this problem can be avoided by changing to the overall memory usage limit.
   - BufferAllocator is a limit to the overall memory usage
   5
   
   These problems can be solved by BufferAllocator to a certain extent, so why 
didn't you consider using BufferAllocator to take over all the memory 
management of Doris, but added a new ChunkAllocator to try to take over?
   
   Impala tries to use BufferPool (based on BufferAllocator) to support all 
runtime memory, instead of MemPools, FreePools, etc. Maybe this is a good choice
   https://issues.apache.org/jira/browse/IMPALA-3200


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] xinyiZzz edited a comment on pull request #1792: Add ChunkAllocator to accelerate chunk allocation

2021-07-14 Thread GitBox


xinyiZzz edited a comment on pull request #1792:
URL: https://github.com/apache/incubator-doris/pull/1792#issuecomment-879684670


   @imay 
   ChunkAllocator and BufferAllocator have many similarities. For example, each 
core has an area, and chunks (buffer) of the same size are in a list. When 
assigning, it will be first obtained from the free list of the area of ​​the 
current core, etc.
   
   Currently ChunkAllocator has some low utilization problems, such as
   1. Memory fragmentation. For example, when a large number of small-size 
chunks are applied at the beginning, these small chunks may no longer be used 
in the future, nor will they be merged or recycled, nor will large chunks be 
split.
   - BufferAllocator will release the current free buffer after the memory 
request reaches the upper limit.
   2. local_core_alloc_count: other_core_alloc_count = 1:541, that is, there is 
a high probability of obtaining chunks from areas of other cores, which is 
locked.
   - BufferAllocator will apply first from the system when there is no buffer 
in the current core area, and will get it from other core areas only when the 
system space is insufficient, so that the effect may be better after running 
for a period of time.
   3. There is no upper and lower limit on the size of the chunk;
   4. The 2G reserved space is easy to fill up when high concurrency, although 
this problem can be avoided by changing to the overall memory usage limit.
   - BufferAllocator is a limit to the overall memory usage
   5
   
   These problems can be solved by BufferAllocator to a certain extent, so why 
didn't you consider using BufferAllocator to take over all the memory 
management of Doris, but added a new ChunkAllocator to try to take over?
   
   Impala tries to use BufferPool (based on BufferAllocator) to support all 
runtime memory, instead of MemPools, FreePools, etc. Maybe this is a good choice
   https://issues.apache.org/jira/browse/IMPALA-3200


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] xinyiZzz edited a comment on pull request #1792: Add ChunkAllocator to accelerate chunk allocation

2021-07-14 Thread GitBox


xinyiZzz edited a comment on pull request #1792:
URL: https://github.com/apache/incubator-doris/pull/1792#issuecomment-879684670


   @imay 
   ChunkAllocator and BufferAllocator have many similarities. For example, each 
core has an area, and chunks (buffer) of the same size are in a list. When 
assigning, it will be first obtained from the free list of the area of ​​the 
current core, etc.
   
   Currently ChunkAllocator has some low utilization problems, such as
   1. Memory fragmentation. For example, when a large number of small-size 
chunks are applied at the beginning, these small chunks may no longer be used 
in the future, nor will they be merged or recycled, nor will large chunks be 
split.
   - BufferAllocator will release the current free buffer after the memory 
request reaches the upper limit.
   2. local_core_alloc_count: other_core_alloc_count = 1:541, that is, there is 
a high probability of obtaining chunks from areas of other cores, which is 
locked.
   - BufferAllocator will apply first from the system when there is no buffer 
in the current core area, and will get it from other core areas only when the 
system space is insufficient, so that the effect may be better after running 
for a period of time.
   3. There is no upper and lower limit on the size of the chunk;
   4. The 2G reserved space is easy to fill up when high concurrency, although 
this problem can be avoided by changing to the overall memory usage limit.
   - BufferAllocator is a limit to the overall memory usage
   5. 
   
   These problems can be solved by BufferAllocator to a certain extent, so why 
didn't you consider using BufferAllocator to take over all the memory 
management of Doris, but added a new ChunkAllocator to try to take over?
   
   Impala tries to use BufferPool (based on BufferAllocator) to support all 
runtime memory, instead of MemPools, FreePools, etc. Maybe this is a good choice
   https://issues.apache.org/jira/browse/IMPALA-3200


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] xinyiZzz edited a comment on pull request #1792: Add ChunkAllocator to accelerate chunk allocation

2021-07-14 Thread GitBox


xinyiZzz edited a comment on pull request #1792:
URL: https://github.com/apache/incubator-doris/pull/1792#issuecomment-879684670


   @imay 
   At present, there are both ChunkAllocator and BufferAllocator in Doris. They 
have many similarities. For example, each core has an area, and chunks (buffer) 
of the same size are in a list. When assigning, it will be first obtained from 
the free list of the current core area. etc.
   
   Currently ChunkAllocator has some low utilization problems, such as
   1. Memory fragmentation. For example, when a large number of small-size 
chunks are applied at the beginning, these small chunks may no longer be used 
in the future, nor will they be merged or recycled, nor will large chunks be 
split.
   - BufferAllocator will release the current free buffer after the memory 
request reaches the upper limit.
   2. local_core_alloc_count: other_core_alloc_count = 1:541, that is, there is 
a high probability of obtaining chunks from areas of other cores, which is 
locked.
   - BufferAllocator will apply first from the system when there is no buffer 
in the current core area, and will get it from other core areas only when the 
system space is insufficient, so that the effect may be better after running 
for a period of time.
   3. There is no upper and lower limit on the size of the chunk;
   4. The 2G reserved space is easy to fill up when high concurrency, although 
this problem can be avoided by changing to the overall memory usage limit.
   - BufferAllocator is a limit to the overall memory usage
   5. 
   
   These problems can be solved by BufferAllocator to a certain extent, so why 
didn't you consider using BufferAllocator to take over all the memory 
management of Doris, but added a new ChunkAllocator to try to take over?
   
   Impala tries to use BufferPool (based on BufferAllocator) to support all 
runtime memory, instead of MemPools, FreePools, etc. Maybe this is a good choice
   https://issues.apache.org/jira/browse/IMPALA-3200


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] tjlygdx opened a new issue #6234: CAST 转 decimal 没有实现预期的精度

2021-07-14 Thread GitBox


tjlygdx opened a new issue #6234:
URL: https://github.com/apache/incubator-doris/issues/6234


   mysql> select cast(10.1112 as decimal(10,2)) ;
   ++
   | CAST(10.1112 AS DECIMAL(10,2)) |
   ++
   |   10.11120 |
   ++
   1 row in set (0.05 sec)
   
   CAST 转 decimal 没有生效,没有实现预期的精度


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg opened a new issue #6235: The query will slow down when there is data skew

2021-07-14 Thread GitBox


yangzhg opened a new issue #6235:
URL: https://github.com/apache/incubator-doris/issues/6235


   When be divides the scanner, according to the method of one scanner for each 
tablet, when the data is skewed, and when one tablet is much larger than other 
tablets, it will appear that a scan node is far behind other nodes.
   Therefore, for the DUPLICATE table, the division method can be converted to 
division according to segment files, and each scanner reads files of the same 
size


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg opened a new pull request #6236: Optimized the reading performance of Duplicate tables with severe data skew

2021-07-14 Thread GitBox


yangzhg opened a new pull request #6236:
URL: https://github.com/apache/incubator-doris/pull/6236


   ## Proposed changes
   
   When dividing the scanner, there is no longer a scanner for each
   tablet, but according to the file size of the tablet, the tablet is
   divided into multiple scanners for reading. By default, each scanner
   reads 1GB. If the data is stored on a SATA disk, this value can be
   increased, and if it is an SSD disk, the value can be decreased.
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [ ] Code refactor (Modify the code structure, format the code, etc...)
   - [x] Optimization. Including functional usability improvements and 
performance improvements.
   - [ ] Dependency. Such as changes related to third-party components.
   - [ ] Other.
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [x] I have created an issue on (Fix #skew) and described the bug/feature 
there in detail
   - [ ] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
d...@doris.apache.org by explaining why you chose the solution you did and what 
alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] huangmengbin commented on a change in pull request #6099: [Enhance] improve performance of init_scan_key by sharing the schema

2021-07-14 Thread GitBox


huangmengbin commented on a change in pull request #6099:
URL: https://github.com/apache/incubator-doris/pull/6099#discussion_r669419768



##
File path: be/src/olap/reader.cpp
##
@@ -548,14 +549,36 @@ OLAPStatus Reader::_init_keys_param(const ReaderParams& 
read_params) {
 
 size_t start_key_size = read_params.start_key.size();
 _keys_param.start_keys.resize(start_key_size, nullptr);
+
+size_t scan_key_size = read_params.start_key.front().size();
+if (scan_key_size > _tablet->tablet_schema().num_columns()) {
+LOG(WARNING)
+<< "Input param are invalid. Column count is bigger than 
num_columns of schema. "
+<< "column_count=" << scan_key_size
+<< ", schema.num_columns=" << 
_tablet->tablet_schema().num_columns();
+return OLAP_ERR_INPUT_PARAMETER_ERROR;
+}
+
+std::vector columns;
+for (size_t i = 0; i < scan_key_size; ++i) {
+columns.push_back(i);
+}

Review comment:
   - Done. 
   - I used the `std::iota` to replace the for-loop.
   - Do I need to extra the "if (scan_key_size > 
_tablet->tablet_schema().num_columns()) { LOG(WARNING)" to a function, and 
wrap it in the macro "RETURN_NOT_OK()" ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] hf200012 closed issue #5371: Flink Doris Connector

2021-07-14 Thread GitBox


hf200012 closed issue #5371:
URL: https://github.com/apache/incubator-doris/issues/5371


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] hf200012 closed issue #3849: Table names are not case sensitive

2021-07-14 Thread GitBox


hf200012 closed issue #3849:
URL: https://github.com/apache/incubator-doris/issues/3849


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] huangmengbin commented on a change in pull request #6099: [Enhance] improve performance of init_scan_key by sharing the schema

2021-07-14 Thread GitBox


huangmengbin commented on a change in pull request #6099:
URL: https://github.com/apache/incubator-doris/pull/6099#discussion_r669421819



##
File path: be/src/olap/reader.cpp
##
@@ -548,14 +549,36 @@ OLAPStatus Reader::_init_keys_param(const ReaderParams& 
read_params) {
 
 size_t start_key_size = read_params.start_key.size();
 _keys_param.start_keys.resize(start_key_size, nullptr);
+
+size_t scan_key_size = read_params.start_key.front().size();
+if (scan_key_size > _tablet->tablet_schema().num_columns()) {
+LOG(WARNING)
+<< "Input param are invalid. Column count is bigger than 
num_columns of schema. "
+<< "column_count=" << scan_key_size
+<< ", schema.num_columns=" << 
_tablet->tablet_schema().num_columns();
+return OLAP_ERR_INPUT_PARAMETER_ERROR;
+}
+
+std::vector columns;
+for (size_t i = 0; i < scan_key_size; ++i) {
+columns.push_back(i);
+}

Review comment:
   > it is better to extract those to a function, this is duplicted with 
line 158
   
   - Done.
   - I used the std::iota to replace the for-loop.
   - Do I need to extra the "if (scan_key_size > 
_tablet->tablet_schema().num_columns()) { LOG(WARNING)" to a function, and 
wrap it in the macro "RETURN_NOT_OK()" ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] huangmengbin commented on a change in pull request #6099: [Enhance] improve performance of init_scan_key by sharing the schema

2021-07-14 Thread GitBox


huangmengbin commented on a change in pull request #6099:
URL: https://github.com/apache/incubator-doris/pull/6099#discussion_r669421819



##
File path: be/src/olap/reader.cpp
##
@@ -548,14 +549,36 @@ OLAPStatus Reader::_init_keys_param(const ReaderParams& 
read_params) {
 
 size_t start_key_size = read_params.start_key.size();
 _keys_param.start_keys.resize(start_key_size, nullptr);
+
+size_t scan_key_size = read_params.start_key.front().size();
+if (scan_key_size > _tablet->tablet_schema().num_columns()) {
+LOG(WARNING)
+<< "Input param are invalid. Column count is bigger than 
num_columns of schema. "
+<< "column_count=" << scan_key_size
+<< ", schema.num_columns=" << 
_tablet->tablet_schema().num_columns();
+return OLAP_ERR_INPUT_PARAMETER_ERROR;
+}
+
+std::vector columns;
+for (size_t i = 0; i < scan_key_size; ++i) {
+columns.push_back(i);
+}

Review comment:
   > it is better to extract those to a function, this is duplicted with 
line 158
   
   - Done.
   - I used the `std::iota` to replace the for-loop.
   - Do I need to extra the "if (scan_key_size > 
_tablet->tablet_schema().num_columns()) { LOG(WARNING)" to a function, and 
wrap it in the macro `RETURN_NOT_OK()` ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] morningman opened a new pull request #6237: [Feature] Support setting concurrency for thread pool token

2021-07-14 Thread GitBox


morningman opened a new pull request #6237:
URL: https://github.com/apache/incubator-doris/pull/6237


   ## Proposed changes
   
   Now we can submit a group of tasks using thread pool token, and limit
   the max concurrency of this task group
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [x] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [ ] Code refactor (Modify the code structure, format the code, etc...)
   - [ ] Optimization. Including functional usability improvements and 
performance improvements.
   - [ ] Dependency. Such as changes related to third-party components.
   - [ ] Other.
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature 
there in detail
   - [ ] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
d...@doris.apache.org by explaining why you chose the solution you did and what 
alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #6209: [BDBJE] Add a tool to view the data in BEBJE

2021-07-14 Thread GitBox


EmmyMiao87 commented on a change in pull request #6209:
URL: https://github.com/apache/incubator-doris/pull/6209#discussion_r669509134



##
File path: 
fe/fe-core/src/main/java/org/apache/doris/http/action/SystemAction.java
##
@@ -17,10 +17,12 @@
 
 package org.apache.doris.http.action;
 
-import org.apache.commons.validator.routines.UrlValidator;
+import io.netty.handler.codec.http.HttpMethod;

Review comment:
   import order




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #6210: [HttpV2] Add more httpv2 APIs

2021-07-14 Thread GitBox


EmmyMiao87 commented on a change in pull request #6210:
URL: https://github.com/apache/incubator-doris/pull/6210#discussion_r669509780



##
File path: 
fe/fe-core/src/main/java/org/apache/doris/httpv2/restv2/ImportAction.java
##
@@ -0,0 +1,228 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.httpv2.restv2;
+
+import lombok.Getter;

Review comment:
   import order




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #6210: [HttpV2] Add more httpv2 APIs

2021-07-14 Thread GitBox


EmmyMiao87 commented on a change in pull request #6210:
URL: https://github.com/apache/incubator-doris/pull/6210#discussion_r669510237



##
File path: 
fe/fe-core/src/test/java/org/apache/doris/common/util/BrokerUtilTest.java
##
@@ -17,8 +17,11 @@
 
 package org.apache.doris.common.util;
 
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.fail;
+import mockit.Expectations;
+import mockit.Injectable;

Review comment:
   same as above




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #6212: [Config] set spark load and odbc table feature enable by default

2021-07-14 Thread GitBox


EmmyMiao87 commented on a change in pull request #6212:
URL: https://github.com/apache/incubator-doris/pull/6212#discussion_r669511610



##
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/BrokerDesc.java
##
@@ -23,11 +23,11 @@
 import org.apache.doris.common.util.PrintableMap;
 import org.apache.doris.thrift.TFileType;
 
-import com.google.common.collect.Maps;
-
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 
+import com.google.common.collect.Maps;

Review comment:
   import order。。




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] EmmyMiao87 commented on pull request #6215: [Bug][RoutineLoad] Fix bug that routine load thread on BE may be blocked

2021-07-14 Thread GitBox


EmmyMiao87 commented on pull request #6215:
URL: https://github.com/apache/incubator-doris/pull/6215#issuecomment-879803065


   Please fix ut


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6237: [Feature] Support setting concurrency for thread pool token

2021-07-14 Thread GitBox


yangzhg commented on a change in pull request #6237:
URL: https://github.com/apache/incubator-doris/pull/6237#discussion_r669527260



##
File path: be/src/util/threadpool.h
##
@@ -362,6 +362,13 @@ class ThreadPoolToken {
 // Returns true if all submissions are complete, false otherwise.
 bool wait_for(const MonoDelta& delta);
 
+bool need_dispatch();

Review comment:
   need_dispatch is a bit weird. Idle may be better or have other better 
names




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6237: [Feature] Support setting concurrency for thread pool token

2021-07-14 Thread GitBox


yangzhg commented on a change in pull request #6237:
URL: https://github.com/apache/incubator-doris/pull/6237#discussion_r669527260



##
File path: be/src/util/threadpool.h
##
@@ -362,6 +362,13 @@ class ThreadPoolToken {
 // Returns true if all submissions are complete, false otherwise.
 bool wait_for(const MonoDelta& delta);
 
+bool need_dispatch();

Review comment:
   `need_dispatch` is a bit weird. `idle` may be better or have other 
better names




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] HappenLee opened a new issue #6238: [Proposal] Vectorization Execution Engine optimization for Doris

2021-07-14 Thread GitBox


HappenLee opened a new issue #6238:
URL: https://github.com/apache/incubator-doris/issues/6238


   ## Motivation
   At present, the underlying storage in Doris is column storage. But query 
execution needs to be transferred to the query layer for execution by 
row-to-column first. Such an implementation maybe cause the performance problem。
   
   * 1. Column-to-row loss.
   * 2. Can not get better CPU performance without vectorized execution.
   
   Currently, vectorized execution has been commonly adopted in mainstream MPP 
databases, which has a significant effect on improving CPU utilization. In this 
paper, we investigate the knowledge of vectorization and give a detailed design 
for implementing vectorization on columnar storage.
   
   ## What is vectorization execution
   
   Organizing data in this batched, columnar fashion is the primary 
prerequisite for using SIMD CPU instructions, which operate on a vector of data 
at a time. Using SIMD instructions is an eventual goal. 
   
   Now suppose there is a table **People**, the data content is as follows:
   |Id|Name|Age|
   |:: |:: |:: |
   |101|Ivan|22|
   115|Peggy|37
   114|Victor|45
   113|Eve|25
   112|Walter|19
   109|Trudy|31
   108|Bob|27
   105|Zoe|29
   104|Charlie|42
   102|Alice|35
   
   Execute the following query on the table:`SELECT Id, Name, Age, (Age - 30) * 
50 AS Bonus FROM People WHERE Age > 30`
   
    Volcano model
   
![image.png](https://upload-images.jianshu.io/upload_images/8552201-675e7939d434aaab.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)
   
   For traditional relational databases, the way it returns data is on a 
per-row basis. The problems are: one function call per line, which interrupts 
CPU flow and is not conducive to branch prediction; high instruction and data 
cache misses; unfriendly compiler, not conducive to loop expansion, and not 
conducive to using CPU acceleration instructions such as SIMD.
   
    Vectorization Execution Engine
   
   
![image.png](https://upload-images.jianshu.io/upload_images/8552201-74329d0444b6d2f9.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)
   
   But for columnar databases, you can actually do it in the form of columns, 
which can reduce the overall cache failure rate. This is equivalent to a 
vectorized execution to achieve such an effect, which is a unique optimization 
technique for columnar databases to reduce CPU consumption and improve the 
overall CPU utilization.
   
   # It can bring us the following benefits:
   * 1. Reduced dummy function calls for branch prediction. 
   
   * 2. Reduces interpretation overhead by getting a batch of results at a time.
   
   * 3. Easy for compiler to do loop pipelining and SIMD optimization.
   
   * 4. Friendly to CPU L1 and L2 Cache.
   
   ## How Vectorization Execution Engine For Doris
   
   ### The key idea and challenge
   
   * Design a new memory structure to replace the original `RowBatch` and 
`Tuple` structure
   * Rewrite all operators to support vectorization/columnar computation
   
   **ClickHouse is an excellent implementation of the vectorized execution 
engine database, so here we have borrowed a lot from its excellent 
implementation in terms of data structure and function implementation. We are 
based on ClickHouse v19.16.2.2 and would like to thank the ClickHouse community 
and developers.**
   
   ### 1. Data Structure
   
   We have implemented all types natively supported by Doris based on 
**Clickhouse's implementation of Column and Block** including:
   
   * tinyint
   * smallint
   * int
   * bigint
   * largeint
   * boolean
   * float
   * double
   * decimal
   * date
   * datetime
   * HLL
   * bitmap
   
   ### 2. Function and AggregateFunction
   
   We have implemented more than 80% of Doris's native support functions based 
on Clickhouse's implementation of Function and AggregateFunction interfaces, 
and have completed a large number of SIMD support.
   
   And for a more efficient SIMD optimization, the type of NULLABLE for each 
function is identified as follows.
   
   
![image.png](https://upload-images.jianshu.io/upload_images/8552201-1535c8756d14b447.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)
   
   The specific function implementation support can be found in the following 
link:[HERE](https://github.com/doris-vectorized/doris-vectorized/issues/49)
   
   ### 3. Operator And Execution System Integration
   
   # Operator
   
   We already implement 
   
   * VSortNode
   * VCrossJoinNode
   * VAggregateNode
   * VOlapScanNode
   
   After the data is read from the storage layer, it is transformed from 
`RowCursor` to `Block` structures, i.e., from row storage to column storage. 
Starting from **VOlapScanNode**, the data is organized in the form of columns.
   
   
![image.png](https://upload-images.jianshu.io/upload_images/8552201-d90cfbfcaae32426.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)
   
   
   # Exe

[GitHub] [incubator-doris] morningman opened a new pull request #6239: [Community] Add an github action to auto add 'approved' label

2021-07-14 Thread GitBox


morningman opened a new pull request #6239:
URL: https://github.com/apache/incubator-doris/pull/6239


   ## Proposed changes
   
   1. If any committer APPROVE a PR, the label 'approved' will be added.
   2. If any other reviewed APPROVE a PR, the label 'reviewed' will be added.
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [ ] Code refactor (Modify the code structure, format the code, etc...)
   - [ ] Optimization. Including functional usability improvements and 
performance improvements.
   - [ ] Dependency. Such as changes related to third-party components.
   - [x] Other.
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature 
there in detail
   - [ ] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
d...@doris.apache.org by explaining why you chose the solution you did and what 
alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] huangmengbin edited a comment on pull request #5369: [Load Parallel][3/3] Support parallel delta writer

2021-07-14 Thread GitBox


huangmengbin edited a comment on pull request #5369:
URL: https://github.com/apache/incubator-doris/pull/5369#issuecomment-879117362


   > ## Proposed changes
   > In the previous broker load, multiple OlapTableSinks would send data to 
the same LoadChannel,
   > and because of the lock granularity problem, LoadChannel could only 
process these requests serially,
   > which made it impossible to make full use of cluster resources.
   > 
   > This CL modifies the related locks so that LoadChannel can process these 
requests in parallel.
   > 
   > In the test, with a size of 20G, the load speed of 334 million rows of 
data in 3 nodes has been
   > increased from 9min to 5min, and after enabling 2 concurrency, it can be 
increased to 3min.
   > 
   > Also modify the profile of load job.
   > 
   > ## Types of changes
   > * [x]  Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   > 
   > ## Checklist
   > * [x]  I have created an issue on (Fix [[Performance] Import the load 
performace #5281](https://github.com/apache/incubator-doris/issues/5281) ) and 
described the bug/feature there in detail
   
   Hi, morningman !
   
请问一下可以更详细地描述一下当时测试时的信息吗?比如列的数量、哪种数据模型(Uniq、Aggregate、Duplicate),等等?我在uniq模型上进行测试(9G数据、9台机器、1.3亿行、246列)
 没有发现其有较明显的提速(13.5min->10min),不知道是否符合代码的预期。(暂时还没找到原因
   Thanks !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] huangmengbin commented on a change in pull request #6099: [Enhance] improve performance of init_scan_key by sharing the schema

2021-07-14 Thread GitBox


huangmengbin commented on a change in pull request #6099:
URL: https://github.com/apache/incubator-doris/pull/6099#discussion_r669421819



##
File path: be/src/olap/reader.cpp
##
@@ -548,14 +549,36 @@ OLAPStatus Reader::_init_keys_param(const ReaderParams& 
read_params) {
 
 size_t start_key_size = read_params.start_key.size();
 _keys_param.start_keys.resize(start_key_size, nullptr);
+
+size_t scan_key_size = read_params.start_key.front().size();
+if (scan_key_size > _tablet->tablet_schema().num_columns()) {
+LOG(WARNING)
+<< "Input param are invalid. Column count is bigger than 
num_columns of schema. "
+<< "column_count=" << scan_key_size
+<< ", schema.num_columns=" << 
_tablet->tablet_schema().num_columns();
+return OLAP_ERR_INPUT_PARAMETER_ERROR;
+}
+
+std::vector columns;
+for (size_t i = 0; i < scan_key_size; ++i) {
+columns.push_back(i);
+}

Review comment:
   > it is better to extract those to a function, this is duplicted with 
line 158
   
   - Done.
   - I used the `std::iota` to replace the for-loop.
   - Do I need to extract the "if (scan_key_size > 
_tablet->tablet_schema().num_columns()) { LOG(WARNING)" to a function, and 
wrap it in the macro `RETURN_NOT_OK()` ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] EmmyMiao87 merged pull request #6239: [Community] Add an github action to auto add 'approved' label

2021-07-14 Thread GitBox


EmmyMiao87 merged pull request #6239:
URL: https://github.com/apache/incubator-doris/pull/6239


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [Community] Add an github action to auto add 'approved' label (#6239)

2021-07-14 Thread lingmiao
This is an automated email from the ASF dual-hosted git repository.

lingmiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new e905dd8  [Community] Add an github action to auto add 'approved' label 
(#6239)
e905dd8 is described below

commit e905dd84c8d6066173ae9cdf071095ea839f1e56
Author: Mingyu Chen 
AuthorDate: Wed Jul 14 21:12:42 2021 +0800

[Community] Add an github action to auto add 'approved' label (#6239)

1. If any committer APPROVE a PR, the label 'approved' will be added.
2. If any other reviewed APPROVE a PR, the label 'reviewed' will be added.
---
 .github/workflows/approve-label.yml | 29 +
 1 file changed, 29 insertions(+)

diff --git a/.github/workflows/approve-label.yml 
b/.github/workflows/approve-label.yml
new file mode 100644
index 000..dfde7db
--- /dev/null
+++ b/.github/workflows/approve-label.yml
@@ -0,0 +1,29 @@
+name: Label when approved
+on: pull_request_review
+
+jobs:
+
+  label-when-approved:
+name: "Label when approved"
+runs-on: ubuntu-latest
+outputs:
+  isApprovedByCommiters: ${{ 
steps.label-when-approved-by-commiters.outputs.isApproved }}
+  isApprovedByAnyone: ${{ 
steps.label-when-approved-by-anyone.outputs.isApproved }}
+steps:
+  - name: Label when approved by commiters
+uses: TobKed/label-when-approved-action@v1.3
+id: label-when-approved-by-commiters
+with:
+  token: ${{ secrets.GITHUB_TOKEN }}
+  label: 'approved'
+  require_committers_approval: 'true'
+  remove_label_when_approval_missing: 'true'
+  comment: 'PR approved by at least one committer and no changes 
requested.'
+  - name: Label when approved by anyone
+uses: TobKed/label-when-approved-action@v1.3
+id: label-when-approved-by-anyone
+with:
+  token: ${{ secrets.GITHUB_TOKEN }}
+  label: 'reviewed'
+  comment: 'PR approved by anyone and no changes requested.'
+

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] xy720 opened a new issue #6240: [Feature] support canal-fe (create/resume/pause/stop/show sync job/idempotent)

2021-07-14 Thread GitBox


xy720 opened a new issue #6240:
URL: https://github.com/apache/incubator-doris/issues/6240


   working.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] boboyunz opened a new issue #6241: my Issue: no valid Basic authorization

2021-07-14 Thread GitBox


boboyunz opened a new issue #6241:
URL: https://github.com/apache/incubator-doris/issues/6241


   i want to add data with steam load ,but get this error **no valid Basic 
authorization**
   this is be log
   I0715 05:07:46.350615 17205 stream_load.cpp:186] new income streaming load 
request.id=7045db7eb458d83a-7180372f6e4996a3, job_id=-1, txn_id=-1, 
label=39c25a5c-7000-496e-a98e-348a264c81dd, db=QHYGJ, tbl=cargis2
   W0715 05:07:46.350641 17205 stream_load.cpp:212] parse basic authorization 
failed.id=7045db7eb458d83a-7180372f6e4996a3, job_id=-1, txn_id=-1, 
label=39c25a5c-7000-496e-a98e-348a264c81dd
   
   my request header:
   {Content-Type: ;charset=UTF-8
   User-Agent: Mozilla/4.0
   Content-Length: 175
   expect: 100-continue
   format: json
   strip_outer_array: true
   jsonpaths: 
["$.cph","$.cpys","$.dt","$.ptbm","$.alarm","$.sta","$.xslc","$.gc","$.sbsd","$.xssd","$.xslc","$.dr","$.lat","$.lon"]
   label: 39c25a5c-7000-496e-a98e-348a264c81dd
   Authorization: Basic cm9vdDoxcWF6IVFBWg==
   
   }
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] boboyunz commented on issue #6241: my Issue: no valid Basic authorization

2021-07-14 Thread GitBox


boboyunz commented on issue #6241:
URL: 
https://github.com/apache/incubator-doris/issues/6241#issuecomment-880267833


   最后我的解决方式如下,供大家参考:
   根据官方文档,fe收到stream load 请求后,只是做了负载均衡,随机选择一个BE,转发http请求,问题就出在转发请求会把Basic 
authorization丢掉,导致访问失败。所以我直接跳过BE,自己维护一个BE列表,随机选择一个来发送请求,这样就把问题解决了。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] boboyunz edited a comment on issue #6241: my Issue: no valid Basic authorization

2021-07-14 Thread GitBox


boboyunz edited a comment on issue #6241:
URL: 
https://github.com/apache/incubator-doris/issues/6241#issuecomment-880267833


   最后我的解决方式如下,供大家参考:
   根据官方文档,fe收到stream load 请求后,只是做了负载均衡,随机选择一个BE,转发http请求,问题就出在转发请求会把Basic 
authorization丢掉,导致访问失败。所以我直接跳过FE,自己维护一个BE列表,随机选择一个来发送请求,这样就把问题解决了。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg opened a new issue #6242: Support more feature of array types

2021-07-14 Thread GitBox


yangzhg opened a new issue #6242:
URL: https://github.com/apache/incubator-doris/issues/6242


   Currently, we have supported ARRAY type of syntax analysis, storage and 
query, but if the array can be really used, it needs more work.
   
   ## Important Feature
   
   Assuming that there is a table test, the data is as follows
   
   ```
   +--++
   | name | arr|
   +--++
   | abc  | [1,  2,  3]|
   +--++
   1 row in set (0.01 sec)
   ```
   
   - [ ] Subscript access: `select arr[0] from test` returns `1`
   
   - [ ] array join: `SELECT name, arr from test ARRAY JOIN arr`
   

   ```
   +--+-+
   | name | arr |
   +--+-+
   | abc  | 1   |
   | abc  | 2   |
   | abc  | 3   |
   +--+-+
   3 rows in set (0.01 sec)
   ```
   
   - [ ] Efficient contains function `select contains(arr, 1) from test`
   
   - [ ] size function `select size(arr) from test`
   
   - [ ] broker load, stream load, routine load support array type import
   
   - [ ] Support importing parquet/orc file array type
   
   ## Other Feature
   
   - [ ] Support nested array
   - [ ] Support array as value column of AGG/unique table, support 
repalce/append aggregation function


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #6211: [s3][bug] Remove log4j1

2021-07-14 Thread GitBox


yangzhg merged pull request #6211:
URL: https://github.com/apache/incubator-doris/pull/6211


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [s3][bug] Remove log4j1 (#6211)

2021-07-14 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 39945ab  [s3][bug] Remove log4j1 (#6211)
39945ab is described below

commit 39945aba1e1c76a1bc7a63490eea7f4ef95ed3c2
Author: Mingyu Chen 
AuthorDate: Thu Jul 15 10:47:59 2021 +0800

[s3][bug] Remove log4j1 (#6211)

There are jar package conflict

Co-authored-by: chenmingyu 
---
 .../src/main/java/org/apache/doris/analysis/StorageBackend.java  | 5 +++--
 fe/fe-core/src/main/java/org/apache/doris/analysis/StorageDesc.java  | 5 +++--
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageBackend.java 
b/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageBackend.java
index e73a90e..933153b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageBackend.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageBackend.java
@@ -27,12 +27,13 @@ import org.apache.doris.thrift.TStorageBackendType;
 import com.google.common.base.Strings;
 
 import org.apache.commons.lang3.StringUtils;
-import org.apache.log4j.Logger;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.util.Map;
 
 public class StorageBackend extends StorageDesc implements ParseNode {
-private static Logger LOG = Logger.getLogger(StorageBackend.class);
+private static final Logger LOG = 
LoggerFactory.getLogger(StorageBackend.class);
 
 private String location;
 private StorageType storageType;
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageDesc.java 
b/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageDesc.java
index 83e5110..914e085 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageDesc.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageDesc.java
@@ -23,14 +23,15 @@ import org.apache.doris.common.Config;
 import org.apache.commons.collections.map.CaseInsensitiveMap;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.http.client.utils.URIBuilder;
-import org.apache.log4j.Logger;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.net.URI;
 import java.net.URISyntaxException;
 import java.util.Map;
 
 public abstract class StorageDesc {
-private static Logger LOG = Logger.getLogger(StorageBackend.class);
+private static final Logger LOG = 
LoggerFactory.getLogger(StorageBackend.class);
 // for dfs
 public static final String HADOOP_JOB_UGI = "hadoop.job.ugi";
 public static final String HADOOP_JOB_GROUP_NAME = "hadoop.job.group.name";

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6223: [ODBC] Support ODBC external table of SQLServer and revise the doc.

2021-07-14 Thread GitBox


yangzhg commented on a change in pull request #6223:
URL: https://github.com/apache/incubator-doris/pull/6223#discussion_r670091008



##
File path: docs/zh-CN/extending-doris/odbc-of-doris.md
##
@@ -208,15 +208,32 @@ set enable_odbc_transcation = true;
 |   DATETIME  | DATETIME |  |
 |   NUMBER  | DECIMAL |  |
 
+### SQLServer
+
+|  MySQL  | Doris  | 替换方案  |

Review comment:
   MYSQL ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6223: [ODBC] Support ODBC external table of SQLServer and revise the doc.

2021-07-14 Thread GitBox


yangzhg commented on a change in pull request #6223:
URL: https://github.com/apache/incubator-doris/pull/6223#discussion_r670091188



##
File path: docs/en/extending-doris/odbc-of-doris.md
##
@@ -212,15 +212,32 @@ There are different data types among different databases. 
Here, the types in eac
 |   DATETIME  | DATETIME |  |
 |   NUMBER  | DECIMAL |  |
 
+### SQLServer
+
+|  MySQL  | Doris  | 替换方案  |

Review comment:
   Mysql ?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6223: [ODBC] Support ODBC external table of SQLServer and revise the doc.

2021-07-14 Thread GitBox


yangzhg commented on a change in pull request #6223:
URL: https://github.com/apache/incubator-doris/pull/6223#discussion_r670091688



##
File path: docs/en/extending-doris/odbc-of-doris.md
##
@@ -212,15 +212,32 @@ There are different data types among different databases. 
Here, the types in eac
 |   DATETIME  | DATETIME |  |
 |   NUMBER  | DECIMAL |  |
 
+### SQLServer
+
+|  MySQL  | Doris  | 替换方案  |
+| :--: | :: | :---: |
+|  BOOLEAN  | BOOLEAN  | |
+|   CHAR   |  CHAR  |Only UTF8 encoding is supported|
+| VARCHAR | VARCHAR |   Only UTF8 encoding is supported  |

Review comment:
   no varchar in sqlserver




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg closed pull request #4655: [array][document] Support array type 4/4. Update array document.

2021-07-14 Thread GitBox


yangzhg closed pull request #4655:
URL: https://github.com/apache/incubator-doris/pull/4655


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg closed pull request #4650: [array][be] Support array type 3/4. Be exec scan array, insert array and sink array type.

2021-07-14 Thread GitBox


yangzhg closed pull request #4650:
URL: https://github.com/apache/incubator-doris/pull/4650


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg closed pull request #4644: [array][fe] Support array type 2/4. FE SQL syntax, ArrayLiteral, meta support

2021-07-14 Thread GitBox


yangzhg closed pull request #4644:
URL: https://github.com/apache/incubator-doris/pull/4644


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg closed pull request #4643: [array][be] Support array type 1/4. add array to tablet_meta, add array expr/function, mysql buffer

2021-07-14 Thread GitBox


yangzhg closed pull request #4643:
URL: https://github.com/apache/incubator-doris/pull/4643


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg closed pull request #4623: Support Array type

2021-07-14 Thread GitBox


yangzhg closed pull request #4623:
URL: https://github.com/apache/incubator-doris/pull/4623


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg closed pull request #2979: Incomplete support Array type (#2871)

2021-07-14 Thread GitBox


yangzhg closed pull request #2979:
URL: https://github.com/apache/incubator-doris/pull/2979


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [Bug-Fix] Fix bug that show view report "Unresolved table reference" error (#6184)

2021-07-14 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 7c34dbb  [Bug-Fix] Fix bug that show view report "Unresolved table 
reference" error (#6184)
7c34dbb is described below

commit 7c34dbbc5b5d155e08ee912a0a18dff3bfd63901
Author: xy720 <22125576+xy...@users.noreply.github.com>
AuthorDate: Thu Jul 15 10:55:15 2021 +0800

[Bug-Fix] Fix bug that show view report "Unresolved table reference" error 
(#6184)
---
 .../Data Manipulation/SHOW STREAM LOAD.md  |  10 +-
 .../Data Manipulation/SHOW STREAM LOAD.md  |  10 +-
 .../java/org/apache/doris/analysis/QueryStmt.java  |   6 +-
 .../java/org/apache/doris/analysis/SelectStmt.java |  23 +-
 .../apache/doris/analysis/SetOperationStmt.java|   6 +-
 .../org/apache/doris/analysis/ShowViewStmt.java|   8 +-
 .../java/org/apache/doris/analysis/WithClause.java |   4 +-
 .../org/apache/doris/analysis/SelectStmtTest.java  |  15 +-
 .../apache/doris/analysis/ShowViewStmtTest.java| 238 +
 .../java/org/apache/doris/utframe/DorisAssert.java |  14 ++
 10 files changed, 255 insertions(+), 79 deletions(-)

diff --git a/docs/en/sql-reference/sql-statements/Data Manipulation/SHOW STREAM 
LOAD.md b/docs/en/sql-reference/sql-statements/Data Manipulation/SHOW STREAM 
LOAD.md
index fc870e7..b2704e3 100644
--- a/docs/en/sql-reference/sql-statements/Data Manipulation/SHOW STREAM 
LOAD.md
+++ b/docs/en/sql-reference/sql-statements/Data Manipulation/SHOW STREAM 
LOAD.md
@@ -57,12 +57,12 @@ SHOW STREAM LOAD FROM example_db WHERE LABEL LIKE 
"2014_01_02" LIMIT 10;
 3. Show the STREAM LOAD task of the specified db, specify label as 
"load_example_db_20140102"
 SHOW STREAM LOAD FROM example_db WHERE LABEL = "load_example_db_20140102";
 
-4. Show the STREAM LOAD task of the specified db, specify status as "success", 
and sort it in descending order by LoadStartTime
-SHOW STREAM LOAD FROM example_db WHERE STATUS = "success" ORDER BY 
LoadStartTime DESC;
+4. Show the STREAM LOAD task of the specified db, specify status as "success", 
and sort it in descending order by StartTime
+SHOW STREAM LOAD FROM example_db WHERE STATUS = "success" ORDER BY StartTime 
DESC;
 
-5. Show the STREAM LOAD task of the specified dB and sort it in descending 
order by LoadStartTime, and display 10 query results starting with offset 5
-SHOW STREAM LOAD FROM example_db ORDER BY LoadStartTime DESC limit 5,10;
-SHOW STREAM LOAD FROM example_db ORDER BY LoadStartTime DESC limit 10 offset 5;
+5. Show the STREAM LOAD task of the specified dB and sort it in descending 
order by StartTime, and display 10 query results starting with offset 5
+SHOW STREAM LOAD FROM example_db ORDER BY StartTime DESC limit 5,10;
+SHOW STREAM LOAD FROM example_db ORDER BY StartTime DESC limit 10 offset 5;
 
 ## keyword
 SHOW,STREAM LOAD
diff --git a/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/SHOW 
STREAM LOAD.md b/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/SHOW 
STREAM LOAD.md
index ec9f570..53f077d 100644
--- a/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/SHOW STREAM 
LOAD.md 
+++ b/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/SHOW STREAM 
LOAD.md 
@@ -57,12 +57,12 @@ under the License.
 3. 展示指定 db 的Stream Load任务,指定 label 为 "load_example_db_20140102"
 SHOW STREAM LOAD FROM example_db WHERE LABEL = 
"load_example_db_20140102";
 
-4. 展示指定 db 的Stream Load任务,指定 status 为 "success", 并按 LoadStartTime 降序排序
-SHOW STREAM LOAD FROM example_db WHERE STATUS = "success" ORDER BY 
LoadStartTime DESC;
+4. 展示指定 db 的Stream Load任务,指定 status 为 "success", 并按 StartTime 降序排序
+SHOW STREAM LOAD FROM example_db WHERE STATUS = "success" ORDER BY 
StartTime DESC;
 
-5. 展示指定 db 的导入任务 并按 LoadStartTime 降序排序,并从偏移量5开始显示10条查询结果
-SHOW STREAM LOAD FROM example_db ORDER BY LoadStartTime DESC limit 
5,10;
-SHOW STREAM LOAD FROM example_db ORDER BY LoadStartTime DESC limit 10 
offset 5;
+5. 展示指定 db 的导入任务 并按 StartTime 降序排序,并从偏移量5开始显示10条查询结果
+SHOW STREAM LOAD FROM example_db ORDER BY StartTime DESC limit 5,10;
+SHOW STREAM LOAD FROM example_db ORDER BY StartTime DESC limit 10 
offset 5;
 
 ## keyword
 SHOW,STREAM LOAD
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/QueryStmt.java 
b/fe/fe-core/src/main/java/org/apache/doris/analysis/QueryStmt.java
index 3460cd5..15bc170 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/QueryStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/QueryStmt.java
@@ -429,9 +429,9 @@ public abstract class QueryStmt extends StatementBase {
 }
 }
 
-public void getWithClauseTableRefs(List tblRefs, Set 
parentViewNameSet) {
+public void getWithClauseTableRefs(Analyzer analyz

[GitHub] [incubator-doris] yangzhg closed issue #6183: [Bug] Show view report "Unresolved table reference" error

2021-07-14 Thread GitBox


yangzhg closed issue #6183:
URL: https://github.com/apache/incubator-doris/issues/6183


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #6184: [Bug-Fix] Fix bug that show view report "Unresolved table reference" error

2021-07-14 Thread GitBox


yangzhg merged pull request #6184:
URL: https://github.com/apache/incubator-doris/pull/6184


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] JNSimba opened a new pull request #6243: [Feature]:Flink-connector supports streamload parameters

2021-07-14 Thread GitBox


JNSimba opened a new pull request #6243:
URL: https://github.com/apache/incubator-doris/pull/6243


   ## Proposed changes
   Flink-connector supports streamload parameters
   #6199 
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [ ] Code refactor (Modify the code structure, format the code, etc...)
   - [ ] Optimization. Including functional usability improvements and 
performance improvements.
   - [ ] Dependency. Such as changes related to third-party components.
   - [ ] Other.
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature 
there in detail
   - [ ] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
d...@doris.apache.org by explaining why you chose the solution you did and what 
alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #6205: [Optimize] Using custom conf dir to save log config of Spring

2021-07-14 Thread GitBox


yangzhg merged pull request #6205:
URL: https://github.com/apache/incubator-doris/pull/6205


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [Optimize] Using custom conf dir to save log config of Spring (#6205)

2021-07-14 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 7e77b5e  [Optimize] Using custom conf dir to save log config of Spring 
(#6205)
7e77b5e is described below

commit 7e77b5ed7f29bf26e66509ba16342a6fa2d98341
Author: Mingyu Chen 
AuthorDate: Thu Jul 15 11:13:51 2021 +0800

[Optimize] Using custom conf dir to save log config of Spring (#6205)

The log4j-config.xml will be generated at startup of FE and also when 
modifying FE config.
But in some deploy environment such as k8s, the conf dir is not writable.

So change the dir of log4j-config.xml to Config.custom_conf_dir.

Also fix some small bugs:

1. Typo "less then" -> "less than"
2. Duplicated `exec_mem_limit` showed in SHOW ROUTINE LOAD
3. Allow MAXVALUE in single partition column table.
4. Add IP info for "intolerate index channel failure" msg.

Change-Id: Ib4e1182084219c41eae44d3a28110c0315fdbd7d

Co-authored-by: chenmingyu 
---
 be/src/exec/tablet_sink.cpp|  3 +++
 .../src/main/java/org/apache/doris/alter/RollupJobV2.java  |  4 ++--
 .../java/org/apache/doris/alter/SchemaChangeJobV2.java |  4 ++--
 .../apache/doris/analysis/CreateMaterializedViewStmt.java  |  2 +-
 .../main/java/org/apache/doris/analysis/InsertStmt.java| 10 +-
 .../java/org/apache/doris/analysis/PartitionKeyDesc.java   | 14 --
 .../src/main/java/org/apache/doris/catalog/Database.java   |  2 +-
 .../src/main/java/org/apache/doris/catalog/OlapTable.java  |  2 +-
 .../src/main/java/org/apache/doris/common/Log4jConfig.java |  5 -
 .../src/main/java/org/apache/doris/httpv2/HttpServer.java  |  3 ++-
 .../org/apache/doris/httpv2/config/SpringLog4j2Config.java |  2 +-
 .../java/org/apache/doris/load/loadv2/LoadLoadingTask.java |  2 +-
 .../org/apache/doris/load/routineload/RoutineLoadJob.java  |  7 +--
 .../doris/load/routineload/RoutineLoadScheduler.java   |  2 +-
 .../java/org/apache/doris/planner/AssertNumRowsNode.java   |  2 +-
 .../main/java/org/apache/doris/qe/ConnectProcessor.java| 11 +--
 .../doris/transaction/TabletQuorumFailedException.java |  2 +-
 .../test/java/org/apache/doris/utframe/UtFrameUtils.java   |  5 +
 18 files changed, 53 insertions(+), 29 deletions(-)

diff --git a/be/src/exec/tablet_sink.cpp b/be/src/exec/tablet_sink.cpp
index d7d24d1..e978f99 100644
--- a/be/src/exec/tablet_sink.cpp
+++ b/be/src/exec/tablet_sink.cpp
@@ -26,6 +26,7 @@
 #include "runtime/row_batch.h"
 #include "runtime/runtime_state.h"
 #include "runtime/tuple_row.h"
+#include "service/backend_options.h"
 #include "service/brpc.h"
 #include "util/brpc_stub_cache.h"
 #include "util/debug/sanitizer_scopes.h"
@@ -473,6 +474,8 @@ Status IndexChannel::add_row(Tuple* tuple, int64_t 
tablet_id) {
 }
 
 if (has_intolerable_failure()) {
+std::stringstream ss;
+ss << "index channel has intolerable failure. " << 
BackendOptions::get_localhost();
 return Status::InternalError(ss.str());
 }
 
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java 
b/fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java
index 470d4c5..dbd0204 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java
@@ -751,8 +751,8 @@ public class RollupJobV2 extends AlterJobV2 implements 
GsonPostProcessable {
 }
 
 /**
- * This method is only used to deserialize the text mate which version is 
less then 86.
- * If the meta version >=86, it will be deserialized by the `read` of 
AlterJobV2 rather then here.
+ * This method is only used to deserialize the text mate which version is 
less than 86.
+ * If the meta version >=86, it will be deserialized by the `read` of 
AlterJobV2 rather than here.
  */
 public static RollupJobV2 read(DataInput in) throws IOException {
 Preconditions.checkState(Catalog.getCurrentCatalogJournalVersion() < 
FeMetaVersion.VERSION_86);
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java 
b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
index bd22ba2..3292d86 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
@@ -979,8 +979,8 @@ public class SchemaChangeJobV2 extends AlterJobV2 {
 }
 
 /**
- * This method is only used to deserialize the text mate which version is 
less then 86.
- * If the meta version >=86, it will be deserialized by the `read` of 
AlterJobV2 rather then here.
+ * This method is only used to deserialize the text mate which version is 
less than 86.
+ * If the

[GitHub] [incubator-doris] yangzhg merged pull request #6208: [Compatibility] Change the response body of load info api in httpv2.

2021-07-14 Thread GitBox


yangzhg merged pull request #6208:
URL: https://github.com/apache/incubator-doris/pull/6208


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [Compatibility] Change the response body of load info api in httpv2. (#6208)

2021-07-14 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new d5cd3ae  [Compatibility] Change the response body of load info api in 
httpv2. (#6208)
d5cd3ae is described below

commit d5cd3ae5ee617ff696e19b2b01fc02a6d571362d
Author: Mingyu Chen 
AuthorDate: Thu Jul 15 11:14:45 2021 +0800

[Compatibility] Change the response body of load info api in httpv2. (#6208)

1. To be compatible with response body of GetLoadInfoAction in httpv1.
2. Not drop partition by force in dynamic partition scheduler.

Change-Id: I50864ddadf1a1c25efa16a465940a1129f937d3d

Co-authored-by: chenmingyu 
---
 .../doris/clone/DynamicPartitionScheduler.java |  2 ++
 .../doris/httpv2/rest/GetLoadInfoAction.java   | 37 ++
 2 files changed, 33 insertions(+), 6 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
 
b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
index 9d10469..0162bc7 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
@@ -285,6 +285,8 @@ public class DynamicPartitionScheduler extends MasterDaemon 
{
 RangeUtils.checkRangeIntersect(reservePartitionKeyRange, 
checkDropPartitionKey);
 if 
(checkDropPartitionKey.upperEndpoint().compareTo(reservePartitionKeyRange.lowerEndpoint())
 <= 0) {
 String dropPartitionName = 
olapTable.getPartition(checkDropPartitionId).getName();
+// Do not drop the partition "by force", or the partition 
will be dropped directly instread of being in
+// catalog recycle bin. This is for safe reason.
 dropPartitionClauses.add(new DropPartitionClause(false, 
dropPartitionName, false, false));
 }
 } catch (DdlException e) {
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/GetLoadInfoAction.java 
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/GetLoadInfoAction.java
index de83bc1..d79cd5b 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/GetLoadInfoAction.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/GetLoadInfoAction.java
@@ -20,7 +20,7 @@ package org.apache.doris.httpv2.rest;
 import org.apache.doris.catalog.Catalog;
 import org.apache.doris.common.DdlException;
 import org.apache.doris.common.MetaNotFoundException;
-import org.apache.doris.httpv2.entity.ResponseEntityBuilder;
+import org.apache.doris.http.rest.RestBaseResult;
 import org.apache.doris.load.Load;
 import org.apache.doris.mysql.privilege.PrivPredicate;
 import org.apache.doris.qe.ConnectContext;
@@ -37,6 +37,20 @@ import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
 
 // Get load information of one load job
+// To be compatible with old api, we return like this:
+// {
+// "status": "OK",
+// "msg": "Success",
+// "jobInfo": {
+// "dbName": "default_cluster:db1",
+// "tblNames": ["tbl1"],
+// "label": "abc",
+// "clusterName": "default_cluster",
+// "state": "FINISHED",
+// "failMsg": "",
+// "trackingUrl": "\\N"
+// }
+// }
 @RestController
 public class GetLoadInfoAction extends RestBaseController {
 
@@ -55,13 +69,13 @@ public class GetLoadInfoAction extends RestBaseController {
 request.getParameter(LABEL_KEY),
 ConnectContext.get().getClusterName());
 if (Strings.isNullOrEmpty(info.dbName)) {
-return ResponseEntityBuilder.badRequest("No database selected");
+return new RestBaseResult("No database selected");
 }
 if (Strings.isNullOrEmpty(info.label)) {
-return ResponseEntityBuilder.badRequest("No label selected");
+return new RestBaseResult("No label selected");
 }
 if (Strings.isNullOrEmpty(info.clusterName)) {
-return ResponseEntityBuilder.badRequest("No cluster selected");
+return new RestBaseResult("No cluster selected");
 }
 
 RedirectView redirectView = redirectToMaster(request, response);
@@ -83,9 +97,20 @@ public class GetLoadInfoAction extends RestBaseController {
 try {
 catalog.getLoadManager().getLoadJobInfo(info);
 } catch (DdlException e1) {
-return 
ResponseEntityBuilder.okWithCommonError(e1.getMessage());
+return new RestBaseResult(e.getMessage());
 }
 }
-return ResponseEntityBuilder.ok(info);
+return new Result(info);
+}
+
+// This is just same as Result in http/rest/Ge

[GitHub] [incubator-doris] yangzhg merged pull request #6216: [Optimize] Use flat_hash_set to replace unorderd_set in InPredicate

2021-07-14 Thread GitBox


yangzhg merged pull request #6216:
URL: https://github.com/apache/incubator-doris/pull/6216


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated (d5cd3ae -> 68f988b)

2021-07-14 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.


from d5cd3ae  [Compatibility] Change the response body of load info api in 
httpv2. (#6208)
 add 68f988b  [Optimize] Use flat_hash_set to replace unorderd_set in 
InPredicate (#6216)

No new revisions were added by this update.

Summary of changes:
 be/src/olap/in_list_predicate.cpp  | 41 --
 be/src/olap/in_list_predicate.h|  7 ++--
 be/src/olap/reader.cpp | 21 +--
 be/test/olap/in_list_predicate_test.cpp| 18 +-
 be/test/olap/rowset/segment_v2/segment_test.cpp|  4 +--
 .../java/org/apache/doris/catalog/Catalog.java |  3 +-
 6 files changed, 42 insertions(+), 52 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #6215: [Bug][RoutineLoad] Fix bug that routine load thread on BE may be blocked

2021-07-14 Thread GitBox


yangzhg merged pull request #6215:
URL: https://github.com/apache/incubator-doris/pull/6215


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated (68f988b -> 409cee0)

2021-07-14 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.


from 68f988b  [Optimize] Use flat_hash_set to replace unorderd_set in 
InPredicate (#6216)
 add 409cee0  [Bug][RoutineLoad] Fix bug that routine load thread on BE may 
be blocked (#6215)

No new revisions were added by this update.

Summary of changes:
 be/src/util/blocking_priority_queue.hpp  |  5 -
 .../java/org/apache/doris/planner/StreamLoadPlanner.java | 12 ++--
 2 files changed, 14 insertions(+), 3 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] pengxiangyu opened a new issue #6244: Add transaction for the operation of insert

2021-07-14 Thread GitBox


pengxiangyu opened a new issue #6244:
URL: https://github.com/apache/incubator-doris/issues/6244


   Add transaction for the operation of insert. Only insert is acceptable, nor 
update or delete.
   e.g.
   commit a transaction:
   begin;
   insert into Tbl values(11, 22, 33);
   commit;
   rollback a transaction:
   begin;
   insert into Tbl values(11, 22, 33);
   rollback;
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] pengxiangyu opened a new pull request #6245: Add transaction for the operation of insert #6244

2021-07-14 Thread GitBox


pengxiangyu opened a new pull request #6245:
URL: https://github.com/apache/incubator-doris/pull/6245


   ## Proposed changes
   
   Add transaction for the operation of insert. It will cost less time than 
non-transaction(it will cost 1/1000 time) when you want to insert a amount of 
rows.
   e.g.
   commit a transaction:
   begin;
   insert into Tbl values(11, 22, 33);
   commit;
   rollback a transaction:
   begin;
   insert into Tbl values(11, 22, 33);
   rollback;
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [x] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [ ] Code refactor (Modify the code structure, format the code, etc...)
   - [ ] Optimization. Including functional usability improvements and 
performance improvements.
   - [ ] Dependency. Such as changes related to third-party components.
   - [ ] Other.
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [x] I have created an issue on (Fix #6244) and described the bug/feature 
there in detail
   - [ ] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #6182: [Docs] Add like, regexp function documents

2021-07-14 Thread GitBox


yangzhg merged pull request #6182:
URL: https://github.com/apache/incubator-doris/pull/6182


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [Docs] Add like, regexp function documents (#6182)

2021-07-14 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 15c5896  [Docs] Add like, regexp function documents (#6182)
15c5896 is described below

commit 15c5896f4110c1c2ac35bdc847b19895b7347f5f
Author: EmmyMiao87 <522274...@qq.com>
AuthorDate: Thu Jul 15 13:16:21 2021 +0800

[Docs] Add like, regexp function documents (#6182)

* [Docs] Add like, regexp function documents

* Reconstruct

* Fix compile error
---
 docs/.vuepress/sidebar/en.js   | 20 ++-
 docs/.vuepress/sidebar/zh-CN.js| 20 ++-
 .../append_trailing_char_if_absent.md  |  4 +-
 .../{regexp_replace.md => like/like.md}| 51 ++--
 .../string-functions/like/not_like.md  | 68 ++
 .../string-functions/regexp/not_regexp.md  | 56 ++
 .../{regexp_replace.md => regexp/regexp.md}| 42 ++---
 .../{ => regexp}/regexp_extract.md |  0
 .../{ => regexp}/regexp_replace.md |  0
 .../append_trailing_char_if_absent.md  |  4 +-
 .../{regexp_replace.md => like/like.md}| 51 ++--
 .../string-functions/like/not_like.md  | 68 ++
 .../{regexp_replace.md => regexp/not_regexp.md}| 42 ++---
 .../{regexp_replace.md => regexp/regexp.md}| 42 ++---
 .../{ => regexp}/regexp_extract.md |  3 -
 .../{ => regexp}/regexp_replace.md |  0
 16 files changed, 362 insertions(+), 109 deletions(-)

diff --git a/docs/.vuepress/sidebar/en.js b/docs/.vuepress/sidebar/en.js
index bf40bd3..ce584d0 100644
--- a/docs/.vuepress/sidebar/en.js
+++ b/docs/.vuepress/sidebar/en.js
@@ -314,8 +314,6 @@ module.exports = [
   "ltrim",
   "money_format",
   "null_or_empty",
-  "regexp_extract",
-  "regexp_replace",
   "repeat",
   "reverse",
   "right",
@@ -324,6 +322,24 @@ module.exports = [
   "starts_with",
   "strleft",
   "strright",
+  {
+title: "fuzzy match",
+directoryPath: "like/",
+children: [
+  "like",
+  "not_like",
+],
+  },
+  {
+title: "regular match",
+directoryPath: "regexp/",
+children: [
+  "regexp",
+  "regexp_extract",
+  "regexp_replace",
+  "not_regexp",
+],
+  },
 ],
   },
   {
diff --git a/docs/.vuepress/sidebar/zh-CN.js b/docs/.vuepress/sidebar/zh-CN.js
index e9ef97b..1e7acc6 100644
--- a/docs/.vuepress/sidebar/zh-CN.js
+++ b/docs/.vuepress/sidebar/zh-CN.js
@@ -319,8 +319,6 @@ module.exports = [
   "ltrim",
   "money_format",
   "null_or_empty",
-  "regexp_extract",
-  "regexp_replace",
   "repeat",
   "reverse",
   "right",
@@ -329,6 +327,24 @@ module.exports = [
   "starts_with",
   "strleft",
   "strright",
+  {
+title: "模糊匹配",
+directoryPath: "like/",
+children: [
+  "like",
+  "not_like",
+],
+  },
+  {
+title: "正则匹配",
+directoryPath: "regexp/",
+children: [
+  "regexp",
+  "regexp_extract",
+  "regexp_replace",
+  "not_regexp",
+],
+  },
 ],
   },
   {
diff --git 
a/docs/en/sql-reference/sql-functions/string-functions/append_trailing_char_if_absent.md
 
b/docs/en/sql-reference/sql-functions/string-functions/append_trailing_char_if_absent.md
index cd24883..6895c3e 100644
--- 
a/docs/en/sql-reference/sql-functions/string-functions/append_trailing_char_if_absent.md
+++ 
b/docs/en/sql-reference/sql-functions/string-functions/append_trailing_char_if_absent.md
@@ -32,8 +32,8 @@ under the License.
 
 `VARCHAR append_trailing_char_if_absent(VARCHAR str, VARCHAR trailing_char)`
 
-If the s string is non-empty and does not contain the c character at the end, 
it appends the c character to the end.
-Trailing_char contains only one character, and it will return NULL if contains 
more than one character
+If the @str string is non-empty and does not contain the @trailing_char 
character at the end, it appends the @trailing_char character to the end.
+@trailing_char contains only one character, and it will retu

[GitHub] [incubator-doris] BiteTheDDDDt closed issue #6195: [Feature] Add trash_used_capacity statistics at 'show backends'

2021-07-14 Thread GitBox


BiteThet closed issue #6195:
URL: https://github.com/apache/incubator-doris/issues/6195


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] BiteTheDDDDt closed pull request #6196: [Feature] Add trash_used_capacity statistics at 'show backends'

2021-07-14 Thread GitBox


BiteThet closed pull request #6196:
URL: https://github.com/apache/incubator-doris/pull/6196


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] BiteTheDDDDt opened a new issue #6246: [Feature] Support for querying the trash used capacity

2021-07-14 Thread GitBox


BiteThet opened a new issue #6246:
URL: https://github.com/apache/incubator-doris/issues/6246


   **Is your feature request related to a problem? Please describe.**
Sometimes, garbage data will accumulate and take up a lot of space, but 
users don't know it.
   
   **Describe the solution you'd like**
   Add a query for display trash used capacity.
   query format: 
   `show trash;`
   `show trash on ("BackendHost1:BackendHeartBeatPort1", 
"BackendHost2:BackendHeartBeatPort2", ...);`
   **Describe alternatives you've considered**
Users can manually delete trash after noticing that trash takes up a lot of 
space
   
   **Additional context**
   
![图片](https://user-images.githubusercontent.com/7939630/125737118-b3e0544a-81fc-44a8-b630-6f651a3e008e.png)
   
![图片](https://user-images.githubusercontent.com/7939630/125737218-78d1ce43-c898-42e8-827e-e52d3ff36f02.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] BiteTheDDDDt opened a new pull request #6247: [Feature] Support for querying the trash used capacity

2021-07-14 Thread GitBox


BiteThet opened a new pull request #6247:
URL: https://github.com/apache/incubator-doris/pull/6247


   ## Proposed changes
   
   Now user can proactively scan trash directory.
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [X] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [ ] Code refactor (Modify the code structure, format the code, etc...)
   - [ ] Optimization. Including functional usability improvements and 
performance improvements.
   - [ ] Dependency. Such as changes related to third-party components.
   - [ ] Other.
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [X] I have created an issue on (Fix #6246) and described the bug/feature 
there in detail
   - [X] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6247: [Feature] Support for querying the trash used capacity

2021-07-14 Thread GitBox


yangzhg commented on a change in pull request #6247:
URL: https://github.com/apache/incubator-doris/pull/6247#discussion_r670184683



##
File path: fe/fe-core/src/main/jflex/sql_scanner.flex
##
@@ -103,6 +103,7 @@ import org.apache.doris.qe.SqlModeHelper;
 keywordMap.put("array", new Integer(SqlParserSymbols.KW_ARRAY));
 keywordMap.put("backend", new Integer(SqlParserSymbols.KW_BACKEND));
 keywordMap.put("backends", new Integer(SqlParserSymbols.KW_BACKENDS));
+keywordMap.put("trash", new Integer(SqlParserSymbols.KW_TRASH));

Review comment:
   keep the alpha order 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6247: [Feature] Support for querying the trash used capacity

2021-07-14 Thread GitBox


yangzhg commented on a change in pull request #6247:
URL: https://github.com/apache/incubator-doris/pull/6247#discussion_r670187091



##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
##
@@ -1690,6 +1698,46 @@ private void handleShowRoles() {
 List> infos = 
Catalog.getCurrentCatalog().getAuth().getRoleInfo();
 resultSet = new ShowResultSet(showStmt.getMetaData(), infos);
 }
+
+private void handleShowTrash() {
+ShowTrashStmt showStmt = (ShowTrashStmt) stmt;
+List> results = Lists.newArrayList();
+
+for (Backend backend : showStmt.getBackends()) {
+BackendService.Client client = null;
+TNetworkAddress address = null;
+Long trashUsedCapacityB = null;
+boolean ok = false;
+try {
+long start = System.currentTimeMillis();
+address = new TNetworkAddress(backend.getHost(), 
backend.getBePort());
+client = ClientPool.backendPool.borrowObject(address);
+trashUsedCapacityB = client.getTrashUsedCapacity();
+LOG.debug("get trash used capacity from backend: {}, 
trashUsedCapacity: {}, cost {}", backend.getId(),
+trashUsedCapacityB, System.currentTimeMillis() - start);
+ok = true;
+} catch (Exception e) {
+LOG.warn("task exec error. backend[{}]", backend.getId(), e);

Review comment:
   Just a log without further processing?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org