[GitHub] [incubator-doris] killxdcj opened a new issue #5548: [Enhancement] StreamLoadPlanner's error log needs to be improved.

2021-03-22 Thread GitBox


killxdcj opened a new issue #5548:
URL: https://github.com/apache/incubator-doris/issues/5548


   StreamLoadPlanner's error log needs to be improved.
   
   1. In OlapTableSink checked partitionIds without error message.
   2021-03-09 17:12:47,464 WARN (thrift-server-pool-42|222) 
[FrontendServiceImpl.streamLoadPut():841] catch unknown result.
   
   > java.lang.IllegalStateException: null
   > at 
com.google.common.base.Preconditions.checkState(Preconditions.java:492) 
~[spark-dpp-1.0.0.jar:?]
   > at 
org.apache.doris.planner.OlapTableSink.(OlapTableSink.java:88) 
~[palo-fe.jar:?]
   > at 
org.apache.doris.planner.StreamLoadPlanner.plan(StreamLoadPlanner.java:145) 
~[palo-fe.jar:?]
   > at 
org.apache.doris.service.FrontendServiceImpl.streamLoadPutImpl(FrontendServiceImpl.java:885)
 ~[palo-fe.jar:?]
   > at 
org.apache.doris.service.FrontendServiceImpl.streamLoadPut(FrontendServiceImpl.java:835)
 [palo-fe.jar:?]
   > at 
org.apache.doris.thrift.FrontendService$Processor$streamLoadPut.getResult(FrontendService.java:1753)
 [palo-fe.jar:?]
   > at 
org.apache.doris.thrift.FrontendService$Processor$streamLoadPut.getResult(FrontendService.java:1738)
 [palo-fe.jar:?]
   > at 
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) 
[libthrift-0.9.3.jar:0.9.3]
   > at 
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
[libthrift-0.9.3.jar:0.9.3]
   > at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
 [libthrift-0.9.3.jar:0.9.3]
   > at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
[?:?]
   > at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
[?:?]
   > at java.lang.Thread.run(Thread.java:834) [?:?]
   
   2. Then we can't see a specific cause of failure on the BE Coordinator and 
Client side.
   
   > W0309 17:12:47.464723 2456184 stream_load.cpp:448] plan streaming load 
failed. errmsg=id=19479a48662e05e6-14a4b93823a73594, job_id=-1, txn_id=958866, 
label=aEGyQWOhLLBrboppmiBT


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] killxdcj opened a new pull request #5549: [FE] Improve StreamLoadPlanner error log.

2021-03-22 Thread GitBox


killxdcj opened a new pull request #5549:
URL: https://github.com/apache/incubator-doris/pull/5549


   ## Proposed changes
   
   StreamLoadPlanner's error log needs to be improved. #5548
   
   ## Types of changes
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [x] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [ ] Code refactor (Modify the code structure, format the code, etc...)
   
   ## Checklist
   
   - [x] I have created an issue on (Fix #5548) and described the bug/feature 
there in detail
   - [x] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   
   More detailed information reference #5548
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] JinLiOnline opened a new issue #5550: [BE] memtracker中parent和limit显示有歧义

2021-03-22 Thread GitBox


JinLiOnline opened a new issue #5550:
URL: https://github.com/apache/incubator-doris/issues/5550


   **Describe the bug**
   doris version:0.14-rc3
BE_IP:BE_HOST/mem_tracker 可以看到详细信息
   
![image](https://user-images.githubusercontent.com/48025020/111951620-e1e39400-8b1e-11eb-9e23-290047206ebc.png)
   
   有些疑惑点,比如:
   1)root节点limit为none
   2)mem_limit 只限制了 ExecEnv Root
   3)在代码中:比如 load channel mgr是根据 ExecEnv中的 
mem_limit来计算的,但在上图中它的父节点是root,却不是ExecEnv root。
   
   
   **Expected behavior**
   我的理解,参数mem_limit限制be总的内存,作为root节点,be其他模块的可用内存上限,从root中继承。
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg opened a new pull request #5551: [DOC] Add docs of load and export using S3 protocol

2021-03-22 Thread GitBox


yangzhg opened a new pull request #5551:
URL: https://github.com/apache/incubator-doris/pull/5551


   ## Proposed changes
   
Add docs of load and export using S3 protocol #5232 
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [x] Documentation Update (if none of the other choices apply)
   - [ ] Code refactor (Modify the code structure, format the code, etc...)
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature 
there in detail
   - [ ] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
d...@doris.apache.org by explaining why you chose the solution you did and what 
alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: remove unused char at end of file (#5545)

2021-03-22 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 95f4f70  remove unused char at end of file (#5545)
95f4f70 is described below

commit 95f4f70fce77b3f5977c8efba4e8b782135541ec
Author: Zhengguo Yang 
AuthorDate: Mon Mar 22 15:44:34 2021 +0800

remove unused char at end of file (#5545)
---
 docker/Dockerfile | 1 -
 1 file changed, 1 deletion(-)

diff --git a/docker/Dockerfile b/docker/Dockerfile
index be40014..c9bbf99 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -123,4 +123,3 @@ ENV DORIS_THIRDPARTY /var/local/thirdparty
 WORKDIR /root
 
 CMD ["/bin/bash"]
-~

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #5545: remove unused char at end of file

2021-03-22 Thread GitBox


yangzhg merged pull request #5545:
URL: https://github.com/apache/incubator-doris/pull/5545


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] xinghuayu007 opened a new pull request #5552: [Refator] Make LikePredicate's operator public

2021-03-22 Thread GitBox


xinghuayu007 opened a new pull request #5552:
URL: https://github.com/apache/incubator-doris/pull/5552


   ## Proposed changes
   
   Currently LikePredicate's operator is not public. It is difficult to build a 
LikePredicate for a test case.
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [x ] Code refactor (Modify the code structure, format the code, etc...)
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature 
there in detail
   - [ ] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
d...@doris.apache.org by explaining why you chose the solution you did and what 
alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] wuyunfeng commented on a change in pull request #5325: [Doris On ES][WIP] Support external ES table with `SSL` secured and configurable node sniffing

2021-03-22 Thread GitBox


wuyunfeng commented on a change in pull request #5325:
URL: https://github.com/apache/incubator-doris/pull/5325#discussion_r598519935



##
File path: 
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsRestClient.java
##
@@ -172,6 +166,13 @@ private String execute(String path) throws 
DorisEsException {
 DorisEsException scratchExceptionForThrow = null;
 OkHttpClient httpClient;
 if (useSslClient) {
+if (sslNetworkClient == null) {

Review comment:
   This would leading to condition competition for `sslNetworkClient`, 
suggest use `synchronized ` method such as `getSSLNetworkClient`

##
File path: 
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsShardPartitions.java
##
@@ -94,16 +94,22 @@ public static EsShardPartitions findShardPartitions(String 
indexName, String sea
 return partitions;
 }
 
-public void addHttpAddress(Map nodesInfo) {
+public void addHttpAddress(Map nodesInfo, boolean 
httpSslEnabled) {
 for (Map.Entry> entry : 
shardRoutings.entrySet()) {
 List shardRoutings = entry.getValue();
 for (EsShardRouting shardRouting : shardRoutings) {
 String nodeId = shardRouting.getNodeId();
+TNetworkAddress httpAddress;
 if (nodesInfo.containsKey(nodeId)) {
-
shardRouting.setHttpAddress(nodesInfo.get(nodeId).getPublishAddress());
+httpAddress = nodesInfo.get(nodeId).getPublishAddress();
 } else {
-shardRouting.setHttpAddress(randomAddress(nodesInfo));
+httpAddress = randomAddress(nodesInfo);
 }
+// If ssl is enabled, determine if the https protocol is 
required
+if (httpSslEnabled && 
!httpAddress.getHostname().startsWith("http")) {
+httpAddress.setHostname("https://"; + 
httpAddress.getHostname());

Review comment:
   why `setHostname` ?? Is `setHttpAddress` does not work??




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] HappenLee commented on issue #5550: [BE] memtracker中parent和limit显示有歧义

2021-03-22 Thread GitBox


HappenLee commented on issue #5550:
URL: 
https://github.com/apache/incubator-doris/issues/5550#issuecomment-803893794


   Hi,3 questions:
   1):none means not limit, now Doris do not hava mem limit of root
   2):  some operation of Doris now do not limit the mem
   3):  load channel mgr is a Independent memory management module  do not have 
relations withe ExecEnv. You can find the answer in the source code. To be 
honest, it's a bad name, we will change the name of it. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] gaodayue opened a new issue #5553: MysqlResultWriter may become a bottleneck for some queries

2021-03-22 Thread GitBox


gaodayue opened a new issue #5553:
URL: https://github.com/apache/incubator-doris/issues/5553


   For queries that return a large number of rows and columns,  we observed 
that `MysqlResultWriter` may become a bottleneck. 
   
   For example, the query below returns 18K rows and 100 bigint columns for 
each row. And we observed that `TupleConvertTime` accounts for **51%** of the 
total execution time.
   
   ```
   Fragment 0:
 Instance 4c71a7a8e4fe447c-9840e16d115713b4:(Active: 363.487ms, % 
non-child: 49.92%)
- AverageThreadTokens: 4607182418800017400.00
- MemoryLimit: 8.00 GB
- PeakMemoryUsage: 15.75 MB
- PeakReservation: 0.00 
- PeakUsedReservation: 0.00 
- RowsProduced: 18.002K (18002)
   DataBufferSender 
(dst_fragment_instance_id=4c71a7a8e4fe447c-9840e16d115713b4):
  - AppendBatchTime: 194.511ms
- ResultRendTime: 5.220ms
- TupleConvertTime: 185.573ms
  - NumSentRows: 18.002K (18002)
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] gaodayue opened a new pull request #5554: Add fmt library to speed up mysql text result serialization

2021-03-22 Thread GitBox


gaodayue opened a new pull request #5554:
URL: https://github.com/apache/incubator-doris/pull/5554


   ## Proposed changes
   
   This CL optimizes the performance of `MysqlResultWriter` in two aspects
   1. Use `fmt::format_int` to do int->string conversion, which is 6x faster 
than `sprintf` according to [this 
blog](https://www.zverovich.net/2020/06/13/fast-int-to-string-revisited.html)
   2. Reduce profiling overhead by measuring convert time at row batch level 
instead of row level
   
   This fixes #5553 . The `TupleConvertTime` dropped from 185.573ms to 27ms 
after this optimization.
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [x] Code refactor (Modify the code structure, format the code, etc...)
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [x] I have created an issue on (Fix #ISSUE) and described the bug/feature 
there in detail
   - [ ] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] EmmyMiao87 opened a new pull request #5555: [Colocate plan][Step2] Colocate aggregation covers more situations

2021-03-22 Thread GitBox


EmmyMiao87 opened a new pull request #:
URL: https://github.com/apache/incubator-doris/pull/


   ## Proposed changes
   
   [Colocate plan][Step2] Colocate aggregation covers more situations
   
   The old colocate aggregation can only cover the case where the child is 
scan.
   In fact, as long as the child's data distribution meets the requirements,
 no matter what the plan node on the child node is, a colocate 
aggregation can be performed.
   
   This PR also fixes the correct data partition attribute of fragment.
   The data partition of fragment which contains scan node is Hash 
Partition rather than Random.
   This modification is mainly to determine the possibility of colocate
 through the correct distribution of child fragments.
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [x] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [ ] Code refactor (Modify the code structure, format the code, etc...)
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [x] I have created an issue on (Fix #5184) and described the bug/feature 
there in detail
   - [x] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   
   unit test is coming.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg closed issue #5542: [Bug] Rowset file being deleted by mistake

2021-03-22 Thread GitBox


yangzhg closed issue #5542:
URL: https://github.com/apache/incubator-doris/issues/5542


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #5541: [Bug] Fix the bug of rowset file being deleted by mistake

2021-03-22 Thread GitBox


yangzhg merged pull request #5541:
URL: https://github.com/apache/incubator-doris/pull/5541


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [Bug] Fix the bug of rowset file being deleted by mistake (#5541)

2021-03-22 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 3a96b6d  [Bug] Fix the bug of rowset file being deleted by mistake 
(#5541)
3a96b6d is described below

commit 3a96b6dfbdd0641d9d7f39e5c54ed2e70a46d294
Author: Mingyu Chen 
AuthorDate: Tue Mar 23 09:33:36 2021 +0800

[Bug] Fix the bug of rowset file being deleted by mistake (#5541)

* [Bug] Fix the bug of rowset file being deleted by mistake
---
 be/src/olap/rowset/unique_rowset_id_generator.cpp | 8 
 be/src/olap/rowset/unique_rowset_id_generator.h   | 6 ++
 .../src/test/java/org/apache/doris/catalog/CreateTableTest.java   | 2 +-
 .../src/test/java/org/apache/doris/planner/PlannerTest.java   | 4 ++--
 4 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/be/src/olap/rowset/unique_rowset_id_generator.cpp 
b/be/src/olap/rowset/unique_rowset_id_generator.cpp
index c21b8ca..9db7faf 100644
--- a/be/src/olap/rowset/unique_rowset_id_generator.cpp
+++ b/be/src/olap/rowset/unique_rowset_id_generator.cpp
@@ -64,6 +64,14 @@ bool UniqueRowsetIdGenerator::id_in_use(const RowsetId& 
rowset_id) const {
 }
 
 void UniqueRowsetIdGenerator::release_id(const RowsetId& rowset_id) {
+// Only release the rowsetid generated after this startup.
+// So we need to check version/mid/low part first
+if (rowset_id.version < _version) {
+return;
+}
+if ((rowset_id.mi != _backend_uid.hi) || (rowset_id.lo != 
_backend_uid.lo)) {
+return;
+}
 std::lock_guard l(_lock);
 _valid_rowset_id_hi.erase(rowset_id.hi);
 }
diff --git a/be/src/olap/rowset/unique_rowset_id_generator.h 
b/be/src/olap/rowset/unique_rowset_id_generator.h
index 0793af0..cb5d566 100644
--- a/be/src/olap/rowset/unique_rowset_id_generator.h
+++ b/be/src/olap/rowset/unique_rowset_id_generator.h
@@ -38,7 +38,13 @@ private:
 mutable SpinLock _lock;
 const UniqueId _backend_uid;
 const int64_t _version = 2; // modify it when create new version id 
generator
+// A monotonically increasing integer generator,
+// This integer will be part of a rowset id.
 std::atomic _inc_id;
+// Save the high part of rowset ids generated since last process startup.
+// Therefore, we cannot strictly rely on _valid_rowset_id_hi
+// to determine whether the rowset id is being used.
+// But to use id_in_use() and release_id() to check it.
 std::unordered_set _valid_rowset_id_hi;
 
 DISALLOW_COPY_AND_ASSIGN(UniqueRowsetIdGenerator);
diff --git 
a/fe/fe-core/src/test/java/org/apache/doris/catalog/CreateTableTest.java 
b/fe/fe-core/src/test/java/org/apache/doris/catalog/CreateTableTest.java
index 6bfdb87..0dabc18 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/CreateTableTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/CreateTableTest.java
@@ -68,7 +68,7 @@ public class CreateTableTest {
 public void testDuplicateCreateTable() throws Exception{
 // test
 Catalog catalog = Catalog.getCurrentCatalog();
-String sql = "create table if not exists test.tbl1\n" + "(k1 int, k2 
int)\n" + "duplicate key(k1)\n"
+String sql = "create table if not exists test.tbl1_colocate\n" + "(k1 
int, k2 int)\n" + "duplicate key(k1)\n"
 + "distributed by hash(k2) buckets 1\n" + 
"properties('replication_num' = '1','colocate_with'='test'); ";
 createTable(sql);
 Set tabletIdSetAfterCreateFirstTable = 
catalog.getTabletInvertedIndex().getReplicaMetaTable().rowKeySet();
diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/PlannerTest.java 
b/fe/fe-core/src/test/java/org/apache/doris/planner/PlannerTest.java
index eb843fc..135a50b 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/planner/PlannerTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/planner/PlannerTest.java
@@ -316,7 +316,7 @@ public class PlannerTest {
 }
 
 @Test
-public void testWithStmtSoltIsAllowNull() throws Exception {
+public void testWithStmtSlotIsAllowNull() throws Exception {
 // union
 String sql1 = "with a as (select NULL as user_id ), " +
 "b as ( select '543' as user_id) " +
@@ -326,7 +326,7 @@ public class PlannerTest {
 stmtExecutor1.execute();
 Planner planner1 = stmtExecutor1.planner();
 List fragments1 = planner1.getFragments();
-String plan1 = planner1.getExplainString(fragments1, new 
ExplainOptions(false, false));
+String plan1 = planner1.getExplainString(fragments1, new 
ExplainOptions(true, false));
 Assert.assertEquals(3, StringUtils.countMatches(plan1, 
"nullIndicatorBit=0"));
 }
 

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.

[incubator-doris] branch master updated: [Docs] fix a mistake in docs basic usage (#5525)

2021-03-22 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 4e0775c  [Docs] fix a mistake in docs basic usage (#5525)
4e0775c is described below

commit 4e0775c3dfc9b76ddcf195d736ea30bf07e22346
Author: Aimiyoo 
AuthorDate: Tue Mar 23 09:33:58 2021 +0800

[Docs] fix a mistake in docs basic usage (#5525)
---
 docs/en/getting-started/basic-usage.md| 2 +-
 docs/zh-CN/getting-started/basic-usage.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/en/getting-started/basic-usage.md 
b/docs/en/getting-started/basic-usage.md
index c4b08ae..fe978e5 100644
--- a/docs/en/getting-started/basic-usage.md
+++ b/docs/en/getting-started/basic-usage.md
@@ -338,7 +338,7 @@ MySQL> SELECT * FROM table1 LIMIT 3;
 |  5 |3 | 'helen'  |3 |
 |  3 |2 | 'tom'|2 |
 ++--+--+--+
-5 rows in set (0.01 sec)
+3 rows in set (0.01 sec)
 
 MySQL> SELECT * FROM table1 ORDER BY citycode;
 ++--+--+--+
diff --git a/docs/zh-CN/getting-started/basic-usage.md 
b/docs/zh-CN/getting-started/basic-usage.md
index 226af76..63ce0c3 100644
--- a/docs/zh-CN/getting-started/basic-usage.md
+++ b/docs/zh-CN/getting-started/basic-usage.md
@@ -337,7 +337,7 @@ MySQL> SELECT * FROM table1 LIMIT 3;
 |  5 |3 | 'helen'  |3 |
 |  3 |2 | 'tom'|2 |
 ++--+--+--+
-5 rows in set (0.01 sec)
+3 rows in set (0.01 sec)
 
 MySQL> SELECT * FROM table1 ORDER BY citycode;
 ++--+--+--+

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #5525: [Docs] fix a mistake in docs basic usage

2021-03-22 Thread GitBox


yangzhg merged pull request #5525:
URL: https://github.com/apache/incubator-doris/pull/5525


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg closed issue #5532: RollupJobV2 support identify cluster when relpay

2021-03-22 Thread GitBox


yangzhg closed issue #5532:
URL: https://github.com/apache/incubator-doris/issues/5532


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #5533: (#5532) RollupJobV2 support identify cluster when replay

2021-03-22 Thread GitBox


yangzhg merged pull request #5533:
URL: https://github.com/apache/incubator-doris/pull/5533


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg closed issue #5537: [Bug] When using multi bytes separator in load process, the last column can not be read

2021-03-22 Thread GitBox


yangzhg closed issue #5537:
URL: https://github.com/apache/incubator-doris/issues/5537


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #5534: [Bug] Fix bug that the last column may be null when using multibytes separator

2021-03-22 Thread GitBox


yangzhg merged pull request #5534:
URL: https://github.com/apache/incubator-doris/pull/5534


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #5535: [Docs] Fix typo in docs zh-CN administrator-guide dynamic-partition.md

2021-03-22 Thread GitBox


yangzhg merged pull request #5535:
URL: https://github.com/apache/incubator-doris/pull/5535


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #5536: [Doc] Modify doc for replace table

2021-03-22 Thread GitBox


yangzhg merged pull request #5536:
URL: https://github.com/apache/incubator-doris/pull/5536


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated (4e0775c -> ae3e9df)

2021-03-22 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.


from 4e0775c  [Docs] fix a mistake in docs basic usage (#5525)
 add ae3e9df  (#5532) RollupJobV2 support identify cluster when replay 
(#5533)

No new revisions were added by this update.

Summary of changes:
 .../src/main/java/org/apache/doris/alter/RollupJobV2.java| 12 ++--
 1 file changed, 10 insertions(+), 2 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg closed issue #5546: [Compile] Some compile warning in gcc > 8

2021-03-22 Thread GitBox


yangzhg closed issue #5546:
URL: https://github.com/apache/incubator-doris/issues/5546


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #5547: [Compile][Refactor] fix some warning in gcc higher than 7

2021-03-22 Thread GitBox


yangzhg merged pull request #5547:
URL: https://github.com/apache/incubator-doris/pull/5547


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [Bug] Fix bug that the last column may be null when using multibytes separator (#5534)

2021-03-22 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new cef3cbc  [Bug] Fix bug that the last column may be null when using 
multibytes separator (#5534)
cef3cbc is described below

commit cef3cbc53ac9a23cd578ac92af9bc4125d9ae837
Author: Mingyu Chen 
AuthorDate: Tue Mar 23 09:35:30 2021 +0800

[Bug] Fix bug that the last column may be null when using multibytes 
separator (#5534)
---
 be/src/exec/broker_scanner.cpp |  44 +---
 be/test/exec/CMakeLists.txt|   1 +
 be/test/exec/broker_scanner_test.cpp   |  42 
 be/test/exec/multi_bytes_separator_test.cpp| 117 +
 .../test_data/broker_scanner/multi_bytes_sep.csv   |   1 +
 .../doris/analysis/CreateRoutineLoadStmt.java  |   2 +-
 6 files changed, 192 insertions(+), 15 deletions(-)

diff --git a/be/src/exec/broker_scanner.cpp b/be/src/exec/broker_scanner.cpp
index e14f1fb..c4254c0 100644
--- a/be/src/exec/broker_scanner.cpp
+++ b/be/src/exec/broker_scanner.cpp
@@ -305,23 +305,39 @@ void BrokerScanner::close() {
 
 void BrokerScanner::split_line(const Slice& line, std::vector* values) {
 const char* value = line.data;
-size_t i = 0;
-// TODO improve the performance
-while (i < line.size) {
-if (i + _value_separator_length <= line.size) {
-if (_value_separator.compare(0, _value_separator_length, line.data 
+ i,
- _value_separator_length) == 0) {
-values->emplace_back(value, line.data + i - value);
-value = line.data + i + _value_separator_length;
-i += _value_separator_length;
-} else {
-++i;
-}
+size_t start = 0; // point to the start pos of next col value.
+size_t curpos= 0; // point to the start pos of separator matching sequence.
+size_t p1 = 0;// point to the current pos of separator matching 
sequence.
+
+// Separator: 
+// 
+//   curpos
+// ▼
+// 
+//   10002000
+//   ▲   ▲
+// Start │
+//   p1
+
+while (curpos < line.size) {
+if (*(value + curpos + p1) != _value_separator[p1]) {
+// Not match, move forward:
+curpos += (p1 == 0 ? 1 : p1);
+p1 = 0;
 } else {
-break;
+p1++;
+if (p1 == _value_separator_length) {
+// Match a separator
+values->emplace_back(value + start, curpos - start);
+start = curpos + _value_separator_length;
+curpos = start;
+p1 = 0;
+}
 }
 }
-values->emplace_back(value, line.data + i - value);
+
+CHECK(curpos == line.size) << curpos << " vs " <<  line.size;
+values->emplace_back(value + start, curpos - start);
 }
 
 void BrokerScanner::fill_fix_length_string(const Slice& value, MemPool* pool, 
char** new_value_p,
diff --git a/be/test/exec/CMakeLists.txt b/be/test/exec/CMakeLists.txt
index 7cd6a93..a35c51f 100644
--- a/be/test/exec/CMakeLists.txt
+++ b/be/test/exec/CMakeLists.txt
@@ -73,3 +73,4 @@ ADD_BE_TEST(unix_odbc_test)
 #ADD_BE_TEST(schema_scanner/schema_collations_scanner_test)
 #ADD_BE_TEST(schema_scanner/schema_charsets_scanner_test)
 ADD_BE_TEST(s3_reader_test)
+ADD_BE_TEST(multi_bytes_separator_test)
diff --git a/be/test/exec/broker_scanner_test.cpp 
b/be/test/exec/broker_scanner_test.cpp
index 98f163b..ade80a1 100644
--- a/be/test/exec/broker_scanner_test.cpp
+++ b/be/test/exec/broker_scanner_test.cpp
@@ -657,6 +657,48 @@ TEST_F(BrokerScannerTest, normal9) {
 ASSERT_TRUE(eof);
 }
 
+TEST_F(BrokerScannerTest, multi_bytes_1) {
+std::vector ranges;
+TBrokerRangeDesc range;
+range.path = "./be/test/exec/test_data/broker_scanner/multi_bytes_sep.csv";
+range.start_offset = 0;
+range.size = 18;
+range.splittable = true;
+range.file_type = TFileType::FILE_LOCAL;
+range.format_type = TFileFormatType::FORMAT_CSV_PLAIN;
+ranges.push_back(range);
+
+_params.column_separator_str = "";
+_params.line_delimiter_str = "BB";
+_params.column_separator_length = 4;
+_params.line_delimiter_length = 2;
+BrokerScanner scanner(&_runtime_state, _profile, _params, ranges, 
_addresses, _pre_filter, &_counter);
+auto st = scanner.open();
+ASSERT_TRUE(st.ok());
+
+MemPool tuple_pool(_tracker.get());
+Tuple* tuple = (Tuple*)tuple_pool.allocate(20);
+bool eof = false;
+// 4,5,6
+st = scanner.get_next(tuple, &tuple_pool, &eof);
+ASSERT_TRUE(st.ok());
+ASSERT_FALSE(eof);
+ASSERT_EQ(4, *(int*)tuple->get_slot(0));
+ASSERT_EQ(5, *(int*)tuple->get_slot(4));
+ASSERT_EQ(6, *(int*)tuple->get_slot(8));
+// 1,2,

[incubator-doris] branch master updated: Update dynamic-partition.md (#5535)

2021-03-22 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 71b6e93  Update dynamic-partition.md (#5535)
71b6e93 is described below

commit 71b6e93424f56fb03489aea21c3039b281f0ec5b
Author: zhangboya1 <49148006+zhangbo...@users.noreply.github.com>
AuthorDate: Tue Mar 23 09:35:51 2021 +0800

Update dynamic-partition.md (#5535)

[Docs] Fix typo in docs zh-CN  administrator-guide dynamic-partition.md
---
 docs/zh-CN/administrator-guide/dynamic-partition.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/zh-CN/administrator-guide/dynamic-partition.md 
b/docs/zh-CN/administrator-guide/dynamic-partition.md
index 5a83b02..2e84276 100644
--- a/docs/zh-CN/administrator-guide/dynamic-partition.md
+++ b/docs/zh-CN/administrator-guide/dynamic-partition.md
@@ -257,7 +257,7 @@ ALTER TABLE tbl1 SET
 );
 ```
 
-某些属性的修改可能会可能会产生冲突。假设之前分区粒度为 DAY,并且已经创建了如下分区:
+某些属性的修改可能会产生冲突。假设之前分区粒度为 DAY,并且已经创建了如下分区:
 
 ```
 p20200519: ["2020-05-19", "2020-05-20")

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [Doc] Modify doc for replace table (#5536)

2021-03-22 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new c8afd56  [Doc] Modify doc for replace table (#5536)
c8afd56 is described below

commit c8afd5646ca0f0711e044f615cb3f978b2cc6234
Author: Mingyu Chen 
AuthorDate: Tue Mar 23 09:36:11 2021 +0800

[Doc] Modify doc for replace table (#5536)
---
 docs/en/administrator-guide/alter-table/alter-table-replace-table.md| 2 +-
 docs/zh-CN/administrator-guide/alter-table/alter-table-replace-table.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git 
a/docs/en/administrator-guide/alter-table/alter-table-replace-table.md 
b/docs/en/administrator-guide/alter-table/alter-table-replace-table.md
index 15d3fe0..0253298 100644
--- a/docs/en/administrator-guide/alter-table/alter-table-replace-table.md
+++ b/docs/en/administrator-guide/alter-table/alter-table-replace-table.md
@@ -53,7 +53,7 @@ Suppose you want to replace table A with table B, and `swap` 
is `true`, the oper
 1. Rename table B to table A.
 2. Rename table A to table B.
 
-If `swap` is `true`, the operation is as follows:
+If `swap` is `false`, the operation is as follows:
 
 1. Drop table A.
 2. Rename table B to table A.
diff --git 
a/docs/zh-CN/administrator-guide/alter-table/alter-table-replace-table.md 
b/docs/zh-CN/administrator-guide/alter-table/alter-table-replace-table.md
index 3deedf7..ce47723 100644
--- a/docs/zh-CN/administrator-guide/alter-table/alter-table-replace-table.md
+++ b/docs/zh-CN/administrator-guide/alter-table/alter-table-replace-table.md
@@ -54,7 +54,7 @@ ALTER TABLE [db.]tbl1 REPLACE WITH TABLE tbl2
 1. 将表 B 重名为表 A。
 2. 将表 A 重名为表 B。
 
-如果 `swap` 为 `true`,则操作如下:
+如果 `swap` 为 `false`,则操作如下:
 
 1. 删除表 A。
 2. 将表 B 重名为表 A。

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [Refactor] fix some warning in gcc higher than 7 make decimal12_t as a POD type (#5547)

2021-03-22 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new bfeb717  [Refactor] fix some warning in gcc higher than 7 make 
decimal12_t as a POD type (#5547)
bfeb717 is described below

commit bfeb717abeb910b1c7614a873f4e601855a6d655
Author: stdpain <34912776+stdp...@users.noreply.github.com>
AuthorDate: Tue Mar 23 09:37:10 2021 +0800

[Refactor] fix some warning in gcc higher than 7 make decimal12_t as a POD 
type (#5547)
---
 be/src/olap/decimal12.h|  19 +--
 be/src/olap/key_coder.h|   2 +-
 be/src/olap/reader.cpp | 180 ++---
 be/src/olap/rowset/column_reader.h |   2 +-
 be/test/olap/comparison_predicate_test.cpp |   6 +-
 be/test/olap/decimal12_test.cpp|   4 +-
 be/test/olap/in_list_predicate_test.cpp|   6 +-
 be/test/olap/key_coder_test.cpp|   8 +-
 be/test/olap/row_cursor_test.cpp   |   6 +-
 .../rowset/segment_v2/bitshuffle_page_test.cpp |   6 +-
 .../bloom_filter_index_reader_writer_test.cpp  |   4 +-
 .../segment_v2/column_reader_writer_test.cpp   |   4 +-
 be/test/olap/schema_change_test.cpp|   2 +-
 be/test/olap/storage_types_test.cpp|   2 +-
 14 files changed, 118 insertions(+), 133 deletions(-)

diff --git a/be/src/olap/decimal12.h b/be/src/olap/decimal12.h
index 2dc4d29..dcd0fab 100644
--- a/be/src/olap/decimal12.h
+++ b/be/src/olap/decimal12.h
@@ -27,16 +27,6 @@ namespace doris {
 
 // the sign of integer must be same as fraction
 struct decimal12_t {
-decimal12_t() : integer(0), fraction(0) {}
-decimal12_t(int64_t int_part, int32_t frac_part) {
-integer = int_part;
-fraction = frac_part;
-}
-
-decimal12_t(const decimal12_t& value) {
-integer = value.integer;
-fraction = value.fraction;
-}
 
 decimal12_t& operator+=(const decimal12_t& value) {
 fraction += value.fraction;
@@ -62,13 +52,6 @@ struct decimal12_t {
 return *this;
 }
 
-// call field::copy
-decimal12_t& operator=(const decimal12_t& value) {
-integer = value.integer;
-fraction = value.fraction;
-return *this;
-}
-
 bool operator<(const decimal12_t& value) const { return cmp(value) < 0; }
 
 bool operator<=(const decimal12_t& value) const { return cmp(value) <= 0; }
@@ -156,6 +139,8 @@ struct decimal12_t {
 int32_t fraction;
 } __attribute__((packed));
 
+static_assert(std::is_trivial::value, "decimal12_t should be a 
POD type");
+
 inline std::ostream& operator<<(std::ostream& os, const decimal12_t& val) {
 os << val.to_string();
 return os;
diff --git a/be/src/olap/key_coder.h b/be/src/olap/key_coder.h
index 634499f..520b2a0 100644
--- a/be/src/olap/key_coder.h
+++ b/be/src/olap/key_coder.h
@@ -186,7 +186,7 @@ public:
 
 static Status decode_ascending(Slice* encoded_key, size_t index_size, 
uint8_t* cell_ptr,
MemPool* pool) {
-decimal12_t decimal_val;
+decimal12_t decimal_val = {0, 0};
 
RETURN_IF_ERROR(KeyCoderTraits::decode_ascending(
 encoded_key, sizeof(decimal_val.integer), 
(uint8_t*)&decimal_val.integer, pool));
 RETURN_IF_ERROR(KeyCoderTraits::decode_ascending(
diff --git a/be/src/olap/reader.cpp b/be/src/olap/reader.cpp
index c56160b..cb2c6e2 100644
--- a/be/src/olap/reader.cpp
+++ b/be/src/olap/reader.cpp
@@ -601,95 +601,95 @@ void Reader::_init_conditions_param(const ReaderParams& 
read_params) {
 }
 }
 
-#define COMPARISON_PREDICATE_CONDITION_VALUE(NAME, PREDICATE)  
\
-ColumnPredicate* Reader::_new_##NAME##_pred(const TabletColumn& column, 
int index, \
-const std::string& cond, bool 
opposite) const {   \
-ColumnPredicate* predicate = nullptr;  
\
-switch (column.type()) {   
\
-case OLAP_FIELD_TYPE_TINYINT: {
\
-std::stringstream ss(cond);
\
-int32_t value = 0; 
\
-ss >> value;   
\
-predicate = new PREDICATE(index, value, opposite); 
  \
-break; 
\
-}  
\
-case OLAP_FIELD_TYPE

[GitHub] [incubator-doris] weihongkai2008 opened a new issue #5556: In some cases, be will do the useless cumulative compaction after restart

2021-03-22 Thread GitBox


weihongkai2008 opened a new issue #5556:
URL: https://github.com/apache/incubator-doris/issues/5556


   When doris rewrites the t-1 day data every day, many tablete versions become 
[0-1],[2-2]. And these data will not be changed in the future. Under the 
current cumulative compaction logic, after doris be restarts, it will 
re-execute the compaction of the rowset of version [2-2]. At the same time, the 
number of such tablets is very large, resulting in a long query delay after be 
restarted. long. These compactions are not necessary because they have been 
done before the restart of dorie be.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] HappenLee commented on pull request #5554: Add fmt library to speed up mysql text result serialization

2021-03-22 Thread GitBox


HappenLee commented on pull request #5554:
URL: https://github.com/apache/incubator-doris/pull/5554#issuecomment-804595075


   Nice job! could you please update the docker env to add fmt lib.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] pigdance opened a new issue #5557: 1、mysql开启binlog,写入kafka消息队列。但是doris 无法解析binlog的数据,有什么好的办法做mysql实时同步到doris吗?

2021-03-22 Thread GitBox


pigdance opened a new issue #5557:
URL: https://github.com/apache/incubator-doris/issues/5557


   Reason: actual column number is less than schema column number. actual 
number: 1 sep:, schema number: 4; . src line: 
[{"data":[{"id":"26","name":"222","status":"22","date":"-00-00"}],"database":"test","es":1616467578000,"id":7,"isDdl":false,"mysqlType":{"id":"bigint(20)","name":"varchar(45)","status":"varchar(45)","date":"date"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":-5,"name":12,"status":12,"date":91},"table":"canal_test","ts":1616467578825,"type":"INSERT"}];
 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] HappenLee commented on a change in pull request #5521: [Colocate plan][Step1] Colocate join covers more situations

2021-03-22 Thread GitBox


HappenLee commented on a change in pull request #5521:
URL: https://github.com/apache/incubator-doris/pull/5521#discussion_r599287848



##
File path: fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
##
@@ -30,15 +30,15 @@
 import org.apache.doris.thrift.TPlan;
 import org.apache.doris.thrift.TPlanNode;
 
-import org.apache.logging.log4j.LogManager;
-import org.apache.logging.log4j.Logger;
-

Review comment:
   OK




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] HappenLee commented on pull request #5521: [Colocate plan][Step1] Colocate join covers more situations

2021-03-22 Thread GitBox


HappenLee commented on pull request #5521:
URL: https://github.com/apache/incubator-doris/pull/5521#issuecomment-804638447


   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] HappenLee removed a comment on pull request #5521: [Colocate plan][Step1] Colocate join covers more situations

2021-03-22 Thread GitBox


HappenLee removed a comment on pull request #5521:
URL: https://github.com/apache/incubator-doris/pull/5521#issuecomment-804638447


   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] gaodayue commented on pull request #5554: Add fmt library to speed up mysql text result serialization

2021-03-22 Thread GitBox


gaodayue commented on pull request #5554:
URL: https://github.com/apache/incubator-doris/pull/5554#issuecomment-804657626


   > Nice job! could you please update the docker env to add fmt lib.
   
   @HappenLee thanks for your reply! In fact I don't know the process about 
adding a new third party library, could you provide some helps? It would be 
better if we documented it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org