[GitHub] [incubator-doris] weizuo93 opened a new pull request #5563: [Refactor] refactor the destructor in olap_meta.cpp

2021-03-24 Thread GitBox


weizuo93 opened a new pull request #5563:
URL: https://github.com/apache/incubator-doris/pull/5563


   ## Proposed changes
   
   refactor the destructor in olap_meta.cpp
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [x ] Bugfix (non-breaking change which fixes an issue)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [x ] Code refactor (Modify the code structure, format the code, etc...)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] acelyc111 commented on a change in pull request #5488: [Feature][Stream Load] Add "show stream_load" function to show the result of stream load

2021-03-24 Thread GitBox


acelyc111 commented on a change in pull request #5488:
URL: https://github.com/apache/incubator-doris/pull/5488#discussion_r600261821



##
File path: be/src/http/action/stream_load.cpp
##
@@ -132,9 +134,19 @@ void StreamLoadAction::handle(HttpRequest* req) {
 str = str + '\n';
 HttpChannel::send_reply(req, str);
 
+auto stream_load_record = 
StorageEngine::instance()->get_stream_load_record();
+if (stream_load_record != nullptr) {
+auto st = 
stream_load_record->put(ToStringFromUnixMicros(ctx->start_micros + 
ctx->load_cost_micros) + "_" + ctx->label, str);
+if (st.ok()) {
+LOG(INFO) << "put stream_load_record rocksdb successfully. label: 
" << ctx->label;

Review comment:
   Better to log the whole key, then we can check it in rocksdb.

##
File path: be/src/olap/olap_server.cpp
##
@@ -118,6 +118,13 @@ Status StorageEngine::start_bg_threads() {
 LOG(INFO) << "path scan/gc threads started. number:" << 
get_stores().size();
 }
 
+// stream load record clean thread
+RETURN_IF_ERROR(Thread::create(

Review comment:
   No need to create a thread to gc expired data, you can create a 
Compaction-Filter on this rocksdb instance.
   https://github.com/facebook/rocksdb/wiki/Compaction-Filter

##
File path: be/src/olap/storage_engine.cpp
##
@@ -225,6 +226,35 @@ Status StorageEngine::_init_store_map() {
 for (auto store : tmp_stores) {
 _store_map.emplace(store->path(), store);
 }
+
+auto st = _init_stream_load_record();
+if (!st.ok()) {
+LOG(WARNING) << "status=" << st.to_string();
+}
+
+return Status::OK();
+}
+
+Status StorageEngine::_init_stream_load_record() {
+std::string stream_load_record_path;
+if (_store_map.size() > 0) {

Review comment:
   Is it valid when _store_map is empty? Would it be better to explicit 
define the path in config file?

##
File path: be/src/runtime/stream_load/stream_load_record.h
##
@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include 
+#include 
+
+#include "rocksdb/db.h"
+
+#pragma once
+
+namespace doris {
+
+class Status;
+
+class StreamLoadRecord {
+public:
+StreamLoadRecord(const std::string& root_path);
+
+virtual ~StreamLoadRecord();
+
+Status init();
+
+Status put(const std::string& key, const std::string& value);
+
+Status get_batch(const std::string& start, const int batch_size, 
std::map &stream_load_records);
+
+Status clean_expired_stream_load_record();
+
+private:
+std::string _root_path;
+rocksdb::DB* _db;
+std::vector _handles;
+
+enum ColumnFamilyIndex {
+DEFAULT_COLUMN_FAMILY_INDEX = 0,

Review comment:
   where does the default column family used for?

##
File path: be/src/http/action/stream_load.cpp
##
@@ -132,9 +134,19 @@ void StreamLoadAction::handle(HttpRequest* req) {
 str = str + '\n';
 HttpChannel::send_reply(req, str);
 
+auto stream_load_record = 
StorageEngine::instance()->get_stream_load_record();
+if (stream_load_record != nullptr) {
+auto st = 
stream_load_record->put(ToStringFromUnixMicros(ctx->start_micros + 
ctx->load_cost_micros) + "_" + ctx->label, str);
+if (st.ok()) {
+LOG(INFO) << "put stream_load_record rocksdb successfully. label: 
" << ctx->label;
+}
+} else {
+LOG(WARNING) << "stream_load_record is null.";

Review comment:
   ```suggestion
   LOG(WARNING) << "put stream_load_record rocksdb failed, status = " 
<< st.ToString();
   ```
   And log the key.

##
File path: be/src/runtime/stream_load/stream_load_record.cpp
##
@@ -0,0 +1,137 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// 

[GitHub] [incubator-doris] caiconghui opened a new pull request #5564: [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout

2021-03-24 Thread GitBox


caiconghui opened a new pull request #5564:
URL: https://github.com/apache/incubator-doris/pull/5564


   ## Proposed changes
   
   This PR is to fix bug that call frontend service failed when rpc_timeout is 
equal with publish_timeout and fix some small issue with log
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [x] Bugfix (non-breaking change which fixes an issue)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [ ] Code refactor (Modify the code structure, format the code, etc...)
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature 
there in detail
   - [ ] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
d...@doris.apache.org by explaining why you chose the solution you did and what 
alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] caiconghui opened a new issue #5565: [Bug] we may see call frontend service failed when rpc_timeout is equal with publish_timeout

2021-03-24 Thread GitBox


caiconghui opened a new issue #5565:
URL: https://github.com/apache/incubator-doris/issues/5565


   **Describe the bug**
   when rpc_timeout is equal with publish_timeout,we may see that call frontend 
service failed  when cluster publish transaction timeout, and then program will 
try to rollback transaction and user will retry stream load again, which will 
bring the cluster more pressure
   
   **To Reproduce**
   1. put many stream loads at the same time
   
   **Expected behavior**
   if publish transaction timeout, we should return publis timeout instead of 
call frontend service failed.
   
   **Screenshots**
   If applicable, add screenshots to help explain your problem.
   
   **Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
   
   **Smartphone (please complete the following information):**
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
   
   **Additional context**
   Add any other context about the problem here.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg commented on a change in pull request #5559: [Refactor] make uint24_t,OLAPIndexFixedHeader as a POD type

2021-03-24 Thread GitBox


yangzhg commented on a change in pull request #5559:
URL: https://github.com/apache/incubator-doris/pull/5559#discussion_r600357835



##
File path: be/src/util/coding.h
##
@@ -171,14 +171,14 @@ extern const uint8_t* decode_varint64_ptr(const uint8_t* 
p, const uint8_t* limit
 
 template 
 inline void put_varint32(T* dst, uint32_t v) {
-uint8_t buf[5];
+uint8_t buf[16];

Review comment:
   #pragma GCC diagnostic ignored "-Warray-bounds" is better




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] acelyc111 commented on a change in pull request #5452: [Audit][Stream Load] Support audit function for stream load

2021-03-24 Thread GitBox


acelyc111 commented on a change in pull request #5452:
URL: https://github.com/apache/incubator-doris/pull/5452#discussion_r600381383



##
File path: be/src/common/config.h
##
@@ -348,6 +348,12 @@ CONF_mInt32(streaming_load_rpc_max_alive_time_sec, "1200");
 CONF_Int32(tablet_writer_open_rpc_timeout_sec, "60");
 // You can ignore brpc error '[E1011]The server is overcrowded' when writing 
data.
 CONF_mBool(tablet_writer_ignore_eovercrowded, "false");
+// batch size of stream load record reported to FE
+CONF_mInt32(stream_load_record_batch_size, "50");
+// expire time of stream load record in rocksdb.
+CONF_mInt32(stream_load_record_expire_time_secs, "28800");

Review comment:
   stream_load_record_expire_time_secs is not configurable at runtime.

##
File path: be/src/runtime/stream_load/stream_load_record.cpp
##
@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/stream_load/stream_load_record.h"
+
+#include "common/config.h"
+#include "common/status.h"
+#include "rocksdb/db.h"
+
+#include "rocksdb/options.h"
+#include "rocksdb/slice.h"
+#include "rocksdb/slice_transform.h"
+#include "rocksdb/utilities/db_ttl.h"
+#include "util/time.h"
+
+
+namespace doris {
+const std::string STREAM_LOAD_POSTFIX = "/stream_load";
+
+StreamLoadRecord::StreamLoadRecord(const std::string& root_path)
+: _root_path(root_path),
+  _db(nullptr),
+  _last_compaction_time(UnixMillis()) {
+}
+
+StreamLoadRecord::~StreamLoadRecord() {
+if (_db != nullptr) {
+for (auto handle : _handles) {
+_db->DestroyColumnFamilyHandle(handle);
+handle = nullptr;
+}
+delete _db;
+_db= nullptr;
+}
+}
+
+Status StreamLoadRecord::init() {
+// init db
+rocksdb::DBOptions options;
+options.IncreaseParallelism();
+options.create_if_missing = true;
+options.create_missing_column_families = true;
+std::string db_path = _root_path + STREAM_LOAD_POSTFIX;
+std::vector column_families;
+// default column family is required
+column_families.emplace_back(DEFAULT_COLUMN_FAMILY, 
rocksdb::ColumnFamilyOptions());
+// stream load column family
+column_families.emplace_back(STREAM_LOAD_COLUMN_FAMILY, 
rocksdb::ColumnFamilyOptions());
+std::vector ttls = {config::stream_load_record_expire_time_secs, 
config::stream_load_record_expire_time_secs};
+rocksdb::Status s = rocksdb::DBWithTTL::Open(options, db_path, 
column_families, &_handles, &_db, ttls);
+
+if (!s.ok() || _db == nullptr) {
+LOG(WARNING) << "rocks db open failed, reason:" << s.ToString();
+return Status::InternalError("Stream load record rocksdb open failed");
+}
+return Status::OK();
+}
+
+Status StreamLoadRecord::put(const std::string& key, const std::string& value) 
{
+rocksdb::ColumnFamilyHandle* handle = _handles[1];
+rocksdb::WriteOptions write_options;
+write_options.sync = false;
+rocksdb::Status s = _db->Put(write_options, handle, rocksdb::Slice(key), 
rocksdb::Slice(value));
+if (!s.ok()) {
+LOG(WARNING) << "rocks db put key:" << key << " failed, reason:" << 
s.ToString();
+return Status::InternalError("Stream load record rocksdb put failed");
+}
+
+if ((UnixMillis() - _last_compaction_time) / 1000 > 
config::clean_stream_load_record_interval_secs) {
+rocksdb::CompactRangeOptions options;
+s = _db->CompactRange(options, _handles[1], nullptr, nullptr);

Review comment:
   CompactRange may consume a long time, better to not invoke it on load 
path.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] JNSimba edited a comment on pull request #5375: [Feature] Add flink datastream connector example (#5371)

2021-03-24 Thread GitBox


JNSimba edited a comment on pull request #5375:
URL: https://github.com/apache/incubator-doris/pull/5375#issuecomment-802749846


   Refer to [User-defined Sources & 
Sinks](https://ci.apache.org/projects/flink/flink-docs-stable/zh/dev/table/sourceSinks.html)
 And flink-connector-jdbc, add flink sql to create doris table, and write doris 
table through streamload。
   
   eg:
   ```java
EnvironmentSettings settings = EnvironmentSettings.newInstance()
   .useBlinkPlanner()
   .inStreamingMode()
   .build();
   TableEnvironment tEnv = TableEnvironment.create(settings);
   tEnv.getConfig().getConfiguration().setString("job.name","test");
   
   tEnv.executeSql(
   "CREATE TABLE doris_test (" +
   "name STRING," +
   "age INT," +
   "price DECIMAL(5,2)," +
   "sale DOUBLE" +
   ") " +
   "WITH (\n" +
   "  'connector' = 'doris',\n" +
   "  'fenodes' = 'FE_IP:8030',\n" +
   "  'table.identifier' = 
'demo.doris_test_source_2',\n" +
   "  'username' = 'root',\n" +
   "  'password' = ''" +
   ")");
   tEnv.executeSql(
   "CREATE TABLE doris_test_sink (" +
   "name STRING," +
   "age INT," +
   "price DECIMAL(5,2)," +
   "sale DOUBLE" +
   ") " +
   "WITH (\n" +
   "  'connector' = 'doris',\n" +
   "  'fenodes' = 'FE_IP:8030',\n" +
   "  'table.identifier' = 'demo.doris_test_sink_2',\n" 
+
   "  'username' = 'root',\n" +
   "  'password' = ''\n" +
   ")");
   
   tEnv.executeSql("INSERT INTO doris_test_sink select 
name,age,price,sale from doris_test");
   ```
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg merged pull request #5394: update gcc to gcc 10 and support c++17

2021-03-24 Thread GitBox


yangzhg merged pull request #5394:
URL: https://github.com/apache/incubator-doris/pull/5394


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: update gcc to gcc 10 and support c++17 (#5394)

2021-03-24 Thread yangzhg
This is an automated email from the ASF dual-hosted git repository.

yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new ad67dd3  update gcc to gcc 10 and support c++17 (#5394)
ad67dd3 is described below

commit ad67dd34a04c1ca960cff38e5b335b30fc7d559f
Author: stdpain <34912776+stdp...@users.noreply.github.com>
AuthorDate: Thu Mar 25 09:30:38 2021 +0800

update gcc to gcc 10 and support c++17 (#5394)

* update gcc to gcc 10 and support c++17
update brpc to 0.9.7
update boost to 1.73
remove third-party boost 1.54 for mysql

* update cmake version

* ignore jdk version

* remove unused patch

* avoid use SYS_getrandom call
---
 be/CMakeLists.txt  |  23 +-
 be/src/exec/CMakeLists.txt |   2 -
 be/src/exec/decompressor.cpp   |   2 +-
 be/src/exec/olap_common.h  |  11 +-
 be/src/exec/scan_node.cpp  |   2 -
 be/src/http/default_path_handlers.cpp  |   6 +-
 be/src/http/ev_http_server.cpp |   2 +-
 be/src/olap/rowset/rowset.h|   1 +
 be/src/olap/rowset/unique_rowset_id_generator.h|   2 +
 be/src/olap/storage_engine.cpp |   2 +
 be/src/olap/task/engine_batch_load_task.cpp|   2 +-
 be/src/runtime/buffered_block_mgr2.cc  |   2 +-
 be/src/runtime/buffered_tuple_stream2.cc   |   2 +-
 be/src/runtime/buffered_tuple_stream3.cc   |   2 +-
 be/src/runtime/bufferpool/buffer_allocator.cc  |   5 +-
 be/src/runtime/bufferpool/buffer_pool.cc   |   8 +-
 be/src/runtime/client_cache.h  |   6 +-
 be/src/runtime/data_stream_mgr.h   |   2 +-
 be/src/runtime/data_stream_recvr.cc|  10 +-
 be/src/runtime/fragment_mgr.cpp|  26 +--
 be/src/runtime/result_buffer_mgr.cpp   |   2 +-
 be/src/runtime/snapshot_loader.cpp |   2 +-
 be/src/runtime/spill_sorter.cc |   2 +-
 be/src/runtime/user_function_cache.cpp |   1 +
 be/src/util/thrift_server.cpp  |   5 +-
 be/test/exprs/topn_function_test.cpp   |   4 +-
 env.sh |  19 ++
 .../main/java/org/apache/doris/common/Config.java  |   2 +-
 thirdparty/build-thirdparty.sh | 109 +-
 thirdparty/download-thirdparty.sh  |  10 +-
 thirdparty/patches/incubator-brpc-0.9.5.patch  | 237 -
 thirdparty/patches/mysql-5.7.18-boost.patch|  20 ++
 thirdparty/vars.sh |  23 +-
 33 files changed, 183 insertions(+), 371 deletions(-)

diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 8be4883..e520b3c 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -15,7 +15,7 @@
 # specific language governing permissions and limitations
 # under the License.
 
-cmake_minimum_required(VERSION 3.12.0)
+cmake_minimum_required(VERSION 3.19.2)
 
 # set CMAKE_C_COMPILER, this must set before project command
 if (DEFINED ENV{DORIS_GCC_HOME})
@@ -130,14 +130,13 @@ endif()
 set(Boost_DEBUG FALSE)
 set(Boost_USE_MULTITHREADED ON)
 set(BOOST_ROOT ${THIRDPARTY_DIR})
+set(Boost_NO_BOOST_CMAKE OFF)
 
 if (NOT APPLE)
-find_package(Boost 1.55.0 REQUIRED COMPONENTS thread regex filesystem 
system date_time program_options)
+find_package(Boost 1.73.0 REQUIRED COMPONENTS regex system filesystem 
thread date_time program_options)
 else()
-find_package(Boost 1.55.0 COMPONENTS thread regex filesystem system 
date_time program_options)
+find_package(Boost 1.73.0 COMPONENTS thread regex system filesystem 
date_time program_options)
 endif()
-include_directories(${Boost_INCLUDE_DIRS})
-message(STATUS ${Boost_LIBRARIES})
 
 set(GPERFTOOLS_HOME "${THIRDPARTY_DIR}/gperftools")
 
@@ -313,10 +312,13 @@ check_function_exists(sched_getcpu HAVE_SCHED_GETCPU)
 #  -fno-omit-frame-pointers: Keep frame pointer for functions in register
 set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -Wall -Wno-sign-compare 
-Wno-unknown-pragmas -pthread")
 set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -fno-strict-aliasing 
-fno-omit-frame-pointer")
-set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -std=gnu++11 -D__STDC_FORMAT_MACROS")
+set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -std=gnu++17 -D__STDC_FORMAT_MACROS")
 set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -Wno-deprecated -Wno-vla")
 set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} 
-DBOOST_DATE_TIME_POSIX_TIME_STD_CONFIG")
 set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -DBOOST_SYSTEM_NO_DEPRECATED")
+# https://github.com/boostorg/uuid/issues/92
+# We need to avoid using SYS_getrandom system calls
+set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} 
-DBOOST_UUID_RANDOM_PROVIDER_FORCE_

[GitHub] [incubator-doris] yangzhg opened a new pull request #5566: [DOC] Update gcc10 compile docs

2021-03-24 Thread GitBox


yangzhg opened a new pull request #5566:
URL: https://github.com/apache/incubator-doris/pull/5566


   ## Proposed changes
   
   Update GCC10 compile docs
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [x] Documentation Update (if none of the other choices apply)
   - [ ] Code refactor (Modify the code structure, format the code, etc...)
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature 
there in detail
   - [ ] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
d...@doris.apache.org by explaining why you chose the solution you did and what 
alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] stdpain commented on a change in pull request #5564: [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout

2021-03-24 Thread GitBox


stdpain commented on a change in pull request #5564:
URL: https://github.com/apache/incubator-doris/pull/5564#discussion_r600993661



##
File path: fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java
##
@@ -94,9 +94,9 @@ public TMasterResult finishTask(TFinishTaskRequest request) {
 // check task status
 // retry task by report process
 TStatus taskStatus = request.getTaskStatus();
-LOG.debug("get task report: {}", request.toString());
+LOG.debug("get task report: {}", request);

Review comment:
   It's a better way to use 
   ```java
   if (LOG.isDebugEnabled()) {
   LOG.debug("get task report: {}", request);
   }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg opened a new issue #5567: Remove some glibc-compatibility file for old kernel

2021-03-24 Thread GitBox


yangzhg opened a new issue #5567:
URL: https://github.com/apache/incubator-doris/issues/5567


   In some old kernel SYS_getrandom and SYS_getcpu is not exist,  and we don't  
use those in our code, so we should remove those files to compatible with old 
kernels


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] yangzhg opened a new pull request #5568: [refactor] Remove some unused file for compatible to older kernels

2021-03-24 Thread GitBox


yangzhg opened a new pull request #5568:
URL: https://github.com/apache/incubator-doris/pull/5568


   ## Proposed changes
   
   Remove some unused file for compatible to older kernels #5567 
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [x] Code refactor (Modify the code structure, format the code, etc...)
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [x] I have created an issue on (Fix #5567 ) and described the bug/feature 
there in detail
   - [ ] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
d...@doris.apache.org by explaining why you chose the solution you did and what 
alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] caiconghui commented on a change in pull request #5564: [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout

2021-03-24 Thread GitBox


caiconghui commented on a change in pull request #5564:
URL: https://github.com/apache/incubator-doris/pull/5564#discussion_r601026010



##
File path: fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java
##
@@ -94,9 +94,9 @@ public TMasterResult finishTask(TFinishTaskRequest request) {
 // check task status
 // retry task by report process
 TStatus taskStatus = request.getTaskStatus();
-LOG.debug("get task report: {}", request.toString());
+LOG.debug("get task report: {}", request);

Review comment:
   ok




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] EmmyMiao87 opened a new pull request #5569: [Colocate plan][Step3] Colocate analytic sort node

2021-03-24 Thread GitBox


EmmyMiao87 opened a new pull request #5569:
URL: https://github.com/apache/incubator-doris/pull/5569


   ## Proposed changes
   
   The underlying sort node of the window function can also be colocate sorted.
   When the data distribution of the child fragment is less than or equal 
to the sorting column,
 the sorting node can perform local sorting.
   This can reduce unnecessary data transmission, thereby improving query 
performance.
   
   For example:
   The table schema: k1, k2, k3, k4
   The table is distrubited by: k1, k2
   Query: select k1, sum(k2) over(partition by k1 order by k2) from 
test_colocate;
   The colocate plan is following:
   
   | PLAN FRAGMENT   1 |
   |   DATA PARTITION: k1, k2  |
   |   |
   |   2:ANALYTIC NODE |
   |   |   |
   |   1:SORT NODE |
   |   |  order by: `k1`, `k2` |
   |   |   |
   |   0:OlapScanNode  |
   |  TABLE: test_colocate |
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [x] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   - [ ] Code refactor (Modify the code structure, format the code, etc...)
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after 
creating the PR. If you're unsure about any of them, don't hesitate to ask. 
We're here to help! This is simply a reminder of what we are going to look for 
before merging your code._
   
   - [x] I have created an issue on (Fix #ISSUE) and described the bug/feature 
there in detail
   - [x] Compiling and unit tests pass locally with my changes
   - [x] I have added tests that prove my fix is effective or that my feature 
works
   - [ ] If these changes need document changes, I have updated the document
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   
   Document and more unit test will be added later.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] acelyc111 commented on a change in pull request #5564: [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout

2021-03-24 Thread GitBox


acelyc111 commented on a change in pull request #5564:
URL: https://github.com/apache/incubator-doris/pull/5564#discussion_r601075266



##
File path: 
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
##
@@ -822,7 +822,7 @@ private boolean loadTxnCommitImpl(TLoadTxnCommitRequest 
request) throws UserExce
 throw new UserException("unknown database, database=" + dbName);
 }
 
-long timeoutMs = request.isSetThriftRpcTimeoutMs() ? 
request.getThriftRpcTimeoutMs() : 5000;
+long timeoutMs = request.isSetThriftRpcTimeoutMs() ? 
request.getThriftRpcTimeoutMs() / 2 : 5000;

Review comment:
   could you please explain why divide by 2? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] caiconghui commented on a change in pull request #5564: [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout

2021-03-24 Thread GitBox


caiconghui commented on a change in pull request #5564:
URL: https://github.com/apache/incubator-doris/pull/5564#discussion_r601094480



##
File path: 
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
##
@@ -822,7 +822,7 @@ private boolean loadTxnCommitImpl(TLoadTxnCommitRequest 
request) throws UserExce
 throw new UserException("unknown database, database=" + dbName);
 }
 
-long timeoutMs = request.isSetThriftRpcTimeoutMs() ? 
request.getThriftRpcTimeoutMs() : 5000;
+long timeoutMs = request.isSetThriftRpcTimeoutMs() ? 
request.getThriftRpcTimeoutMs() / 2 : 5000;

Review comment:
   This is an experience value, here we just ensure that the rpc_timeout 
value is larger enough than  timeoutMs  to keep timeout setting consistent with 
before, to reduce the "call frontend service failed error" occur times. But I 
think there is still some improvement we can do that we should compute the left 
time that  timeoutMillis minus commit time cost to improve the  Probability of 
publish timeout instead of "call frontend service failed" when commit or 
publish version performance is poor




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] morningman commented on a change in pull request #5521: [Colocate plan][Step1] Colocate join covers more situations

2021-03-24 Thread GitBox


morningman commented on a change in pull request #5521:
URL: https://github.com/apache/incubator-doris/pull/5521#discussion_r601037722



##
File path: 
fe/fe-core/src/main/java/org/apache/doris/system/SystemInfoService.java
##
@@ -781,11 +782,16 @@ public void releaseBackends(String clusterName, boolean 
isReplay) {
 }
 }
 
+
 // if more than one backend exists in same host, select a backend at 
random
 List backends = Lists.newArrayList();
 for (List list : backendMaps.values()) {
-Collections.shuffle(list);
-backends.add(list.get(0));
+if (FeConstants.runningUnitTest) {
+backends.addAll(list);

Review comment:
   why add all?

##
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/Expr.java
##
@@ -1334,7 +1334,26 @@ public Type castBinaryOp(Type compatibleType) throws 
AnalysisException {
 @Override
 public String toString() {
 return MoreObjects.toStringHelper(this.getClass()).add("id", 
id).add("type", type).add("sel",
-  selectivity).add("#distinct", numDistinctValues).add("scale", 
outputScale).toString();
+selectivity).add("#distinct", numDistinctValues).add("scale", 
outputScale).toString();
+}
+
+public SlotRef getSrcSlotRef() {

Review comment:
   add some comment

##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
##
@@ -235,8 +235,8 @@
 @VariableMgr.VarAttr(name = DISABLE_STREAMING_PREAGGREGATIONS)
 public boolean disableStreamPreaggregations = false;
 
-@VariableMgr.VarAttr(name = DISABLE_COLOCATE_JOIN)
-public boolean disableColocateJoin = false;
+@VariableMgr.VarAttr(name = DISABLE_COLOCATE_PLAN)
+public boolean disableColocatePlan = false;

Review comment:
   Why changing the name?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org