[GitHub] [incubator-doris] weizuo93 opened a new pull request #5563: [Refactor] refactor the destructor in olap_meta.cpp
weizuo93 opened a new pull request #5563: URL: https://github.com/apache/incubator-doris/pull/5563 ## Proposed changes refactor the destructor in olap_meta.cpp ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [x ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [x ] Code refactor (Modify the code structure, format the code, etc...) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] acelyc111 commented on a change in pull request #5488: [Feature][Stream Load] Add "show stream_load" function to show the result of stream load
acelyc111 commented on a change in pull request #5488: URL: https://github.com/apache/incubator-doris/pull/5488#discussion_r600261821 ## File path: be/src/http/action/stream_load.cpp ## @@ -132,9 +134,19 @@ void StreamLoadAction::handle(HttpRequest* req) { str = str + '\n'; HttpChannel::send_reply(req, str); +auto stream_load_record = StorageEngine::instance()->get_stream_load_record(); +if (stream_load_record != nullptr) { +auto st = stream_load_record->put(ToStringFromUnixMicros(ctx->start_micros + ctx->load_cost_micros) + "_" + ctx->label, str); +if (st.ok()) { +LOG(INFO) << "put stream_load_record rocksdb successfully. label: " << ctx->label; Review comment: Better to log the whole key, then we can check it in rocksdb. ## File path: be/src/olap/olap_server.cpp ## @@ -118,6 +118,13 @@ Status StorageEngine::start_bg_threads() { LOG(INFO) << "path scan/gc threads started. number:" << get_stores().size(); } +// stream load record clean thread +RETURN_IF_ERROR(Thread::create( Review comment: No need to create a thread to gc expired data, you can create a Compaction-Filter on this rocksdb instance. https://github.com/facebook/rocksdb/wiki/Compaction-Filter ## File path: be/src/olap/storage_engine.cpp ## @@ -225,6 +226,35 @@ Status StorageEngine::_init_store_map() { for (auto store : tmp_stores) { _store_map.emplace(store->path(), store); } + +auto st = _init_stream_load_record(); +if (!st.ok()) { +LOG(WARNING) << "status=" << st.to_string(); +} + +return Status::OK(); +} + +Status StorageEngine::_init_stream_load_record() { +std::string stream_load_record_path; +if (_store_map.size() > 0) { Review comment: Is it valid when _store_map is empty? Would it be better to explicit define the path in config file? ## File path: be/src/runtime/stream_load/stream_load_record.h ## @@ -0,0 +1,57 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include +#include + +#include "rocksdb/db.h" + +#pragma once + +namespace doris { + +class Status; + +class StreamLoadRecord { +public: +StreamLoadRecord(const std::string& root_path); + +virtual ~StreamLoadRecord(); + +Status init(); + +Status put(const std::string& key, const std::string& value); + +Status get_batch(const std::string& start, const int batch_size, std::map &stream_load_records); + +Status clean_expired_stream_load_record(); + +private: +std::string _root_path; +rocksdb::DB* _db; +std::vector _handles; + +enum ColumnFamilyIndex { +DEFAULT_COLUMN_FAMILY_INDEX = 0, Review comment: where does the default column family used for? ## File path: be/src/http/action/stream_load.cpp ## @@ -132,9 +134,19 @@ void StreamLoadAction::handle(HttpRequest* req) { str = str + '\n'; HttpChannel::send_reply(req, str); +auto stream_load_record = StorageEngine::instance()->get_stream_load_record(); +if (stream_load_record != nullptr) { +auto st = stream_load_record->put(ToStringFromUnixMicros(ctx->start_micros + ctx->load_cost_micros) + "_" + ctx->label, str); +if (st.ok()) { +LOG(INFO) << "put stream_load_record rocksdb successfully. label: " << ctx->label; +} +} else { +LOG(WARNING) << "stream_load_record is null."; Review comment: ```suggestion LOG(WARNING) << "put stream_load_record rocksdb failed, status = " << st.ToString(); ``` And log the key. ## File path: be/src/runtime/stream_load/stream_load_record.cpp ## @@ -0,0 +1,137 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +//
[GitHub] [incubator-doris] caiconghui opened a new pull request #5564: [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout
caiconghui opened a new pull request #5564: URL: https://github.com/apache/incubator-doris/pull/5564 ## Proposed changes This PR is to fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout and fix some small issue with log ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui opened a new issue #5565: [Bug] we may see call frontend service failed when rpc_timeout is equal with publish_timeout
caiconghui opened a new issue #5565: URL: https://github.com/apache/incubator-doris/issues/5565 **Describe the bug** when rpc_timeout is equal with publish_timeout,we may see that call frontend service failed when cluster publish transaction timeout, and then program will try to rollback transaction and user will retry stream load again, which will bring the cluster more pressure **To Reproduce** 1. put many stream loads at the same time **Expected behavior** if publish transaction timeout, we should return publis timeout instead of call frontend service failed. **Screenshots** If applicable, add screenshots to help explain your problem. **Desktop (please complete the following information):** - OS: [e.g. iOS] - Browser [e.g. chrome, safari] - Version [e.g. 22] **Smartphone (please complete the following information):** - Device: [e.g. iPhone6] - OS: [e.g. iOS8.1] - Browser [e.g. stock browser, safari] - Version [e.g. 22] **Additional context** Add any other context about the problem here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #5559: [Refactor] make uint24_t,OLAPIndexFixedHeader as a POD type
yangzhg commented on a change in pull request #5559: URL: https://github.com/apache/incubator-doris/pull/5559#discussion_r600357835 ## File path: be/src/util/coding.h ## @@ -171,14 +171,14 @@ extern const uint8_t* decode_varint64_ptr(const uint8_t* p, const uint8_t* limit template inline void put_varint32(T* dst, uint32_t v) { -uint8_t buf[5]; +uint8_t buf[16]; Review comment: #pragma GCC diagnostic ignored "-Warray-bounds" is better -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] acelyc111 commented on a change in pull request #5452: [Audit][Stream Load] Support audit function for stream load
acelyc111 commented on a change in pull request #5452: URL: https://github.com/apache/incubator-doris/pull/5452#discussion_r600381383 ## File path: be/src/common/config.h ## @@ -348,6 +348,12 @@ CONF_mInt32(streaming_load_rpc_max_alive_time_sec, "1200"); CONF_Int32(tablet_writer_open_rpc_timeout_sec, "60"); // You can ignore brpc error '[E1011]The server is overcrowded' when writing data. CONF_mBool(tablet_writer_ignore_eovercrowded, "false"); +// batch size of stream load record reported to FE +CONF_mInt32(stream_load_record_batch_size, "50"); +// expire time of stream load record in rocksdb. +CONF_mInt32(stream_load_record_expire_time_secs, "28800"); Review comment: stream_load_record_expire_time_secs is not configurable at runtime. ## File path: be/src/runtime/stream_load/stream_load_record.cpp ## @@ -0,0 +1,123 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "runtime/stream_load/stream_load_record.h" + +#include "common/config.h" +#include "common/status.h" +#include "rocksdb/db.h" + +#include "rocksdb/options.h" +#include "rocksdb/slice.h" +#include "rocksdb/slice_transform.h" +#include "rocksdb/utilities/db_ttl.h" +#include "util/time.h" + + +namespace doris { +const std::string STREAM_LOAD_POSTFIX = "/stream_load"; + +StreamLoadRecord::StreamLoadRecord(const std::string& root_path) +: _root_path(root_path), + _db(nullptr), + _last_compaction_time(UnixMillis()) { +} + +StreamLoadRecord::~StreamLoadRecord() { +if (_db != nullptr) { +for (auto handle : _handles) { +_db->DestroyColumnFamilyHandle(handle); +handle = nullptr; +} +delete _db; +_db= nullptr; +} +} + +Status StreamLoadRecord::init() { +// init db +rocksdb::DBOptions options; +options.IncreaseParallelism(); +options.create_if_missing = true; +options.create_missing_column_families = true; +std::string db_path = _root_path + STREAM_LOAD_POSTFIX; +std::vector column_families; +// default column family is required +column_families.emplace_back(DEFAULT_COLUMN_FAMILY, rocksdb::ColumnFamilyOptions()); +// stream load column family +column_families.emplace_back(STREAM_LOAD_COLUMN_FAMILY, rocksdb::ColumnFamilyOptions()); +std::vector ttls = {config::stream_load_record_expire_time_secs, config::stream_load_record_expire_time_secs}; +rocksdb::Status s = rocksdb::DBWithTTL::Open(options, db_path, column_families, &_handles, &_db, ttls); + +if (!s.ok() || _db == nullptr) { +LOG(WARNING) << "rocks db open failed, reason:" << s.ToString(); +return Status::InternalError("Stream load record rocksdb open failed"); +} +return Status::OK(); +} + +Status StreamLoadRecord::put(const std::string& key, const std::string& value) { +rocksdb::ColumnFamilyHandle* handle = _handles[1]; +rocksdb::WriteOptions write_options; +write_options.sync = false; +rocksdb::Status s = _db->Put(write_options, handle, rocksdb::Slice(key), rocksdb::Slice(value)); +if (!s.ok()) { +LOG(WARNING) << "rocks db put key:" << key << " failed, reason:" << s.ToString(); +return Status::InternalError("Stream load record rocksdb put failed"); +} + +if ((UnixMillis() - _last_compaction_time) / 1000 > config::clean_stream_load_record_interval_secs) { +rocksdb::CompactRangeOptions options; +s = _db->CompactRange(options, _handles[1], nullptr, nullptr); Review comment: CompactRange may consume a long time, better to not invoke it on load path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] JNSimba edited a comment on pull request #5375: [Feature] Add flink datastream connector example (#5371)
JNSimba edited a comment on pull request #5375: URL: https://github.com/apache/incubator-doris/pull/5375#issuecomment-802749846 Refer to [User-defined Sources & Sinks](https://ci.apache.org/projects/flink/flink-docs-stable/zh/dev/table/sourceSinks.html) And flink-connector-jdbc, add flink sql to create doris table, and write doris table through streamload。 eg: ```java EnvironmentSettings settings = EnvironmentSettings.newInstance() .useBlinkPlanner() .inStreamingMode() .build(); TableEnvironment tEnv = TableEnvironment.create(settings); tEnv.getConfig().getConfiguration().setString("job.name","test"); tEnv.executeSql( "CREATE TABLE doris_test (" + "name STRING," + "age INT," + "price DECIMAL(5,2)," + "sale DOUBLE" + ") " + "WITH (\n" + " 'connector' = 'doris',\n" + " 'fenodes' = 'FE_IP:8030',\n" + " 'table.identifier' = 'demo.doris_test_source_2',\n" + " 'username' = 'root',\n" + " 'password' = ''" + ")"); tEnv.executeSql( "CREATE TABLE doris_test_sink (" + "name STRING," + "age INT," + "price DECIMAL(5,2)," + "sale DOUBLE" + ") " + "WITH (\n" + " 'connector' = 'doris',\n" + " 'fenodes' = 'FE_IP:8030',\n" + " 'table.identifier' = 'demo.doris_test_sink_2',\n" + " 'username' = 'root',\n" + " 'password' = ''\n" + ")"); tEnv.executeSql("INSERT INTO doris_test_sink select name,age,price,sale from doris_test"); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg merged pull request #5394: update gcc to gcc 10 and support c++17
yangzhg merged pull request #5394: URL: https://github.com/apache/incubator-doris/pull/5394 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: update gcc to gcc 10 and support c++17 (#5394)
This is an automated email from the ASF dual-hosted git repository. yangzhg pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new ad67dd3 update gcc to gcc 10 and support c++17 (#5394) ad67dd3 is described below commit ad67dd34a04c1ca960cff38e5b335b30fc7d559f Author: stdpain <34912776+stdp...@users.noreply.github.com> AuthorDate: Thu Mar 25 09:30:38 2021 +0800 update gcc to gcc 10 and support c++17 (#5394) * update gcc to gcc 10 and support c++17 update brpc to 0.9.7 update boost to 1.73 remove third-party boost 1.54 for mysql * update cmake version * ignore jdk version * remove unused patch * avoid use SYS_getrandom call --- be/CMakeLists.txt | 23 +- be/src/exec/CMakeLists.txt | 2 - be/src/exec/decompressor.cpp | 2 +- be/src/exec/olap_common.h | 11 +- be/src/exec/scan_node.cpp | 2 - be/src/http/default_path_handlers.cpp | 6 +- be/src/http/ev_http_server.cpp | 2 +- be/src/olap/rowset/rowset.h| 1 + be/src/olap/rowset/unique_rowset_id_generator.h| 2 + be/src/olap/storage_engine.cpp | 2 + be/src/olap/task/engine_batch_load_task.cpp| 2 +- be/src/runtime/buffered_block_mgr2.cc | 2 +- be/src/runtime/buffered_tuple_stream2.cc | 2 +- be/src/runtime/buffered_tuple_stream3.cc | 2 +- be/src/runtime/bufferpool/buffer_allocator.cc | 5 +- be/src/runtime/bufferpool/buffer_pool.cc | 8 +- be/src/runtime/client_cache.h | 6 +- be/src/runtime/data_stream_mgr.h | 2 +- be/src/runtime/data_stream_recvr.cc| 10 +- be/src/runtime/fragment_mgr.cpp| 26 +-- be/src/runtime/result_buffer_mgr.cpp | 2 +- be/src/runtime/snapshot_loader.cpp | 2 +- be/src/runtime/spill_sorter.cc | 2 +- be/src/runtime/user_function_cache.cpp | 1 + be/src/util/thrift_server.cpp | 5 +- be/test/exprs/topn_function_test.cpp | 4 +- env.sh | 19 ++ .../main/java/org/apache/doris/common/Config.java | 2 +- thirdparty/build-thirdparty.sh | 109 +- thirdparty/download-thirdparty.sh | 10 +- thirdparty/patches/incubator-brpc-0.9.5.patch | 237 - thirdparty/patches/mysql-5.7.18-boost.patch| 20 ++ thirdparty/vars.sh | 23 +- 33 files changed, 183 insertions(+), 371 deletions(-) diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt index 8be4883..e520b3c 100644 --- a/be/CMakeLists.txt +++ b/be/CMakeLists.txt @@ -15,7 +15,7 @@ # specific language governing permissions and limitations # under the License. -cmake_minimum_required(VERSION 3.12.0) +cmake_minimum_required(VERSION 3.19.2) # set CMAKE_C_COMPILER, this must set before project command if (DEFINED ENV{DORIS_GCC_HOME}) @@ -130,14 +130,13 @@ endif() set(Boost_DEBUG FALSE) set(Boost_USE_MULTITHREADED ON) set(BOOST_ROOT ${THIRDPARTY_DIR}) +set(Boost_NO_BOOST_CMAKE OFF) if (NOT APPLE) -find_package(Boost 1.55.0 REQUIRED COMPONENTS thread regex filesystem system date_time program_options) +find_package(Boost 1.73.0 REQUIRED COMPONENTS regex system filesystem thread date_time program_options) else() -find_package(Boost 1.55.0 COMPONENTS thread regex filesystem system date_time program_options) +find_package(Boost 1.73.0 COMPONENTS thread regex system filesystem date_time program_options) endif() -include_directories(${Boost_INCLUDE_DIRS}) -message(STATUS ${Boost_LIBRARIES}) set(GPERFTOOLS_HOME "${THIRDPARTY_DIR}/gperftools") @@ -313,10 +312,13 @@ check_function_exists(sched_getcpu HAVE_SCHED_GETCPU) # -fno-omit-frame-pointers: Keep frame pointer for functions in register set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -Wall -Wno-sign-compare -Wno-unknown-pragmas -pthread") set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -fno-strict-aliasing -fno-omit-frame-pointer") -set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -std=gnu++11 -D__STDC_FORMAT_MACROS") +set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -std=gnu++17 -D__STDC_FORMAT_MACROS") set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -Wno-deprecated -Wno-vla") set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -DBOOST_DATE_TIME_POSIX_TIME_STD_CONFIG") set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -DBOOST_SYSTEM_NO_DEPRECATED") +# https://github.com/boostorg/uuid/issues/92 +# We need to avoid using SYS_getrandom system calls +set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -DBOOST_UUID_RANDOM_PROVIDER_FORCE_
[GitHub] [incubator-doris] yangzhg opened a new pull request #5566: [DOC] Update gcc10 compile docs
yangzhg opened a new pull request #5566: URL: https://github.com/apache/incubator-doris/pull/5566 ## Proposed changes Update GCC10 compile docs ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] stdpain commented on a change in pull request #5564: [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout
stdpain commented on a change in pull request #5564: URL: https://github.com/apache/incubator-doris/pull/5564#discussion_r600993661 ## File path: fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java ## @@ -94,9 +94,9 @@ public TMasterResult finishTask(TFinishTaskRequest request) { // check task status // retry task by report process TStatus taskStatus = request.getTaskStatus(); -LOG.debug("get task report: {}", request.toString()); +LOG.debug("get task report: {}", request); Review comment: It's a better way to use ```java if (LOG.isDebugEnabled()) { LOG.debug("get task report: {}", request); } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg opened a new issue #5567: Remove some glibc-compatibility file for old kernel
yangzhg opened a new issue #5567: URL: https://github.com/apache/incubator-doris/issues/5567 In some old kernel SYS_getrandom and SYS_getcpu is not exist, and we don't use those in our code, so we should remove those files to compatible with old kernels -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg opened a new pull request #5568: [refactor] Remove some unused file for compatible to older kernels
yangzhg opened a new pull request #5568: URL: https://github.com/apache/incubator-doris/pull/5568 ## Proposed changes Remove some unused file for compatible to older kernels #5567 ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [x] Code refactor (Modify the code structure, format the code, etc...) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have created an issue on (Fix #5567 ) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a change in pull request #5564: [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout
caiconghui commented on a change in pull request #5564: URL: https://github.com/apache/incubator-doris/pull/5564#discussion_r601026010 ## File path: fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java ## @@ -94,9 +94,9 @@ public TMasterResult finishTask(TFinishTaskRequest request) { // check task status // retry task by report process TStatus taskStatus = request.getTaskStatus(); -LOG.debug("get task report: {}", request.toString()); +LOG.debug("get task report: {}", request); Review comment: ok -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 opened a new pull request #5569: [Colocate plan][Step3] Colocate analytic sort node
EmmyMiao87 opened a new pull request #5569: URL: https://github.com/apache/incubator-doris/pull/5569 ## Proposed changes The underlying sort node of the window function can also be colocate sorted. When the data distribution of the child fragment is less than or equal to the sorting column, the sorting node can perform local sorting. This can reduce unnecessary data transmission, thereby improving query performance. For example: The table schema: k1, k2, k3, k4 The table is distrubited by: k1, k2 Query: select k1, sum(k2) over(partition by k1 order by k2) from test_colocate; The colocate plan is following: | PLAN FRAGMENT 1 | | DATA PARTITION: k1, k2 | | | | 2:ANALYTIC NODE | | | | | 1:SORT NODE | | | order by: `k1`, `k2` | | | | | 0:OlapScanNode | | TABLE: test_colocate | ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [x] Compiling and unit tests pass locally with my changes - [x] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments Document and more unit test will be added later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] acelyc111 commented on a change in pull request #5564: [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout
acelyc111 commented on a change in pull request #5564: URL: https://github.com/apache/incubator-doris/pull/5564#discussion_r601075266 ## File path: fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java ## @@ -822,7 +822,7 @@ private boolean loadTxnCommitImpl(TLoadTxnCommitRequest request) throws UserExce throw new UserException("unknown database, database=" + dbName); } -long timeoutMs = request.isSetThriftRpcTimeoutMs() ? request.getThriftRpcTimeoutMs() : 5000; +long timeoutMs = request.isSetThriftRpcTimeoutMs() ? request.getThriftRpcTimeoutMs() / 2 : 5000; Review comment: could you please explain why divide by 2? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a change in pull request #5564: [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout
caiconghui commented on a change in pull request #5564: URL: https://github.com/apache/incubator-doris/pull/5564#discussion_r601094480 ## File path: fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java ## @@ -822,7 +822,7 @@ private boolean loadTxnCommitImpl(TLoadTxnCommitRequest request) throws UserExce throw new UserException("unknown database, database=" + dbName); } -long timeoutMs = request.isSetThriftRpcTimeoutMs() ? request.getThriftRpcTimeoutMs() : 5000; +long timeoutMs = request.isSetThriftRpcTimeoutMs() ? request.getThriftRpcTimeoutMs() / 2 : 5000; Review comment: This is an experience value, here we just ensure that the rpc_timeout value is larger enough than timeoutMs to keep timeout setting consistent with before, to reduce the "call frontend service failed error" occur times. But I think there is still some improvement we can do that we should compute the left time that timeoutMillis minus commit time cost to improve the Probability of publish timeout instead of "call frontend service failed" when commit or publish version performance is poor -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #5521: [Colocate plan][Step1] Colocate join covers more situations
morningman commented on a change in pull request #5521: URL: https://github.com/apache/incubator-doris/pull/5521#discussion_r601037722 ## File path: fe/fe-core/src/main/java/org/apache/doris/system/SystemInfoService.java ## @@ -781,11 +782,16 @@ public void releaseBackends(String clusterName, boolean isReplay) { } } + // if more than one backend exists in same host, select a backend at random List backends = Lists.newArrayList(); for (List list : backendMaps.values()) { -Collections.shuffle(list); -backends.add(list.get(0)); +if (FeConstants.runningUnitTest) { +backends.addAll(list); Review comment: why add all? ## File path: fe/fe-core/src/main/java/org/apache/doris/analysis/Expr.java ## @@ -1334,7 +1334,26 @@ public Type castBinaryOp(Type compatibleType) throws AnalysisException { @Override public String toString() { return MoreObjects.toStringHelper(this.getClass()).add("id", id).add("type", type).add("sel", - selectivity).add("#distinct", numDistinctValues).add("scale", outputScale).toString(); +selectivity).add("#distinct", numDistinctValues).add("scale", outputScale).toString(); +} + +public SlotRef getSrcSlotRef() { Review comment: add some comment ## File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java ## @@ -235,8 +235,8 @@ @VariableMgr.VarAttr(name = DISABLE_STREAMING_PREAGGREGATIONS) public boolean disableStreamPreaggregations = false; -@VariableMgr.VarAttr(name = DISABLE_COLOCATE_JOIN) -public boolean disableColocateJoin = false; +@VariableMgr.VarAttr(name = DISABLE_COLOCATE_PLAN) +public boolean disableColocatePlan = false; Review comment: Why changing the name? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org