[GitHub] [incubator-doris] xy720 opened a new issue #7060: [Feature] Clean up old sync jobs regularly
xy720 opened a new issue #7060: URL: https://github.com/apache/incubator-doris/issues/7060 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description We should clean up old sync jobs regularly to save Fe memory space. Each job that has been stopped for more than 3 days(set with Config.label_keep_max_second) will be permanently cleaned up. ### Use case _No response_ ### Related issues #6287 ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #7032: [Heartbeat] Support cancel outdated query after FE restart
morningman commented on pull request #7032: URL: https://github.com/apache/incubator-doris/pull/7032#issuecomment-963904504 And FE ut failed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a change in pull request #7032: [Heartbeat] Support cancel outdated query after FE restart
xinyiZzz commented on a change in pull request #7032: URL: https://github.com/apache/incubator-doris/pull/7032#discussion_r745379083 ## File path: be/src/runtime/exec_env.h ## @@ -219,6 +225,9 @@ class ExecEnv { HeartbeatFlags* _heartbeat_flags = nullptr; PluginMgr* _plugin_mgr = nullptr; + +std::map _frontends_start_time; Review comment: Thanks for your advice, But `_frontends_start_time` I understand that there is no need to lock, it will only be modified when the BE is processing the heartbeat, and a BE process has only one thread to process the heartbeat. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zuochunwei commented on a change in pull request #6971: hll optimize: trace memory usage, new explicit data when really need
zuochunwei commented on a change in pull request #6971: URL: https://github.com/apache/incubator-doris/pull/6971#discussion_r745396863 ## File path: be/src/olap/hll.h ## @@ -33,6 +33,7 @@ class Slice; const static int HLL_COLUMN_PRECISION = 14; const static int HLL_ZERO_COUNT_BITS = (64 - HLL_COLUMN_PRECISION); const static int HLL_EXPLICIT_INT64_NUM = 160; +const static int HLL_EXPLICIT_INT64_NUM_DOUBLE = HLL_EXPLICIT_INT64_NUM*2; Review comment: do you means i should add space between HLL_EXPLICIT_INT64_NUM and *? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zuochunwei commented on a change in pull request #6971: hll optimize: trace memory usage, new explicit data when really need
zuochunwei commented on a change in pull request #6971: URL: https://github.com/apache/incubator-doris/pull/6971#discussion_r745398893 ## File path: be/src/olap/hll.h ## @@ -154,7 +174,7 @@ class HyperLogLog { HllDataType _type = HLL_DATA_EMPTY; uint32_t _explicit_data_num = 0; -uint64_t _explicit_data[HLL_EXPLICIT_INT64_NUM * 2]; +uint64_t* _explicit_data = nullptr; Review comment: no need replace with unique_ptr, keep style consistency with uint8_t *_registers -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zuochunwei commented on a change in pull request #6971: hll optimize: trace memory usage, new explicit data when really need
zuochunwei commented on a change in pull request #6971: URL: https://github.com/apache/incubator-doris/pull/6971#discussion_r745402142 ## File path: be/src/olap/hll.h ## @@ -33,6 +33,7 @@ class Slice; const static int HLL_COLUMN_PRECISION = 14; const static int HLL_ZERO_COUNT_BITS = (64 - HLL_COLUMN_PRECISION); const static int HLL_EXPLICIT_INT64_NUM = 160; +const static int HLL_EXPLICIT_INT64_NUM_DOUBLE = HLL_EXPLICIT_INT64_NUM*2; Review comment: fixed it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] qidaye commented on a change in pull request #6569: [Feature] Support query hive table
qidaye commented on a change in pull request #6569: URL: https://github.com/apache/incubator-doris/pull/6569#discussion_r745402643 ## File path: fe/fe-core/src/main/java/org/apache/doris/planner/HiveScanNode.java ## @@ -0,0 +1,186 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.planner; + +import org.apache.doris.analysis.Analyzer; +import org.apache.doris.analysis.Expr; +import org.apache.doris.analysis.ImportColumnDesc; +import org.apache.doris.analysis.TupleDescriptor; +import org.apache.doris.catalog.HiveMetaStoreClientHelper; +import org.apache.doris.catalog.HiveTable; +import org.apache.doris.common.DdlException; +import org.apache.doris.common.UserException; +import org.apache.doris.thrift.TBrokerFileStatus; +import org.apache.doris.thrift.TExplainLevel; + +import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.hadoop.hive.metastore.api.Table; +import org.apache.hadoop.hive.ql.plan.ExprNodeDesc; +import org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc; +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +import com.google.common.base.Strings; + +import java.util.ArrayList; +import java.util.List; +import java.util.ListIterator; +import java.util.Map; + +public class HiveScanNode extends BrokerScanNode{ +private static final Logger LOG = LogManager.getLogger(HiveScanNode.class); + +private static final String HIVE_DEFAULT_COLUMN_SEPARATOR = "\001"; +private static final String HIVE_DEFAULT_LINE_DELIMITER = "\n"; + +private HiveTable hiveTable; +// partition column predicates of hive table +private List hivePredicates = new ArrayList<>(); +private ExprNodeGenericFuncDesc hivePartitionPredicate; +private List parsedColumnExprList = new ArrayList<>(); +private String hdfsUri; + +private Table remoteHiveTable; + +/* hive table properties */ +private String columnSeparator; +private String lineDelimiter; +private String fileFormat; +private String path; +private List partitionKeys = new ArrayList<>(); +/* hive table properties */ + +public String getHdfsUri() { +return hdfsUri; +} + +public List getParsedColumnExprList() { +return parsedColumnExprList; +} + +public String getColumnSeparator() { +return columnSeparator; +} + +public String getLineDelimiter() { +return lineDelimiter; +} + +public String getFileFormat() { +return fileFormat; +} + +public String getPath() { +return path; +} + +public List getPartitionKeys() { +return partitionKeys; +} + +public HiveScanNode(PlanNodeId id, TupleDescriptor destTupleDesc, String planNodeName, +List> fileStatusesList, int filesAdded) { +super(id, destTupleDesc, planNodeName, fileStatusesList, filesAdded); +this.hiveTable = (HiveTable) destTupleDesc.getTable(); +} + +@Override +public void init(Analyzer analyzer) throws UserException { +initHiveTblProperties(); +if (partitionKeys.size() > 0) { +extractHivePartitionPredicate(analyzer); +} +analyzeColumnFromPath(); +super.init(analyzer); Review comment: Refactored -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xy720 opened a new pull request #7061: [Feature] Clean up old sync jobs regularly
xy720 opened a new pull request #7061: URL: https://github.com/apache/incubator-doris/pull/7061 ## Proposed changes #7060 Each job that has been stopped for more than 3 days(set with Config.label_keep_max_second) will be permanently cleaned up. ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [x] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have created an issue on (Fix #7060 ) and described the bug/feature there in detail - [x] Compiling and unit tests pass locally with my changes - [x] I have added tests that prove my fix is effective or that my feature works ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Bug] Fix docs website home page last news icon loading failure (#7057)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new 906c305 [Bug] Fix docs website home page last news icon loading failure (#7057) 906c305 is described below commit 906c305a190d53cd279c25c3e436f66c8c88c902 Author: wangyongfeng <943155...@qq.com> AuthorDate: Tue Nov 9 17:34:42 2021 +0800 [Bug] Fix docs website home page last news icon loading failure (#7057) * Fix last news icon loading failure Co-authored-by: 943155336 Co-authored-by: jiafeng.zhang --- docs/.vuepress/theme/components/Home.vue | 19 ++- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/.vuepress/theme/components/Home.vue b/docs/.vuepress/theme/components/Home.vue index e2a90a8..2c0ba6a 100644 --- a/docs/.vuepress/theme/components/Home.vue +++ b/docs/.vuepress/theme/components/Home.vue @@ -54,10 +54,13 @@ under the License. class="action-button" :item="actionLink" /> - + + + +
[GitHub] [incubator-doris] hf200012 merged pull request #7057: [Bug] Fix docs website home page last news icon loading failure
hf200012 merged pull request #7057: URL: https://github.com/apache/incubator-doris/pull/7057 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: Chinese annotation modification (#6958)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new 088a16d Chinese annotation modification (#6958) 088a16d is described below commit 088a16d33b78e4978f6ff0a8f0028d8aba49da3b Author: jiafeng.zhang AuthorDate: Tue Nov 9 18:00:14 2021 +0800 Chinese annotation modification (#6958) * Modify Chinese comment (#6951) --- be/src/exec/olap_scan_node.cpp | 16 +-- be/src/olap/rowset/bit_field_writer.h| 2 +- be/src/olap/rowset/column_reader.h | 139 +++ be/src/olap/rowset/column_writer.cpp | 34 +++--- be/src/olap/rowset/column_writer.h | 40 +++ be/src/olap/rowset/run_length_integer_reader.cpp | 2 +- be/src/olap/rowset/segment_reader.cpp| 32 +++--- be/src/olap/rowset/segment_reader.h | 124 ++-- be/src/olap/serialize.h | 53 - be/src/olap/storage_engine.h | 10 +- be/src/olap/stream_index_common.h| 35 +++--- be/test/exec/csv_scan_node_test.cpp | 8 +- be/test/exec/parquet_scanner_test.cpp| 10 +- 13 files changed, 254 insertions(+), 251 deletions(-) diff --git a/be/src/exec/olap_scan_node.cpp b/be/src/exec/olap_scan_node.cpp index 746e553..9157e1b 100644 --- a/be/src/exec/olap_scan_node.cpp +++ b/be/src/exec/olap_scan_node.cpp @@ -1337,14 +1337,14 @@ void OlapScanNode::transfer_thread(RuntimeState* state) { ThreadPoolToken* thread_token = state->get_query_fragments_ctx()->get_token(); /* - * 优先级调度基本策略: - * 1. 通过查询拆分的Range个数来确定初始nice值 - *Range个数越多,越倾向于认定为大查询,nice值越小 - * 2. 通过查询累计读取的数据量来调整nice值 - *读取的数据越多,越倾向于认定为大查询,nice值越小 - * 3. 通过nice值来判断查询的优先级 - *nice值越大的,越优先获得的查询资源 - * 4. 定期提高队列内残留任务的优先级,避免大查询完全饿死 + * The basic strategy of priority scheduling: + * 1. Determine the initial nice value by querying the number of split ranges + *The more the number of Ranges, the more likely it is to be recognized as a large query, and the smaller the nice value + * 2. Adjust the nice value by querying the accumulated data volume + *The more data read, the more likely it is to be regarded as a large query, and the smaller the nice value + * 3. Judge the priority of the query by the nice value + *The larger the nice value, the more preferentially obtained query resources + * 4. Regularly increase the priority of the remaining tasks in the queue to avoid starvation for large queries */ PriorityThreadPool* thread_pool = state->exec_env()->scan_thread_pool(); _total_assign_num = 0; diff --git a/be/src/olap/rowset/bit_field_writer.h b/be/src/olap/rowset/bit_field_writer.h index 32a40e4..6812833 100644 --- a/be/src/olap/rowset/bit_field_writer.h +++ b/be/src/olap/rowset/bit_field_writer.h @@ -31,7 +31,7 @@ public: explicit BitFieldWriter(OutStream* output); ~BitFieldWriter(); OLAPStatus init(); -// 写入一个bit, bit_value为true表示写入1, false表示写入0 +// Write a bit, bit_value is true means writing 1, false means writing 0 OLAPStatus write(bool bit_value); OLAPStatus flush(); void get_position(PositionEntryWriter* index_entry) const; diff --git a/be/src/olap/rowset/column_reader.h b/be/src/olap/rowset/column_reader.h index f0eeffd..a57a3eb 100644 --- a/be/src/olap/rowset/column_reader.h +++ b/be/src/olap/rowset/column_reader.h @@ -38,7 +38,7 @@ class BitFieldReader; class RowIndexEntryMessage; class ColumnEncodingMessage; -// 解出流 +// Solution flow inline ReadOnlyFileStream* extract_stream(uint32_t column_unique_id, StreamInfoMessage::Kind kind, std::map* streams) { StreamName stream_name(column_unique_id, kind); @@ -56,34 +56,34 @@ typedef std::unordered_map UniqueIdPositionProviderM // Unique id -> ColumnEncodingMessage typedef std::map UniqueIdEncodingMap; -// Integer和String的读取器。 -// 这些读取器虽然也冠有Reader之名,但注意并不从ColumnReader继承 -// 因此不考虑空值的情况。 +// Readers for Integer and String. +// Although these readers are also named Reader, note that they do not inherit from ColumnReader +// Therefore, the case of null values is not considered. -// 对于SHORT/INT/LONG类型的数据,统一使用int64作为存储的数据 -// 由于使用变长编码,所以不会造成浪费 -// IntegerColumnReader是读取底层int64数据的reader,上层返回时 -// 使用IntColumnReaderWrapper转为具体的数据类型 +// For SHORT/INT/LONG type data, use int64 as the stored data uniformly +// Due to the use of variable length coding, it will not cause waste +// IntegerColumnReader is a reader that reads the int64 data of the bottom layer, when the upper layer returns +// Use IntColumnReaderWrapper to convert to a specific data type /
[GitHub] [incubator-doris] hf200012 merged pull request #6958: Chinese annotation modification
hf200012 merged pull request #6958: URL: https://github.com/apache/incubator-doris/pull/6958 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wayofeng opened a new pull request #7062: [Bug] Fix version nav button loaded multiple times in docs website header
wayofeng opened a new pull request #7062: URL: https://github.com/apache/incubator-doris/pull/7062 ## Proposed changes Fix version nav button loaded multiple times in docs website header. ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7062: [Bug] Fix version nav button loaded multiple times in docs website header
github-actions[bot] commented on pull request #7062: URL: https://github.com/apache/incubator-doris/pull/7062#issuecomment-964004786 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7062: [Bug] Fix version nav button loaded multiple times in docs website header
github-actions[bot] commented on pull request #7062: URL: https://github.com/apache/incubator-doris/pull/7062#issuecomment-964004810 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 merged pull request #7062: [Bug] Fix version nav button loaded multiple times in docs website header
hf200012 merged pull request #7062: URL: https://github.com/apache/incubator-doris/pull/7062 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Bug] Fix version nav button loaded multiple times in docs website header (#7062)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new 4c6cbdf [Bug] Fix version nav button loaded multiple times in docs website header (#7062) 4c6cbdf is described below commit 4c6cbdf4635e48a3465bdfd7302cd3a0faabba42 Author: wangyongfeng <943155...@qq.com> AuthorDate: Tue Nov 9 18:23:44 2021 +0800 [Bug] Fix version nav button loaded multiple times in docs website header (#7062) * Fix version nav button loaded multiple times Co-authored-by: 943155336 Co-authored-by: jiafeng.zhang --- docs/.vuepress/theme/layouts/Layout.vue | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/.vuepress/theme/layouts/Layout.vue b/docs/.vuepress/theme/layouts/Layout.vue index 5793625..98ad313 100644 --- a/docs/.vuepress/theme/layouts/Layout.vue +++ b/docs/.vuepress/theme/layouts/Layout.vue @@ -42,9 +42,9 @@ export default { .get("/versions.json") .then(res => { Object.keys(this.$site.themeConfig.locales).forEach(k => { - this.$site.themeConfig.locales[k].nav = this.$site.themeConfig.locales[k].nav.concat( -res.data[k.replace(/\//gi, "")] || [] - ) + const nav = res.data[k.replace(/\//gi, "")] || [] + const localNav = nav.filter(item => this.$site.themeConfig.locales[k].nav.every(v => v.text !== item.text)) + this.$site.themeConfig.locales[k].nav = this.$site.themeConfig.locales[k].nav.concat(localNav) }) this.renderIndex = 1 }) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] killxdcj opened a new issue #7063: [Feature]
killxdcj opened a new issue #7063: URL: https://github.com/apache/incubator-doris/issues/7063 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description For high throughput scenarios, after turn on the local_replica_selection, the performance improvement is obvious. But local_replica_selection have a problem, all frontends can only use local replicas to do the query, so even if the be fails for a short time, a large number of queries will fail. A reasonable solution is to select a non-local BE to service the query when the local be is unavailable, whether to fallback to non-local BE could be control by configure. ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] killxdcj opened a new pull request #7064: [#2794437]. Add degradate strategy for local_replica_selection.
killxdcj opened a new pull request #7064: URL: https://github.com/apache/incubator-doris/pull/7064 ## Proposed changes When local_replica_selection is turned on, support select a non-local BE to service the query when the local be is unavailable ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [x] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have created an issue on (Fix #7063) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments Please refer to #7063 for details -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] GoGoWen opened a new pull request #7065: fix log error for ActionController
GoGoWen opened a new pull request #7065: URL: https://github.com/apache/incubator-doris/pull/7065 ## Proposed changes This pr just to fix the log error in ActionController.java. - private static final Logger LOG = LogManager.getLogger(LoadAction.class); + private static final Logger LOG = LogManager.getLogger(ActionController.class); ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [X ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments NO -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7065: fix log error for ActionController
github-actions[bot] commented on pull request #7065: URL: https://github.com/apache/incubator-doris/pull/7065#issuecomment-964047967 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a change in pull request #7032: [Heartbeat] Support cancel outdated query after FE restart
xinyiZzz commented on a change in pull request #7032: URL: https://github.com/apache/incubator-doris/pull/7032#discussion_r745535484 ## File path: gensrc/thrift/FrontendService.thrift ## @@ -695,6 +695,7 @@ struct TFrontendPingFrontendResult { 4: required i32 rpcPort 5: required i64 replayedJournalId 6: required string version +7: required i64 startTime Review comment: done ## File path: gensrc/thrift/HeartbeatService.thrift ## @@ -31,6 +31,13 @@ struct TMasterInfo { 8: optional i64 backend_id } +// Contain master +struct TFrontendInfo { +1: required Types.TNetworkAddress network_address Review comment: done ## File path: gensrc/thrift/HeartbeatService.thrift ## @@ -46,5 +53,5 @@ struct THeartbeatResult { } service HeartbeatService { -THeartbeatResult heartbeat(1:TMasterInfo master_info); -} +THeartbeatResult heartbeat(1:TMasterInfo master_info, 2:list frontend_info); Review comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] killxdcj opened a new issue #7066: [Feature] Support config scheduling strategy for scannode.
killxdcj opened a new issue #7066: URL: https://github.com/apache/incubator-doris/issues/7066 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description When schedule ScanNode, the current strategy will schedule scannodes to as many backends as possible, which can efficiently utilize cluster resources and improve query performance. However, for high throughput point query scenarios, the current scheduling strategy has some problems 1. Point query itself does not require many resources, so it is of little significance to mobilize all backends to serve query. 2. The thundering herd will be serious, which will affect the query performance and query parallelism of the cluster 3. It will increase the RPC and data interaction between backends, which will have a great impact on the performance Therefore, maybe we should support a new scheduling strategy for high-throughput point query scenarios, select as few backends as possible to serve query. ### Use case Suppose you have a table with many partitions, and the query scenario is high-throughput point query (each query will specify the distribute key) ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] killxdcj opened a new pull request #7067: [Feature] Support config scheduling strategy.
killxdcj opened a new pull request #7067: URL: https://github.com/apache/incubator-doris/pull/7067 ## Proposed changes Support config scheduling strategy. More detail in: #7066 ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [x] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments More detail in: #7066 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] pengxiangyu commented on a change in pull request #7061: [Feature] Clean up old sync jobs regularly
pengxiangyu commented on a change in pull request #7061: URL: https://github.com/apache/incubator-doris/pull/7061#discussion_r745579688 ## File path: fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncChecker.java ## @@ -73,5 +73,7 @@ private void process() throws UserException { job.cancel(msgType, exception.getMessage()); } } + +this.syncJobManager.cleanOldSyncJobs(); Review comment: cleanOldSyncJobs() 放在runAfterCatalogReady()里更好些,新建与删除最好分开在两个函数。 ## File path: fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncJobManager.java ## @@ -279,6 +280,40 @@ public void updateNeedSchedule() throws UserException { } } +// Remove old sync jobs. Called periodically. +// Stopped jobs will be removed after Config.label_keep_max_second. +public void cleanOldSyncJobs() { +LOG.debug("begin to clean old sync jobs "); +long currentTimeMs = System.currentTimeMillis(); +writeLock(); +try { +Iterator> iterator = idToSyncJob.entrySet().iterator(); +while (iterator.hasNext()) { +SyncJob syncJob = iterator.next().getValue(); +if (syncJob.isExpired(currentTimeMs)) { +Map> map = dbIdToJobNameToSyncJobs.get(syncJob.getDbId()); Review comment: 使用之前判断一下key是否存在吧。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7048: [Compile] Fix spark-connector compile problem
github-actions[bot] commented on pull request #7048: URL: https://github.com/apache/incubator-doris/pull/7048#issuecomment-964162957 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #7069: [Bug] Fix bug that NPE thrown when adding partition for table with MV
morningman opened a new pull request #7069: URL: https://github.com/apache/incubator-doris/pull/7069 ## Proposed changes The `defineExpr` in `Column` must be analyzed before calling its `treeToThrift` method. And fro CreateReplicaTask, no need to set `defineExpr` in TColumn. ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #7068 ) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7065: fix log error for ActionController
github-actions[bot] commented on pull request #7065: URL: https://github.com/apache/incubator-doris/pull/7065#issuecomment-964708365 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7020: [Colocate] Fix bug that colocate group can not be redistributed after dropping a backend
github-actions[bot] commented on pull request #7020: URL: https://github.com/apache/incubator-doris/pull/7020#issuecomment-964716912 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #7036: [Manager] Adapt to manger product prototype
morningman commented on pull request #7036: URL: https://github.com/apache/incubator-doris/pull/7036#issuecomment-964718751 cc @songchuangyuan And please leave a message when this PR is ready to merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6986: [RoutineLoad] And "runningTxns" fields in SHOW ROUTINE LOAD result.
github-actions[bot] commented on pull request #6986: URL: https://github.com/apache/incubator-doris/pull/6986#issuecomment-964728996 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #7061: [Feature] Clean up old sync jobs regularly
xy720 commented on a change in pull request #7061: URL: https://github.com/apache/incubator-doris/pull/7061#discussion_r746206761 ## File path: fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncChecker.java ## @@ -73,5 +73,7 @@ private void process() throws UserException { job.cancel(msgType, exception.getMessage()); } } + +this.syncJobManager.cleanOldSyncJobs(); Review comment: ok. done ## File path: fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncJobManager.java ## @@ -279,6 +280,40 @@ public void updateNeedSchedule() throws UserException { } } +// Remove old sync jobs. Called periodically. +// Stopped jobs will be removed after Config.label_keep_max_second. +public void cleanOldSyncJobs() { +LOG.debug("begin to clean old sync jobs "); +long currentTimeMs = System.currentTimeMillis(); +writeLock(); +try { +Iterator> iterator = idToSyncJob.entrySet().iterator(); +while (iterator.hasNext()) { +SyncJob syncJob = iterator.next().getValue(); +if (syncJob.isExpired(currentTimeMs)) { +Map> map = dbIdToJobNameToSyncJobs.get(syncJob.getDbId()); Review comment: ok -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7061: [Feature] Clean up old sync jobs regularly
github-actions[bot] commented on pull request #7061: URL: https://github.com/apache/incubator-doris/pull/7061#issuecomment-964736879 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on pull request #7025: Add tablets number limit to affend wrong usage
tianhui5 commented on pull request #7025: URL: https://github.com/apache/incubator-doris/pull/7025#issuecomment-964737271 The previous replica check can only check the history quota, I add the current operation quota check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #7030: [Feature] Print peak memory use of all backend after query in audit log
morningman commented on a change in pull request #7030: URL: https://github.com/apache/incubator-doris/pull/7030#discussion_r746202859 ## File path: be/src/runtime/query_statistics.h ## @@ -27,25 +27,49 @@ namespace doris { class QueryStatisticsRecvr; +class NodeStatistics { +public: +NodeStatistics() : peak_memory_bytes(0) {}; + +void add_peak_memory(int64_t peak_memory) { this->peak_memory_bytes += peak_memory; } + +void merge(const NodeStatistics& other); + +void to_pb(PNodeStatistics* nodeStatistics); + +void from_pb(const PNodeStatistics& nodeStatistics); + +private: +int64_t peak_memory_bytes; +}; + // This is responsible for collecting query statistics, usually it consists of // two parts, one is current fragment or plan's statistics, the other is sub fragment // or plan's statistics and QueryStatisticsRecvr is responsible for collecting it. class QueryStatistics { public: QueryStatistics() : scan_rows(0), scan_bytes(0), cpu_ms(0), returned_rows(0) {} -void merge(const QueryStatistics& other) { -scan_rows += other.scan_rows; -scan_bytes += other.scan_bytes; -cpu_ms += other.cpu_ms; -} +void merge(const QueryStatistics& other); void add_scan_rows(int64_t scan_rows) { this->scan_rows += scan_rows; } void add_scan_bytes(int64_t scan_bytes) { this->scan_bytes += scan_bytes; } void add_cpu_ms(int64_t cpu_ms) { this->cpu_ms += cpu_ms; } +NodeStatistics* add_nodes_statistics(int64_t node_id) { +NodeStatistics* nodeStatistics = nullptr; +auto iter = nodes_statistics_map.find(node_id); +if (iter == nodes_statistics_map.end()) { +nodeStatistics = new NodeStatistics; Review comment: You need to add `~QueryStatistics()` to clear the objects in `nodes_statistics_map`, or there will be memory leak. ## File path: be/src/runtime/query_statistics.h ## @@ -78,6 +93,10 @@ class QueryStatistics { // number rows returned by query. // only set once by result sink when closing. int64_t returned_rows; + +// The statistics of the query on each backend. +typedef std::unordered_map NodeStatisticsMap; +NodeStatisticsMap nodes_statistics_map; Review comment: ```suggestion NodeStatisticsMap _nodes_statistics_map; ``` ## File path: be/src/runtime/plan_fragment_executor.cpp ## @@ -81,6 +81,7 @@ Status PlanFragmentExecutor::prepare(const TExecPlanFragmentParams& request, RETURN_IF_ERROR(_runtime_state->init_mem_trackers(_query_id)); _runtime_state->set_be_number(request.backend_num); +_runtime_state->set_backend_id(request.backend_id); Review comment: Need to check if `backend_Id` is set in `request`, for forward compatibility. Or set a default value for `backend_id` in proto file. ## File path: fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java ## @@ -123,6 +124,15 @@ private void auditAfterExec(String origStmt, StatementBase parsedStmt, Data.PQue .setStmtId(ctx.getStmtId()) .setQueryId(ctx.queryId() == null ? "NaN" : DebugUtil.printId(ctx.queryId())); +if (statistics != null && statistics.getNodesStatisticsCount() != 0) { Review comment: I think this can be merge on BE side? Or FE will become the bottleneck ## File path: be/src/runtime/query_statistics.cpp ## @@ -19,6 +19,54 @@ namespace doris { +void NodeStatistics::merge(const NodeStatistics& other) { +peak_memory_bytes += other.peak_memory_bytes; +} + +void NodeStatistics::to_pb(PNodeStatistics* nodeStatistics) { Review comment: ```suggestion void NodeStatistics::to_pb(PNodeStatistics* node_statistics) { ``` ## File path: fe/fe-core/src/main/java/org/apache/doris/plugin/AuditEvent.java ## @@ -79,6 +83,8 @@ public String stmt = ""; @AuditField(value = "CpuTimeMS") public long cpuTimeMs = -1; +@AuditField(value = "PeakMemory") Review comment: Add to the end, for compatibility. And better use `long` instead of string, so that it can be easily analyzed by application ## File path: be/src/runtime/query_statistics.h ## @@ -55,21 +79,12 @@ class QueryStatistics { scan_bytes = 0; cpu_ms = 0; returned_rows = 0; +nodes_statistics_map.clear(); Review comment: This clear() will not release the memory allocated for `NodeStatistics` in `nodes_statistics_map` ## File path: be/src/runtime/query_statistics.cpp ## @@ -19,6 +19,54 @@ namespace doris { +void NodeStatistics::merge(const NodeStatistics& other) { +peak_memory_bytes += other.peak_memory_bytes; +} + +void NodeStatistics::to_pb(PNodeStatistics* nodeStatistics) { +DCHECK(nodeStatistics != nullptr); +nodeStatistics->set_peak_memory_bytes(peak_memory_bytes); +} + +void NodeStatistics::from_pb(const PNodeStat
[GitHub] [incubator-doris] EmmyMiao87 commented on pull request #7069: [Bug] Fix bug that NPE thrown when adding partition for table with MV
EmmyMiao87 commented on pull request #7069: URL: https://github.com/apache/incubator-doris/pull/7069#issuecomment-964744822 Please add a precondition in the SlotRef.isNullable(); -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #7069: [Bug] Fix bug that NPE thrown when adding partition for table with MV
morningman commented on pull request #7069: URL: https://github.com/apache/incubator-doris/pull/7069#issuecomment-964752381 > Please add a precondition in the SlotRef.isNullable(); done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #7025: Add tablets number limit to affend wrong usage
morningman commented on a change in pull request #7025: URL: https://github.com/apache/incubator-doris/pull/7025#discussion_r746224515 ## File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java ## @@ -1384,6 +1384,12 @@ @ConfField(mutable = true, masterOnly = true) public static long default_db_data_quota_bytes = 1024L * 1024 * 1024 * 1024 * 1024L; // 1PB +/** + * Used to set default db replica quota num. + */ +@ConfField(mutable = true, masterOnly = true) +public static long default_db_replica_quota_size = 1024 * 1024 * 1024; Review comment: Add this new config in ``` docs/zh-CN/administrator-guide/config/fe_config.md docs/en/administrator-guide/config/fe_config.md ``` ## File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java ## @@ -3254,6 +3254,16 @@ public void addPartition(Database db, String tableName, AddPartitionClause addPa DataProperty dataProperty = singlePartitionDesc.getPartitionDataProperty(); Preconditions.checkNotNull(dataProperty); +// check replica quota if this operation done Review comment: The check here, and check at line L3851-3858 and L3895-3907 can be done in `Catalog.createPartitionWithIndices()`, before `createTablets()` https://github.com/apache/incubator-doris/blob/4c6cbdf4635e48a3465bdfd7302cd3a0faabba42/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java#L3527-L3537 ## File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java ## @@ -6691,6 +6726,19 @@ public void truncateTable(TruncateTableStmt truncateTableStmt) throws DdlExcepti // tabletIdSet to save all newly created tablet ids. Set tabletIdSet = Sets.newHashSet(); try { +// check replica quota if this operation done +long totalReplicaNum = 0; Review comment: For `truncate` operation. I don't think we should check replica quota. Otherwise user may not be able to truncate table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a change in pull request #7032: [Heartbeat] Support cancel outdated query after FE restart
xinyiZzz commented on a change in pull request #7032: URL: https://github.com/apache/incubator-doris/pull/7032#discussion_r746277869 ## File path: be/src/runtime/exec_env.h ## @@ -219,6 +225,9 @@ class ExecEnv { HeartbeatFlags* _heartbeat_flags = nullptr; PluginMgr* _plugin_mgr = nullptr; + +std::map _frontends_start_time; Review comment: done, Thats right -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on a change in pull request #7025: Add tablets number limit to affend wrong usage
tianhui5 commented on a change in pull request #7025: URL: https://github.com/apache/incubator-doris/pull/7025#discussion_r746279532 ## File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java ## @@ -3254,6 +3254,16 @@ public void addPartition(Database db, String tableName, AddPartitionClause addPa DataProperty dataProperty = singlePartitionDesc.getPartitionDataProperty(); Preconditions.checkNotNull(dataProperty); +// check replica quota if this operation done Review comment: L3895-3907,when there are multi partitions, I don't think we can check quota in single partition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] tinkerrrr opened a new pull request #7074: add thrift dir for spark connector
tinke opened a new pull request #7074: URL: https://github.com/apache/incubator-doris/pull/7074 ## Proposed changes Add thrift dir for spark connector, to fix error when building `spark-doris-connector` ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ x ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ x ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #7045: [Feature] make BinaryPredicate do not cast date to datetime/varchar
HappenLee commented on a change in pull request #7045: URL: https://github.com/apache/incubator-doris/pull/7045#discussion_r746303304 ## File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java ## @@ -1022,30 +1027,42 @@ public static Type getCmpType(Type t1, Type t2) { } // int family type and char family type should cast to char family type -if ((t1ResultType.isFixedPointType() && t2ResultType.isCharFamily()) || -(t2ResultType.isFixedPointType() && t1ResultType.isCharFamily())) { -return t1.isStringType() ? t1 : t2; +if ((t1ResultType.isFixedPointType() && t2ResultType.isCharFamily()) +|| (t2ResultType.isFixedPointType() && t1ResultType.isCharFamily())) { +return t1.isStringType() ? t1 : t2; } if (t1ResultType == PrimitiveType.BIGINT && t2ResultType == PrimitiveType.BIGINT) { return getAssignmentCompatibleType(t1, t2, false); } -if ((t1ResultType == PrimitiveType.BIGINT -|| t1ResultType == PrimitiveType.DECIMALV2) -&& (t2ResultType == PrimitiveType.BIGINT -|| t2ResultType == PrimitiveType.DECIMALV2)) { +if ((t1ResultType == PrimitiveType.BIGINT || t1ResultType == PrimitiveType.DECIMALV2) +&& (t2ResultType == PrimitiveType.BIGINT || t2ResultType == PrimitiveType.DECIMALV2)) { return Type.DECIMALV2; } -if ((t1ResultType == PrimitiveType.BIGINT -|| t1ResultType == PrimitiveType.LARGEINT) -&& (t2ResultType == PrimitiveType.BIGINT -|| t2ResultType == PrimitiveType.LARGEINT)) { +if ((t1ResultType == PrimitiveType.BIGINT || t1ResultType == PrimitiveType.LARGEINT) +&& (t2ResultType == PrimitiveType.BIGINT || t2ResultType == PrimitiveType.LARGEINT)) { return Type.LARGEINT; } return Type.DOUBLE; } public static boolean canCompareDate(PrimitiveType t1, PrimitiveType t2) { +if (t1 == PrimitiveType.DATE) { +if (t2 == PrimitiveType.DATE || t2.isStringType() || t2.isIntegerType()) { Review comment: Int compare date shoule cast int, because int are 4byte, date is 16byte。 int is more chance to SIMD -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on issue #7008: [Enhance] use predicate if in bitmap_union_count() can not hit rollup.
EmmyMiao87 commented on issue #7008: URL: https://github.com/apache/incubator-doris/issues/7008#issuecomment-964852704 Now the selector cannot parse the detail column and aggregate column required by the query from expr So it cannot know the k2 is required detail column and the v1 is required bitmap agg column. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] JNSimba opened a new pull request #7075: [Feature] Support Flink and Spark connector support String type
JNSimba opened a new pull request #7075: URL: https://github.com/apache/incubator-doris/pull/7075 ## Proposed changes Support Flink and Spark connector support String type ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on issue #7037: [Bug] sesstion parameter 'query_timeout' setting too large causes overflow
HappenLee commented on issue #7037: URL: https://github.com/apache/incubator-doris/issues/7037#issuecomment-964856549 Thank you, Now master version of doris had fix this problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 opened a new pull request #7076: Add session variable to close join order
EmmyMiao87 opened a new pull request #7076: URL: https://github.com/apache/incubator-doris/pull/7076 ## Proposed changes The new session variable 'close_join_reorder' is used to turn off all automatic join reorder algorithms. If close_join_reorder is true, the Doris will execute query by the order in the original query. ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org