[GitHub] [incubator-doris] liutang123 opened a new issue #6320: Disk monitor hold StorageEngine's _store_lock too long
liutang123 opened a new issue #6320: URL: https://github.com/apache/incubator-doris/issues/6320 1. `StorageEngine::_delete_tablets_on_unused_root_path` will try to obtain tablet shard write lock in `TabletManager` ``` StorageEngine::_delete_tablets_on_unused_root_path TabletManager::drop_tablets_on_error_root_path obtain each tablet shard's write lock ``` 2. `TabletManager::build_all_report_tablets_info` and other methods will obtain tablet shard read lock frequently. So, `StorageEngine::_delete_tablets_on_unused_root_path` will hold `_store_lock` for a long time. This will make it difficult for other threads to get write `_store_lock`, such as `StorageEngine::get_stores_for_create_tablet` `drop_tablets_on_error_root_path` is a small probability event, `TabletManager::drop_tablets_on_error_root_path` should return when its param `tablet_info_vec` is empty -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] liutang123 opened a new pull request #6321: Fix 6320 Optimize lock when check error storage
liutang123 opened a new pull request #6321: URL: https://github.com/apache/incubator-doris/pull/6321 ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [x] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #6321: Fix 6320 Optimize lock when check error storage
morningman commented on a change in pull request #6321:
URL: https://github.com/apache/incubator-doris/pull/6321#discussion_r676120836
##
File path: be/src/olap/tablet_manager.cpp
##
@@ -562,13 +562,22 @@ OLAPStatus TabletManager::_drop_tablet_unlocked(TTabletId
tablet_id, SchemaHash
OLAPStatus TabletManager::drop_tablets_on_error_root_path(
const std::vector& tablet_info_vec) {
OLAPStatus res = OLAP_SUCCESS;
-for (int32 i = 0; i < _tablets_shards_size; i++) {
+if (tablet_info_vec.empty()) { // This is a high probability event
+LOG(INFO) << "tablet_info_vec is empty, return directly.";
Review comment:
I think this log can be removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] liutang123 commented on a change in pull request #6321: Fix 6320 Optimize lock when check error storage
liutang123 commented on a change in pull request #6321:
URL: https://github.com/apache/incubator-doris/pull/6321#discussion_r676121453
##
File path: be/src/olap/tablet_manager.cpp
##
@@ -562,13 +562,22 @@ OLAPStatus TabletManager::_drop_tablet_unlocked(TTabletId
tablet_id, SchemaHash
OLAPStatus TabletManager::drop_tablets_on_error_root_path(
const std::vector& tablet_info_vec) {
OLAPStatus res = OLAP_SUCCESS;
-for (int32 i = 0; i < _tablets_shards_size; i++) {
+if (tablet_info_vec.empty()) { // This is a high probability event
+LOG(INFO) << "tablet_info_vec is empty, return directly.";
Review comment:
ok， i will test this pr in our cluster. And then remove this log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #6314: [Bug] Fix show load like match
morningman commented on pull request #6314: URL: https://github.com/apache/incubator-doris/pull/6314#issuecomment-886179479 This is a historical issue, because the previous `like` operation did not implement wildcards. If you change this way, all online operations using the previous method will fail. Unless we can implement a compatible solution, such as automatically adding `%` symbols. eg: auto convert `like 'abc'` to `like "%abc%"` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #6192: ADD: support sql block rule
morningman commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r676148390
##
File path:
fe/fe-core/src/main/java/org/apache/doris/analysis/AlterSqlBlockRuleStmt.java
##
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.Map;
+
+public class AlterSqlBlockRuleStmt extends DdlStmt {
+
+private final String ruleName;
+
+private String sql;
+
+private String sqlHash;
+
+private Boolean global;
+
+private Boolean enable;
+
+private final Map properties;
+
+public AlterSqlBlockRuleStmt(String ruleName, Map
properties) {
+this.ruleName = ruleName;
+this.properties = properties;
+}
+
+@Override
+public void analyze(Analyzer analyzer) throws UserException {
+super.analyze(analyzer);
+// check auth
+if
(!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(),
PrivPredicate.ADMIN)) {
+
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
"ADMIN");
+}
+// check properties
+CreateSqlBlockRuleStmt.checkCommonProperties(properties);
+setProperties(properties);
+}
+
+private void setProperties(Map properties) {
+this.sql = properties.get(CreateSqlBlockRuleStmt.SQL_PROPERTY);
+this.sqlHash =
properties.get(CreateSqlBlockRuleStmt.SQL_HASH_PROPERTY);
+// allow null, represents no modification
+String globalStr =
properties.get(CreateSqlBlockRuleStmt.GLOBAL_PROPERTY);
+this.global = StringUtils.isNotEmpty(globalStr) ?
Boolean.parseBoolean(globalStr) : null;
+String enableStr =
properties.get(CreateSqlBlockRuleStmt.ENABLE_PROPERTY);
+this.enable = StringUtils.isNotEmpty(enableStr) ?
Boolean.parseBoolean(enableStr) : null;
+}
+
+public String getRuleName() {
+return ruleName;
+}
+
+public String getSql() {
+return sql;
+}
+
+public Boolean getGlobal() {
+return global;
+}
+
+public Boolean getEnable() {
+return enable;
+}
+
+public String getSqlHash() {
+return sqlHash;
+}
Review comment:
Override `toSql()` for all statements
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
##
@@ -172,6 +176,13 @@ private void handleQuery() {
ctx.getState().setError("Unsupported character set(UTF-8)");
return;
}
+try {
+
Catalog.getCurrentCatalog().getSqlBlockRuleMgr().matchSql(originStmt,
ctx.getQualifiedUser());
+} catch (AnalysisException e) {
+LOG.error(e.getMessage());
Review comment:
use warn log
##
File path:
fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.blockrule;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.Cr
[GitHub] [incubator-doris] morningman commented on a change in pull request #6301: Doris developers quickly use the development sample code framework,
morningman commented on a change in pull request #6301: URL: https://github.com/apache/incubator-doris/pull/6301#discussion_r676152510 ## File path: samples/doris-demo/README.md ## @@ -0,0 +1,23 @@ + Review comment: license header -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6279: [Community] Fix PR labeling github action workflow
morningman merged pull request #6279: URL: https://github.com/apache/incubator-doris/pull/6279 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Community] Fix PR labeling github action workflow (#6279)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new 8d1c1ef [Community] Fix PR labeling github action workflow (#6279) 8d1c1ef is described below commit 8d1c1ef1e6b223d1e335f16999823aa4199a16b7 Author: Tobiasz Kędzierski AuthorDate: Sun Jul 25 16:22:09 2021 +0200 [Community] Fix PR labeling github action workflow (#6279) My change is the fix and improvement for github action which labels approved PRs (introduced in this [PR](https://github.com/apache/incubator-doris/pull/6239)). It is inspired by solution introduced and tested in [Apache Airflow](https://github.com/apache/airflow) (thanks @potiuk @ashb 🚀 ) Corresponding Apache Airflow workflows on which I based this PR: - https://github.com/apache/airflow/blob/main/.github/workflows/label_when_reviewed.yml - https://github.com/apache/airflow/blob/main/.github/workflows/label_when_reviewed_workflow_run.yml Problems which were solved in this PR: - **Permissions**. @morningman opened a related bug: [[Help] Error: Resource not accessible by integration](https://github.com/TobKed/label-when-approved-action/issues/7). It is related to limited permissions of workflows being triggered by `pull_request_review` (`GITHUB_TOKEN` has read-only permissions). More information about it you can find in the article: [Keeping your GitHub Actions and workflows secure: Preventing pwn requests](https://securitylab.github.com/research/github-actions-preventing-p [...] TL;DR: On pull request review event (`on: pull_request_review` ) "dummy" workflow `Label when reviewed` triggers another workflow `Label when approved workflow run` which has sufficient permissions (`on: workflow_run: workflows: ["Label when reviewed"]`). - **Safe use of 3rd-party Github Actions by using submodules pattern.** It is decribed in: https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status > NEVER use 3rd-party actions directly in your workflows - use the "submodule" pattern. This pattern is successfully used by projects like: - [Apache Airflow](https://github.com/apache/airflow) ([PR](https://github.com/apache/airflow/pull/13514#)) - [Apache Beam](https://github.com/apache/beam) ([PR](https://github.com/apache/beam/pull/13736)) - [Apache Superset](https://github.com/apache/superset) ([PR](https://github.com/apache/superset/pull/12709)) --- .github/actions/get-workflow-origin | 1 + .github/actions/label-when-approved-action | 1 + .github/workflows/approve-label-trigger.yml | 28 + .github/workflows/approve-label.yml | 48 ++--- .gitmodules | 6 5 files changed, 79 insertions(+), 5 deletions(-) diff --git a/.github/actions/get-workflow-origin b/.github/actions/get-workflow-origin new file mode 16 index 000..588cc14 --- /dev/null +++ b/.github/actions/get-workflow-origin @@ -0,0 +1 @@ +Subproject commit 588cc14f9f1cdf1b8be3db816855e96422204fec diff --git a/.github/actions/label-when-approved-action b/.github/actions/label-when-approved-action new file mode 16 index 000..0058d00 --- /dev/null +++ b/.github/actions/label-when-approved-action @@ -0,0 +1 @@ +Subproject commit 0058d0094da27e116fad6e0da516ebe1107f26de diff --git a/.github/workflows/approve-label-trigger.yml b/.github/workflows/approve-label-trigger.yml new file mode 100644 index 000..1a9cb84 --- /dev/null +++ b/.github/workflows/approve-label-trigger.yml @@ -0,0 +1,28 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: Label when reviewed +on: pull_request_review +jobs: + + label-when-reviewed: +name: "Label PRs when reviewed" +runs-on: ubuntu-latest +steps: + - name: "Do nothing. Only trigger corresponding workflow_run event" +run: echo diff --git a/.github/workflows/approve-label.yml b/.github/workflows/approve-label.yml index dfde7db..f022578 100644 -
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6306: [Metrics][Load] Add metrics to monitor the size of queued tasks in load thread pool
github-actions[bot] commented on pull request #6306: URL: https://github.com/apache/incubator-doris/pull/6306#issuecomment-886209826 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #6298: [Doc]Modify the storage path configuration instructions in the installation and BE configuration documents
morningman commented on a change in pull request #6298: URL: https://github.com/apache/incubator-doris/pull/6298#discussion_r676154678 ## File path: docs/zh-CN/installing/install-deploy.md ## @@ -159,7 +162,23 @@ BROKER 当前没有，也不需要 priority\_networks 这个选项。Broker 的 * 修改所有 BE 的配置 -修改 be/conf/be.conf。主要是配置 `storage_root_path`：数据存放目录。默认在be/storage下，需要**手动创建**该目录。多个路径之间使用 `;` 分隔（最后一个目录后不要加 `;`）。如果be部署在hadoop集群中，注意调整be.conf中的`webserver_port = 8040`,以免造成断开冲突 +修改 be/conf/be.conf。主要是配置 `storage_root_path`：数据存放目录。默认在be/storage下，需要**手动创建**该目录。多个路径之间使用英文状态的分号 `;` 分隔（**最后一个目录后不要加 `;`**）。可以通过路径区别存储目录的介质，HDD或SSD。可以添加容量限制在每个路径的末尾，通过英文状态逗号`,`隔开。 + +**注意：如果是SSD磁盘要在目录后面加上`.SSD`,HDD磁盘在目录后面加`.HDD`** + +示例如下： + + `storage_root_path=/home/disk1/doris.HDD,50;/home/disk2/doris.SSD,10;/home/disk2/doris` + +示例说明 + +- /home/disk1/doris.HDD, 50 ： 表示存储限制为50GB, HDD; +- /home/disk2/doris.SSD 10，存储限制为10GB，SSD； +- /home/disk2/doris，存储限制为磁盘最大容量，默认为HDD + +* BE webserver_port端口配置 + +如果 be 部署在 hadoop 集群中，注意调整 be.conf 中的 `webserver_port = 8040` ,以免造成断开冲突 Review comment: ```suggestion 如果 be 部署在 hadoop 集群中，注意调整 be.conf 中的 `webserver_port = 8040` ,以免造成端口冲突 ``` ## File path: docs/zh-CN/installing/install-deploy.md ## @@ -159,7 +162,23 @@ BROKER 当前没有，也不需要 priority\_networks 这个选项。Broker 的 * 修改所有 BE 的配置 -修改 be/conf/be.conf。主要是配置 `storage_root_path`：数据存放目录。默认在be/storage下，需要**手动创建**该目录。多个路径之间使用 `;` 分隔（最后一个目录后不要加 `;`）。如果be部署在hadoop集群中，注意调整be.conf中的`webserver_port = 8040`,以免造成断开冲突 +修改 be/conf/be.conf。主要是配置 `storage_root_path`：数据存放目录。默认在be/storage下，需要**手动创建**该目录。多个路径之间使用英文状态的分号 `;` 分隔（**最后一个目录后不要加 `;`**）。可以通过路径区别存储目录的介质，HDD或SSD。可以添加容量限制在每个路径的末尾，通过英文状态逗号`,`隔开。 + +**注意：如果是SSD磁盘要在目录后面加上`.SSD`,HDD磁盘在目录后面加`.HDD`** + +示例如下： + + `storage_root_path=/home/disk1/doris.HDD,50;/home/disk2/doris.SSD,10;/home/disk2/doris` + +示例说明 + +- /home/disk1/doris.HDD, 50 ： 表示存储限制为50GB, HDD; Review comment: ```suggestion - /home/disk1/doris.HDD, 50，表示存储限制为50GB, HDD; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6297: [Doc]Compile and automatically create fe doris-meta and be storage directories
github-actions[bot] commented on pull request #6297: URL: https://github.com/apache/incubator-doris/pull/6297#issuecomment-886210694 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #6279: [Community] Fix PR labeling github action workflow
morningman commented on pull request #6279: URL: https://github.com/apache/incubator-doris/pull/6279#issuecomment-886210803 Thanks @TobKed , it works! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6294: [UT] Fix MaterializedViewFunctionTest run failed
github-actions[bot] commented on pull request #6294: URL: https://github.com/apache/incubator-doris/pull/6294#issuecomment-886211155 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #6292: [Optimize] Make light schema change complete more faster
morningman commented on a change in pull request #6292: URL: https://github.com/apache/incubator-doris/pull/6292#discussion_r676156008 ## File path: be/src/common/config.h ## @@ -87,7 +87,7 @@ CONF_Int32(clear_transaction_task_worker_count, "1"); // the count of thread to delete CONF_Int32(delete_worker_count, "3"); // the count of thread to alter table -CONF_Int32(alter_tablet_worker_count, "3"); +CONF_Int32(alter_tablet_worker_count, "6"); Review comment: better not changing the default value if it is not that much necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a change in pull request #6292: [Optimize] Make light schema change complete more faster
caiconghui commented on a change in pull request #6292: URL: https://github.com/apache/incubator-doris/pull/6292#discussion_r676157556 ## File path: be/src/common/config.h ## @@ -87,7 +87,7 @@ CONF_Int32(clear_transaction_task_worker_count, "1"); // the count of thread to delete CONF_Int32(delete_worker_count, "3"); // the count of thread to alter table -CONF_Int32(alter_tablet_worker_count, "3"); +CONF_Int32(alter_tablet_worker_count, "6"); Review comment: ok. maybe it is better decided by user's business scenario, I will restore the value to its original state -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on pull request #6292: [Optimize] Make light schema change complete more faster
caiconghui commented on pull request #6292: URL: https://github.com/apache/incubator-doris/pull/6292#issuecomment-886213876 > I don't get it, why it get faster than before? > I notice that you put a limit number of active jobs in a new thread pool and let them run concurrently? on our production environment, there are usually some tables added columns at the same time, and now these table are loaded data all time by stream load, so We hope that adding column operation can be complished as soon as possible, or stream load can be more slower when it should write two copies of data. now, it waste time to sleep and schema change jobs will affect each other, so here I change the interval for schema change check and add threadpool to make schema change jobs wouldn't affect each other. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui edited a comment on pull request #6292: [Optimize] Make light schema change complete more faster
caiconghui edited a comment on pull request #6292: URL: https://github.com/apache/incubator-doris/pull/6292#issuecomment-886213876 > I don't get it, why it get faster than before? > I notice that you put a limit number of active jobs in a new thread pool and let them run concurrently? on our production environment, there are usually some tables added columns at the same time, and now these table are loaded data all time by stream load, so We hope that adding column operation can be complished as soon as possible, or stream load can be more slower when it should write two copies of data. now, it waste time to sleep and schema change jobs will affect each other, so here I change the interval for schema change check and add threadpool to make schema change jobs wouldn't affect each other. and if alter worker thread count is big enough, even big table with many tablet, adding column operation can be accomplished in short time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui edited a comment on pull request #6292: [Optimize] Make light schema change complete more faster
caiconghui edited a comment on pull request #6292: URL: https://github.com/apache/incubator-doris/pull/6292#issuecomment-886213876 > I don't get it, why it get faster than before? > I notice that you put a limit number of active jobs in a new thread pool and let them run concurrently? on our production environment, there are usually some tables added columns at the same time, and now these table are loaded data all time by stream load, so We hope that adding column operation can be complished as soon as possible, or stream load can be more slower when it should write two copies of data. now, it waste time to sleep and schema change jobs will affect each other, so here I change the interval for schema change check and add threadpool to make schema change jobs wouldn't affect each other. and if alter worker thread count is big enough, even big table with many tablets, adding column operation can be accomplished in short time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui edited a comment on pull request #6292: [Optimize] Make light schema change complete more faster
caiconghui edited a comment on pull request #6292: URL: https://github.com/apache/incubator-doris/pull/6292#issuecomment-886213876 > I don't get it, why it get faster than before? > I notice that you put a limit number of active jobs in a new thread pool and let them run concurrently? on our production environment, there are usually some tables added columns at the same time, and now these table are loaded data all time by stream load, so We hope that adding column operation can be accomplished as soon as possible, or stream load can be more slower when it should write two copies of data. now, it waste time to sleep and schema change jobs will affect each other, so here I change the interval for schema change check and add threadpool to make schema change jobs wouldn't affect each other. and if alter worker thread count is big enough, even big table with many tablets, adding column operation can be accomplished in short time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui closed pull request #5681: [Config] Add config to limit the num of history load job num or history transaction under db to avoid too much memory usage
caiconghui closed pull request #5681: URL: https://github.com/apache/incubator-doris/pull/5681 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6253: [Improvement][Vectorized] Lower/upper case tranfer function vertorized
morningman merged pull request #6253: URL: https://github.com/apache/incubator-doris/pull/6253 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (8d1c1ef -> 13ef2c9)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from 8d1c1ef [Community] Fix PR labeling github action workflow (#6279)
add 13ef2c9 [Function][Enhance] lower/upper case transfer function
vectorized (#6253)
No new revisions were added by this update.
Summary of changes:
be/src/exprs/string_functions.cpp | 14 ++---
.../util/{spinlock.cc => vectorized-tool/lower.h} | 26 -
be/src/util/vectorized-tool/lower_upper_impl.h | 61 ++
.../util/{date_func.h => vectorized-tool/upper.h} | 29 +-
be/test/exprs/string_functions_test.cpp| 19 +++
5 files changed, 113 insertions(+), 36 deletions(-)
copy be/src/util/{spinlock.cc => vectorized-tool/lower.h} (71%)
create mode 100644 be/src/util/vectorized-tool/lower_upper_impl.h
copy be/src/util/{date_func.h => vectorized-tool/upper.h} (68%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (13ef2c9 -> 75d954c)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 13ef2c9 [Function][Enhance] lower/upper case transfer function vectorized (#6253) add 75d954c [Feature] Modify the cost evaluation algorithm of Broadcast and Shuffle Join (#6274) No new revisions were added by this update. Summary of changes: .../apache/doris/planner/DistributedPlanner.java | 53 +-- .../apache/doris/planner/JoinCostEvaluation.java | 153 + .../org/apache/doris/planner/OlapScanNode.java | 24 ++-- .../doris/planner/JoinCostEvaluationTest.java | 115 4 files changed, 282 insertions(+), 63 deletions(-) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/planner/JoinCostEvaluation.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/planner/JoinCostEvaluationTest.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6274: [Feature] Modify the cost evaluation algorithm of Broadcast and Shuffle Join
morningman merged pull request #6274: URL: https://github.com/apache/incubator-doris/pull/6274 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #6272: [Proposal] Modify the cost evaluation algorithm of Broadcast and Shuffle Join
morningman closed issue #6272: URL: https://github.com/apache/incubator-doris/issues/6272 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #6309: change a config called min_bytes_per_broker_scanner to max_bytes_per_broker_scanner as it should be.
morningman closed issue #6309: URL: https://github.com/apache/incubator-doris/issues/6309 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6310: [fix] change min_bytes_per_broker_scanner's Chinese translation.
morningman merged pull request #6310: URL: https://github.com/apache/incubator-doris/pull/6310 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (75d954c -> bf0ba1d)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 75d954c [Feature] Modify the cost evaluation algorithm of Broadcast and Shuffle Join (#6274) add bf0ba1d [Doc-fix] change min_bytes_per_broker_scanner's Chinese translation. (#6310) No new revisions were added by this update. Summary of changes: docs/en/administrator-guide/config/fe_config.md| 4 ++-- docs/zh-CN/administrator-guide/config/fe_config.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6315: fix doc issue
morningman merged pull request #6315: URL: https://github.com/apache/incubator-doris/pull/6315 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (bf0ba1d -> 7b44e7f)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from bf0ba1d [Doc-fix] change min_bytes_per_broker_scanner's Chinese translation. (#6310) add 7b44e7f [Doc-fix] fix doc issue (#6315) No new revisions were added by this update. Summary of changes: docs/zh-CN/administrator-guide/config/be_config.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6305: [Bug] Fix core dump in BloomFilter while build Runtime Filter right table string column contains null
morningman merged pull request #6305: URL: https://github.com/apache/incubator-doris/pull/6305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (7b44e7f -> 4c0fdd2)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 7b44e7f [Doc-fix] fix doc issue (#6315) add 4c0fdd2 [Bug] Fix core dump in BloomFilter while build Runtime Filter right table string column contains null (#6305) No new revisions were added by this update. Summary of changes: be/src/exprs/bloomfilter_predicate.h | 9 - be/src/exprs/runtime_filter.h | 6 -- be/test/exprs/bloom_filter_predicate_test.cpp | 2 ++ 3 files changed, 14 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #6304: [BUG] coredump in RuntimeFilter BloomFilter
morningman closed issue #6304: URL: https://github.com/apache/incubator-doris/issues/6304 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [UT] Fix MaterializedViewFunctionTest run failed (#6294)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new e3db773 [UT] Fix MaterializedViewFunctionTest run failed (#6294)
e3db773 is described below
commit e3db7731493a54b3a9ed1cc7791d691f67333abf
Author: caiconghui <55968745+caicong...@users.noreply.github.com>
AuthorDate: Mon Jul 26 09:42:39 2021 +0800
[UT] Fix MaterializedViewFunctionTest run failed (#6294)
Co-authored-by: caiconghui
---
.../planner/MaterializedViewFunctionTest.java | 22 ++
.../java/org/apache/doris/utframe/DorisAssert.java | 6 +-
2 files changed, 15 insertions(+), 13 deletions(-)
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/planner/MaterializedViewFunctionTest.java
b/fe/fe-core/src/test/java/org/apache/doris/planner/MaterializedViewFunctionTest.java
index c6ed638..790ff59 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/planner/MaterializedViewFunctionTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/planner/MaterializedViewFunctionTest.java
@@ -82,9 +82,9 @@ public class MaterializedViewFunctionTest {
@After
public void afterMethod() throws Exception {
-dorisAssert.dropTable(EMPS_TABLE_NAME);
-dorisAssert.dropTable(DEPTS_TABLE_NAME);
-dorisAssert.dropTable(USER_TAG_TABLE_NAME);
+dorisAssert.dropTable(EMPS_TABLE_NAME, true);
+dorisAssert.dropTable(DEPTS_TABLE_NAME, true);
+dorisAssert.dropTable(USER_TAG_TABLE_NAME, true);
}
@AfterClass
@@ -607,7 +607,7 @@ public class MaterializedViewFunctionTest {
+ "k1,k2;";
String query = "select k1 from " + TEST_TABLE_NAME + " group by k1
having max(v1) > 10;";
dorisAssert.withMaterializedView(createK1K2MV).query(query).explainWithout("k1_k2");
-dorisAssert.dropTable(TEST_TABLE_NAME);
+dorisAssert.dropTable(TEST_TABLE_NAME, true);
}
@Test
@@ -621,7 +621,7 @@ public class MaterializedViewFunctionTest {
+ "k1,k2;";
String query = "select k1 from " + TEST_TABLE_NAME + " group by k1
order by max(v1);";
dorisAssert.withMaterializedView(createK1K2MV).query(query).explainWithout("k1_k2");
-dorisAssert.dropTable(TEST_TABLE_NAME);
+dorisAssert.dropTable(TEST_TABLE_NAME, true);
}
@Test
@@ -635,7 +635,7 @@ public class MaterializedViewFunctionTest {
+ "k1,k2;";
String query = "select k1 , sum(k2) over (partition by v1 ) from " +
TEST_TABLE_NAME + ";";
dorisAssert.withMaterializedView(createK1K2MV).query(query).explainWithout("k1_k2");
-dorisAssert.dropTable(TEST_TABLE_NAME);
+dorisAssert.dropTable(TEST_TABLE_NAME, true);
}
@Test
@@ -647,9 +647,7 @@ public class MaterializedViewFunctionTest {
+ "k1;";
String query = "select * from " + TEST_TABLE_NAME + ";";
dorisAssert.withMaterializedView(createK1K2MV).query(query).explainContains(TEST_TABLE_NAME);
-dorisAssert.dropTable(TEST_TABLE_NAME);
-
-
+dorisAssert.dropTable(TEST_TABLE_NAME, true);
}
@Test
@@ -717,7 +715,7 @@ public class MaterializedViewFunctionTest {
dorisAssert.withTable(aggTable);
String query = "select k1, count(distinct v1) from " + TEST_TABLE_NAME
+ " group by k1;";
dorisAssert.query(query).explainContains(TEST_TABLE_NAME,
"bitmap_union_count");
-dorisAssert.dropTable(TEST_TABLE_NAME);
+dorisAssert.dropTable(TEST_TABLE_NAME, true);
}
@Test
@@ -727,7 +725,7 @@ public class MaterializedViewFunctionTest {
dorisAssert.withTable(aggTable);
String query = "select k1, count(distinct v1) from " + TEST_TABLE_NAME
+ " group by k1;";
dorisAssert.query(query).explainContains(TEST_TABLE_NAME,
"hll_union_agg");
-dorisAssert.dropTable(TEST_TABLE_NAME);
+dorisAssert.dropTable(TEST_TABLE_NAME, true);
}
@Test
@@ -831,6 +829,6 @@ public class MaterializedViewFunctionTest {
dorisAssert.withMaterializedView(createMVSQL);
String query = "select count(distinct salary) from agg_table;";
dorisAssert.query(query).explainContains("mv");
-dorisAssert.dropTable("agg_table");
+dorisAssert.dropTable("agg_table", true);
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java
b/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java
index 758a81e..df7d23a 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java
@@ -88,8 +88,12 @@ public class DorisAssert {
}
public DorisAssert dropTable(String tableName) throws Exception {
+return dropTable(tableName, false);
[GitHub] [incubator-doris] morningman merged pull request #6294: [UT] Fix MaterializedViewFunctionTest run failed
morningman merged pull request #6294: URL: https://github.com/apache/incubator-doris/pull/6294 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #6295: [UT] Run MaterializedViewFunctionTest failed
morningman closed issue #6295: URL: https://github.com/apache/incubator-doris/issues/6295 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6230: [Update] Support update syntax
github-actions[bot] commented on pull request #6230: URL: https://github.com/apache/incubator-doris/pull/6230#issuecomment-886310907 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6214: [Profile] Support show load profile for broker load job
github-actions[bot] commented on pull request #6214: URL: https://github.com/apache/incubator-doris/pull/6214#issuecomment-886316433 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6301: Doris developers quickly use the development sample code framework,
morningman merged pull request #6301: URL: https://github.com/apache/incubator-doris/pull/6301 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6301: Doris developers quickly use the development sample code framework,
github-actions[bot] commented on pull request #6301: URL: https://github.com/apache/incubator-doris/pull/6301#issuecomment-886321757 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (e3db773 -> 88a96d6)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from e3db773 [UT] Fix MaterializedViewFunctionTest run failed (#6294) add 88a96d6 [Sample] Doris developers quickly use the development sample code framework, (#6301) No new revisions were added by this update. Summary of changes: .../doris-demo/README.md | 43 ++- samples/doris-demo/flink-demo/pom.xml | 49 ++ samples/doris-demo/pom.xml | 44 +++ .../doris-demo/spark-demo/pom.xml | 47 + samples/doris-demo/spring-jdbc-demo/pom.xml| 47 + samples/doris-demo/stream-load-demo/pom.xml| 49 ++ 6 files changed, 220 insertions(+), 59 deletions(-) copy docs/en/administrator-guide/http-actions/fe/logout-action.md => samples/doris-demo/README.md (52%) create mode 100644 samples/doris-demo/flink-demo/pom.xml create mode 100644 samples/doris-demo/pom.xml copy docs/en/sql-reference/sql-functions/string-functions/replace.md => samples/doris-demo/spark-demo/pom.xml (54%) create mode 100644 samples/doris-demo/spring-jdbc-demo/pom.xml create mode 100644 samples/doris-demo/stream-load-demo/pom.xml - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new issue #6322: [Feature] Support for cleaning the trash actively
BiteThet opened a new issue #6322:
URL: https://github.com/apache/incubator-doris/issues/6322
**Is your feature request related to a problem? Please describe.**
Sometimes, garbage data will accumulate and take up a lot of space, users
who want to clean up trash can only delete files by themselves.
**Describe the solution you'd like**
Add a sql statement for cleaning the trash actively.
format:
`clean trash;`
`clean trash on ("BackendHost1:BackendHeartBeatPort1",
"BackendHost2:BackendHeartBeatPort2", ...);`
**Additional context**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new pull request #6323: [Feature] Support for cleaning the trash actively
BiteThet opened a new pull request #6323: URL: https://github.com/apache/incubator-doris/pull/6323 ## Proposed changes Support for cleaning the trash actively. User can use 'CLEAN TRASH' to clean trash. ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [X] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [X] I have created an issue on (Fix #6322) and described the bug/feature there in detail - [X] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [X] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on pull request #6323: [Feature] Support for cleaning the trash actively
EmmyMiao87 commented on pull request #6323: URL: https://github.com/apache/incubator-doris/pull/6323#issuecomment-886332546 Please enrich your commit msg -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #6323: [Feature] Support for cleaning the trash actively
EmmyMiao87 commented on a change in pull request #6323:
URL: https://github.com/apache/incubator-doris/pull/6323#discussion_r676260313
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/DdlExecutor.java
##
@@ -237,6 +238,8 @@ public static void execute(Catalog catalog, DdlStmt
ddlStmt) throws Exception {
catalog.getResourceMgr().createResource((CreateResourceStmt)
ddlStmt);
} else if (ddlStmt instanceof DropResourceStmt) {
catalog.getResourceMgr().dropResource((DropResourceStmt) ddlStmt);
+} else if(ddlStmt instanceof CleanTrashStmt) {
Review comment:
```suggestion
} else if (ddlStmt instanceof CleanTrashStmt) {
```
##
File path: docs/zh-CN/sql-reference/sql-statements/Administration/CLEAN TRASH.md
##
@@ -0,0 +1,47 @@
+---
+{
+"title": "CLEAN TRASH",
+"language": "zh-CN"
+}
+---
+
+
+
+# CLEAN TRASH
+## description
+该语句用于清理 backend 内的垃圾数据。
Review comment:
Will this statement clean up both trash and snapshot?
Will cleaning up the snapshot involve the snapshot being restored?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt commented on a change in pull request #6323: [Feature] Support for cleaning the trash actively
BiteThet commented on a change in pull request #6323:
URL: https://github.com/apache/incubator-doris/pull/6323#discussion_r676265235
##
File path: docs/zh-CN/sql-reference/sql-statements/Administration/CLEAN TRASH.md
##
@@ -0,0 +1,47 @@
+---
+{
+"title": "CLEAN TRASH",
+"language": "zh-CN"
+}
+---
+
+
+
+# CLEAN TRASH
+## description
+该语句用于清理 backend 内的垃圾数据。
Review comment:
>
>
> Will this statement clean up both trash and snapshot?
> Will cleaning up the snapshot involve the snapshot being restored?
Yes, this statement will clean up both trash and snapshot.
This statement will call `StorageEngine::start_trash_sweep`.
It will only clean up expired data (define at
`config::snapshot_expire_time_sec`/`config::trash_file_expire_time_sec`).
And this function will be automatically called periodically, so I think its
cleanup is harmless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on issue #6303: DataX-dorisdb-writer load data failed
hf200012 commented on issue #6303: URL: https://github.com/apache/incubator-doris/issues/6303#issuecomment-886342793 The specific software of dorisdb, dorisdb and apache doris are not the same product, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #6323: [Feature] Support for cleaning the trash actively
EmmyMiao87 commented on a change in pull request #6323: URL: https://github.com/apache/incubator-doris/pull/6323#discussion_r676266112 ## File path: docs/.vuepress/sidebar/en.js ## @@ -451,6 +451,7 @@ module.exports = [ "SHOW MIGRATIONS", "SHOW PLUGINS", "SHOW TABLE STATUS", + "CLEAN TRASH", Review comment: You need to add both guide and sql reference -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt commented on a change in pull request #6323: [Feature] Support for cleaning the trash actively
BiteThet commented on a change in pull request #6323: URL: https://github.com/apache/incubator-doris/pull/6323#discussion_r676319331 ## File path: docs/.vuepress/sidebar/en.js ## @@ -451,6 +451,7 @@ module.exports = [ "SHOW MIGRATIONS", "SHOW PLUGINS", "SHOW TABLE STATUS", + "CLEAN TRASH", Review comment: > > > You need to add both guide and sql reference https://github.com/apache/incubator-doris/pull/6323/commits/c31dd3e5d111da8e8cde3f00bd9367af49c04a96 I add description at `/administrator-guide/operation/disk-capacity.md` . At the same time, I found that there is no corresponding English version of this document. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
