(doris) branch cp_1213_2 updated (4131d3c111d -> 318087e17ac)
This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a change to branch cp_1213_2 in repository https://gitbox.apache.org/repos/asf/doris.git from 4131d3c111d make sink operator process eos signals after wake_up_early add 318087e17ac Revert "[fix](memory) Disable Jemalloc Hook and loading libjvm at runtime (#45210)" No new revisions were added by this update. Summary of changes: be/src/http/default_path_handlers.cpp | 2 +- be/src/runtime/CMakeLists.txt | 4 +++- be/src/runtime/memory/heap_profiler.cpp | 8 be/src/util/mem_info.cpp| 2 +- be/src/util/mem_info.h | 10 +- be/src/vec/common/allocator.h | 2 +- thirdparty/build-thirdparty.sh | 4 +--- 7 files changed, 16 insertions(+), 16 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Cp 1213 2 [doris]
BiteThet commented on PR #45400: URL: https://github.com/apache/doris/pull/45400#issuecomment-2540817099 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](cloud) Make get tablet stats and update delete bitmap update lock be able to be in different fdb txns [doris]
github-actions[bot] commented on code in PR #45206:
URL: https://github.com/apache/doris/pull/45206#discussion_r1883522012
##
cloud/test/meta_service_test.cpp:
##
@@ -4530,6 +4579,231 @@ TEST(MetaServiceTest, GetDeleteBitmapUpdateLock) {
ASSERT_EQ(res.status().code(), MetaServiceCode::OK);
}
+TEST(MetaServiceTest, GetDeleteBitmapUpdateLockTabletStatsNormal) {
+auto meta_service = get_meta_service();
+
+std::string instance_id = "test_get_delete_bitmap_update_lock_normal";
+[[maybe_unused]] auto* sp = SyncPoint::get_instance();
+std::unique_ptr> defer((int*)0x01, [](int*)
{
+SyncPoint::get_instance()->disable_processing();
+SyncPoint::get_instance()->clear_all_call_backs();
+});
+sp->set_call_back("get_instance_id", [&](auto&& args) {
+auto* ret = try_any_cast_ret(args);
+ret->first = instance_id;
+ret->second = true;
+});
+sp->enable_processing();
+
+int64_t db_id = 1000;
+int64_t table_id = 2001;
+int64_t index_id = 3001;
+// [(partition_id, tablet_id)]
+std::vector> tablet_idxes {{70001, 12345}, {80001,
3456}, {90001, 6789}};
+
+add_tablet_stats(meta_service.get(), instance_id, table_id, index_id,
tablet_idxes);
+
+GetDeleteBitmapUpdateLockResponse res;
+get_delete_bitmap_update_lock(meta_service.get(), res, db_id, table_id,
index_id, tablet_idxes,
+ 5, 99, -1, true);
+ASSERT_EQ(res.status().code(), MetaServiceCode::OK);
+
+ASSERT_EQ(res.base_compaction_cnts().size(), tablet_idxes.size());
+for (const auto& base_compaction_cnt : res.base_compaction_cnts()) {
+ASSERT_EQ(base_compaction_cnt, 10);
+}
+ASSERT_EQ(res.cumulative_compaction_cnts().size(), tablet_idxes.size());
+for (const auto& cumu_compaction_cnt : res.cumulative_compaction_cnts()) {
+ASSERT_EQ(cumu_compaction_cnt, 20);
+}
+ASSERT_EQ(res.cumulative_points().size(), tablet_idxes.size());
+for (const auto& cumulative_point : res.cumulative_points()) {
+ASSERT_EQ(cumulative_point, 30);
+}
+}
+
+TEST(MetaServiceTest, GetDeleteBitmapUpdateLockTabletStatsLockExpired) {
Review Comment:
warning: function 'TEST' exceeds recommended size/complexity thresholds
[readability-function-size]
```cpp
TEST(MetaServiceTest, GetDeleteBitmapUpdateLockTabletStatsLockExpired) {
^
```
Additional context
**cloud/test/meta_service_test.cpp:4624:** 81 lines including whitespace and
comments (threshold 80)
```cpp
TEST(MetaServiceTest, GetDeleteBitmapUpdateLockTabletStatsLockExpired) {
^
```
##
cloud/test/meta_service_test.cpp:
##
@@ -4530,6 +4579,231 @@
ASSERT_EQ(res.status().code(), MetaServiceCode::OK);
}
+TEST(MetaServiceTest, GetDeleteBitmapUpdateLockTabletStatsNormal) {
+auto meta_service = get_meta_service();
+
+std::string instance_id = "test_get_delete_bitmap_update_lock_normal";
+[[maybe_unused]] auto* sp = SyncPoint::get_instance();
+std::unique_ptr> defer((int*)0x01, [](int*)
{
+SyncPoint::get_instance()->disable_processing();
+SyncPoint::get_instance()->clear_all_call_backs();
+});
+sp->set_call_back("get_instance_id", [&](auto&& args) {
+auto* ret = try_any_cast_ret(args);
+ret->first = instance_id;
+ret->second = true;
+});
+sp->enable_processing();
+
+int64_t db_id = 1000;
+int64_t table_id = 2001;
+int64_t index_id = 3001;
+// [(partition_id, tablet_id)]
+std::vector> tablet_idxes {{70001, 12345}, {80001,
3456}, {90001, 6789}};
+
+add_tablet_stats(meta_service.get(), instance_id, table_id, index_id,
tablet_idxes);
+
+GetDeleteBitmapUpdateLockResponse res;
+get_delete_bitmap_update_lock(meta_service.get(), res, db_id, table_id,
index_id, tablet_idxes,
+ 5, 99, -1, true);
+ASSERT_EQ(res.status().code(), MetaServiceCode::OK);
+
+ASSERT_EQ(res.base_compaction_cnts().size(), tablet_idxes.size());
+for (const auto& base_compaction_cnt : res.base_compaction_cnts()) {
+ASSERT_EQ(base_compaction_cnt, 10);
+}
+ASSERT_EQ(res.cumulative_compaction_cnts().size(), tablet_idxes.size());
+for (const auto& cumu_compaction_cnt : res.cumulative_compaction_cnts()) {
+ASSERT_EQ(cumu_compaction_cnt, 20);
+}
+ASSERT_EQ(res.cumulative_points().size(), tablet_idxes.size());
+for (const auto& cumulative_point : res.cumulative_points()) {
+ASSERT_EQ(cumulative_point, 30);
+}
+}
+
+TEST(MetaServiceTest, GetDeleteBitmapUpdateLockTabletStatsLockExpired) {
+auto meta_service = get_meta_service();
+
+{
+// 2.1 abnormal path, lock has been expired and taken by another
load/compaction during
+// the reading of tablet stats
+std::string instance_id =
"test_get_delete_bitmap_update_lock_abnormal1";
+[[maybe_unused]] auto* sp = Syn
(doris-website) branch master updated: [feat]:searchbar in mobile (#1517)
This is an automated email from the ASF dual-hosted git repository.
jeffreyh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new fcb4a82eb4 [feat]:searchbar in mobile (#1517)
fcb4a82eb4 is described below
commit fcb4a82eb4f09783e0c80380d9ff02ccd2e7d707
Author: yangon <2689991...@qq.com>
AuthorDate: Fri Dec 13 16:08:02 2024 +0800
[feat]:searchbar in mobile (#1517)
-
Co-authored-by: liyang
---
src/scss/common.scss | 1 -
src/scss/components/navbar.scss| 2 +-
src/scss/components/search.scss| 3 -
src/theme/Layout/index.tsx | 64 +++--
src/theme/Navbar/Content/components/NavbarDocs.tsx | 35 ++-
src/theme/Navbar/Content/index.tsx | 19 +-
src/theme/Navbar/Layout/index.tsx | 5 +-
src/theme/Navbar/MobileSidebar/Layout/index.tsx| 40 +--
src/theme/Navbar/MobileSidebar/Layout/style.scss | 25 ++
src/theme/Navbar/Search/index.tsx | 2 +-
src/theme/Navbar/Search/styles.module.css | 9 +-
src/theme/SearchBar/SearchBar.jsx | 305 +++--
src/theme/SearchBar/SearchBar.module.css | 17 +-
start-docs.sh | 65 -
static/images/icon/close.svg | 6 +
worker.js | 7 -
16 files changed, 319 insertions(+), 286 deletions(-)
diff --git a/src/scss/common.scss b/src/scss/common.scss
index d675602913..bfa4a29cd0 100644
--- a/src/scss/common.scss
+++ b/src/scss/common.scss
@@ -2,7 +2,6 @@
.docs-wrapper {
.navbar__inner {
.navbar__items--right {
-.docs-search,
.custom-navbar-item {
display: none;
}
diff --git a/src/scss/components/navbar.scss b/src/scss/components/navbar.scss
index 0680fc9201..12588c4c46 100644
--- a/src/scss/components/navbar.scss
+++ b/src/scss/components/navbar.scss
@@ -328,7 +328,7 @@
height: 3.75rem;
&.docs,
&.community {
-height: 8.1rem;
+height: 6.5rem;
flex-direction: column;
.navbar__inner {
height: 3.75rem;
diff --git a/src/scss/components/search.scss b/src/scss/components/search.scss
index d8a739112f..2864f134ba 100644
--- a/src/scss/components/search.scss
+++ b/src/scss/components/search.scss
@@ -56,9 +56,6 @@ p[class*='searchResultItemSummary'] {
}
@media screen and (max-width: 996px) {
-.docs-search {
-display: none;
-}
.docs-subnav-searchbar {
.navbar-search {
position: relative !important;
diff --git a/src/theme/Layout/index.tsx b/src/theme/Layout/index.tsx
index 8fdbac9145..a9c3a379a5 100644
--- a/src/theme/Layout/index.tsx
+++ b/src/theme/Layout/index.tsx
@@ -1,4 +1,4 @@
-import React, { useEffect } from 'react';
+import React, { useEffect, createContext, useContext, useState, useRef } from
'react';
import clsx from 'clsx';
import ErrorBoundary from '@docusaurus/ErrorBoundary';
import { PageMetadata, SkipToContentFallbackId, ThemeClassNames } from
'@docusaurus/theme-common';
@@ -9,10 +9,19 @@ import Navbar from '@theme/Navbar';
import Footer from '@theme/Footer';
import LayoutProvider from '@theme/Layout/Provider';
import ErrorPageContent from '@theme/ErrorPageContent';
+import NavbarSearch from '../Navbar/Search';
+import SearchBar from '@theme/SearchBar';
import type { Props } from '@theme/Layout';
import styles from './styles.module.css';
import { useHistory } from '@docusaurus/router';
+interface DataType {
+showSearchPageMobile: boolean;
+setShowSearchPageMobile: React.Dispatch>;
+}
+
+export const DataContext = createContext(null);
+
export default function Layout(props: Props): JSX.Element {
const {
children,
@@ -23,37 +32,54 @@ export default function Layout(props: Props): JSX.Element {
description,
} = props;
const history = useHistory();
+const [showSearchPageMobile, setShowSearchPageMobile] = useState(false);
+const searchPageDom = useRef(null);
useKeyboardNavigation();
useEffect(() => {
if (
history.location.pathname?.length > 1 &&
history.location.pathname[history.location.pathname.length - 1]
=== '/'
-){
+) {
const params = location.href.split(history.location.pathname)[1];
history.replace(history.location.pathname.slice(0, -1) + params);
}
-
}, [history.location]);
-return (
-
-
-
-
-
-
+useEffect(() => {
+if (showSearchPageMobile)
Re: [PR] [feature](backup) Backup and restore cooldown data [doris]
justfortaste commented on PR #44993: URL: https://github.com/apache/doris/pull/44993#issuecomment-2540830012 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat]:searchbar in mobile [doris-website]
jeffreys-cat merged PR #1517: URL: https://github.com/apache/doris-website/pull/1517 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cluster key) add some case [doris]
doris-robot commented on PR #45383: URL: https://github.com/apache/doris/pull/45383#issuecomment-2540830386 TeamCity be ut coverage result: Function Coverage: 38.75% (10108/26082) Line Coverage: 29.69% (84853/285767) Region Coverage: 28.76% (43561/151447) Branch Coverage: 25.32% (22130/87414) Coverage Report: http://coverage.selectdb-in.cc/coverage/e274b4d67b7f0d9df8f93eb05413aba7dd406bb9_e274b4d67b7f0d9df8f93eb05413aba7dd406bb9/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](cloud) Make get tablet stats and update delete bitmap update lock be able to be in different fdb txns [doris]
bobhan1 commented on PR #45206: URL: https://github.com/apache/doris/pull/45206#issuecomment-2540893561 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](datalake) Add BucketShuffleJoin support for bucketed hive tables [doris]
github-actions[bot] commented on PR #27784: URL: https://github.com/apache/doris/pull/27784#issuecomment-2540893941 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Insert-Job)Resolving Task Status Discrepancies in Query Execution Records [doris]
github-actions[bot] commented on PR #45342: URL: https://github.com/apache/doris/pull/45342#issuecomment-2540895451 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Insert-Job)Resolving Task Status Discrepancies in Query Execution Records [doris]
github-actions[bot] commented on PR #45342: URL: https://github.com/apache/doris/pull/45342#issuecomment-2540895352 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) fix cases unstable when using profile to check is nereids execution [doris]
github-actions[bot] commented on PR #45384: URL: https://github.com/apache/doris/pull/45384#issuecomment-2540891925 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](iceberg) Fill in the detailed error information [doris]
hello-stephen commented on PR #45415: URL: https://github.com/apache/doris/pull/45415#issuecomment-2541299302 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Exec](expr) Opt the compound pred performace [doris]
github-actions[bot] commented on PR #45414: URL: https://github.com/apache/doris/pull/45414#issuecomment-2541288739 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Exec](expr) Opt the compound pred performace [doris]
github-actions[bot] commented on PR #45414: URL: https://github.com/apache/doris/pull/45414#issuecomment-2541288808 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](iceberg) Fill in the detailed error information [doris]
wuwenchi opened a new pull request, #45415: URL: https://github.com/apache/doris/pull/45415 ### What problem does this PR solve? Related PR: #45285 Problem Summary: When dropping a database, fill in the detailed error information. ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](iceberg) Fill in the detailed error information [doris]
wuwenchi commented on PR #45415: URL: https://github.com/apache/doris/pull/45415#issuecomment-2541299756 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix]Remove be special handling of date types and bugs in registration functions [doris]
koarz commented on PR #45159: URL: https://github.com/apache/doris/pull/45159#issuecomment-2541311561 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](regression) regression out file print sql [doris]
yujun777 commented on PR #44887: URL: https://github.com/apache/doris/pull/44887#issuecomment-2540881910 > 1. how about set it as suite level, not sql level? and dump by default. > 2. it is useful for groovy script, not sql script. Make sure only works for groovy cases 1. not support suitlevel for groovy script 2. sql script also support dump sql. by default, sql script will dump sql to its out file. if user doesn't want dump sql into its out file, the script file name need contains '_no_dump_sql' -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve](nereids) use hash set replace three set in DiscreteValue to improve in predicate performance [doris]
dqz123 commented on PR #45181: URL: https://github.com/apache/doris/pull/45181#issuecomment-2540884322 run performance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Enhancement] (nereids)implement RefreshTableCommand in nereids [doris]
vinlee19 commented on issue #42578: URL: https://github.com/apache/doris/issues/42578#issuecomment-2541245735 Please assign it to me! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Exec](expr) Opt the compound pred performace [doris]
github-actions[bot] commented on code in PR #45414:
URL: https://github.com/apache/doris/pull/45414#discussion_r1883809470
##
be/src/vec/exprs/vcompound_pred.h:
##
@@ -231,6 +243,32 @@ class VCompoundPred : public VectorizedFnCall {
return null_map_data;
};
+auto vector_vector = [&]() {
+if (lhs_mem_can_reuse) {
+*result_column_id = lhs_id;
+} else if (rhs_mem_can_reuse) {
+*result_column_id = rhs_id;
+
+auto tmp_column = rhs_data_column;
Review Comment:
warning: 'auto tmp_column' can be declared as 'auto *tmp_column'
[readability-qualified-auto]
```suggestion
auto *tmp_column = rhs_data_column;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in profile [doris]
gavinchou commented on code in PR #44863:
URL: https://github.com/apache/doris/pull/44863#discussion_r1883824758
##
be/src/io/io_common.h:
##
@@ -60,6 +61,7 @@ struct IOContext {
int64_t expiration_time = 0;
const TUniqueId* query_id = nullptr; // Ref
FileCacheStatistics* file_cache_stats = nullptr; // Ref
+bool is_inverted_index = false;
Review Comment:
is this field used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: branch-3.0: [fix](coordinator) Fix wrong bucket assignments by coordinator #45365 (#45401)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new f53bb5d7ff7 branch-3.0: [fix](coordinator) Fix wrong bucket
assignments by coordinator #45365 (#45401)
f53bb5d7ff7 is described below
commit f53bb5d7ff72f917db8e89d5cf51f2e7ed840352
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Dec 13 17:10:24 2024 +0800
branch-3.0: [fix](coordinator) Fix wrong bucket assignments by coordinator
#45365 (#45401)
Cherry-picked from #45365
Co-authored-by: Gabriel
---
.../main/java/org/apache/doris/qe/Coordinator.java | 85 +++-
.../test_colocate_join_with_different_tablets.out | 12 +++
...est_colocate_join_with_different_tablets.groovy | 109 +
3 files changed, 182 insertions(+), 24 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
index 0f6bc0212d1..2889e943e93 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
@@ -2734,7 +2734,8 @@ public class Coordinator implements CoordInterface {
* 1. `parallelExecInstanceNum * numBackends` is larger than scan
ranges.
* 2. Use Nereids planner.
*/
-boolean ignoreStorageDataDistribution = params.fragment != null &&
params.fragment.useSerialSource(context);
+boolean ignoreStorageDataDistribution = scanNodes != null &&
!scanNodes.isEmpty()
+&& params.fragment != null &&
params.fragment.useSerialSource(context);
FragmentScanRangeAssignment assignment = params.scanRangeAssignment;
for (Map.Entry addressScanRange
@@ -2744,33 +2745,69 @@ public class Coordinator implements CoordInterface {
= findOrInsert(assignment, addressScanRange.getKey(), new
HashMap<>());
if (ignoreStorageDataDistribution) {
-FInstanceExecParam instanceParam = new FInstanceExecParam(
-null, addressScanRange.getKey(), 0, params);
-
-for (Pair>>
nodeScanRangeMap : scanRange) {
-for (Map.Entry>
nodeScanRange
-: nodeScanRangeMap.second.entrySet()) {
-if
(!instanceParam.perNodeScanRanges.containsKey(nodeScanRange.getKey())) {
-range.put(nodeScanRange.getKey(),
Lists.newArrayList());
-
instanceParam.perNodeScanRanges.put(nodeScanRange.getKey(),
Lists.newArrayList());
+List perInstanceScanRanges
+= ListUtil.splitBySize(scanRange,
parallelExecInstanceNum);
+/**
+ * Split scan ranges evenly into `parallelExecInstanceNum`
instances.
+ *
+ *
+ * For a fragment contains co-located join,
+ *
+ * scan (id = 0) -> join build (id = 2)
+ * |
+ * scan (id = 1) -> join probe (id = 2)
+ *
+ * If both of `scan (id = 0)` and `scan (id = 1)` are serial
operators, we will plan local exchanger
+ * after them:
+ *
+ * scan (id = 0) -> local exchange -> join build (id = 2)
+ * |
+ * scan (id = 1) -> local exchange -> join probe (id = 2)
+ *
+ *
+ * And there is another more complicated scenario, for
example, `scan (id = 0)` has 10 partitions and
+ * 3 buckets which means 3 * 10 tablets and `scan (id = 1)`
has 3 buckets and no partition which means
+ * 3 tablets totally. If expected parallelism is 8, we will
get a serial scan (id = 0) and a
+ * non-serial scan (id = 1). For this case, we will plan
another plan with local exchange:
+ *
+ * scan (id = 0) -> local exchange -> join build (id = 2)
+ * |
+ * scan (id = 1) -> join probe (id = 2)
+ */
+FInstanceExecParam firstInstanceParam = null;
+for (List>>>
perInstanceScanRange
+: perInstanceScanRanges) {
+FInstanceExecParam instanceParam = new FInstanceExecParam(
+null, addressScanRange.getKey(), 0, params);
+
+if (firstInstanceParam == null) {
+firstInstanceParam = instanceParam;
+
Re: [PR] branch-3.0: [fix](coordinator) Fix wrong bucket assignments by coordinator #45365 [doris]
Gabriel39 merged PR #45401: URL: https://github.com/apache/doris/pull/45401 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-45365-branch-3.0 deleted (was bce90819466)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-45365-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git was bce90819466 [fix](coordinator) Fix wrong bucket assignments by coordinator (#45365) The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](shuffle)(part III) Use local exchanger for both exchange sink and lo… [doris]
github-actions[bot] commented on PR #45375: URL: https://github.com/apache/doris/pull/45375#issuecomment-2540944912 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](pipeline) make sink operator process eos signals after wake_up_early [doris]
BiteThet commented on code in PR #45207:
URL: https://github.com/apache/doris/pull/45207#discussion_r1883606855
##
be/src/pipeline/exec/exchange_sink_operator.cpp:
##
@@ -186,9 +186,6 @@ void
ExchangeSinkLocalState::on_channel_finished(InstanceLoId channel_id) {
_finished_channels.emplace(channel_id);
if (_working_channels_count.fetch_sub(1) == 1) {
set_reach_limit();
-if (_finish_dependency) {
Review Comment:
set_reach_limit 实际上会使pipeline
task触发is_finished的逻辑，这个pr中触发is_finished会直接set_wake_up_early，以统一的方式来set
dependency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](meta-service) Support dynamic configuration for meta service [doris]
TangSiyang2001 commented on PR #45394: URL: https://github.com/apache/doris/pull/45394#issuecomment-2540960142 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](pipeline) make sink operator process eos signals after wake_up_early [doris]
BiteThet commented on code in PR #45207:
URL: https://github.com/apache/doris/pull/45207#discussion_r1883617408
##
be/src/pipeline/exec/hashjoin_build_sink.cpp:
##
@@ -135,26 +135,17 @@ Status HashJoinBuildSinkLocalState::close(RuntimeState*
state, Status exec_statu
}
}};
-if (!_runtime_filter_slots || _runtime_filters.empty() ||
state->is_cancelled()) {
+if (!_runtime_filter_slots || _runtime_filters.empty() ||
state->is_cancelled() ||
+!p.get_local_state(state)._eos) {
return Base::close(state, exec_status);
}
try {
-if (state->get_task()->wake_up_by_downstream()) {
-if (_should_build_hash_table) {
-// partitial ignore rf to make global rf work
-RETURN_IF_ERROR(
-_runtime_filter_slots->send_filter_size(state, 0,
_finish_dependency));
-RETURN_IF_ERROR(_runtime_filter_slots->ignore_all_filters());
-} else {
-// do not publish filter coz local rf not inited and useless
-return Base::close(state, exec_status);
-}
+if (state->get_task()->wake_up_early()) {
Review Comment:
例如build shared hash table的broad cast场景，非build shared hash
table的instance自己先提前close了，此时它虽然在sink中有eos，但是没获取到shared context，不能正常的做rf
publish，需要进到这个分支里ignore所有filter再publish。
wake_up_early意味着rf对应的部分数据已经不需要了，而且此时也不一定能正确处理构造（比如没有sync filter
size/没有获取到shared context），所以需要ignore all filters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](pipeline) make sink operator process eos signals after wake_up_early [doris]
yiguolei commented on code in PR #45207:
URL: https://github.com/apache/doris/pull/45207#discussion_r1883605372
##
be/src/pipeline/pipeline_task.cpp:
##
@@ -480,7 +470,10 @@ Status PipelineTask::close(Status exec_status) {
Status s;
{
SCOPED_RAW_TIMER(&close_ns);
-s = _sink->close(_state, exec_status);
+if (close_sink) {
+_task_profile->add_info_string("WakeUpEarly", wake_up_early() ?
"true" : "false");
+s = _sink->close(_state, exec_status);
+}
for (auto& op : _operators) {
auto tem = op->close(_state);
Review Comment:
这个close 函数，如果被调用2次，中间这些update statistics 都不一定能够重入。
我们这么改：
if (!close_sink) {
for (auto& op : _operators) {}
} else {
int64_t close_ns = 0;
Defer defer {[&]() {
if (_task_queue) {
_task_queue->update_statistics(this, close_ns);
}
}};
Status s;
{
SCOPED_RAW_TIMER(&close_ns);
if (close_sink) {
_task_profile->add_info_string("WakeUpEarly", wake_up_early() ?
"true" : "false");
s = _sink->close(_state, exec_status);
}
}
if (_opened) {
_fresh_profile_counter();
COUNTER_SET(_close_timer, close_ns);
COUNTER_UPDATE(_task_profile->total_time_counter(), close_ns);
}
return s;
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regression-test](framework) disable defining global variable in test groovy files [doris]
hello-stephen merged PR #44576: URL: https://github.com/apache/doris/pull/44576 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) range inference convert isnull to EmptyValue [doris]
github-actions[bot] commented on PR #45310: URL: https://github.com/apache/doris/pull/45310#issuecomment-2540905906 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset [doris]
gavinchou merged PR #43466: URL: https://github.com/apache/doris/pull/43466 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43466-branch-3.0 created (now 122d5745fc3)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43466-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at 122d5745fc3 branch-3.0: [test](index compaction)Add index compaction exception fault injection cases #45127 (#45396) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset (#43466)
This is an automated email from the ASF dual-hosted git repository.
gavinchou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 55f450cd00e [Enhancement](compaction) Do not set failure time when
cumulative compaction dealing with delete rowset (#43466)
55f450cd00e is described below
commit 55f450cd00ead0728b592abc2b1020aee83ca009
Author: abmdocrt
AuthorDate: Fri Dec 13 16:50:44 2024 +0800
[Enhancement](compaction) Do not set failure time when cumulative
compaction dealing with delete rowset (#43466)
Before this PR, in cases where there is an alternating distribution of
data rowset -> delete rowset -> data rowset -> delete rowset, cumulative
compaction would only move the cumulative point forward to allow base
compaction to handle the delete rowset. Cumulative compaction itself
would not process the data and would return be marked as failure. This
would cause the compaction submission task process to pause for 5
seconds, impacting efficiency.
This PR modifies the return status to OK for such cases, which improves
the efficiency of the compaction submission task.
---
be/src/cloud/cloud_cumulative_compaction.cpp | 4 +
be/src/cloud/cloud_storage_engine.cpp | 3 +-
be/src/common/config.cpp | 1 +
be/src/common/config.h | 2 +
be/src/olap/tablet.cpp | 13 ++-
.../test_cumu_compaction_with_delete.out | 5 +
.../test_cumu_compaction_with_delete.groovy| 122 +
7 files changed, 148 insertions(+), 2 deletions(-)
diff --git a/be/src/cloud/cloud_cumulative_compaction.cpp
b/be/src/cloud/cloud_cumulative_compaction.cpp
index 6b4dbe360da..1acf8efe32e 100644
--- a/be/src/cloud/cloud_cumulative_compaction.cpp
+++ b/be/src/cloud/cloud_cumulative_compaction.cpp
@@ -92,6 +92,10 @@ PREPARE_TRY_AGAIN:
// plus 1 to skip the delete version.
// NOTICE: after that, the cumulative point may be larger than max
version of this tablet, but it doesn't matter.
update_cumulative_point();
+if (!config::enable_sleep_between_delete_cumu_compaction) {
+st = Status::Error(
+"_last_delete_version.first not equal to -1");
+}
}
return st;
}
diff --git a/be/src/cloud/cloud_storage_engine.cpp
b/be/src/cloud/cloud_storage_engine.cpp
index b66a9cfbdb2..650909a2915 100644
--- a/be/src/cloud/cloud_storage_engine.cpp
+++ b/be/src/cloud/cloud_storage_engine.cpp
@@ -677,7 +677,8 @@ Status
CloudStorageEngine::_submit_cumulative_compaction_task(const CloudTabletS
auto st = compaction->prepare_compact();
if (!st.ok()) {
long now =
duration_cast(system_clock::now().time_since_epoch()).count();
-if (st.is()) {
+if (st.is() &&
+st.msg() != "_last_delete_version.first not equal to -1") {
// Backoff strategy if no suitable version
tablet->last_cumu_no_suitable_version_ms = now;
}
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index b70f492a29d..95a3e61fb55 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -1402,6 +1402,7 @@ DEFINE_mBool(enable_delete_bitmap_merge_on_compaction,
"false");
// Enable validation to check the correctness of table size.
DEFINE_Bool(enable_table_size_correctness_check, "false");
DEFINE_Bool(force_regenerate_rowsetid_on_start_error, "false");
+DEFINE_mBool(enable_sleep_between_delete_cumu_compaction, "false");
// clang-format off
#ifdef BE_TEST
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 984024cab7c..f8a9c3f7480 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -1487,6 +1487,8 @@ DECLARE_Bool(force_regenerate_rowsetid_on_start_error);
DECLARE_mBool(enable_delete_bitmap_merge_on_compaction);
// Enable validation to check the correctness of table size.
DECLARE_Bool(enable_table_size_correctness_check);
+// Enable sleep 5s between delete cumulative compaction.
+DECLARE_mBool(enable_sleep_between_delete_cumu_compaction);
#ifdef BE_TEST
// test s3
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index a1a56507ffc..705d0ad5309 100644
--- a/be/src/olap/tablet.cpp
+++ b/be/src/olap/tablet.cpp
@@ -1750,8 +1750,13 @@ Status Tablet::prepare_compaction_and_calculate_permits(
}
if (!res.ok()) {
-tablet->set_last_cumu_compaction_failure_time(UnixMillis());
permits = 0;
+// if we meet a delete version, should increase the cumulative
point to let base compaction handle the delete version.
+// no need to wait 5s.
+if (!(res.msg() == "_last_delete_version.first not equal to -1") ||
+config::enable_sleep_between_delete_
(doris) branch auto-pick-43466-branch-3.0 updated (122d5745fc3 -> 1ccd088f271)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch auto-pick-43466-branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
from 122d5745fc3 branch-3.0: [test](index compaction)Add index compaction
exception fault injection cases #45127 (#45396)
add 1ccd088f271 [Enhancement](compaction) Do not set failure time when
cumulative compaction dealing with delete rowset (#43466)
No new revisions were added by this update.
Summary of changes:
be/src/cloud/cloud_cumulative_compaction.cpp | 4 +
be/src/cloud/cloud_storage_engine.cpp | 3 +-
be/src/common/config.cpp | 1 +
be/src/common/config.h | 2 +
be/src/olap/tablet.cpp | 13 ++-
.../test_cumu_compaction_with_delete.out} | 0
.../test_cumu_compaction_with_delete.groovy| 122 +
7 files changed, 143 insertions(+), 2 deletions(-)
copy regression-test/data/{delete_p0/test_new_delete.out =>
compaction/test_cumu_compaction_with_delete.out} (100%)
create mode 100644
regression-test/suites/compaction/test_cumu_compaction_with_delete.groovy
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset #43466 [doris]
hello-stephen commented on PR #45404: URL: https://github.com/apache/doris/pull/45404#issuecomment-2540903953 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset #43466 [doris]
dataroaring closed pull request #45404: branch-3.0: [Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset #43466 URL: https://github.com/apache/doris/pull/45404 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset #43466 [doris]
hello-stephen commented on PR #45404: URL: https://github.com/apache/doris/pull/45404#issuecomment-2540904072 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-3.0: [Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset #43466 [doris]
github-actions[bot] opened a new pull request, #45404: URL: https://github.com/apache/doris/pull/45404 Cherry-picked from #43466 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43466-branch-2.1 created (now 384a1a4bd70)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43466-branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git at 384a1a4bd70 [branch-2.1]log more detail when query hits policy (#44685) (#45366) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/02: improve log
This is an automated email from the ASF dual-hosted git repository.
jacktengg pushed a commit to branch spill_and_reserve
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 7982d9c69098540ec11535b6469f3db708012a67
Author: jacktengg
AuthorDate: Thu Dec 12 23:06:53 2024 +0800
improve log
---
.../exec/partitioned_hash_join_sink_operator.cpp | 38 ++--
be/src/pipeline/pipeline_task.cpp | 40 ++
be/src/runtime/query_context.cpp | 19 +-
3 files changed, 55 insertions(+), 42 deletions(-)
diff --git a/be/src/pipeline/exec/partitioned_hash_join_sink_operator.cpp
b/be/src/pipeline/exec/partitioned_hash_join_sink_operator.cpp
index abf81a068e7..09e13eb465d 100644
--- a/be/src/pipeline/exec/partitioned_hash_join_sink_operator.cpp
+++ b/be/src/pipeline/exec/partitioned_hash_join_sink_operator.cpp
@@ -29,6 +29,7 @@
#include "pipeline/pipeline_task.h"
#include "runtime/fragment_mgr.h"
#include "util/mem_info.h"
+#include "util/pretty_printer.h"
#include "util/runtime_profile.h"
#include "vec/spill/spill_stream.h"
#include "vec/spill/spill_stream_manager.h"
@@ -286,8 +287,8 @@ Status
PartitionedHashJoinSinkLocalState::_revoke_unpartitioned_block(
Status PartitionedHashJoinSinkLocalState::revoke_memory(
RuntimeState* state, const std::shared_ptr&
spill_context) {
SCOPED_TIMER(_spill_total_timer);
-VLOG_DEBUG << "Query: " << print_id(state->query_id()) << ", task: " <<
state->task_id()
- << " hash join sink " << _parent->node_id() << " revoke_memory"
+VLOG_DEBUG << "Query: " << print_id(state->query_id()) << ", task " <<
state->task_id()
+ << " sink " << _parent->node_id() << " revoke_memory"
<< ", eos: " << _child_eos;
DCHECK_EQ(_spilling_task_count, 0);
CHECK_EQ(_spill_dependency->is_blocked_by(nullptr), nullptr);
@@ -312,9 +313,9 @@ Status PartitionedHashJoinSinkLocalState::revoke_memory(
Status status;
if (_child_eos) {
-VLOG_DEBUG << "Query:" << print_id(this->state()->query_id()) <<
", hash join sink "
- << _parent->node_id() << " set_ready_to_read"
- << ", task id: " << state->task_id();
+VLOG_DEBUG << "Query:" << print_id(this->state()->query_id()) <<
", task "
+ << state->task_id() << " sink " << _parent->node_id()
+ << " set_ready_to_read";
std::for_each(_shared_state->partitioned_build_blocks.begin(),
_shared_state->partitioned_build_blocks.end(),
[&](auto& block) {
if (block) {
@@ -398,9 +399,8 @@ Status PartitionedHashJoinSinkLocalState::revoke_memory(
}
if (_child_eos) {
-VLOG_DEBUG << "Query:" << print_id(state->query_id()) << ", hash join
sink "
- << _parent->node_id() << " set_ready_to_read"
- << ", task id: " << state->task_id();
+VLOG_DEBUG << "Query:" << print_id(state->query_id()) << ", task " <<
state->task_id()
+ << " sink " << _parent->node_id() << " set_ready_to_read";
std::for_each(_shared_state->partitioned_build_blocks.begin(),
_shared_state->partitioned_build_blocks.end(), [&](auto&
block) {
if (block) {
@@ -587,9 +587,9 @@ Status PartitionedHashJoinSinkOperatorX::sink(RuntimeState*
state, vectorized::B
const auto need_to_spill = local_state._shared_state->need_to_spill;
if (rows == 0) {
if (eos) {
-VLOG_DEBUG << "Query: " << print_id(state->query_id()) << ", hash
join sink "
- << node_id() << " sink eos, set_ready_to_read"
- << ", task id: " << state->task_id() << ", need spill:
" << need_to_spill;
+VLOG_DEBUG << "Query: " << print_id(state->query_id()) << ", task
" << state->task_id()
+ << " sink " << node_id() << " eos, set_ready_to_read"
+ << ", need spill: " << need_to_spill;
if (need_to_spill) {
return revoke_memory(state, nullptr);
@@ -605,11 +605,11 @@ Status
PartitionedHashJoinSinkOperatorX::sink(RuntimeState* state, vectorized::B
Defer defer {[&]() { local_state.update_memory_usage(); }};
RETURN_IF_ERROR(_inner_sink_operator->sink(
local_state._shared_state->inner_runtime_state.get(),
in_block, eos));
-VLOG_DEBUG << "Query: " << print_id(state->query_id()) <<
"hash join sink "
- << node_id() << " sink eos, set_ready_to_read"
- << ", task id: " << state->task_id() << ", nonspill
build usage: "
- << _inner_sink_operator->get_memory_usage(
-
local_state._shared_state->inner_runtime_state.get());
+
Re: [PR] [opt](hms table)Some optimizations for hms external table [doris]
wuwenchi commented on PR #44909: URL: https://github.com/apache/doris/pull/44909#issuecomment-2540909245 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 02/02: fix problem of cannot spill afterr hash join sink build table table finishe
This is an automated email from the ASF dual-hosted git repository.
jacktengg pushed a commit to branch spill_and_reserve
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c68e3293befea482303fe33aa7d5b0ee184af311
Author: jacktengg
AuthorDate: Fri Dec 13 16:32:01 2024 +0800
fix problem of cannot spill afterr hash join sink build table table finishe
---
be/src/pipeline/pipeline_task.cpp | 16 +++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/be/src/pipeline/pipeline_task.cpp
b/be/src/pipeline/pipeline_task.cpp
index a099ac66421..a8b7dd06b27 100644
--- a/be/src/pipeline/pipeline_task.cpp
+++ b/be/src/pipeline/pipeline_task.cpp
@@ -396,6 +396,7 @@ Status PipelineTask::execute(bool* eos) {
return status;
});
+auto workload_group = _state->get_query_ctx()->workload_group();
if (_pending_block) [[unlikely]] {
LOG(INFO) << "Query: " << print_id(query_id)
<< " has pending block, size: " <<
_pending_block->allocated_bytes();
@@ -414,7 +415,6 @@ Status PipelineTask::execute(bool* eos) {
const auto reserve_size = _root->get_reserve_mem_size(_state);
_root->reset_reserve_mem_size(_state);
-auto workload_group = _state->get_query_ctx()->workload_group();
if (workload_group && _state->enable_reserve_memory() &&
reserve_size > 0) {
auto st = thread_context()->try_reserve_memory(reserve_size);
@@ -484,6 +484,20 @@ Status PipelineTask::execute(bool* eos) {
*eos = false;
continue;
}
+if (workload_group) {
+bool is_low_watermark = false;
+bool is_high_watermark = false;
+workload_group->check_mem_used(&is_low_watermark,
&is_high_watermark);
+// for hash join build sink, if it's eos at this reserve,
it will build hash table and
+// it will not be able to spill later even if memory is
low, and will cause cancel of queries.
+// So make a check here, if it's low watermark after
reserve and if reserved memory is too many,
+// then trigger revoke memory.
+if (is_low_watermark &&
+sink_reserve_size >= workload_group->memory_limit() *
0.05) {
+RETURN_IF_ERROR(_sink->revoke_memory(_state, nullptr));
+continue;
+}
+}
}
// Define a lambda function to catch sink exception, because sink
will check
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch spill_and_reserve updated (57446b70f43 -> c68e3293bef)
This is an automated email from the ASF dual-hosted git repository. jacktengg pushed a change to branch spill_and_reserve in repository https://gitbox.apache.org/repos/asf/doris.git from 57446b70f43 fix local exchanger low mem mode new 7982d9c6909 improve log new c68e3293bef fix problem of cannot spill afterr hash join sink build table table finishe The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../exec/partitioned_hash_join_sink_operator.cpp | 38 +++ be/src/pipeline/pipeline_task.cpp | 56 -- be/src/runtime/query_context.cpp | 19 3 files changed, 70 insertions(+), 43 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Fix](Insert-Job)Resolving Task Status Discrepancies in Query Execution Records (#45342)
This is an automated email from the ASF dual-hosted git repository.
kirs pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 41e554dd7ef [Fix](Insert-Job)Resolving Task Status Discrepancies in
Query Execution Records (#45342)
41e554dd7ef is described below
commit 41e554dd7efca2762244f2cc27eed936b31e63f8
Author: Calvin Kirs
AuthorDate: Fri Dec 13 16:57:10 2024 +0800
[Fix](Insert-Job)Resolving Task Status Discrepancies in Query Execution
Records (#45342)
### What problem does this PR solve?
When querying the execution records of tasks, the current system has two
sources that may describe task status:
Task status in the scheduling module: Represents the processing status
of tasks within the scheduling system.
Task status in LoadManager: Specifically refers to the status of
Insert-type tasks, containing more detailed execution information.
When the data in the LoadManager is deleted or expires, the completion
status will revert to the task status from the scheduling system, which
could cause confusion. Therefore, consistency needs to be maintained.
Since these two sources may have inconsistencies in their status
information, they can easily cause confusion for users and developers.
---
.../main/java/org/apache/doris/job/extensions/insert/InsertTask.java | 2 +-
regression-test/suites/job_p0/test_base_insert_job.groovy | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/InsertTask.java
b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/InsertTask.java
index 23a367d5d6e..a577250dc86 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/InsertTask.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/InsertTask.java
@@ -244,7 +244,7 @@ public class InsertTask extends AbstractTask {
trow.addToColumnValue(new
TCell().setStringVal(String.valueOf(getJobId(;
trow.addToColumnValue(new TCell().setStringVal(jobName));
trow.addToColumnValue(new TCell().setStringVal(getJobId() +
LABEL_SPLITTER + getTaskId()));
-trow.addToColumnValue(new
TCell().setStringVal(jobInfo.getState().name()));
+trow.addToColumnValue(new TCell().setStringVal(getStatus().name()));
trow.addToColumnValue(new TCell().setStringVal(getErrorMsg()));
// create time
trow.addToColumnValue(new
TCell().setStringVal(TimeUtils.longToTimeString(getCreateTimeMs(;
diff --git a/regression-test/suites/job_p0/test_base_insert_job.groovy
b/regression-test/suites/job_p0/test_base_insert_job.groovy
index 2bdf96cd5bd..33ae28443b2 100644
--- a/regression-test/suites/job_p0/test_base_insert_job.groovy
+++ b/regression-test/suites/job_p0/test_base_insert_job.groovy
@@ -116,7 +116,7 @@ suite("test_base_insert_job") {
def taskStatus = sql """select status from tasks("type"="insert") where
JobName ='${jobName}'"""
for (int i = 0; i < taskStatus.size(); i++) {
-assert taskStatus.get(i).get(0) =="CANCELED" ||
taskStatus.get(i).get(0) =="FINISHED"
+assert taskStatus.get(i).get(0) =="CANCELED" ||
taskStatus.get(i).get(0) =="SUCCESS"
}
sql """
CREATE JOB ${jobMixedName} ON SCHEDULE every 1 second DO insert into
${tableName} (timestamp, type, user_id) values ('2023-03-18','1','12213');
@@ -168,7 +168,7 @@ suite("test_base_insert_job") {
// table should have one record after job finished
assert datas.size() == 1
// one time job only has one task. when job finished, task status should
be FINISHED
-assert datas.get(0).get(0) == "FINISHED"
+assert datas.get(0).get(0) == "SUCCESS"
// check table data
def dataCount1 = sql """select count(1) from ${tableName} where
user_id=1001"""
assert dataCount1.get(0).get(0) == 1
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-45342-branch-3.0 created (now 122d5745fc3)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-45342-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at 122d5745fc3 branch-3.0: [test](index compaction)Add index compaction exception fault injection cases #45127 (#45396) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Insert-Job)Resolving Task Status Discrepancies in Query Execution Records [doris]
CalvinKirs merged PR #45342: URL: https://github.com/apache/doris/pull/45342 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-45342-branch-3.0 updated (122d5745fc3 -> efc5fa13cfc)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-45342-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from 122d5745fc3 branch-3.0: [test](index compaction)Add index compaction exception fault injection cases #45127 (#45396) add efc5fa13cfc [Fix](Insert-Job)Resolving Task Status Discrepancies in Query Execution Records (#45342) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/job/extensions/insert/InsertTask.java | 2 +- regression-test/suites/job_p0/test_base_insert_job.groovy | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Fix](Insert-Job)Resolving Task Status Discrepancies in Query Execution Records #45342 [doris]
Thearas commented on PR #45405: URL: https://github.com/apache/doris/pull/45405#issuecomment-2540915266 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-3.0: [Fix](Insert-Job)Resolving Task Status Discrepancies in Query Execution Records #45342 [doris]
github-actions[bot] opened a new pull request, #45405: URL: https://github.com/apache/doris/pull/45405 Cherry-picked from #45342 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Fix](Insert-Job)Resolving Task Status Discrepancies in Query Execution Records #45342 [doris]
Thearas commented on PR #45405: URL: https://github.com/apache/doris/pull/45405#issuecomment-2540915377 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-45342-branch-2.1 created (now 384a1a4bd70)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-45342-branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git at 384a1a4bd70 [branch-2.1]log more detail when query hits policy (#44685) (#45366) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Fix](Insert-Job)Resolving Task Status Discrepancies in Query Execution Records #45342 [doris]
dataroaring closed pull request #45405: branch-3.0: [Fix](Insert-Job)Resolving Task Status Discrepancies in Query Execution Records #45342 URL: https://github.com/apache/doris/pull/45405 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](datalake) Add BucketShuffleJoin support for bucketed hive tables [doris]
github-actions[bot] commented on PR #27784: URL: https://github.com/apache/doris/pull/27784#issuecomment-2540917175 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](shuffle)(part III) Use local exchanger for both exchange sink and lo… [doris]
Gabriel39 commented on PR #45375: URL: https://github.com/apache/doris/pull/45375#issuecomment-2540928269 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](shuffle)(part III) Use local exchanger for both exchange sink and lo… [doris]
Gabriel39 commented on PR #45375: URL: https://github.com/apache/doris/pull/45375#issuecomment-2540932486 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](pipeline) make sink operator process eos signals after wake_up_early [doris]
yiguolei commented on code in PR #45207:
URL: https://github.com/apache/doris/pull/45207#discussion_r1883594208
##
be/src/pipeline/exec/exchange_sink_operator.cpp:
##
@@ -186,9 +186,6 @@ void
ExchangeSinkLocalState::on_channel_finished(InstanceLoId channel_id) {
_finished_channels.emplace(channel_id);
if (_working_channels_count.fetch_sub(1) == 1) {
set_reach_limit();
-if (_finish_dependency) {
Review Comment:
why remove this code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Fix](Insert-Job)Resolving Task Status Discrepancies in Query Execution Records #45342 [doris]
doris-robot commented on PR #45405: URL: https://github.com/apache/doris/pull/45405#issuecomment-2540988724 TPC-H: Total hot run time: 40082 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit efc5fa13cfc2892a3f5e8cdd445fe21d4e2df729, data reload: false -- Round 1 -- q1 17667 734872557255 q2 2056171 167 167 q3 10720 105011211050 q4 10547 714 664 664 q5 7725280027692769 q6 241 147 149 147 q7 969 607 592 592 q8 9573190819811908 q9 7038633963316331 q10 6937226622712266 q11 454 258 260 258 q12 400 212 205 205 q13 17784 295829742958 q14 229 217 203 203 q15 550 514 526 514 q16 652 594 613 594 q17 954 601 520 520 q18 7176660165036503 q19 1546997 962 962 q20 492 190 191 190 q21 3965310330603060 q22 1084979 966 966 Total cold run time: 108759 ms Total hot run time: 40082 ms - Round 2, with runtime_filter_mode=off - q1 7275722271947194 q2 323 235 231 231 q3 2828280028292800 q4 2024174118051741 q5 5624569357025693 q6 222 137 139 137 q7 2187175317851753 q8 3283352734603460 q9 8662880487768776 q10 3507350034893489 q11 609 510 496 496 q12 762 602 622 602 q13 16449 314931303130 q14 306 269 275 269 q15 566 512 518 512 q16 697 669 667 667 q17 1816161616241616 q18 8222785275147514 q19 4664153614371437 q20 2069185618341834 q21 5474520652395206 q22 10701029988 988 Total cold run time: 78639 ms Total hot run time: 59545 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](pipeline) make sink operator process eos signals after wake_up_early [doris]
BiteThet commented on code in PR #45207:
URL: https://github.com/apache/doris/pull/45207#discussion_r1883634420
##
be/src/pipeline/pipeline_task.cpp:
##
@@ -480,7 +470,10 @@ Status PipelineTask::close(Status exec_status) {
Status s;
{
SCOPED_RAW_TIMER(&close_ns);
-s = _sink->close(_state, exec_status);
+if (close_sink) {
+_task_profile->add_info_string("WakeUpEarly", wake_up_early() ?
"true" : "false");
+s = _sink->close(_state, exec_status);
+}
for (auto& op : _operators) {
auto tem = op->close(_state);
Review Comment:
我改了一下，现在应该可以重入了，直接return的话会漏统计一些时间
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [fix](txn insert) txn insert show error url #45254 [doris]
dataroaring merged PR #45381: URL: https://github.com/apache/doris/pull/45381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: branch-2.1: [fix](txn insert) txn insert show error url #45254 (#45381)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 6dabdd6a8ce branch-2.1: [fix](txn insert) txn insert show error url
#45254 (#45381)
6dabdd6a8ce is described below
commit 6dabdd6a8ce38aeed2033b65cfa7a2181c7569ef
Author: meiyi
AuthorDate: Fri Dec 13 17:36:02 2024 +0800
branch-2.1: [fix](txn insert) txn insert show error url #45254 (#45381)
Cherry-picked from https://github.com/apache/doris/pull/45254
---
be/src/runtime/stream_load/stream_load_executor.cpp | 9 +++--
.../suites/insert_p0/transaction/txn_insert.groovy| 15 ++-
2 files changed, 21 insertions(+), 3 deletions(-)
diff --git a/be/src/runtime/stream_load/stream_load_executor.cpp
b/be/src/runtime/stream_load/stream_load_executor.cpp
index e285d38208e..cb165a51f3d 100644
--- a/be/src/runtime/stream_load/stream_load_executor.cpp
+++ b/be/src/runtime/stream_load/stream_load_executor.cpp
@@ -85,13 +85,18 @@ Status
StreamLoadExecutor::execute_plan_fragment(std::shared_ptrnumber_unselected_rows = state->num_rows_load_unselected();
ctx->loaded_bytes = state->num_bytes_load_total();
int64_t num_selected_rows = ctx->number_total_rows -
ctx->number_unselected_rows;
+ctx->error_url =
to_load_error_http_path(state->get_error_log_file_path());
if (!ctx->group_commit && num_selected_rows > 0 &&
(double)ctx->number_filtered_rows / num_selected_rows >
ctx->max_filter_ratio) {
// NOTE: Do not modify the error message here, for historical
reasons,
// some users may rely on this error message.
-*status = Status::DataQualityError("too many filtered rows");
+if (ctx->need_commit_self) {
+*status =
+Status::DataQualityError("too many filtered rows, url:
" + ctx->error_url);
+} else {
+*status = Status::DataQualityError("too many filtered rows");
+}
}
-ctx->error_url =
to_load_error_http_path(state->get_error_log_file_path());
if (status->ok()) {
DorisMetrics::instance()->stream_receive_bytes_total->increment(ctx->receive_bytes);
diff --git a/regression-test/suites/insert_p0/transaction/txn_insert.groovy
b/regression-test/suites/insert_p0/transaction/txn_insert.groovy
index ef3d46141a7..b98dd51e0cd 100644
--- a/regression-test/suites/insert_p0/transaction/txn_insert.groovy
+++ b/regression-test/suites/insert_p0/transaction/txn_insert.groovy
@@ -82,7 +82,7 @@ suite("txn_insert") {
sql """ DROP TABLE IF EXISTS $tableMV """
sql """
create table $tableMV (
-id int default '10',
+id int not null,
c1 int default '10'
) distributed by hash(id, c1)
properties('replication_num'="1");
@@ -97,6 +97,19 @@ suite("txn_insert") {
order_qt_select5 """select * from $tableMV"""
order_qt_select6 """select c1 from $tableMV"""
} while (0);
+do {
+try {
+sql "begin"
+sql """insert into $tableMV values(9, 2), (10, 4)"""
+sql """insert into $tableMV values(null, 6)"""
+sql "commit"
+} catch (Exception e) {
+sql "rollback"
+logger.info("insert into $tableMV failed: " + e.getMessage())
+assertTrue(e.getMessage().contains("too many filtered rows"))
+assertTrue(e.getMessage().contains("url"))
+}
+} while (0);
// --- insert into select ---
for (int j = 0; j < 3; j++) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](hive) fix block decompressor bug #45289 [doris]
dataroaring closed pull request #45377: branch-3.0: [fix](hive) fix block decompressor bug #45289 URL: https://github.com/apache/doris/pull/45377 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](hive) fix block decompressor bug #45289 [doris]
dataroaring commented on PR #45377: URL: https://github.com/apache/doris/pull/45377#issuecomment-2540997133 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](coordinator) Fix wrong bucket assignments by coordinator #45365 [doris]
doris-robot commented on PR #45401: URL: https://github.com/apache/doris/pull/45401#issuecomment-2540834283 TPC-H: Total hot run time: 40308 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit bce90819492f7744dd0f3b695643e20a286d, data reload: false -- Round 1 -- q1 17600 736173127312 q2 2068166 155 155 q3 10759 189711381138 q4 10554 732 725 725 q5 7710284528062806 q6 233 145 146 145 q7 978 622 605 605 q8 9565191419091909 q9 8122633563906335 q10 6963228822942288 q11 448 259 254 254 q12 408 215 209 209 q13 17765 299229392939 q14 236 206 207 206 q15 553 513 494 494 q16 706 624 599 599 q17 965 527 588 527 q18 7154667264396439 q19 14791063945 945 q20 464 197 194 194 q21 4032315831033103 q22 10511002981 981 Total cold run time: 109813 ms Total hot run time: 40308 ms - Round 2, with runtime_filter_mode=off - q1 7355724872957248 q2 321 229 235 229 q3 2955289428742874 q4 2042182917461746 q5 5650570357065703 q6 243 149 145 145 q7 2180177317741773 q8 3330350933463346 q9 8826882388418823 q10 3536353034553455 q11 599 499 500 499 q12 796 607 602 602 q13 16486 313431223122 q14 312 274 275 274 q15 569 519 521 519 q16 714 680 664 664 q17 1834159315621562 q18 8199773773467346 q19 5093159814761476 q20 2117187318151815 q21 5405508552395085 q22 1126104010271027 Total cold run time: 79688 ms Total hot run time: 59333 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](backup) backup and restore privileges catalogs workloadgroup [doris]
justfortaste commented on PR #44905: URL: https://github.com/apache/doris/pull/44905#issuecomment-2540831904 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](nereids) add session variable disable_nereids_expression_rules [doris]
yujun777 commented on PR #44831: URL: https://github.com/apache/doris/pull/44831#issuecomment-2540833081 run performance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](tvf) Tvf supports to parse the enclose character in csv files [doris]
BePPPower commented on PR #45407: URL: https://github.com/apache/doris/pull/45407#issuecomment-2541060489 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](tvf) Tvf supports to parse the enclose character in csv files [doris]
hello-stephen commented on PR #45407: URL: https://github.com/apache/doris/pull/45407#issuecomment-2541058787 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) range inference convert isnull to EmptyValue [doris]
morrySnow commented on code in PR #45310:
URL: https://github.com/apache/doris/pull/45310#discussion_r1883674353
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/AddMinMax.java:
##
@@ -326,7 +326,10 @@ private Map
getExprMinMaxValues(UnknownValue valueDesc)
for (int i = 1; i < sourceValues.size(); i++) {
// process in sourceValues[i]
Map minMaxValues =
getExprMinMaxValues(sourceValues.get(i));
-for (Map.Entry entry :
minMaxValues.entrySet()) {
+List> minMaxValueList =
minMaxValues.entrySet().stream()
+.sorted((a, b) ->
Integer.compare(a.getValue().exprOrderIndex, b.getValue().exprOrderIndex))
+.collect(Collectors.toList());
+for (Map.Entry entry : minMaxValueList) {
Review Comment:
add these comment into code please
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/RangeInference.java:
##
@@ -461,6 +475,9 @@ public boolean isAnd() {
@Override
public ValueDesc union(ValueDesc other) {
+if (other instanceof EmptyValue) {
+return other.union(this);
+}
Review Comment:
add these comment into code please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
svn commit: r73672 - /dev/doris/flink-connector/24.1.0/ /release/doris/flink-connector/24.1.0/
Author: diwu Date: Fri Dec 13 10:02:09 2024 New Revision: 73672 Log: move doris flink connector 24.1.0 to release Added: release/doris/flink-connector/24.1.0/ - copied from r73671, dev/doris/flink-connector/24.1.0/ Removed: dev/doris/flink-connector/24.1.0/ - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) range inference convert isnull to EmptyValue [doris]
yujun777 commented on code in PR #45310:
URL: https://github.com/apache/doris/pull/45310#discussion_r1883686340
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/AddMinMax.java:
##
@@ -326,7 +326,10 @@ private Map
getExprMinMaxValues(UnknownValue valueDesc)
for (int i = 1; i < sourceValues.size(); i++) {
// process in sourceValues[i]
Map minMaxValues =
getExprMinMaxValues(sourceValues.get(i));
-for (Map.Entry entry :
minMaxValues.entrySet()) {
+List> minMaxValueList =
minMaxValues.entrySet().stream()
+.sorted((a, b) ->
Integer.compare(a.getValue().exprOrderIndex, b.getValue().exprOrderIndex))
+.collect(Collectors.toList());
+for (Map.Entry entry : minMaxValueList) {
Review Comment:
had add
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) range inference convert isnull to EmptyValue [doris]
yujun777 commented on code in PR #45310:
URL: https://github.com/apache/doris/pull/45310#discussion_r1883686643
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/RangeInference.java:
##
@@ -461,6 +475,9 @@ public boolean isAnd() {
@Override
public ValueDesc union(ValueDesc other) {
+if (other instanceof EmptyValue) {
+return other.union(this);
+}
Review Comment:
had add
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](tvf) Tvf supports to parse the enclose character in csv files [doris]
BePPPower opened a new pull request, #45407: URL: https://github.com/apache/doris/pull/45407 ### What problem does this PR solve? Problem Summary: Tvf supports to parse the enclose character in csv files ### Check List (For Author) - Test - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [x] Yes. - Does this need documentation? - [ ] No. - [x] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](regression) regression out file print sql [doris]
yujun777 commented on PR #44887: URL: https://github.com/apache/doris/pull/44887#issuecomment-2541058236 > why need dump sql to out file? @morrySnow 两种场景 1.有些同学一个groovy里写了几十个sql，但没有给每个sql 区分标签，这样某个sql错了，就很难查找 2.如果groovy里有很多循环等逻辑操作，很难确定跑出问题的sql具体内容（sql 是拼接得来的） -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](hive) fix block decompressor bug #45289 [doris]
doris-robot commented on PR #45377: URL: https://github.com/apache/doris/pull/45377#issuecomment-2541078217 TPC-H: Total hot run time: 40464 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 0f23d3ef78c04841272bc469107cec9b42287adb, data reload: false -- Round 1 -- q1 17578 739074267390 q2 2067170 176 170 q3 10855 144111451145 q4 10561 775 741 741 q5 7728278127592759 q6 235 152 147 147 q7 965 611 601 601 q8 9584193018961896 q9 8133636864216368 q10 70102315 q11 456 251 251 251 q12 394 215 206 206 q13 17796 298129792979 q14 244 201 208 201 q15 557 525 531 525 q16 698 604 613 604 q17 946 595 560 560 q18 7124652365156515 q19 1698106110031003 q20 467 195 189 189 q21 3881317830023002 q22 1102990 1002990 Total cold run time: 110079 ms Total hot run time: 40464 ms - Round 2, with runtime_filter_mode=off - q1 7366720172167201 q2 320 233 229 229 q3 2911284028452840 q4 2042176717721767 q5 5654566556885665 q6 226 146 145 145 q7 2210177017591759 q8 3276346734353435 q9 8859878787628762 q10 3515354234963496 q11 581 503 519 503 q12 810 611 606 606 q13 16494 315731423142 q14 314 282 274 274 q15 568 529 533 529 q16 710 658 682 658 q17 1832159515721572 q18 8183769475557555 q19 2149156715201520 q20 2008185518191819 q21 5475537152245224 q22 1141103410051005 Total cold run time: 76644 ms Total hot run time: 59706 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](tvf) Tvf supports to parse the enclose character in csv files [doris]
github-actions[bot] commented on PR #45407: URL: https://github.com/apache/doris/pull/45407#issuecomment-2541078509 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](external)fix split and get the schema [doris]
Thearas commented on PR #45408: URL: https://github.com/apache/doris/pull/45408#issuecomment-2541085792 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](external)fix split and get the schema [doris]
wuwenchi commented on PR #45408: URL: https://github.com/apache/doris/pull/45408#issuecomment-2541085867 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](external)fix split and get the schema [doris]
wuwenchi opened a new pull request, #45408: URL: https://github.com/apache/doris/pull/45408 ### What problem does this PR solve? Related PR: #39116 Problem Summary: Split and get the schema according to `://` and `:/`. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](hive) fix block decompressor bug #45289 [doris]
doris-robot commented on PR #45377: URL: https://github.com/apache/doris/pull/45377#issuecomment-2541103968 TPC-DS: Total hot run time: 195864 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 0f23d3ef78c04841272bc469107cec9b42287adb, data reload: false query1 1233931 907 907 query2 6232205020012001 query3 10893 422942234223 query4 66493 30268 23436 23436 query5 5325456 458 456 query6 500 175 170 170 query7 6006307 311 307 query8 329 229 235 229 query9 9045268026542654 query10 475 269 243 243 query11 17614 15284 15828 15284 query12 179 99 102 99 query13 1589424 438 424 query14 10819 690772576907 query15 210 171 166 166 query16 7256466 501 466 query17 1266568 561 561 query18 1782341 300 300 query19 189 149 151 149 query20 113 111 106 106 query21 64 42 42 42 query22 4735477944864486 query23 34570 34104 34708 34104 query24 5887288729462887 query25 517 393 386 386 query26 659 172 164 164 query27 1933303 297 297 query28 4206251524752475 query29 656 450 435 435 query30 247 163 159 159 query31 986 839 833 833 query32 69 57 54 54 query33 414 287 289 287 query34 901 505 491 491 query35 858 739 748 739 query36 1059955 951 951 query37 112 72 68 68 query38 4118401340254013 query39 1523148214521452 query40 138 80 78 78 query41 49 45 45 45 query42 111 97 100 97 query43 531 491 500 491 query44 1146798 804 798 query45 183 167 167 167 query46 1136706 739 706 query47 2011186919151869 query48 468 372 371 371 query49 723 382 371 371 query50 836 400 409 400 query51 7474719370357035 query52 99 89 92 89 query53 252 183 184 183 query54 552 448 441 441 query55 78 73 71 71 query56 248 236 234 234 query57 1210109010891089 query58 218 200 206 200 query59 3292285429952854 query60 290 249 250 249 query61 108 107 104 104 query62 775 655 676 655 query63 215 187 192 187 query64 1420653 624 624 query65 3288319431623162 query66 722 297 297 297 query67 15692 15538 15299 15299 query68 4542547 551 547 query69 405 273 260 260 query70 1081107911341079 query71 347 249 251 249 query72 6298407641244076 query73 752 339 339 339 query74 9998895988988898 query75 3330263526382635 query76 1878103710021002 query77 487 261 265 261 query78 10618 980295139513 query79 3049595 603 595 query80 1866437 434 434 query81 551 243 239 239 query82 850 123 121 121 query83 257 146 150 146 query84 302 90 79 79 query85 1262373 398 373 query86 457 302 296 296 query87 4348419143204191 query88 5093239223812381 query89 422 296 287 287 query90 2054183 183 183 query91 177 141 142 141 query92 58 48 50 48 query93 5421538 529 529 query94 901 274 284 274 query95 343 241 244 241 query96 623 280 277 277 query97 3318311132133111 query98 224 204 188 188 query99 1734129413311294 Total cold run time: 327373 ms Total hot run time: 195864 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t
[PR] [Fix] remove useless test [doris]
wangbo opened a new pull request, #45409: URL: https://github.com/apache/doris/pull/45409 ### What problem does this PR solve? remove useless test ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Enhancement](compaction) enable the compaction producer to generate multiple compaction tasks in a single run [doris]
luwei16 opened a new pull request, #45411: URL: https://github.com/apache/doris/pull/45411 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](inverted index) Time Series Compaction Level 2 Logical Optimization [doris]
hello-stephen commented on PR #45412: URL: https://github.com/apache/doris/pull/45412#issuecomment-2541161946 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](compaction) enable the compaction producer to generate multiple compaction tasks in a single run [doris]
luwei16 commented on PR #45411: URL: https://github.com/apache/doris/pull/45411#issuecomment-2541163569 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [opt](inverted index) Time Series Compaction Level 2 Logical Optimization [doris]
zzzxl1993 opened a new pull request, #45412: URL: https://github.com/apache/doris/pull/45412 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](compaction) enable the compaction producer to generate multiple compaction tasks in a single run [doris]
github-actions[bot] commented on code in PR #45411:
URL: https://github.com/apache/doris/pull/45411#discussion_r1883746147
##
be/src/olap/tablet_manager.cpp:
##
@@ -736,6 +736,11 @@ void TabletManager::get_tablet_stat(TTabletStatResult*
result) {
result->__set_tablet_stat_list(*local_cache);
}
+struct TabletScore {
+TabletSharedPtr tablet_ptr;
+int score;
+};
+
std::vector TabletManager::find_best_tablets_to_compaction(
Review Comment:
warning: function 'find_best_tablets_to_compaction' has cognitive complexity
of 61 (threshold 50) [readability-function-cognitive-complexity]
```cpp
std::vector TabletManager::find_best_tablets_to_compaction(
^
```
Additional context
**be/src/olap/tablet_manager.cpp:750:** +1, including nesting penalty of 0,
nesting level increased to 1
```cpp
compaction_type == CompactionType::BASE_COMPACTION ? "base" :
"cumulative";
^
```
**be/src/olap/tablet_manager.cpp:756:** nesting level increased to 1
```cpp
auto cmp = [](TabletScore left, TabletScore right) { return left.score >
right.score; };
^
```
**be/src/olap/tablet_manager.cpp:759:** nesting level increased to 1
```cpp
auto handler = [&](const TabletSharedPtr& tablet_ptr) {
^
```
**be/src/olap/tablet_manager.cpp:760:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if
(tablet_ptr->tablet_meta()->tablet_schema()->disable_auto_compaction()) {
^
```
**be/src/olap/tablet_manager.cpp:767:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (config::enable_skip_tablet_compaction &&
^
```
**be/src/olap/tablet_manager.cpp:771:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (!tablet_ptr->can_do_compaction(data_dir->path_hash(),
compaction_type)) {
^
```
**be/src/olap/tablet_manager.cpp:776:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (search != tablet_submitted_compaction.end()) {
^
```
**be/src/olap/tablet_manager.cpp:781:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (compaction_type == CompactionType::BASE_COMPACTION) {
^
```
**be/src/olap/tablet_manager.cpp:784:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (now_ms - last_failure_ms <= 5000) {
^
```
**be/src/olap/tablet_manager.cpp:792:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (compaction_type == CompactionType::BASE_COMPACTION) {
^
```
**be/src/olap/tablet_manager.cpp:795:** +3, including nesting penalty of 2,
nesting level increased to 3
```cpp
if (!lock.owns_lock()) {
^
```
**be/src/olap/tablet_manager.cpp:799:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/tablet_manager.cpp:802:** +3, including nesting penalty of 2,
nesting level increased to 3
```cpp
if (!lock.owns_lock()) {
^
```
**be/src/olap/tablet_manager.cpp:810:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (current_compaction_score < 5) {
^
```
**be/src/olap/tablet_manager.cpp:815:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (current_compaction_score > single_compact_highest_score &&
^
```
**be/src/olap/tablet_manager.cpp:819:** +3, including nesting penalty of 2,
nesting level increased to 3
```cpp
if (ret) {
^
```
**be/src/olap/tablet_manager.cpp:825:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (config::compaction_num_per_round > 1 &&
!tablet_ptr->should_fetch_from_peer()) {
^
```
**be/src/olap/tablet_manager.cpp:829:** +3, including nesting penalty of 2,
nesting level increased to 3
```cpp
if ((top_tablets.size() >= config::compaction_num_per_round &&
current_compaction_score > top_tablets.top().score)
^
```
**be/src/olap/tablet_manager.cpp:833:** +4, including nesting penalty of 3,
nesting level increased to 4
```cpp
if (ret) {
^
```
**be/src/olap/tablet_manager.cpp:835:** +5, including nesting penalty of 4,
nesting level increased to 5
```cpp
if (top_tablets.size() >
config::compaction_num_per_round) {
^
```
**be/src/olap/tablet_manager.cpp:838:** +5, including nesting penalty of 4,
nesting level increased
[PR] [Fix](ccr) Wrong result of tosql for show create table [doris]
wyxxxcat opened a new pull request, #45413: URL: https://github.com/apache/doris/pull/45413 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](nereids) simplify comparison predicate rule add check data type limit [doris]
morrySnow merged PR #44732: URL: https://github.com/apache/doris/pull/44732 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](ccr) Wrong result of tosql for show create table [doris]
hello-stephen commented on PR #45413: URL: https://github.com/apache/doris/pull/45413#issuecomment-2541180889 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](inverted index) Time Series Compaction Level 2 Logical Optimization [doris]
github-actions[bot] commented on PR #45412: URL: https://github.com/apache/doris/pull/45412#issuecomment-2541180616 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](Nereids) lock table in ascending order of table IDs [doris]
zddr commented on code in PR #45045:
URL: https://github.com/apache/doris/pull/45045#discussion_r1883720606
##
fe/fe-core/src/main/java/org/apache/doris/common/proc/PartitionsProcDir.java:
##
@@ -243,6 +243,7 @@ private List, TRow>>
getPartitionInfosInrernal() throws An
List, TRow>> partitionInfos = new
ArrayList, TRow>>();
Map> partitionsUnSyncTables = null;
String mtmvPartitionSyncErrorMsg = null;
+olapTable.readLock();
Review Comment:
need lock of relatedTable?
##
fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java:
##
@@ -381,40 +352,31 @@ default void replayDropConstraint(String name) {
}
default void dropConstraint(String name, boolean replay) {
-writeLock();
-try {
-Map constraintMap = getConstraintsMapUnsafe();
-if (!constraintMap.containsKey(name)) {
-throw new AnalysisException(
-String.format("Unknown constraint %s on table %s.",
name, this.getName()));
-}
-Constraint constraint = constraintMap.get(name);
-constraintMap.remove(name);
-if (constraint instanceof PrimaryKeyConstraint) {
-((PrimaryKeyConstraint) constraint).getForeignTables()
-.forEach(t -> t.dropFKReferringPK(this,
(PrimaryKeyConstraint) constraint));
-}
-if (!replay) {
-Env.getCurrentEnv().getEditLog().logDropConstraint(new
AlterConstraintLog(constraint, this));
-}
-} finally {
-writeUnlock();
+Map constraintMap = getConstraintsMapUnsafe();
+if (!constraintMap.containsKey(name)) {
+throw new AnalysisException(
+String.format("Unknown constraint %s on table %s.", name,
this.getName()));
+}
+Constraint constraint = constraintMap.get(name);
+constraintMap.remove(name);
+if (constraint instanceof PrimaryKeyConstraint) {
+((PrimaryKeyConstraint) constraint).getForeignTables()
+.forEach(t -> t.dropFKReferringPK(this,
(PrimaryKeyConstraint) constraint));
+}
+if (!replay) {
+Env.getCurrentEnv().getEditLog().logDropConstraint(new
AlterConstraintLog(constraint, this));
}
}
default void dropFKReferringPK(TableIf table, PrimaryKeyConstraint
constraint) {
-writeLock();
-try {
-Map constraintMap = getConstraintsMapUnsafe();
-Set fkName = constraintMap.entrySet().stream()
-.filter(e -> e.getValue() instanceof ForeignKeyConstraint
-&& ((ForeignKeyConstraint)
e.getValue()).isReferringPK(table, constraint))
-.map(Entry::getKey)
-.collect(Collectors.toSet());
-fkName.forEach(constraintMap::remove);
-} finally {
-writeUnlock();
-}
+Map constraintMap = getConstraintsMapUnsafe();
Review Comment:
Where did the logic of locking move to
##
fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java:
##
@@ -844,22 +845,28 @@ private static TFetchSchemaTableDataResult
mtmvMetadataResult(TMetadataTableRequ
}
MTMV mv = (MTMV) table;
if (LOG.isDebugEnabled()) {
-LOG.debug("mv: " + mv.toInfoString());
+LOG.debug("mv: {}", mv.toInfoString());
}
TRow trow = new TRow();
-trow.addToColumnValue(new TCell().setLongVal(mv.getId()));
-trow.addToColumnValue(new TCell().setStringVal(mv.getName()));
-trow.addToColumnValue(new
TCell().setStringVal(mv.getJobInfo().getJobName()));
-trow.addToColumnValue(new
TCell().setStringVal(mv.getStatus().getState().name()));
-trow.addToColumnValue(new
TCell().setStringVal(mv.getStatus().getSchemaChangeDetail()));
-trow.addToColumnValue(new
TCell().setStringVal(mv.getStatus().getRefreshState().name()));
-trow.addToColumnValue(new
TCell().setStringVal(mv.getRefreshInfo().toString()));
-trow.addToColumnValue(new
TCell().setStringVal(mv.getQuerySql()));
-trow.addToColumnValue(new
TCell().setStringVal(mv.getMvProperties().toString()));
-trow.addToColumnValue(new
TCell().setStringVal(mv.getMvPartitionInfo().toNameString()));
-trow.addToColumnValue(new
TCell().setBoolVal(MTMVPartitionUtil.isMTMVSync(mv)));
-if (LOG.isDebugEnabled()) {
-LOG.debug("mvend: " + mv.getName());
+mv.readLock();
Review Comment:
need readLock of relatedTable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
Re: [PR] [opt](inverted index) Time Series Compaction Level 2 Logical Optimization [doris]
zzzxl1993 commented on PR #45412: URL: https://github.com/apache/doris/pull/45412#issuecomment-2541179682 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](Nereids) lock table in ascending order of table IDs [doris]
zddr commented on code in PR #45045:
URL: https://github.com/apache/doris/pull/45045#discussion_r1883714598
##
fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java:
##
@@ -381,40 +352,31 @@ default void replayDropConstraint(String name) {
}
default void dropConstraint(String name, boolean replay) {
-writeLock();
-try {
-Map constraintMap = getConstraintsMapUnsafe();
-if (!constraintMap.containsKey(name)) {
-throw new AnalysisException(
-String.format("Unknown constraint %s on table %s.",
name, this.getName()));
-}
-Constraint constraint = constraintMap.get(name);
-constraintMap.remove(name);
-if (constraint instanceof PrimaryKeyConstraint) {
-((PrimaryKeyConstraint) constraint).getForeignTables()
-.forEach(t -> t.dropFKReferringPK(this,
(PrimaryKeyConstraint) constraint));
-}
-if (!replay) {
-Env.getCurrentEnv().getEditLog().logDropConstraint(new
AlterConstraintLog(constraint, this));
-}
-} finally {
-writeUnlock();
+Map constraintMap = getConstraintsMapUnsafe();
+if (!constraintMap.containsKey(name)) {
+throw new AnalysisException(
+String.format("Unknown constraint %s on table %s.", name,
this.getName()));
+}
+Constraint constraint = constraintMap.get(name);
+constraintMap.remove(name);
+if (constraint instanceof PrimaryKeyConstraint) {
+((PrimaryKeyConstraint) constraint).getForeignTables()
+.forEach(t -> t.dropFKReferringPK(this,
(PrimaryKeyConstraint) constraint));
+}
+if (!replay) {
+Env.getCurrentEnv().getEditLog().logDropConstraint(new
AlterConstraintLog(constraint, this));
}
}
default void dropFKReferringPK(TableIf table, PrimaryKeyConstraint
constraint) {
-writeLock();
-try {
-Map constraintMap = getConstraintsMapUnsafe();
-Set fkName = constraintMap.entrySet().stream()
-.filter(e -> e.getValue() instanceof ForeignKeyConstraint
-&& ((ForeignKeyConstraint)
e.getValue()).isReferringPK(table, constraint))
-.map(Entry::getKey)
-.collect(Collectors.toSet());
-fkName.forEach(constraintMap::remove);
-} finally {
-writeUnlock();
-}
+Map constraintMap = getConstraintsMapUnsafe();
Review Comment:
Where did the logic of locking move to
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (2ca753feb84 -> 475571b1520)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 2ca753feb84 [fix](Nereids) fix cases unstable when using profile to check is nereids execution (#45384) add 475571b1520 [feat](nereids) simplify comparison predicate rule add check data type limit (#44732) No new revisions were added by this update. Summary of changes: .../rules/SimplifyComparisonPredicate.java | 184 +- .../apache/doris/nereids/util/ExpressionUtils.java | 16 ++ .../doris/nereids/util/TypeCoercionUtils.java | 44 .../rules/SimplifyComparisonPredicateTest.java | 263 - .../infer_predicate/pull_up_predicate_literal.out | 110 + .../predicate_infer/infer_predicate.out| 10 +- .../predicate_infer/infer_predicate.groovy | 7 +- 7 files changed, 462 insertions(+), 172 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](compaction) enable the compaction producer to generate multiple compaction tasks in a single run [doris]
github-actions[bot] commented on code in PR #45411:
URL: https://github.com/apache/doris/pull/45411#discussion_r1883749522
##
be/src/olap/tablet_manager.cpp:
##
@@ -736,6 +736,11 @@ void TabletManager::get_tablet_stat(TTabletStatResult*
result) {
result->__set_tablet_stat_list(*local_cache);
}
+struct TabletScore {
+TabletSharedPtr tablet_ptr;
+int score;
+};
+
std::vector TabletManager::find_best_tablets_to_compaction(
Review Comment:
warning: function 'find_best_tablets_to_compaction' has cognitive complexity
of 61 (threshold 50) [readability-function-cognitive-complexity]
```cpp
std::vector TabletManager::find_best_tablets_to_compaction(
^
```
Additional context
**be/src/olap/tablet_manager.cpp:750:** +1, including nesting penalty of 0,
nesting level increased to 1
```cpp
compaction_type == CompactionType::BASE_COMPACTION ? "base" :
"cumulative";
^
```
**be/src/olap/tablet_manager.cpp:756:** nesting level increased to 1
```cpp
auto cmp = [](TabletScore left, TabletScore right) { return left.score >
right.score; };
^
```
**be/src/olap/tablet_manager.cpp:759:** nesting level increased to 1
```cpp
auto handler = [&](const TabletSharedPtr& tablet_ptr) {
^
```
**be/src/olap/tablet_manager.cpp:760:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if
(tablet_ptr->tablet_meta()->tablet_schema()->disable_auto_compaction()) {
^
```
**be/src/olap/tablet_manager.cpp:767:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (config::enable_skip_tablet_compaction &&
^
```
**be/src/olap/tablet_manager.cpp:771:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (!tablet_ptr->can_do_compaction(data_dir->path_hash(),
compaction_type)) {
^
```
**be/src/olap/tablet_manager.cpp:776:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (search != tablet_submitted_compaction.end()) {
^
```
**be/src/olap/tablet_manager.cpp:781:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (compaction_type == CompactionType::BASE_COMPACTION) {
^
```
**be/src/olap/tablet_manager.cpp:784:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (now_ms - last_failure_ms <= 5000) {
^
```
**be/src/olap/tablet_manager.cpp:792:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (compaction_type == CompactionType::BASE_COMPACTION) {
^
```
**be/src/olap/tablet_manager.cpp:795:** +3, including nesting penalty of 2,
nesting level increased to 3
```cpp
if (!lock.owns_lock()) {
^
```
**be/src/olap/tablet_manager.cpp:799:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/tablet_manager.cpp:802:** +3, including nesting penalty of 2,
nesting level increased to 3
```cpp
if (!lock.owns_lock()) {
^
```
**be/src/olap/tablet_manager.cpp:810:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (current_compaction_score < 5) {
^
```
**be/src/olap/tablet_manager.cpp:815:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (current_compaction_score > single_compact_highest_score &&
^
```
**be/src/olap/tablet_manager.cpp:819:** +3, including nesting penalty of 2,
nesting level increased to 3
```cpp
if (ret) {
^
```
**be/src/olap/tablet_manager.cpp:825:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
if (config::compaction_num_per_round > 1 &&
!tablet_ptr->should_fetch_from_peer()) {
^
```
**be/src/olap/tablet_manager.cpp:829:** +3, including nesting penalty of 2,
nesting level increased to 3
```cpp
if ((top_tablets.size() >= config::compaction_num_per_round &&
^
```
**be/src/olap/tablet_manager.cpp:834:** +4, including nesting penalty of 3,
nesting level increased to 4
```cpp
if (ret) {
^
```
**be/src/olap/tablet_manager.cpp:836:** +5, including nesting penalty of 4,
nesting level increased to 5
```cpp
if (top_tablets.size() >
config::compaction_num_per_round) {
^
```
**be/src/olap/tablet_manager.cpp:839:** +5, including nesting penalty of 4,
nesting level increased to 5
```cpp
if (current_co
Re: [PR] [opt](inverted index) Time Series Compaction Level 2 Logical Optimization [doris]
github-actions[bot] commented on PR #45412: URL: https://github.com/apache/doris/pull/45412#issuecomment-2541187491 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](tvf) Tvf supports to parse the enclose character in csv files [doris]
doris-robot commented on PR #45407: URL: https://github.com/apache/doris/pull/45407#issuecomment-2541196749 TeamCity be ut coverage result: Function Coverage: 38.76% (10109/26082) Line Coverage: 29.69% (84846/285766) Region Coverage: 28.76% (43563/151446) Branch Coverage: 25.32% (22133/87414) Coverage Report: http://coverage.selectdb-in.cc/coverage/2ec488dd977635b39c09967e6122745c042cb18a_2ec488dd977635b39c09967e6122745c042cb18a/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](shuffle)(part III) Use local exchanger for both exchange sink and lo… [doris]
doris-robot commented on PR #45375: URL: https://github.com/apache/doris/pull/45375#issuecomment-2541196404 TeamCity be ut coverage result: Function Coverage: 38.71% (10106/26105) Line Coverage: 29.67% (84818/285839) Region Coverage: 28.76% (43547/151414) Branch Coverage: 25.31% (22123/87416) Coverage Report: http://coverage.selectdb-in.cc/coverage/37d21bfce66cc756a304ca9baa90ebd8416e153f_37d21bfce66cc756a304ca9baa90ebd8416e153f/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
