[doris] branch master updated (295b26db00 -> 5bd00626be)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 295b26db00 [chore](fe) update aspectj-maven-plugin to 1.14.0 version (#17890) add 5bd00626be [doc](releasenote) 1.2.3 release note (#17926) No new revisions were added by this update. Summary of changes: README.md | 2 +- docs/en/docs/releasenotes/release-1.2.3.md| 109 + docs/sidebars.json| 1 + docs/zh-CN/docs/releasenotes/release-1.2.3.md | 111 ++ 4 files changed, 222 insertions(+), 1 deletion(-) create mode 100644 docs/en/docs/releasenotes/release-1.2.3.md create mode 100644 docs/zh-CN/docs/releasenotes/release-1.2.3.md - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #17926: [doc](releasenote) 1.2.3 release note
yiguolei merged PR #17926: URL: https://github.com/apache/doris/pull/17926 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17929: [fix](tests) add order by for test_multiply and test_schema_change_with_delete
github-actions[bot] commented on PR #17929: URL: https://github.com/apache/doris/pull/17929#issuecomment-1475133347 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17929: [fix](tests) add order by for test_multiply and test_schema_change_with_delete
github-actions[bot] commented on PR #17929: URL: https://github.com/apache/doris/pull/17929#issuecomment-1475133354 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu opened a new pull request, #17930: [Docs](multi-catalog) add `yarn.resourcemanager.principal` for hive c…
dutyu opened a new pull request, #17930: URL: https://github.com/apache/doris/pull/17930 …atalog with kerberos enabled. # Proposed changes Add neccessary `yarn.resourcemanager.principal` config for hive catalog with kerberos enabled. ## Problem summary When creating a hive catalog with a kerberos enabled hive cluster, the config `yarn.resourcemanager.principal` is required, because hdfs client needs this principal otherwise we will encounter an exception like this:  ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 opened a new pull request, #17931: [vectorized](function) support array_exists lambda function
ZhangYu0123 opened a new pull request, #17931: URL: https://github.com/apache/doris/pull/17931 # Proposed changes Support array_exists lambda function. https://user-images.githubusercontent.com/67053339/226160665-6bea93ae-626b-4aef-8392-bb7ab23a209c.png";> Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on pull request #17916: [improve](nereids)adjust distribution stats derive and fix bug in join estimation
englefly commented on PR #17916: URL: https://github.com/apache/doris/pull/17916#issuecomment-1475138557 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17931: [vectorized](function) support array_exists lambda function
github-actions[bot] commented on PR #17931: URL: https://github.com/apache/doris/pull/17931#issuecomment-1475138792 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on pull request #17916: [improve](nereids)adjust distribution stats derive and fix bug in join estimation
englefly commented on PR #17916: URL: https://github.com/apache/doris/pull/17916#issuecomment-1475138860 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 commented on pull request #17931: [vectorized](function) support array_exists lambda function
ZhangYu0123 commented on PR #17931: URL: https://github.com/apache/doris/pull/17931#issuecomment-1475140909 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17684: [improvement](predicate) Cache the dict code in ComparisonPredicate
hello-stephen commented on PR #17684: URL: https://github.com/apache/doris/pull/17684#issuecomment-1475144624 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.84 seconds stream load tsv: 466 seconds loaded 74807831229 Bytes, about 153 MB/s stream load json: 24 seconds loaded 2358488459 Bytes, about 93 MB/s stream load orc: 76 seconds loaded 1101869774 Bytes, about 13 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230319080516_clickbench_pr_116719.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu opened a new pull request, #17932: [Fix](multi-catalog) invalidates the file cache when table is non-par…
dutyu opened a new pull request, #17932: URL: https://github.com/apache/doris/pull/17932 …titioned. # Proposed changes Invalidate the file cache when refresh a non-partitioned external hive table. ## Problem summary File cache of `HiveMetaStoreCache` may be created even the table is a non-partitioned table, so the `RefreshTableStmt` should consider this scene and handle it. ## Checklist(Required) * [x] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu commented on pull request #17932: [Fix](multi-catalog) invalidates the file cache when table is non-par…
dutyu commented on PR #17932: URL: https://github.com/apache/doris/pull/17932#issuecomment-1475145287 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu commented on pull request #17932: [Fix](multi-catalog) invalidates the file cache when table is non-par…
dutyu commented on PR #17932: URL: https://github.com/apache/doris/pull/17932#issuecomment-1475145575 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17928: [fix](nereids) Analyze failed for SQL that has count distinct with same col.
hello-stephen commented on PR #17928: URL: https://github.com/apache/doris/pull/17928#issuecomment-1475153994 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.07 seconds stream load tsv: 452 seconds loaded 74807831229 Bytes, about 157 MB/s stream load json: 24 seconds loaded 2358488459 Bytes, about 93 MB/s stream load orc: 73 seconds loaded 1101869774 Bytes, about 14 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230319084452_clickbench_pr_116749.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu opened a new pull request, #17933: [Fix](multi-catalog) add handler for hms INSERT EVENT.
dutyu opened a new pull request, #17933: URL: https://github.com/apache/doris/pull/17933 # Proposed changes ## Problem summary When we use a hive client to submit a `INSERT INTO TBL SELECT * FROM ...` or `INSERT INTO TBL VALUES ...` sql and the table is non-partitioned table, the hms will generate an insert event. The insert stmt may changed the hdfs file distribution of this table, but currently we do not handle this, so the file cache of this table may be inaccurate. ## Checklist(Required) * [x] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu commented on pull request #17933: [Fix](multi-catalog) add handler for hms INSERT EVENT.
dutyu commented on PR #17933: URL: https://github.com/apache/doris/pull/17933#issuecomment-1475159745 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17916: [improve](nereids)adjust distribution stats derive and fix bug in join estimation
hello-stephen commented on PR #17916: URL: https://github.com/apache/doris/pull/17916#issuecomment-1475161405 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.8 seconds stream load tsv: 471 seconds loaded 74807831229 Bytes, about 151 MB/s stream load json: 25 seconds loaded 2358488459 Bytes, about 89 MB/s stream load orc: 76 seconds loaded 1101869774 Bytes, about 13 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230319090323_clickbench_pr_116766.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17931: [vectorized](function) support array_exists lambda function
hello-stephen commented on PR #17931: URL: https://github.com/apache/doris/pull/17931#issuecomment-1475162680 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.03 seconds stream load tsv: 454 seconds loaded 74807831229 Bytes, about 157 MB/s stream load json: 24 seconds loaded 2358488459 Bytes, about 93 MB/s stream load orc: 73 seconds loaded 1101869774 Bytes, about 14 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230319090721_clickbench_pr_116771.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on pull request #17741: [enhancement](nereids)support topN opt in nereids
englefly commented on PR #17741: URL: https://github.com/apache/doris/pull/17741#issuecomment-1475162903 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #17833: [ehancement](statistics) remove lock in statistics cache loader
Kikyou1997 commented on PR #17833: URL: https://github.com/apache/doris/pull/17833#issuecomment-1475164336 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #17586: [refactor](file-system) refactor file sysmte on BE and remove storage_backend
morningman commented on PR #17586: URL: https://github.com/apache/doris/pull/17586#issuecomment-1475165024 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer commented on a diff in pull request #17874: [improvement](compaction)Support segcompaction for inverted index
freemandealer commented on code in PR #17874: URL: https://github.com/apache/doris/pull/17874#discussion_r1141314968 ## be/src/olap/rowset/beta_rowset_writer.cpp: ## @@ -255,13 +279,35 @@ Status BetaRowsetWriter::_rename_compacted_segment_plain(uint64_t seg_id) { _segid_statistics_map.emplace(_num_segcompacted, org); _clear_statistics_for_deleting_segments_unsafe(seg_id, seg_id); } -++_num_segcompacted; ret = rename(src_seg_path.c_str(), dst_seg_path.c_str()); if (ret) { LOG(WARNING) << "failed to rename " << src_seg_path << " to " << dst_seg_path << ". ret:" << ret << " errno:" << errno; return Status::Error(); } +// rename remaining inverted index files Review Comment: the following code is similar to the code beginning with line 227 except for the path logic. Can we make a common function here to eliminate duplication? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on pull request #17684: [improvement](predicate) Cache the dict code in ComparisonPredicate
yiguolei commented on PR #17684: URL: https://github.com/apache/doris/pull/17684#issuecomment-1475175737 skip failed p0 because it is a know unstable test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #17684: [improvement](predicate) Cache the dict code in ComparisonPredicate
yiguolei merged PR #17684: URL: https://github.com/apache/doris/pull/17684 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](predicate) Cache the dict code in ComparisonPredicate (#17684)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 1d26b4d6c2 [improvement](predicate) Cache the dict code in ComparisonPredicate (#17684) 1d26b4d6c2 is described below commit 1d26b4d6c288bbe012bf4fc92585aeed194eeded Author: Jerry Hu AuthorDate: Sun Mar 19 17:37:28 2023 +0800 [improvement](predicate) Cache the dict code in ComparisonPredicate (#17684) --- be/src/olap/column_predicate.h | 5 ++ be/src/olap/comparison_predicate.h | 72 -- be/src/olap/rowset/segment_v2/segment_iterator.cpp | 16 - be/src/olap/rowset/segment_v2/segment_iterator.h | 2 + 4 files changed, 73 insertions(+), 22 deletions(-) diff --git a/be/src/olap/column_predicate.h b/be/src/olap/column_predicate.h index f3987a006e..6cc6ec8ad2 100644 --- a/be/src/olap/column_predicate.h +++ b/be/src/olap/column_predicate.h @@ -192,6 +192,11 @@ public: ", opposite=" + (_opposite ? "true" : "false"); } +/// Some predicates need to be cloned for each segment. +virtual bool need_to_clone() const { return false; } + +virtual void clone(ColumnPredicate** to) const { LOG(FATAL) << "clone not supported"; } + std::shared_ptr predicate_params() { return _predicate_params; } const std::string pred_type_string(PredicateType type) { diff --git a/be/src/olap/comparison_predicate.h b/be/src/olap/comparison_predicate.h index 26d97a27ed..e2ca003b9b 100644 --- a/be/src/olap/comparison_predicate.h +++ b/be/src/olap/comparison_predicate.h @@ -32,7 +32,15 @@ class ComparisonPredicateBase : public ColumnPredicate { public: using T = typename PredicatePrimitiveTypeTraits::PredicateFieldType; ComparisonPredicateBase(uint32_t column_id, const T& value, bool opposite = false) -: ColumnPredicate(column_id, opposite), _value(value) {} +: ColumnPredicate(column_id, opposite), + _cached_code(_InvalidateCodeValue), + _value(value) {} + +void clone(ColumnPredicate** to) const override { +*to = new ComparisonPredicateBase(_column_id, _value, _opposite); +} + +bool need_to_clone() const override { return true; } PredicateType type() const override { return PT; } @@ -258,13 +266,20 @@ public: auto* dict_column_ptr = vectorized::check_and_get_column( nested_column); -auto dict_code = _is_range() ? dict_column_ptr->find_code_by_bound( - _value, _is_greater(), _is_eq()) - : dict_column_ptr->find_code(_value); -auto* data_array = dict_column_ptr->get_data().data(); -_base_loop_vec(size, flags, null_map.data(), data_array, - dict_code); +auto dict_code = _find_code_from_dictionary_column(*dict_column_ptr); +do { +if constexpr (PT == PredicateType::EQ) { +if (dict_code == -2) { +memset(flags, 0, size); +break; +} +} +auto* data_array = dict_column_ptr->get_data().data(); + +_base_loop_vec(size, flags, null_map.data(), data_array, + dict_code); +} while (false); } else { LOG(FATAL) << "column_dictionary must use StringRef predicate."; } @@ -281,12 +296,18 @@ public: if constexpr (std::is_same_v) { auto* dict_column_ptr = vectorized::check_and_get_column(column); -auto dict_code = _is_range() ? dict_column_ptr->find_code_by_bound( - _value, _is_greater(), _is_eq()) - : dict_column_ptr->find_code(_value); -auto* data_array = dict_column_ptr->get_data().data(); - -_base_loop_vec(size, flags, nullptr, data_array, dict_code); +auto dict_code = _find_code_from_dictionary_column(*dict_column_ptr); +do { +if constexpr (PT == PredicateType::EQ) { +if (dict_code == -2) { +memset(flags, 0, size); +break; +} +} +auto* data_array = dict_column_ptr->g
[GitHub] [doris] yiguolei merged pull request #17929: [fix](tests) add order by for test_multiply and test_schema_change_with_delete
yiguolei merged PR #17929: URL: https://github.com/apache/doris/pull/17929 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (1d26b4d6c2 -> bf814411d8)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 1d26b4d6c2 [improvement](predicate) Cache the dict code in ComparisonPredicate (#17684) add bf814411d8 [fix](tests) add order by for test_multiply and test_schema_change_with_delete (#17929) No new revisions were added by this update. Summary of changes: .../query_p0/sql_functions/math_functions/test_multiply.groovy | 5 +++-- .../suites/schema_change/test_schema_change_with_delete.groovy | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei opened a new pull request, #17934: [regression-test](enhancement) fix unstable alter table column regression test
yiguolei opened a new pull request, #17934: URL: https://github.com/apache/doris/pull/17934 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on pull request #17934: [regression-test](enhancement) fix unstable alter table column regression test
yiguolei commented on PR #17934: URL: https://github.com/apache/doris/pull/17934#issuecomment-1475185818 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17873: [compaction](check) add some row check in compaction
github-actions[bot] commented on PR #17873: URL: https://github.com/apache/doris/pull/17873#issuecomment-1475186817 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #17912: [Bug](planner) fix core dump when lateral view above union node and have predicate
yiguolei commented on code in PR #17912: URL: https://github.com/apache/doris/pull/17912#discussion_r1141321661 ## regression-test/suites/query_p0/lateral_view/test_with_view.groovy: ## @@ -32,4 +32,29 @@ suite("lateral_view_with_inline_view") { example1 lateral view explode([1,2,3]) tmp1 as e1 where col4
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #17586: [refactor](file-system) refactor file sysmte on BE and remove storage_backend
github-actions[bot] commented on code in PR #17586: URL: https://github.com/apache/doris/pull/17586#discussion_r1141321731 ## be/test/io/fs/file_system_test.cpp: ## @@ -0,0 +1,643 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include + +#include "io/fs/broker_file_system.h" +#include "io/fs/file_reader.h" +#include "io/fs/file_writer.h" +#include "io/fs/hdfs_file_system.h" +#include "io/fs/local_file_system.h" +#include "io/fs/s3_file_system.h" +#include "io/hdfs_builder.h" +#include "util/s3_uri.h" + +namespace doris { + +#ifndef CHECK_STATUS_OK +#define CHECK_STATUS_OK(stmt) \ +do {\ +Status _status_ = (stmt); \ +ASSERT_TRUE(_status_.ok()) << _status_; \ +} while (false) +#endif + +// set your own info +// s3 +static std::string ak = ""; +static std::string sk = ""; +static std::string endpoint = "http://cos.ap-beijing.myqcloud.com";; +static std::string region = "ap-beijing"; +static std::string s3_location = ""; + +// hdfs +static std::string fs_name = "hdfs://my_nameservice"; +static std::string username = "hadoop"; +static std::string nameservices = "my_nameservice"; +static std::string nn = "nn1,nn2"; +static std::string rpc1 = "172.21.0.1:4007"; +static std::string rpc2 = "172.21.0.2:4007"; +static std::string provider = + "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"; +static std::string hdfs_location = "/user/doris/"; + +// broker +static std::string broker_ip = "127.0.0.1"; +static int broker_port = 8008; +static std::string broker_location = "hdfs://my_nameservice/user/doris"; + +// commend out to enable specified test +#define TestHdfsFileSystem DISABLED_TestHdfsFileSystem +#define TestS3FileSystem DISABLED_TestS3FileSystem +#define TestBrokerFileSystem DISABLED_TestBrokerFileSystem + +class FileSystemTest : public testing::Test { +public: +virtual void SetUp() { +s3_prop.emplace("AWS_ACCESS_KEY", ak); +s3_prop.emplace("AWS_SECRET_KEY", sk); +s3_prop.emplace("AWS_ENDPOINT", endpoint); +s3_prop.emplace("AWS_REGION", region); + +hdfs_prop.emplace("fs.defaultFS", fs_name); +hdfs_prop.emplace("hadoop.username", username); +hdfs_prop.emplace("username", username); // for broker hdfs +hdfs_prop.emplace("dfs.nameservices", nameservices); +hdfs_prop.emplace("dfs.ha.namenodes." + nameservices, nn); +hdfs_prop.emplace("dfs.namenode.rpc-address." + nameservices + ".nn1", rpc1); +hdfs_prop.emplace("dfs.namenode.rpc-address." + nameservices + ".nn2", rpc2); +hdfs_prop.emplace("dfs.client.failover.proxy.provider." + nameservices, provider); + +broker_addr.__set_hostname(broker_ip); +broker_addr.__set_port(broker_port); +} + +virtual void TearDown() {} + +private: +std::map s3_prop; +std::map hdfs_prop; +TNetworkAddress broker_addr; +}; + +TEST_F(FileSystemTest, TestBrokerFileSystem) { +std::shared_ptr fs; +CHECK_STATUS_OK(io::BrokerFileSystem::create(broker_addr, hdfs_prop, 0, &fs)); + +// delete directory +io::Path delete_path = broker_location + "/tmp1"; +CHECK_STATUS_OK(fs->delete_directory(delete_path)); +io::Path delete_path2 = broker_location + "/tmp2"; +CHECK_STATUS_OK(fs->delete_directory(delete_path2)); +// create directory not implemented +// io::Path create_path = delete_path; +// CHECK_STATUS_OK(fs->create_directory(create_path)); +// write file +std::string file1 = broker_location + "/tmp1/file1.txt"; +io::FileWriterPtr writer; +CHECK_STATUS_OK(fs->create_file(file1, &writer)); +CHECK_STATUS_OK(writer->append({"content"})); +CHECK_STATUS_OK(writer->close()); +// read file +io::FileReaderSPtr reader; +CHECK_STATUS_OK(fs->open_file(file1, &reader)); +char read_buf[10]; +size_t bytes_read = 0; +CHECK_STATUS_OK(reader->read_at(0, {read_buf, 10}, &bytes_read)); +ASSERT_EQ(7, bytes_read); + +// exist +bool exists = false; +CHECK_STATUS_OK(fs->exists(file1, &exists)); +ASSERT_TRUE(exists); +std::string file_non_exist = broker_location + "/
[GitHub] [doris] bin41215 opened a new issue, #17935: [Enhancement] sort result by BackendId when show backends like 'show backends'
bin41215 opened a new issue, #17935: URL: https://github.com/apache/doris/issues/17935 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description when show backends,i get backend ids unsort,for example: +---+ | BackendId | +---+ | 99412 | | 11003 | | 11002 | | 60616 | | 60673 | +--+ i want to get result sorted by backendId when show backends. ### Solution when show backends, sort result by backendIds. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17932: [Fix](multi-catalog) invalidates the file cache when table is non-par…
hello-stephen commented on PR #17932: URL: https://github.com/apache/doris/pull/17932#issuecomment-1475195331 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.17 seconds stream load tsv: 471 seconds loaded 74807831229 Bytes, about 151 MB/s stream load json: 26 seconds loaded 2358488459 Bytes, about 86 MB/s stream load orc: 75 seconds loaded 1101869774 Bytes, about 14 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230319102626_clickbench_pr_116777.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #17936: [fix](test) add order by to test_alter_table_column
dataroaring opened a new pull request, #17936: URL: https://github.com/apache/doris/pull/17936 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring commented on pull request #17936: [fix](test) add order by to test_alter_table_column
dataroaring commented on PR #17936: URL: https://github.com/apache/doris/pull/17936#issuecomment-1475196541 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] gitccl commented on pull request #17808: [feature](json) add json_extract function
gitccl commented on PR #17808: URL: https://github.com/apache/doris/pull/17808#issuecomment-1475198900 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17808: [feature](json) add json_extract function
github-actions[bot] commented on PR #17808: URL: https://github.com/apache/doris/pull/17808#issuecomment-1475200102 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 commented on pull request #17931: [vectorized](function) support array_exists lambda function
ZhangYu0123 commented on PR #17931: URL: https://github.com/apache/doris/pull/17931#issuecomment-1475204607 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17936: [fix](test) add order by to test_alter_table_column
github-actions[bot] commented on PR #17936: URL: https://github.com/apache/doris/pull/17936#issuecomment-1475213171 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17808: [feature](json) add json_extract function
hello-stephen commented on PR #17808: URL: https://github.com/apache/doris/pull/17808#issuecomment-1475213743 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.46 seconds stream load tsv: 471 seconds loaded 74807831229 Bytes, about 151 MB/s stream load json: 23 seconds loaded 2358488459 Bytes, about 97 MB/s stream load orc: 75 seconds loaded 1101869774 Bytes, about 14 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230319111313_clickbench_pr_116826.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17936: [fix](test) add order by to test_alter_table_column
github-actions[bot] commented on PR #17936: URL: https://github.com/apache/doris/pull/17936#issuecomment-1475218666 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 commented on pull request #17913: [fix](nereids)add a project node above sort node to eliminate unused order by keys
starocean999 commented on PR #17913: URL: https://github.com/apache/doris/pull/17913#issuecomment-1475219257 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 commented on pull request #17831: [fix](planner)project should be done inside inlineview
starocean999 commented on PR #17831: URL: https://github.com/apache/doris/pull/17831#issuecomment-1475219304 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 commented on pull request #17807: [fix](nereids)bucket shuffle and colocate join is not correctly recognized
starocean999 commented on PR #17807: URL: https://github.com/apache/doris/pull/17807#issuecomment-1475219332 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SaintBacchus commented on pull request #17745: [BugFix](DOE) Add http prefix when it's not set in hosts properties.
SaintBacchus commented on PR #17745: URL: https://github.com/apache/doris/pull/17745#issuecomment-1475221680 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SaintBacchus commented on pull request #17775: [Feature-Wip](MySQL LOAD)Add trim quotes property for mysql load
SaintBacchus commented on PR #17775: URL: https://github.com/apache/doris/pull/17775#issuecomment-1475221797 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hanyisong commented on pull request #17207: [Feature](schema change) Lightweight schema change of modify column with varchar length
hanyisong commented on PR #17207: URL: https://github.com/apache/doris/pull/17207#issuecomment-1475222118 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hanyisong commented on pull request #17207: [Feature](schema change) Lightweight schema change of modify column with varchar length
hanyisong commented on PR #17207: URL: https://github.com/apache/doris/pull/17207#issuecomment-1475222518 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #17837: [fix](timeout) fix set timeout failure and simplify timeout logic
TangSiyang2001 commented on PR #17837: URL: https://github.com/apache/doris/pull/17837#issuecomment-1475233006 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #17704: [feature](table-metadata) support altering the property "light_schema_change" for the tables which created before 1.2
TangSiyang2001 commented on PR #17704: URL: https://github.com/apache/doris/pull/17704#issuecomment-1475233100 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #17704: [feature](table-metadata) support altering the property "light_schema_change" for the tables which created before 1.2
TangSiyang2001 commented on PR #17704: URL: https://github.com/apache/doris/pull/17704#issuecomment-147527 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17837: [fix](timeout) fix set timeout failure and simplify timeout logic
github-actions[bot] commented on PR #17837: URL: https://github.com/apache/doris/pull/17837#issuecomment-1475233644 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17704: [feature](table-metadata) support altering the property "light_schema_change" for the tables which created before 1.2
github-actions[bot] commented on PR #17704: URL: https://github.com/apache/doris/pull/17704#issuecomment-1475233859 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed pull request #17934: [regression-test](enhancement) fix unstable alter table column regression test
yiguolei closed pull request #17934: [regression-test](enhancement) fix unstable alter table column regression test URL: https://github.com/apache/doris/pull/17934 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [nerids](statistics) remove lock in statistics cache loader #17833
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 74dfdc00dc [nerids](statistics) remove lock in statistics cache loader #17833 74dfdc00dc is described below commit 74dfdc00dcc1ee7066dcdea22bc482116ca57404 Author: AKIRA <33112463+kikyou1...@users.noreply.github.com> AuthorDate: Sun Mar 19 21:30:21 2023 +0900 [nerids](statistics) remove lock in statistics cache loader #17833 remove the redandunt lock in the CacheLoader, since it use the forkjoinpool in default Add execute time log for collect stats Avoid submit duplicate task, when there already has a task to load for the same column --- .../doris/statistics/StatisticsCacheLoader.java| 108 ++--- 1 file changed, 54 insertions(+), 54 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCacheLoader.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCacheLoader.java index 73d38b7db8..c592a9b4eb 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCacheLoader.java +++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCacheLoader.java @@ -33,6 +33,8 @@ import java.util.List; import java.util.Map; import java.util.concurrent.CompletableFuture; import java.util.concurrent.CompletionException; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; import java.util.concurrent.Executor; public class StatisticsCacheLoader implements AsyncCacheLoader { @@ -47,70 +49,68 @@ public class StatisticsCacheLoader implements AsyncCacheLoader> +inProgressing = new ConcurrentHashMap<>(); -// TODO: Maybe we should trigger a analyze job when the required ColumnStatistic doesn't exists. @Override public @NonNull CompletableFuture asyncLoad(@NonNull StatisticsCacheKey key, @NonNull Executor executor) { -synchronized (LOCK) { -if (CUR_RUNNING_LOAD > StatisticConstants.LOAD_TASK_LIMITS) { -try { -LOCK.wait(); -} catch (InterruptedException e) { -LOG.warn("Ignore interruption", e); -} -} -CUR_RUNNING_LOAD++; -return CompletableFuture.supplyAsync(() -> { + +CompletableFuture future = inProgressing.get(key); +if (future != null) { +return future; +} +future = CompletableFuture.supplyAsync(() -> { +long startTime = System.currentTimeMillis(); +try { +LOG.info("Query BE for column stats:{}-{} start time:{}", key.tableId, key.colName, +startTime); ColumnLevelStatisticCache statistic = new ColumnLevelStatisticCache(); +Map params = new HashMap<>(); +params.put("tblId", String.valueOf(key.tableId)); +params.put("idxId", String.valueOf(key.idxId)); +params.put("colId", String.valueOf(key.colName)); +List columnStatistics; +List columnResult = +StatisticsUtil.execStatisticQuery(new StringSubstitutor(params) +.replace(QUERY_COLUMN_STATISTICS)); try { -Map params = new HashMap<>(); -params.put("tblId", String.valueOf(key.tableId)); -params.put("idxId", String.valueOf(key.idxId)); -params.put("colId", String.valueOf(key.colName)); - -List columnStatistics; -List columnResult = -StatisticsUtil.execStatisticQuery(new StringSubstitutor(params) -.replace(QUERY_COLUMN_STATISTICS)); -try { -columnStatistics = StatisticsUtil.deserializeToColumnStatistics(columnResult); -} catch (Exception e) { -LOG.warn("Failed to deserialize column statistics", e); -throw new CompletionException(e); -} -if (CollectionUtils.isEmpty(columnStatistics)) { -statistic.setColumnStatistic(ColumnStatistic.UNKNOWN); -} else { -statistic.setColumnStatistic(columnStatistics.get(0)); -} - -List histogramStatistics; -List histogramResult = -StatisticsUtil.execStatisticQuery(new StringSubstitutor(params) -.replace(QUERY_HISTOGRAM_STATISTICS)); -try { -histogramStatistics = StatisticsUtil.deserializeToHisto
[GitHub] [doris] yiguolei merged pull request #17833: [nerids](statistics) remove lock in statistics cache loader
yiguolei merged PR #17833: URL: https://github.com/apache/doris/pull/17833 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on pull request #17809: [refactor](Nereids): refactor Join Reorder Rule.
jackwener commented on PR #17809: URL: https://github.com/apache/doris/pull/17809#issuecomment-1475240182 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on pull request #17765: [feature](Nereids): pull up SEMI JOIN from INNER JOIN
jackwener commented on PR #17765: URL: https://github.com/apache/doris/pull/17765#issuecomment-1475240267 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #17586: [refactor](file-system) refactor file sysmte on BE and remove storage_backend
morningman commented on PR #17586: URL: https://github.com/apache/doris/pull/17586#issuecomment-1475240375 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #17704: [feature](table-metadata) support altering the property "light_schema_change" for the tables which created before 1.2
TangSiyang2001 commented on PR #17704: URL: https://github.com/apache/doris/pull/17704#issuecomment-1475241755 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17704: [feature](table-metadata) support altering the property "light_schema_change" for the tables which created before 1.2
github-actions[bot] commented on PR #17704: URL: https://github.com/apache/doris/pull/17704#issuecomment-1475242579 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #17884: [Feature](multi-catalog) add refresh for catalog if it needs
morningman commented on code in PR #17884: URL: https://github.com/apache/doris/pull/17884#discussion_r1141352733 ## fe/fe-core/src/main/java/org/apache/doris/PaloFe.java: ## @@ -173,6 +176,12 @@ public static void start(String dorisHomeDir, String pidDir, String[] args, Star qeService.start(); } +ScheduledThreadPoolExecutor refreshTimer = ThreadPoolManager.newDaemonScheduledThreadPool(1, +"refresh-timer-pool", true); +Integer refreshTime = 20; Review Comment: What does `20` mean? It is a magic number to me ## fe/fe-core/src/main/java/org/apache/doris/PaloFe.java: ## @@ -173,6 +176,12 @@ public static void start(String dorisHomeDir, String pidDir, String[] args, Star qeService.start(); } +ScheduledThreadPoolExecutor refreshTimer = ThreadPoolManager.newDaemonScheduledThreadPool(1, Review Comment: This `ScheduledThreadPoolExecutor` should be in `RefreshMgr`. And there is already a `CatalogMgr` is `Env.java`, why new a `CatalogMgr` here? Currently, there are 2 main manager: - RefreshMgr - CatalogMgr All refresh method(refresh catalog/db/table/partition) should be in RefreshMgr. But currently, there is a mistake that "refresh catalog method" is in CatalogMgr. I think you can do a refactor to move the "refresh catalog method" from CatalogMgr to RefreshMgr. And this newly added refresh timer thread should be in RefreshMgr too. ## fe/fe-core/src/main/java/org/apache/doris/PaloFe.java: ## @@ -173,6 +176,12 @@ public static void start(String dorisHomeDir, String pidDir, String[] args, Star qeService.start(); } +ScheduledThreadPoolExecutor refreshTimer = ThreadPoolManager.newDaemonScheduledThreadPool(1, Review Comment: Please change your IDEA formatter to only format the code you modified. Not changing other code line. ## fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java: ## @@ -88,17 +91,25 @@ public class CatalogMgr implements Writable, GsonPostProcessable { private final Map idToCatalog = Maps.newConcurrentMap(); // this map will be regenerated from idToCatalog, so not need to persist. private final Map nameToCatalog = Maps.newConcurrentMap(); + +private static final Map refreshMap = Maps.newConcurrentMap(); Review Comment: Add some comment in code to explain the key value in refreshMap -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 commented on pull request #17931: [vectorized](function) support array_exists lambda function
ZhangYu0123 commented on PR #17931: URL: https://github.com/apache/doris/pull/17931#issuecomment-1475244953 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangstar333 commented on pull request #17832: [vectorized](function) support array_filter function
zhangstar333 commented on PR #17832: URL: https://github.com/apache/doris/pull/17832#issuecomment-1475247648 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17832: [vectorized](function) support array_filter function
github-actions[bot] commented on PR #17832: URL: https://github.com/apache/doris/pull/17832#issuecomment-1475248708 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] q763562998 commented on a diff in pull request #17884: [Feature](multi-catalog) add refresh for catalog if it needs
q763562998 commented on code in PR #17884: URL: https://github.com/apache/doris/pull/17884#discussion_r1141356531 ## fe/fe-core/src/main/java/org/apache/doris/PaloFe.java: ## @@ -173,6 +176,12 @@ public static void start(String dorisHomeDir, String pidDir, String[] args, Star qeService.start(); } +ScheduledThreadPoolExecutor refreshTimer = ThreadPoolManager.newDaemonScheduledThreadPool(1, Review Comment: 1.The reason why I want to create a new `CatalogMgr` is because the `CatalogMgr` needs a timing refresh time, because when the `CatalogMgr` of `Env` is initialized, there is no such timing refresh time, in order to reuse I will subsequently use the `CatalogMgr` of `Env` and assign this refresh time to the` CatalogMg`r of `Env` when it is initialized and constructed 2.Ok,I will move the "refresh catalog method" from `CatalogMgr` to `RefreshMgr` and putthe refresh timer thread in `RefreshMgr` 3.Ok, I will not change other lines of code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] q763562998 commented on a diff in pull request #17884: [Feature](multi-catalog) add refresh for catalog if it needs
q763562998 commented on code in PR #17884: URL: https://github.com/apache/doris/pull/17884#discussion_r1141356769 ## fe/fe-core/src/main/java/org/apache/doris/PaloFe.java: ## @@ -173,6 +176,12 @@ public static void start(String dorisHomeDir, String pidDir, String[] args, Star qeService.start(); } +ScheduledThreadPoolExecutor refreshTimer = ThreadPoolManager.newDaemonScheduledThreadPool(1, +"refresh-timer-pool", true); +Integer refreshTime = 20; Review Comment: Indeed it pops up, it is a timer refresh time that will go after every 20 seconds to refresh the catalog that needs to be refreshed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] q763562998 commented on a diff in pull request #17884: [Feature](multi-catalog) add refresh for catalog if it needs
q763562998 commented on code in PR #17884: URL: https://github.com/apache/doris/pull/17884#discussion_r1141356826 ## fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java: ## @@ -88,17 +91,25 @@ public class CatalogMgr implements Writable, GsonPostProcessable { private final Map idToCatalog = Maps.newConcurrentMap(); // this map will be regenerated from idToCatalog, so not need to persist. private final Map nameToCatalog = Maps.newConcurrentMap(); + +private static final Map refreshMap = Maps.newConcurrentMap(); Review Comment: ok,I will add some comment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] q763562998 commented on a diff in pull request #17884: [Feature](multi-catalog) add refresh for catalog if it needs
q763562998 commented on code in PR #17884: URL: https://github.com/apache/doris/pull/17884#discussion_r1141356826 ## fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java: ## @@ -88,17 +91,25 @@ public class CatalogMgr implements Writable, GsonPostProcessable { private final Map idToCatalog = Maps.newConcurrentMap(); // this map will be regenerated from idToCatalog, so not need to persist. private final Map nameToCatalog = Maps.newConcurrentMap(); + +private static final Map refreshMap = Maps.newConcurrentMap(); Review Comment: Ok,I will add some comment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #17671: [refactor](Nereids) expression translate no longer rely on legacy planner code
morrySnow commented on PR #17671: URL: https://github.com/apache/doris/pull/17671#issuecomment-1475254035 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #17813: [fix](planner) should not bind slot on brother's tuple in subquery
morrySnow commented on PR #17813: URL: https://github.com/apache/doris/pull/17813#issuecomment-1475254105 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #17885: [fix](planner) should always execute projection plan
morrySnow commented on PR #17885: URL: https://github.com/apache/doris/pull/17885#issuecomment-1475254167 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #17914: [enhencement](planner) update and delete support use alias for target table
morrySnow commented on PR #17914: URL: https://github.com/apache/doris/pull/17914#issuecomment-1475254247 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #17892: [enhancement](planner) condition delete support between predicate
morrySnow commented on PR #17892: URL: https://github.com/apache/doris/pull/17892#issuecomment-1475254213 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17928: [fix](nereids) Analyze failed for SQL that has count distinct with same col.
github-actions[bot] commented on PR #17928: URL: https://github.com/apache/doris/pull/17928#issuecomment-1475256236 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17928: [fix](nereids) Analyze failed for SQL that has count distinct with same col.
github-actions[bot] commented on PR #17928: URL: https://github.com/apache/doris/pull/17928#issuecomment-1475256250 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #17928: [fix](nereids) Analyze failed for SQL that has count distinct with same col.
morrySnow merged PR #17928: URL: https://github.com/apache/doris/pull/17928 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](nereids) Analyze failed for SQL that has count distinct with same col (#17928)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 5c990fb737 [fix](nereids) Analyze failed for SQL that has count distinct with same col (#17928) 5c990fb737 is described below commit 5c990fb737f665be63f7a8d733aed97dc9e0bf7e Author: AKIRA <33112463+kikyou1...@users.noreply.github.com> AuthorDate: Sun Mar 19 22:31:47 2023 +0900 [fix](nereids) Analyze failed for SQL that has count distinct with same col (#17928) This problem is caused by the slots with same hashcodes was put in the hashset results into the wrong rules was selected.Use list instead of set as return type of getDistinctArguments method --- .../nereids/rules/implementation/AggregateStrategies.java | 8 .../org/apache/doris/nereids/stats/StatsErrorEstimator.java| 10 +- .../apache/doris/nereids/trees/plans/algebra/Aggregate.java| 6 +++--- .../data/nereids_syntax_p0/aggregate_strategies.out| 7 +++ .../suites/nereids_syntax_p0/aggregate_strategies.groovy | 2 ++ 5 files changed, 25 insertions(+), 8 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/AggregateStrategies.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/AggregateStrategies.java index f56bec8d21..ff3fe6d421 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/AggregateStrategies.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/AggregateStrategies.java @@ -71,6 +71,7 @@ import com.google.common.collect.ImmutableMap; import com.google.common.collect.ImmutableSet; import com.google.common.collect.Lists; +import java.util.Collection; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -395,8 +396,7 @@ public class AggregateStrategies implements ImplementationRuleFactory { private List> twoPhaseAggregateWithCountDistinctMulti( LogicalAggregate logicalAgg, CascadesContext cascadesContext) { AggregateParam inputToBufferParam = new AggregateParam(AggPhase.LOCAL, AggMode.INPUT_TO_BUFFER); - -Set countDistinctArguments = logicalAgg.getDistinctArguments(); +Collection countDistinctArguments = logicalAgg.getDistinctArguments(); List localAggGroupBy = ImmutableList.copyOf(ImmutableSet.builder() .addAll(logicalAgg.getGroupByExpressions()) @@ -513,7 +513,7 @@ public class AggregateStrategies implements ImplementationRuleFactory { LogicalAggregate logicalAgg, CascadesContext cascadesContext) { AggregateParam inputToBufferParam = new AggregateParam(AggPhase.LOCAL, AggMode.INPUT_TO_BUFFER); -Set countDistinctArguments = logicalAgg.getDistinctArguments(); +Collection countDistinctArguments = logicalAgg.getDistinctArguments(); List localAggGroupBy = ImmutableList.copyOf(ImmutableSet.builder() .addAll(logicalAgg.getGroupByExpressions()) @@ -1134,7 +1134,7 @@ public class AggregateStrategies implements ImplementationRuleFactory { RequireProperties.of(PhysicalProperties.GATHER), anyLocalAgg); if (logicalAgg.getGroupByExpressions().isEmpty()) { -Set distinctArguments = logicalAgg.getDistinctArguments(); +Collection distinctArguments = logicalAgg.getDistinctArguments(); RequireProperties requireDistinctHash = RequireProperties.of(PhysicalProperties.createHash( distinctArguments, ShuffleType.AGGREGATE)); PhysicalHashAggregate hashLocalGatherGlobalAgg = anyLocalGatherGlobalAgg diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsErrorEstimator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsErrorEstimator.java index 27da0af30f..6966fc97ea 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsErrorEstimator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsErrorEstimator.java @@ -23,6 +23,7 @@ import org.apache.doris.common.util.ProfileManager; import org.apache.doris.nereids.trees.plans.AbstractPlan; import org.apache.doris.persist.gson.GsonUtils; import org.apache.doris.planner.PlanNode; +import org.apache.doris.statistics.Statistics; import org.apache.doris.thrift.TReportExecStatusParams; import org.apache.doris.thrift.TRuntimeProfileNode; import org.apache.doris.thrift.TUniqueId; @@ -50,8 +51,15 @@ public class StatsErrorEstimator { legacyPlanIdStats = new HashMap<>(); } +/** + * Map plan id to stats. + */ public void updateLegacyPlanIdToPhysicalPlan(PlanNode planNode, AbstractPlan physicalPlan) { -legacyPlanIdStats.put(planNode.getId().asInt(), Pair.of(physicalPlan.getStats().ge
[GitHub] [doris] TangSiyang2001 commented on pull request #17837: [fix](timeout) fix set timeout failure and simplify timeout logic
TangSiyang2001 commented on PR #17837: URL: https://github.com/apache/doris/pull/17837#issuecomment-1475259987 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon commented on pull request #17696: [feature](array_zip) Support array_zip function
eldenmoon commented on PR #17696: URL: https://github.com/apache/doris/pull/17696#issuecomment-1475260670 rebase please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangstar333 commented on pull request #16563: [fix](fold_constant) add defer in fold constant to close
zhangstar333 commented on PR #16563: URL: https://github.com/apache/doris/pull/16563#issuecomment-1475261219 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] AdarshRawat1 commented on issue #17902: [Enhancement] Explanation and presentation of different sizes in BrokerLoad & StreamLoad
AdarshRawat1 commented on issue #17902: URL: https://github.com/apache/doris/issues/17902#issuecomment-1475261548 Requesting permission to contribute to this feature. . Can you please advise on how I can get involved? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16563: [fix](fold_constant) add defer in fold constant to close
github-actions[bot] commented on PR #16563: URL: https://github.com/apache/doris/pull/16563#issuecomment-1475261750 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SaintBacchus commented on pull request #17745: [BugFix](DOE) Add http prefix when it's not set in hosts properties.
SaintBacchus commented on PR #17745: URL: https://github.com/apache/doris/pull/17745#issuecomment-1475265074 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SaintBacchus commented on pull request #17775: [Feature-Wip](MySQL LOAD)Add trim quotes property for mysql load
SaintBacchus commented on PR #17775: URL: https://github.com/apache/doris/pull/17775#issuecomment-1475264449 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #17704: [feature](table-metadata) support altering the property "light_schema_change" for the tables which created before 1.2
TangSiyang2001 commented on PR #17704: URL: https://github.com/apache/doris/pull/17704#issuecomment-1475276090 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17704: [feature](table-metadata) support altering the property "light_schema_change" for the tables which created before 1.2
github-actions[bot] commented on PR #17704: URL: https://github.com/apache/doris/pull/17704#issuecomment-1475276836 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on pull request #17741: [enhancement](nereids)support topN opt in nereids
englefly commented on PR #17741: URL: https://github.com/apache/doris/pull/17741#issuecomment-1475279602 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #17937: [fix](ctas) fix show load throw NPE after ctas
morningman opened a new pull request, #17937: URL: https://github.com/apache/doris/pull/17937 # Proposed changes Issue Number: close #xxx ## Problem summary Missing userinfo ``` java.lang.NullPointerException: null at org.apache.doris.load.loadv2.LoadJob.getShowInfo(LoadJob.java:816) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.load.loadv2.LoadManager.getLoadJobInfosByDb(LoadManager.java:557) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ShowExecutor.handleShowLoad(ShowExecutor.java:1094) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ShowExecutor.execute(ShowExecutor.java:280) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleShow(StmtExecutor.java:1862) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:619) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:435) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:414) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:558) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:799) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131] ``` ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #17937: [fix](ctas) fix show load throw NPE after ctas
morningman commented on PR #17937: URL: https://github.com/apache/doris/pull/17937#issuecomment-1475280054 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17704: [feature](table-metadata) support altering the property "light_schema_change" for the tables which created before 1.2
hello-stephen commented on PR #17704: URL: https://github.com/apache/doris/pull/17704#issuecomment-1475286492 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.07 seconds stream load tsv: 450 seconds loaded 74807831229 Bytes, about 158 MB/s stream load json: 24 seconds loaded 2358488459 Bytes, about 93 MB/s stream load orc: 73 seconds loaded 1101869774 Bytes, about 14 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230319150724_clickbench_pr_116899.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #17837: [fix](timeout) fix set timeout failure and simplify timeout logic
TangSiyang2001 commented on PR #17837: URL: https://github.com/apache/doris/pull/17837#issuecomment-1475291409 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17837: [fix](timeout) fix set timeout failure and simplify timeout logic
github-actions[bot] commented on PR #17837: URL: https://github.com/apache/doris/pull/17837#issuecomment-1475292434 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17885: [fix](planner) should always execute projection plan
hello-stephen commented on PR #17885: URL: https://github.com/apache/doris/pull/17885#issuecomment-1475301474 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.73 seconds stream load tsv: 470 seconds loaded 74807831229 Bytes, about 151 MB/s stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s stream load orc: 73 seconds loaded 1101869774 Bytes, about 14 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230319155220_clickbench_pr_116926.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17892: [enhancement](planner) condition delete support between predicate
hello-stephen commented on PR #17892: URL: https://github.com/apache/doris/pull/17892#issuecomment-1475301597 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.21 seconds stream load tsv: 451 seconds loaded 74807831229 Bytes, about 158 MB/s stream load json: 24 seconds loaded 2358488459 Bytes, about 93 MB/s stream load orc: 73 seconds loaded 1101869774 Bytes, about 14 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230319155257_clickbench_pr_116928.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17924: [Fix](parquet-reader) Fix dict_filter crashed caused by VDirectInPredicate checking expr result is not nullable.
github-actions[bot] commented on PR #17924: URL: https://github.com/apache/doris/pull/17924#issuecomment-1475304010 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed issue #17923: [Bug] Be crashed in parquet dict_filter function caused by VDirectInPredicate checking expr result is not nullable.
morningman closed issue #17923: [Bug] Be crashed in parquet dict_filter function caused by VDirectInPredicate checking expr result is not nullable. URL: https://github.com/apache/doris/issues/17923 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #17924: [Fix](parquet-reader) Fix dict_filter crashed caused by VDirectInPredicate checking expr result is not nullable.
morningman merged PR #17924: URL: https://github.com/apache/doris/pull/17924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org