[GitHub] [doris] yuxuan-luo opened a new pull request, #19233: [Enhancement](load) Add the size during process.
yuxuan-luo opened a new pull request, #19233: URL: https://github.com/apache/doris/pull/19233 # Proposed changes Issue Number: close #17902 ## Problem summary Add the size during process. ## Checklist(Required) * [x ] Does it affect the original behavior * [ x] Has unit tests been added * [ x] Has document been added or modified * [ x] Does it need to update dependencies * [ x] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19233: [Enhancement](load) Add the size during process.
github-actions[bot] commented on PR #19233: URL: https://github.com/apache/doris/pull/19233#issuecomment-1529480424 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #19228: [vectorized](function) add some check about result type in array map
yiguolei merged PR #19228: URL: https://github.com/apache/doris/pull/19228 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [vectorized](function) add some check about result type in array map (#19228)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new eac61dc410 [vectorized](function) add some check about result type in
array map (#19228)
eac61dc410 is described below
commit eac61dc41043e4bad935d5609e426ed23a7eb1cb
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Mon May 1 16:28:11 2023 +0800
[vectorized](function) add some check about result type in array map
(#19228)
---
be/src/vec/exprs/lambda_function/lambda_function.h | 2 +-
.../lambda_function/varray_filter_function.cpp | 2 +-
.../exprs/lambda_function/varray_map_function.cpp | 24 +-
gensrc/script/doris_builtins_functions.py | 2 +-
4 files changed, 26 insertions(+), 4 deletions(-)
diff --git a/be/src/vec/exprs/lambda_function/lambda_function.h
b/be/src/vec/exprs/lambda_function/lambda_function.h
index a1eb173725..a7f2fb8f69 100644
--- a/be/src/vec/exprs/lambda_function/lambda_function.h
+++ b/be/src/vec/exprs/lambda_function/lambda_function.h
@@ -32,7 +32,7 @@ public:
virtual std::string get_name() const = 0;
virtual doris::Status execute(VExprContext* context,
doris::vectorized::Block* block,
- int* result_column_id, DataTypePtr
result_type,
+ int* result_column_id, const DataTypePtr&
result_type,
const std::vector& children) = 0;
};
diff --git a/be/src/vec/exprs/lambda_function/varray_filter_function.cpp
b/be/src/vec/exprs/lambda_function/varray_filter_function.cpp
index 59d6dcb851..9fae6ed634 100644
--- a/be/src/vec/exprs/lambda_function/varray_filter_function.cpp
+++ b/be/src/vec/exprs/lambda_function/varray_filter_function.cpp
@@ -60,7 +60,7 @@ public:
std::string get_name() const override { return name; }
doris::Status execute(VExprContext* context, doris::vectorized::Block*
block,
- int* result_column_id, DataTypePtr result_type,
+ int* result_column_id, const DataTypePtr&
result_type,
const std::vector& children) override {
///* array_filter(array, array) *///
diff --git a/be/src/vec/exprs/lambda_function/varray_map_function.cpp
b/be/src/vec/exprs/lambda_function/varray_map_function.cpp
index 2d305af7b6..2f57ed8976 100644
--- a/be/src/vec/exprs/lambda_function/varray_map_function.cpp
+++ b/be/src/vec/exprs/lambda_function/varray_map_function.cpp
@@ -60,7 +60,7 @@ public:
std::string get_name() const override { return name; }
doris::Status execute(VExprContext* context, doris::vectorized::Block*
block,
- int* result_column_id, DataTypePtr result_type,
+ int* result_column_id, const DataTypePtr&
result_type,
const std::vector& children) override {
///* array_map(lambda,arg1,arg2,.) *///
@@ -138,6 +138,12 @@ public:
"R" +
array_column_type_name.name};
lambda_block.insert(std::move(data_column));
}
+//check nullable(array(nullable(nested)))
+DCHECK(result_type->is_nullable() &&
+
is_array(((DataTypeNullable*)result_type.get())->get_nested_type()))
+<< "array_map result type is error, now must be
nullable(array): "
+<< result_type->get_name()
+<< " ,and block structure is: " << block->dump_structure();
//3. child[0]->execute(new_block)
RETURN_IF_ERROR(children[0]->execute(context, &lambda_block,
result_column_id));
@@ -156,6 +162,7 @@ public:
result_type, res_name};
} else {
+// deal with eg: select array_map(x -> x is null, [null, 1, 2]);
// need to create the nested column null map for column array
auto nested_null_map = ColumnUInt8::create(res_col->size(), 0);
result_arr = {ColumnNullable::create(
@@ -167,6 +174,21 @@ public:
}
block->insert(std::move(result_arr));
*result_column_id = block->columns() - 1;
+//check nullable(nested)
+DCHECK((assert_cast(
+
(((DataTypeNullable*)result_type.get())->get_nested_type().get(
+ ->get_nested_type()
+ ->equals(*make_nullable(res_type)))
+<< " array_map function FE given result type is: " <<
result_type->get_name()
+<< " get nested is: "
+<< (assert_cast(
+
(((DataTypeNullable*)result_type.get())->get_nested_type().get(
+ ->get_nested_type()
+ ->get_name()
+<< " and now actual nested
[GitHub] [doris] ZhangYu0123 commented on pull request #19232: [Feature](ngram index) support `=` and `in` predicate for query in ngram index
ZhangYu0123 commented on PR #19232: URL: https://github.com/apache/doris/pull/19232#issuecomment-1529488067 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19232: [Feature](ngram index) support `=` and `in` predicate for query in ngram index
github-actions[bot] commented on PR #19232: URL: https://github.com/apache/doris/pull/19232#issuecomment-1529489427 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 commented on pull request #19232: [Feature](ngram index) support `=` and `in` predicate for query in ngram index
ZhangYu0123 commented on PR #19232: URL: https://github.com/apache/doris/pull/19232#issuecomment-1529502581 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19232: [Feature](ngram index) support `=` and `in` predicate for query in ngram index
github-actions[bot] commented on PR #19232: URL: https://github.com/apache/doris/pull/19232#issuecomment-1529505379 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 commented on pull request #19232: [Feature](ngram index) support `=` and `in` predicate for accelerating query when using ngram index
ZhangYu0123 commented on PR #19232: URL: https://github.com/apache/doris/pull/19232#issuecomment-1529608239 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19232: [Feature](ngram index) support `=` and `in` predicate for accelerating query when using ngram index
github-actions[bot] commented on PR #19232: URL: https://github.com/apache/doris/pull/19232#issuecomment-1529609482 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 commented on pull request #19232: [Feature](ngram index) support `=` and `in` predicate for accelerating query when using ngram index
ZhangYu0123 commented on PR #19232: URL: https://github.com/apache/doris/pull/19232#issuecomment-1529609932 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19232: [Feature](ngram index) support `=` and `in` predicate for accelerating query when using ngram index
github-actions[bot] commented on PR #19232: URL: https://github.com/apache/doris/pull/19232#issuecomment-152969 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-2.0-alpha updated: change version to 2.0.0-alpha1
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-2.0-alpha
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0-alpha by this push:
new 86a99d58ac change version to 2.0.0-alpha1
86a99d58ac is described below
commit 86a99d58ac136d7f4b74d20f6f9099a326a939f7
Author: Kang
AuthorDate: Mon May 1 12:48:30 2023 +0800
change version to 2.0.0-alpha1
---
gensrc/script/gen_build_version.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/gensrc/script/gen_build_version.sh
b/gensrc/script/gen_build_version.sh
index c2fc94fcf7..bd9f7dda46 100755
--- a/gensrc/script/gen_build_version.sh
+++ b/gensrc/script/gen_build_version.sh
@@ -28,10 +28,10 @@
set -eo pipefail
build_version_prefix="doris"
-build_version_major=0
+build_version_major=2
build_version_minor=0
build_version_patch=0
-build_version_rc_version="trunk"
+build_version_rc_version="alpha1"
build_version="${build_version_prefix}-${build_version_major}.${build_version_minor}.${build_version_patch}-${build_version_rc_version}"
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] annotated tag 2.0.0-alpha1 updated (86a99d58ac -> 7424e2213d)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to annotated tag 2.0.0-alpha1 in repository https://gitbox.apache.org/repos/asf/doris.git *** WARNING: tag 2.0.0-alpha1 was modified! *** from 86a99d58ac (commit) to 7424e2213d (tag) tagging 86a99d58ac136d7f4b74d20f6f9099a326a939f7 (commit) by morningman on Mon May 1 22:10:53 2023 +0800 - Log - 2.0.0-alpha1 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on pull request #19098: [refactor](fs)(step3)use filesystem instead of old storage, new storage just access remote object storage
wsjz commented on PR #19098: URL: https://github.com/apache/doris/pull/19098#issuecomment-1529754728 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed issue #18762: Release Note 1.2.4.1
morningman closed issue #18762: Release Note 1.2.4.1 URL: https://github.com/apache/doris/issues/18762 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #19217: [doc](thrift) update doc for thrift 0.16
morningman commented on PR #19217: URL: https://github.com/apache/doris/pull/19217#issuecomment-1529782260 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19167: [feature](table) implement the round robin selection be when create tablet
github-actions[bot] commented on PR #19167: URL: https://github.com/apache/doris/pull/19167#issuecomment-1529789977 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19167: [feature](table) implement the round robin selection be when create tablet
github-actions[bot] commented on PR #19167: URL: https://github.com/apache/doris/pull/19167#issuecomment-1529790020 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #19120: [Fix](multi-catalog) fix FE abnormal exit when replay OP_REFRESH_EXTERNAL_TABLE
morningman commented on PR #19120: URL: https://github.com/apache/doris/pull/19120#issuecomment-1529792256 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19120: [Fix](multi-catalog) fix FE abnormal exit when replay OP_REFRESH_EXTERNAL_TABLE
github-actions[bot] commented on PR #19120: URL: https://github.com/apache/doris/pull/19120#issuecomment-1529792676 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19120: [Fix](multi-catalog) fix FE abnormal exit when replay OP_REFRESH_EXTERNAL_TABLE
github-actions[bot] commented on PR #19120: URL: https://github.com/apache/doris/pull/19120#issuecomment-1529792714 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19119: [enhance](be)add more profile in prefetched buffered reader
github-actions[bot] commented on PR #19119: URL: https://github.com/apache/doris/pull/19119#issuecomment-1529794253 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19119: [enhance](be)add more profile in prefetched buffered reader
github-actions[bot] commented on PR #19119: URL: https://github.com/apache/doris/pull/19119#issuecomment-1529794284 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #19048: [docker](hudi) add hudi docker compose
morningman commented on PR #19048: URL: https://github.com/apache/doris/pull/19048#issuecomment-1529795683 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed pull request #19048: [docker](hudi) add hudi docker compose
morningman closed pull request #19048: [docker](hudi) add hudi docker compose URL: https://github.com/apache/doris/pull/19048 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #19048: [docker](hudi) add hudi docker compose
morningman commented on PR #19048: URL: https://github.com/apache/doris/pull/19048#issuecomment-1529795869 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19048: [docker](hudi) add hudi docker compose
github-actions[bot] commented on PR #19048: URL: https://github.com/apache/doris/pull/19048#issuecomment-1529796158 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19048: [docker](hudi) add hudi docker compose
github-actions[bot] commented on PR #19048: URL: https://github.com/apache/doris/pull/19048#issuecomment-1529796212 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #19090: [feature](analysis) insert overwrite table
morningman commented on code in PR #19090:
URL: https://github.com/apache/doris/pull/19090#discussion_r1181634672
##
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java:
##
@@ -1341,6 +1341,50 @@ public void replayCreateTable(String dbName, Table
table) throws MetaNotFoundExc
}
}
+public void addPartitionLike(Database db, String tableName,
AddPartitionLikeClause addPartitionLikeClause)
+throws DdlException {
+try {
+Table table = db.getTableOrDdlException(tableName);
+
+if (table.getType() == TableType.VIEW) {
Review Comment:
```suggestion
if (table.getType() != TableType.OLAP) {
```
##
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java:
##
@@ -1341,6 +1341,50 @@ public void replayCreateTable(String dbName, Table
table) throws MetaNotFoundExc
}
}
+public void addPartitionLike(Database db, String tableName,
AddPartitionLikeClause addPartitionLikeClause)
+throws DdlException {
+try {
+Table table = db.getTableOrDdlException(tableName);
+
+if (table.getType() == TableType.VIEW) {
+throw new DdlException("Not support create partition from a
View");
Review Comment:
```suggestion
throw new DdlException("Only support create partition from a
OLAP table");
```
##
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java:
##
@@ -1341,6 +1341,50 @@ public void replayCreateTable(String dbName, Table
table) throws MetaNotFoundExc
}
}
+public void addPartitionLike(Database db, String tableName,
AddPartitionLikeClause addPartitionLikeClause)
+throws DdlException {
+try {
+Table table = db.getTableOrDdlException(tableName);
+
+if (table.getType() == TableType.VIEW) {
+throw new DdlException("Not support create partition from a
View");
+}
+
+table.readLock();
+try {
+if (table.getType().equals(TableType.OLAP)) {
+String partitionName =
addPartitionLikeClause.getPartitionName();
+String existedName =
addPartitionLikeClause.getExistedPartitionName();
+OlapTable olapTable = (OlapTable) table;
+Partition part = olapTable.getPartition(existedName);
+PartitionInfo partitionInfo = olapTable.getPartitionInfo();
+PartitionDesc partitionDesc =
partitionInfo.toPartitionDesc((OlapTable) table);
+SinglePartitionDesc oldPartitionDesc =
partitionDesc.getSinglePartitionDescByName(existedName);
+if (oldPartitionDesc == null) {
+throw new DdlException("Failed to ADD PARTITION" +
partitionName + " LIKE "
++ existedName + ". Reason: " + "partition " +
existedName + "not exist");
+}
+DistributionDesc distributionDesc =
part.getDistributionInfo().toDistributionDesc();
+SinglePartitionDesc newPartitionDesc = new
SinglePartitionDesc(false, partitionName,
+oldPartitionDesc.getPartitionKeyDesc(),
oldPartitionDesc.getProperties());
+Map properties =
newPartitionDesc.getProperties();
+AddPartitionClause clause = new
AddPartitionClause(newPartitionDesc, distributionDesc,
+properties,
addPartitionLikeClause.getIsTempPartition());
+table.readUnlock();
Review Comment:
This `table.readUnlock();` should be in a `finally` block.
You split the whole logic into 2 parts:
One is for creating an `AddPartitionClause`, the other is for calling
`addPartition()`.
And refactor the current `try..catch`
##
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java:
##
@@ -1341,6 +1341,50 @@ public void replayCreateTable(String dbName, Table
table) throws MetaNotFoundExc
}
}
+public void addPartitionLike(Database db, String tableName,
AddPartitionLikeClause addPartitionLikeClause)
+throws DdlException {
+try {
+Table table = db.getTableOrDdlException(tableName);
+
+if (table.getType() == TableType.VIEW) {
+throw new DdlException("Not support create partition from a
View");
+}
+
+table.readLock();
+try {
+if (table.getType().equals(TableType.OLAP)) {
Review Comment:
Remove this `if`
##
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##
@@ -2145,6 +2163,160 @@ private void handleCtasRollback(TableName table) {
}
}
+private void handleIotStmt() {
+InsertOverwriteTableStmt iotStmt = (Inser
[GitHub] [doris] morningman commented on pull request #19090: [feature](analysis) insert overwrite table
morningman commented on PR #19090: URL: https://github.com/apache/doris/pull/19090#issuecomment-1529822079 Need to add more test cases for these feature: 1. insert overwrite empty table/partition 2. insert overwrite with wrong table/partition name 3. insert overwrite table/partition with wrong data(data not in partition's range, or data is unqualified). 4. insert overwrite different partition type(range partition & list partition) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19026: [Fix](load) fix request_slave_tablet_pull_rowset get wrong url in case of ipv6 address
github-actions[bot] commented on PR #19026: URL: https://github.com/apache/doris/pull/19026#issuecomment-1529829213 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on issue #17176: [Good First Issue] Doris' Future
morningman commented on issue #17176: URL: https://github.com/apache/doris/issues/17176#issuecomment-1529836403 > Hi. I want to do issue #86. Could you assign it to me? Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
svn commit: r61565 - /dev/doris/2.0.0-alpha1/
Author: morningman Date: Mon May 1 15:29:09 2023 New Revision: 61565 Log: doris 2.0.0 alpha1 release Added: dev/doris/2.0.0-alpha1/ dev/doris/2.0.0-alpha1/apache-doris-2.0.0-alpha1-src.tar.gz (with props) dev/doris/2.0.0-alpha1/apache-doris-2.0.0-alpha1-src.tar.gz.asc (with props) dev/doris/2.0.0-alpha1/apache-doris-2.0.0-alpha1-src.tar.gz.sha512 dev/doris/2.0.0-alpha1/apache-doris-be-2.0.0-alpha1-bin-x86_64.tar.xz (with props) dev/doris/2.0.0-alpha1/apache-doris-be-2.0.0-alpha1-bin-x86_64.tar.xz.asc (with props) dev/doris/2.0.0-alpha1/apache-doris-be-2.0.0-alpha1-bin-x86_64.tar.xz.sha512 dev/doris/2.0.0-alpha1/apache-doris-dependencies-2.0.0-alpha1-bin-x86_64.tar.xz (with props) dev/doris/2.0.0-alpha1/apache-doris-dependencies-2.0.0-alpha1-bin-x86_64.tar.xz.asc (with props) dev/doris/2.0.0-alpha1/apache-doris-dependencies-2.0.0-alpha1-bin-x86_64.tar.xz.sha512 dev/doris/2.0.0-alpha1/apache-doris-fe-2.0.0-alpha1-bin-x86_64.tar.xz (with props) dev/doris/2.0.0-alpha1/apache-doris-fe-2.0.0-alpha1-bin-x86_64.tar.xz.asc (with props) dev/doris/2.0.0-alpha1/apache-doris-fe-2.0.0-alpha1-bin-x86_64.tar.xz.sha512 Added: dev/doris/2.0.0-alpha1/apache-doris-2.0.0-alpha1-src.tar.gz == Binary file - no diff available. Propchange: dev/doris/2.0.0-alpha1/apache-doris-2.0.0-alpha1-src.tar.gz -- svn:mime-type = application/x-gzip Added: dev/doris/2.0.0-alpha1/apache-doris-2.0.0-alpha1-src.tar.gz.asc == Binary file - no diff available. Propchange: dev/doris/2.0.0-alpha1/apache-doris-2.0.0-alpha1-src.tar.gz.asc -- svn:mime-type = application/pgp-signature Added: dev/doris/2.0.0-alpha1/apache-doris-2.0.0-alpha1-src.tar.gz.sha512 == --- dev/doris/2.0.0-alpha1/apache-doris-2.0.0-alpha1-src.tar.gz.sha512 (added) +++ dev/doris/2.0.0-alpha1/apache-doris-2.0.0-alpha1-src.tar.gz.sha512 Mon May 1 15:29:09 2023 @@ -0,0 +1 @@ +fd1927c53137e4d41260272ac876a69079a0bdb7b0589487ae5df101dea3014bdc86efc99a0d25df23d834669df829e8629df59562d1d15808c79217f8d00823 apache-doris-2.0.0-alpha1-src.tar.gz Added: dev/doris/2.0.0-alpha1/apache-doris-be-2.0.0-alpha1-bin-x86_64.tar.xz == Binary file - no diff available. Propchange: dev/doris/2.0.0-alpha1/apache-doris-be-2.0.0-alpha1-bin-x86_64.tar.xz -- svn:mime-type = application/x-xz Added: dev/doris/2.0.0-alpha1/apache-doris-be-2.0.0-alpha1-bin-x86_64.tar.xz.asc == Binary file - no diff available. Propchange: dev/doris/2.0.0-alpha1/apache-doris-be-2.0.0-alpha1-bin-x86_64.tar.xz.asc -- svn:mime-type = application/pgp-signature Added: dev/doris/2.0.0-alpha1/apache-doris-be-2.0.0-alpha1-bin-x86_64.tar.xz.sha512 == --- dev/doris/2.0.0-alpha1/apache-doris-be-2.0.0-alpha1-bin-x86_64.tar.xz.sha512 (added) +++ dev/doris/2.0.0-alpha1/apache-doris-be-2.0.0-alpha1-bin-x86_64.tar.xz.sha512 Mon May 1 15:29:09 2023 @@ -0,0 +1 @@ +91b630800499057588369c88110734884145347b959eb738ef432879f016ffabe1041592de06c1d58d3627865946d7fa224698c57dd7ba3a2721256ee96e6a9d apache-doris-be-2.0.0-alpha1-bin-x86_64.tar.xz Added: dev/doris/2.0.0-alpha1/apache-doris-dependencies-2.0.0-alpha1-bin-x86_64.tar.xz == Binary file - no diff available. Propchange: dev/doris/2.0.0-alpha1/apache-doris-dependencies-2.0.0-alpha1-bin-x86_64.tar.xz -- svn:mime-type = application/x-xz Added: dev/doris/2.0.0-alpha1/apache-doris-dependencies-2.0.0-alpha1-bin-x86_64.tar.xz.asc == Binary file - no diff available. Propchange: dev/doris/2.0.0-alpha1/apache-doris-dependencies-2.0.0-alpha1-bin-x86_64.tar.xz.asc -- svn:mime-type = application/pgp-signature Added: dev/doris/2.0.0-alpha1/apache-doris-dependencies-2.0.0-alpha1-bin-x86_64.tar.xz.sha512 == --- dev/doris/2.0.0-alpha1/apache-doris-dependencies-2.0.0-alpha1-bin-x86_64.tar.xz.sha512 (added) +++ dev/doris/2.0.0-alpha1/apache-doris-dependencies-2.0.0
[GitHub] [doris] morningman commented on issue #17176: [Good First Issue] Doris' Future
morningman commented on issue #17176: URL: https://github.com/apache/doris/issues/17176#issuecomment-1529837727 > Hi. I want to do issue #86. Could you assign it to me? Sorry that #86 has been taken by @nextdreamblue . I have updated the sheet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #19229: [community](collaborator) add more collaborators
morningman merged PR #19229: URL: https://github.com/apache/doris/pull/19229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [community](collaborator) add more collaborators (#19229)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 43803940f5 [community](collaborator) add more collaborators (#19229) 43803940f5 is described below commit 43803940f5d0cd07a17ae2cf060a9c60c928fa31 Author: abmdocrt AuthorDate: Mon May 1 23:34:06 2023 +0800 [community](collaborator) add more collaborators (#19229) Add @TangSiyang2001 as collaborator, and he helped a lot in good first issue. --- .asf.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/.asf.yaml b/.asf.yaml index f6ee3d31fd..9e37611dbe 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -87,6 +87,7 @@ github: - zy-kkk - Yukang-Lian - xiaokang +- TangSiyang2001 notifications: pullrequests_status: commits@doris.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #19170: [Enhancement](Broker Load) New progress manager for showing loading progress status
morningman commented on code in PR #19170:
URL: https://github.com/apache/doris/pull/19170#discussion_r1181664051
##
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/ProgressManager.java:
##
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.load.loadv2;
+
+import org.apache.doris.thrift.TUniqueId;
+
+import com.google.common.collect.HashBasedTable;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Table;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Map;
+
+/**
+ * ProgressManager manage the progress of loading and exporting tasks
+ */
+public class ProgressManager {
+private static final Logger LOG = LogManager.getLogger(LoadManager.class);
Review Comment:
```suggestion
private static final Logger LOG =
LogManager.getLogger(ProgressManager.class);
```
##
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/ProgressManager.java:
##
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.load.loadv2;
+
+import org.apache.doris.thrift.TUniqueId;
+
+import com.google.common.collect.HashBasedTable;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Table;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Map;
+
+/**
+ * ProgressManager manage the progress of loading and exporting tasks
+ */
+public class ProgressManager {
+private static final Logger LOG = LogManager.getLogger(LoadManager.class);
+
+private Map idToProgress = Maps.newConcurrentMap();
+
+public void registerProgress(Long id, int scannerNum) {
+LOG.info("create " + id + " with initial scannerNum " + scannerNum);
Review Comment:
```suggestion
LOG.debug("create {} with initial scannerNum {}", id, scannerNum);
```
##
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/ProgressManager.java:
##
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.load.loadv2;
+
+import org.apache.doris.thrift.TUniqueId;
+
+import com.google.common.collect.HashBasedTable;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Table;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Map;
+
+/**
+ * ProgressManager manage the progress of loading and exporting tasks
+ */
+public class ProgressManager {
+private static final Logger LOG = LogManager.getLogger(LoadManager.class);
+
+private Map idToProgress = Maps.newConcurrentMa
[GitHub] [doris] morningman commented on pull request #18712: [improvement] TOperationStatus determines that a null pointer is redundant.
morningman commented on PR #18712: URL: https://github.com/apache/doris/pull/18712#issuecomment-1529866960 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18712: [improvement] TOperationStatus determines that a null pointer is redundant.
github-actions[bot] commented on PR #18712: URL: https://github.com/apache/doris/pull/18712#issuecomment-1529868106 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18712: [improvement] TOperationStatus determines that a null pointer is redundant.
github-actions[bot] commented on PR #18712: URL: https://github.com/apache/doris/pull/18712#issuecomment-1529868218 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] reswqa opened a new pull request, #19234: [feat](mv lifecycle) Support drop partition from index.
reswqa opened a new pull request, #19234: URL: https://github.com/apache/doris/pull/19234 # Proposed changes Issue Number: partially close #18505 ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [x] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments 1. Support deleting only partial indices of partition. 2. When the last index of a partition is deleted, delete this partition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #19216: [BugFix](vectorization) Fix some regression bugs
zclllyybb commented on PR #19216: URL: https://github.com/apache/doris/pull/19216#issuecomment-1530264512 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19216: [BugFix](vectorization) Fix some regression bugs
github-actions[bot] commented on PR #19216: URL: https://github.com/apache/doris/pull/19216#issuecomment-1530271749 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #19217: [doc](thrift) update doc for thrift 0.16
morningman commented on PR #19217: URL: https://github.com/apache/doris/pull/19217#issuecomment-1530758021 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #19120: [Fix](multi-catalog) fix FE abnormal exit when replay OP_REFRESH_EXTERNAL_TABLE
morningman merged PR #19120: URL: https://github.com/apache/doris/pull/19120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Fix](multi-catalog) fix FE abnormal exit when replay OP_REFRESH_EXTERNAL_TABLE (#19120)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 05beb8538e [Fix](multi-catalog) fix FE abnormal exit when replay
OP_REFRESH_EXTERNAL_TABLE (#19120)
05beb8538e is described below
commit 05beb8538eefaa83f0b11f8ee32b91f33d186f41
Author: Xiangyu Wang
AuthorDate: Tue May 2 09:53:20 2023 +0800
[Fix](multi-catalog) fix FE abnormal exit when replay
OP_REFRESH_EXTERNAL_TABLE (#19120)
When salve FE nodes replay OP_REFRESH_EXTERNAL_TABLE log, it will invoke
`org.apache.doris.datasource.hive.HiveMetaStoreCache#invalidateTableCache`,
but if the table is a non-partitioned table, it will invoke
`catalog.getClient().getTable`.
If some network problem occurs or this table is not existed, an exception
will be thrown and FE will exit right away.
The solution is that we can use a dummy key as the file cache key which
only contains db name and table name.
And when slave FE nodes replay OP_REFRESH_EXTERNAL_TABLE log,
it will not rely on the hms client and there will not any exception occurs.
---
.../doris/datasource/hive/HiveMetaStoreCache.java | 34 +-
.../doris/datasource/hive/HivePartition.java | 18 ++--
.../doris/planner/external/HiveSplitter.java | 3 +-
3 files changed, 44 insertions(+), 11 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
index f859f9cc7d..44d2379188 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
@@ -59,7 +59,6 @@ import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.HdfsConfiguration;
import org.apache.hadoop.hive.metastore.api.Partition;
import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
-import org.apache.hadoop.hive.metastore.api.Table;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.InputFormat;
import org.apache.hadoop.mapred.InputSplit;
@@ -261,7 +260,7 @@ public class HiveMetaStoreCache {
sd.getInputFormat(), sd.getLocation(), key,
catalog.getName());
}
// TODO: more info?
-return new HivePartition(sd.getInputFormat(), sd.getLocation(),
key.values);
+return new HivePartition(key.dbName, key.tblName, false,
sd.getInputFormat(), sd.getLocation(), key.values);
}
private FileCacheValue loadFiles(FileCacheKey key) {
@@ -360,7 +359,10 @@ public class HiveMetaStoreCache {
long start = System.currentTimeMillis();
List keys =
Lists.newArrayListWithExpectedSize(partitions.size());
partitions.stream().forEach(p -> {
-FileCacheKey fileCacheKey = new FileCacheKey(p.getPath(),
p.getInputFormat(), p.getPartitionValues());
+FileCacheKey fileCacheKey = p.isDummyPartition()
+? FileCacheKey.createDummyCacheKey(p.getDbName(),
p.getTblName(), p.getPath(),
+p.getInputFormat(), useSelfSplitter)
+: new FileCacheKey(p.getPath(), p.getInputFormat(),
p.getPartitionValues());
fileCacheKey.setUseSelfSplitter(useSelfSplitter);
keys.add(fileCacheKey);
});
@@ -438,12 +440,13 @@ public class HiveMetaStoreCache {
* A file cache entry can be created reference to
* {@link
org.apache.doris.planner.external.HiveSplitter#getSplits},
* so we need to invalidate it if this is a non-partitioned table.
- *
+ * We use {@link
org.apache.doris.datasource.hive.HiveMetaStoreCache.FileCacheKey#createDummyCacheKey}
+ * to avoid invocation by Hms Client, because this method may be
invoked when salve FE replay journal logs,
+ * and FE will exit if some network problems occur.
* */
-Table table = catalog.getClient().getTable(dbName, tblName);
-// we just need to assign the `location` filed because the
`equals` method of `FileCacheKey`
-// just compares the value of `location`
-fileCacheRef.get().invalidate(new
FileCacheKey(table.getSd().getLocation(), null, null));
+FileCacheKey fileCacheKey = FileCacheKey.createDummyCacheKey(
+dbName, tblName, null, null, false);
+fileCacheRef.get().invalidate(fileCacheKey);
}
}
@@ -699,6 +702,7 @@ public class HiveMetaStoreCache {
@Data
public static class FileCacheKey {
+private String dummyKey;
private String location;
// not in key
private String inputFormat;
@@ -717,6 +721,14 @@ public class HiveMetaStoreCache
[GitHub] [doris] morningman merged pull request #19119: [enhance](be)add more profile in prefetched buffered reader
morningman merged PR #19119: URL: https://github.com/apache/doris/pull/19119 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhance](be)add more profile in prefetched buffered reader (#19119)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b0c215e694 [enhance](be)add more profile in prefetched buffered reader
(#19119)
b0c215e694 is described below
commit b0c215e694811822898919c5cfe94e525c41bb14
Author: AlexYue
AuthorDate: Tue May 2 09:53:39 2023 +0800
[enhance](be)add more profile in prefetched buffered reader (#19119)
---
be/src/io/fs/buffered_reader.cpp | 54 +-
be/src/io/fs/buffered_reader.h | 24 +++
be/src/service/internal_service.cpp| 5 +++-
be/test/io/fs/buffered_reader_test.cpp | 12 +---
4 files changed, 77 insertions(+), 18 deletions(-)
diff --git a/be/src/io/fs/buffered_reader.cpp b/be/src/io/fs/buffered_reader.cpp
index dab5ab8f61..f57aa16f86 100644
--- a/be/src/io/fs/buffered_reader.cpp
+++ b/be/src/io/fs/buffered_reader.cpp
@@ -406,8 +406,13 @@ void PrefetchBuffer::prefetch_buffer() {
buf_size = merge_small_ranges(_offset, read_range_index);
}
-s = _reader->read_at(_offset, Slice {_buf.get(), buf_size}, &_len,
_io_ctx);
+{
+SCOPED_RAW_TIMER(&_statis.read_time);
+s = _reader->read_at(_offset, Slice {_buf.get(), buf_size}, &_len,
_io_ctx);
+}
g_bytes_downloaded << _len;
+_statis.prefetch_request_io += 1;
+_statis.prefetch_request_bytes += _len;
std::unique_lock lck {_lock};
_prefetched.wait(lck, [this]() { return _buffer_status ==
BufferStatus::PENDING; });
if (!s.ok() && _offset < _reader->size()) {
@@ -506,8 +511,13 @@ Status PrefetchBuffer::read_buffer(size_t off, const char*
out, size_t buf_len,
}
// [0]: maximum len trying to read, [1] maximum length buffer can provide,
[2] actual len buffer has
size_t read_len = std::min({buf_len, _offset + _size - off, _offset + _len
- off});
-memcpy((void*)out, _buf.get() + (off - _offset), read_len);
+{
+SCOPED_RAW_TIMER(&_statis.copy_time);
+memcpy((void*)out, _buf.get() + (off - _offset), read_len);
+}
*bytes_read = read_len;
+_statis.request_io += 1;
+_statis.request_bytes += read_len;
if (off + *bytes_read == _offset + _len) {
reset_offset(_offset + _whole_buffer_size);
}
@@ -520,11 +530,15 @@ void PrefetchBuffer::close() {
_prefetched.wait(lck, [this]() { return _buffer_status !=
BufferStatus::PENDING; });
_buffer_status = BufferStatus::CLOSED;
_prefetched.notify_all();
+if (_sync_profile != nullptr) {
+_sync_profile(*this);
+}
}
// buffered reader
-PrefetchBufferedReader::PrefetchBufferedReader(io::FileReaderSPtr reader,
PrefetchRange file_range,
- const IOContext* io_ctx,
int64_t buffer_size)
+PrefetchBufferedReader::PrefetchBufferedReader(RuntimeProfile* profile,
io::FileReaderSPtr reader,
+ PrefetchRange file_range, const
IOContext* io_ctx,
+ int64_t buffer_size)
: _reader(std::move(reader)), _file_range(file_range), _io_ctx(io_ctx)
{
if (buffer_size == -1L) {
buffer_size = config::remote_storage_read_buffer_mb * 1024 * 1024;
@@ -533,12 +547,35 @@
PrefetchBufferedReader::PrefetchBufferedReader(io::FileReaderSPtr reader, Prefet
_whole_pre_buffer_size = buffer_size;
_file_range.end_offset = std::min(_file_range.end_offset, _size);
int buffer_num = buffer_size > s_max_pre_buffer_size ? buffer_size /
s_max_pre_buffer_size : 1;
+std::function sync_buffer = nullptr;
+if (profile != nullptr) {
+const char* prefetch_buffered_reader = "PrefetchBufferedReader";
+ADD_TIMER(profile, prefetch_buffered_reader);
+auto copy_time = ADD_CHILD_TIMER(profile, "CopyTime",
prefetch_buffered_reader);
+auto read_time = ADD_CHILD_TIMER(profile, "ReadTime",
prefetch_buffered_reader);
+auto prefetch_request_io =
+ADD_CHILD_COUNTER(profile, "PreRequestIO", TUnit::UNIT,
prefetch_buffered_reader);
+auto prefetch_request_bytes = ADD_CHILD_COUNTER(profile,
"PreRequestBytes", TUnit::BYTES,
+
prefetch_buffered_reader);
+auto request_io =
+ADD_CHILD_COUNTER(profile, "RequestIO", TUnit::UNIT,
prefetch_buffered_reader);
+auto request_bytes =
+ADD_CHILD_COUNTER(profile, "RequestBytes", TUnit::BYTES,
prefetch_buffered_reader);
+sync_buffer = [=](PrefetchBuffer& buf) {
+COUNTER_UPDATE(copy_time, buf._statis.copy_time);
+COUNTER_UPDATE(read_time, buf._statis.read_time);
+COUNTER_UPDATE(prefetch_request_io,
buf._statis.prefetch_request_io);
+COUNTER_UPDATE(prefetch_request_bytes,
buf._statis.pr
[GitHub] [doris] morningman merged pull request #19048: [docker](hudi) add hudi docker compose
morningman merged PR #19048: URL: https://github.com/apache/doris/pull/19048 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [docker](hudi) add hudi docker compose (#19048)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 224bca3794 [docker](hudi) add hudi docker compose (#19048) 224bca3794 is described below commit 224bca379470b3a396d40e680dd40436c9f6d2b0 Author: hechao <73096722+hechao-u...@users.noreply.github.com> AuthorDate: Tue May 2 09:54:52 2023 +0800 [docker](hudi) add hudi docker compose (#19048) --- docker/thirdparties/docker-compose/hudi/hadoop.env | 52 .../thirdparties/docker-compose/hudi/hudi.yaml.tpl | 267 + .../hudi/scripts/config/base.properties| 25 ++ .../hudi/scripts/config/dfs-source.properties | 31 +++ .../hudi/scripts/config/hoodie-incr.properties | 34 +++ .../hudi/scripts/config/hoodie-schema.avsc | 146 +++ .../hudi/scripts/config/kafka-source.properties| 30 +++ .../hudi/scripts/config/log4j2.properties | 61 + .../docker-compose/hudi/scripts/config/schema.avsc | 59 + .../hudi/scripts/config/spark-defaults.conf| 30 +++ .../docker-compose/hudi/scripts/run_sync_tool.sh | 56 + .../hudi/scripts/setup_demo_container_adhoc_1.sh | 31 +++ .../hudi/scripts/setup_demo_container_adhoc_2.sh | 77 ++ docker/thirdparties/run-thirdparties-docker.sh | 31 ++- .../developer-guide/regression-testing.md | 127 +++--- 15 files changed, 1021 insertions(+), 36 deletions(-) diff --git a/docker/thirdparties/docker-compose/hudi/hadoop.env b/docker/thirdparties/docker-compose/hudi/hadoop.env new file mode 100644 index 00..28ef46c3eb --- /dev/null +++ b/docker/thirdparties/docker-compose/hudi/hadoop.env @@ -0,0 +1,52 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +HIVE_SITE_CONF_javax_jdo_option_ConnectionURL=jdbc:postgresql://hive-metastore-postgresql/metastore +HIVE_SITE_CONF_javax_jdo_option_ConnectionDriverName=org.postgresql.Driver +HIVE_SITE_CONF_javax_jdo_option_ConnectionUserName=hive +HIVE_SITE_CONF_javax_jdo_option_ConnectionPassword=hive +HIVE_SITE_CONF_datanucleus_autoCreateSchema=false +HIVE_SITE_CONF_hive_metastore_uris=thrift://hivemetastore:9083 + +HDFS_CONF_dfs_namenode_datanode_registration_ip___hostname___check=false +HDFS_CONF_dfs_webhdfs_enabled=true +HDFS_CONF_dfs_permissions_enabled=false +#HDFS_CONF_dfs_client_use_datanode_hostname=true +#HDFS_CONF_dfs_namenode_use_datanode_hostname=true +HDFS_CONF_dfs_replication=1 + +CORE_CONF_fs_defaultFS=hdfs://namenode:8020 +CORE_CONF_hadoop_http_staticuser_user=root +CORE_CONF_hadoop_proxyuser_hue_hosts=* +CORE_CONF_hadoop_proxyuser_hue_groups=* + +YARN_CONF_yarn_log___aggregation___enable=true +YARN_CONF_yarn_resourcemanager_recovery_enabled=true +YARN_CONF_yarn_resourcemanager_store_class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore +YARN_CONF_yarn_resourcemanager_fs_state___store_uri=/rmstate +YARN_CONF_yarn_nodemanager_remote___app___log___dir=/app-logs +YARN_CONF_yarn_log_server_url=http://historyserver:8188/applicationhistory/logs/ +YARN_CONF_yarn_timeline___service_enabled=true +YARN_CONF_yarn_timeline___service_generic___application___history_enabled=true +YARN_CONF_yarn_resourcemanager_system___metrics___publisher_enabled=true +YARN_CONF_yarn_resourcemanager_hostname=resourcemanager +YARN_CONF_yarn_timeline___service_hostname=historyserver +YARN_CONF_yarn_resourcemanager_address=resourcemanager:8032 +YARN_CONF_yarn_resourcemanager_scheduler_address=resourcemanager:8030 +YARN_CONF_yarn_resourcemanager_resource___tracker_address=resourcemanager:8031 +YARN_CONF_yarn_nodemanager_vmem___check___enabled=false diff --git a/docker/thirdparties/docker-compose/hudi/hudi.yaml.tpl b/docker/thirdparties/docker-compose/hudi/hudi.yaml.tpl new file mode 100644 index 00..f0878e452b --- /dev/null +++ b/docker/thirdparties/docker-compose/hudi/hudi.yaml.tpl @@ -0,0 +1,267 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright own
[GitHub] [doris] morningman merged pull request #19026: [Fix](load) fix request_slave_tablet_pull_rowset get wrong url in case of ipv6 address
morningman merged PR #19026: URL: https://github.com/apache/doris/pull/19026 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed issue #19025: [Bug] request_slave_tablet_pull_rowset get wrong url in case of ipv6 address
morningman closed issue #19025: [Bug] request_slave_tablet_pull_rowset get wrong url in case of ipv6 address URL: https://github.com/apache/doris/issues/19025 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Fix](load) fix request_slave_tablet_pull_rowset get wrong url in case of ipv6 address (#19026)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 145b94531f [Fix](load) fix request_slave_tablet_pull_rowset get wrong
url in case of ipv6 address (#19026)
145b94531f is described below
commit 145b94531fb4f75015fe0ed300832a63d17d6a1a
Author: TsukiokaKogane
AuthorDate: Tue May 2 09:55:09 2023 +0800
[Fix](load) fix request_slave_tablet_pull_rowset get wrong url in case of
ipv6 address (#19026)
---
be/src/service/internal_service.cpp | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/be/src/service/internal_service.cpp
b/be/src/service/internal_service.cpp
index ab639e6f29..084edf7079 100644
--- a/be/src/service/internal_service.cpp
+++ b/be/src/service/internal_service.cpp
@@ -89,6 +89,7 @@
#include "util/doris_metrics.h"
#include "util/md5.h"
#include "util/metrics.h"
+#include "util/network_util.h"
#include "util/proto_util.h"
#include "util/ref_count_closure.h"
#include "util/runtime_profile.h"
@@ -1213,9 +1214,9 @@ void
PInternalServiceImpl::request_slave_tablet_pull_rowset(
}
std::stringstream ss;
-ss << "http://"; << host << ":" << http_port <<
"/api/_tablet/_download?token=" << token
- << "&file=" << rowset_path << "/" << remote_rowset_id << "_" <<
segment.first
- << ".dat";
+ss << "http://"; << get_host_port(host, http_port)
+ << "/api/_tablet/_download?token=" << token << "&file=" <<
rowset_path << "/"
+ << remote_rowset_id << "_" << segment.first << ".dat";
std::string remote_file_url = ss.str();
ss.str("");
ss << tablet->tablet_path() << "/" << rowset_meta->rowset_id() <<
"_" << segment.first
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #18421: [enhancement](load) merge single-replica related services as non-standalone
TangSiyang2001 commented on PR #18421: URL: https://github.com/apache/doris/pull/18421#issuecomment-1530770286 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18421: [enhancement](load) merge single-replica related services as non-standalone
github-actions[bot] commented on PR #18421: URL: https://github.com/apache/doris/pull/18421#issuecomment-1530772770 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HHoflittlefish777 commented on pull request #18874: [enhancement](storage) lazy-open necessary partitions when load
HHoflittlefish777 commented on PR #18874: URL: https://github.com/apache/doris/pull/18874#issuecomment-1530785350 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on pull request #19098: [refactor](fs)(step3)use filesystem instead of old storage, new storage just access remote object storage
wsjz commented on PR #19098: URL: https://github.com/apache/doris/pull/19098#issuecomment-1530878449 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on pull request #19098: [refactor](fs)(step3)use filesystem instead of old storage, new storage just access remote object storage
wsjz commented on PR #19098: URL: https://github.com/apache/doris/pull/19098#issuecomment-1530964943 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
