[GitHub] [incubator-doris] yiguolei commented on issue #8374: [Feature] support TiKV as an option for metadata storage
yiguolei commented on issue #8374: URL: https://github.com/apache/incubator-doris/issues/8374#issuecomment-1060291918 Currently, doris store rowset meta and tablet meta in local rocksdb. Could you explain more detail why need to store it to a external global KV store? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on issue #8374: [Feature] support TiKV as an option for metadata storage
wangbo commented on issue #8374: URL: https://github.com/apache/incubator-doris/issues/8374#issuecomment-1060298717 Doris's FE and BE both has metadata, so which part do you want to replace? Can you offer more detailed design? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on pull request #8292: (improving) refactor column::insert_many_dict_data
wangbo commented on pull request #8292: URL: https://github.com/apache/incubator-doris/pull/8292#issuecomment-1060312019 We did a performance test for this pr in SSB, no significant improvement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] dataroaring opened a new pull request #8375: add an aggregation test from trino.
dataroaring opened a new pull request #8375: URL: https://github.com/apache/incubator-doris/pull/8375 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on pull request #8358: [Doc][Vectorized] Add vectorized execution engine docs
yangzhg commented on pull request #8358: URL: https://github.com/apache/incubator-doris/pull/8358#issuecomment-1060327441 It is better to redraw those two images, the source of those two images is unknown -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8363: [fix](broker-load) fix bug that a cancelled job's state is LOADING
github-actions[bot] commented on pull request #8363: URL: https://github.com/apache/incubator-doris/pull/8363#issuecomment-1060329065 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new issue #8376: [Bug] fe start fail
BiteThet opened a new issue #8376: URL: https://github.com/apache/incubator-doris/issues/8376 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? java.io.IOException: failed to read session variable: Can not set int field org.apache.doris.qe.SessionVariable.queryTimeoutS to java.lang.Long at org.apache.doris.qe.SessionVariable.readFromJson(SessionVariable.java:1023) at org.apache.doris.qe.SessionVariable.readFields(SessionVariable.java:981) at org.apache.doris.qe.VariableMgr.read(VariableMgr.java:312) at org.apache.doris.catalog.Catalog.loadGlobalVariable(Catalog.java:1902) at org.apache.doris.common.MetaReader.read(MetaReader.java:89) at org.apache.doris.catalog.Catalog.loadImage(Catalog.java:1618) at org.apache.doris.catalog.Catalog.initialize(Catalog.java:848) at org.apache.doris.PaloFe.start(PaloFe.java:125) at org.apache.doris.PaloFe.main(PaloFe.java:63) ### What You Expected? fix it ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new pull request #8377: [Bug] fe start fail
BiteThet opened a new pull request #8377: URL: https://github.com/apache/incubator-doris/pull/8377 # Proposed changes Issue Number: close #8376 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz closed pull request #7911: [Help] gcc11 compiles `thread_local` variable, BE start: version `GLIBC_2.18' not found
xinyiZzz closed pull request #7911: URL: https://github.com/apache/incubator-doris/pull/7911 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morrySnow opened a new issue #8378: [Enhancement] [doc] there are some dead link in fe doc pages
morrySnow opened a new issue #8378: URL: https://github.com/apache/incubator-doris/issues/8378 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description when i read doris developer docs, i found some dead link: 1. link for [problems with mounting] in https://doris.incubator.apache.org/developer-guide/docker-dev.html 2. link for [挂载的问题] in https://doris.incubator.apache.org/zh-CN/developer-guide/docker-dev.html i don't know which is the correct link. so just raise an issue here. if anybody know the correct link. i'd like to commit a pr to fix it. ### Solution _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a change in pull request #8322: [refactor] Impl of MemTracker, and related use
xinyiZzz commented on a change in pull request #8322: URL: https://github.com/apache/incubator-doris/pull/8322#discussion_r820504386 ## File path: be/src/exec/es/es_scroll_parser.cpp ## @@ -410,11 +411,12 @@ Status ScrollParser::fill_tuple(const TupleDescriptor* tuple_desc, Tuple* tuple, } } size_t val_size = val.length(); -char* buffer = reinterpret_cast(tuple_pool->try_allocate_unaligned(val_size)); +Status rst; +char* buffer = reinterpret_cast(tuple_pool->try_allocate_unaligned(val_size, &rst)); Review comment: There are so many places to call MemPool allocate, "return Status and use buffer as parameter" will cause a lot of changes, so your suggestion is whether to change it. The purpose of getting the status here is to print out the real reason of allocate failure externally, such as the details of tracker exceed, not just the location of the failure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morrySnow opened a new pull request #8379: Fix dead link in build docker environment in developer guide
morrySnow opened a new pull request #8379: URL: https://github.com/apache/incubator-doris/pull/8379 # Proposed changes Issue Number: close #8378 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (Yes) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8379: Fix dead link in build docker environment in developer guide
github-actions[bot] commented on pull request #8379: URL: https://github.com/apache/incubator-doris/pull/8379#issuecomment-1060399237 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on pull request #8358: [Doc][Vectorized] Add vectorized execution engine docs
HappenLee commented on pull request #8358: URL: https://github.com/apache/incubator-doris/pull/8358#issuecomment-1060407124 > It is better to redraw those two images, the source of those two images is unknown had set quoted from [Column-Oriented Database Systems](https://web.stanford.edu/class/cs346/2015/notes/old/column.pdf -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8379: Fix dead link in build docker environment in developer guide
github-actions[bot] commented on pull request #8379: URL: https://github.com/apache/incubator-doris/pull/8379#issuecomment-1060413099 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #8373: [Refactor] to_bitmap(-1) return NULL instead of return parse failed error_message
HappenLee commented on a change in pull request #8373:
URL: https://github.com/apache/incubator-doris/pull/8373#discussion_r820543213
##
File path: be/src/exprs/bitmap_function.cpp
##
@@ -358,21 +358,17 @@ BigIntVal BitmapFunctions::bitmap_min(FunctionContext*
ctx, const StringVal& src
StringVal BitmapFunctions::to_bitmap(doris_udf::FunctionContext* ctx,
const doris_udf::StringVal& src) {
+if(src.is_null) {
Review comment:
`if (`
##
File path: be/src/vec/functions/function_bitmap.cpp
##
@@ -125,23 +151,48 @@ struct BitmapHash {
}
};
-struct NameBitmapCount {
+class FunctionBitmapCount : public IFunction {
Review comment:
why bitmap count is always nullable?
##
File path: be/src/vec/functions/function_bitmap.cpp
##
@@ -125,23 +151,48 @@ struct BitmapHash {
}
};
-struct NameBitmapCount {
+class FunctionBitmapCount : public IFunction {
+public:
static constexpr auto name = "bitmap_count";
-};
-struct BitmapCount {
-using ReturnType = DataTypeInt64;
-static constexpr auto TYPE_INDEX = TypeIndex::BitMap;
-using Type = DataTypeBitMap::FieldType;
-using ReturnColumnType = ColumnVector;
-using ReturnColumnContainer = ColumnVector::Container;
+String get_name() const override { return name; }
-static Status vector(const std::vector& data,
ReturnColumnContainer& res) {
-size_t size = data.size();
-res.reserve(size);
-for (size_t i = 0; i < size; ++i) {
-res.push_back(data[i].cardinality());
+static FunctionPtr create() { return
std::make_shared(); }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+return std::make_shared();
+}
+
+size_t get_number_of_arguments() const override { return 1; }
+
+bool use_default_implementation_for_nulls() const override { return false;
}
+
+bool use_default_implementation_for_constants() const override { return
true; }
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) override {
+auto res_data_column = ColumnInt64::create();
+auto& res = res_data_column->get_data();
+auto data_null_map = ColumnUInt8::create(input_rows_count, 0);
+auto& null_map = data_null_map->get_data();
+
+auto column =
block.get_by_position(arguments[0]).column->convert_to_full_column_if_const();
+if (auto* nullable = check_and_get_column(*column)) {
+VectorizedUtils::update_null_map(null_map,
nullable->get_null_map_data());
+column = nullable->get_nested_column_ptr();
+}
+auto str_col = assert_cast(column.get());
+const auto& col_data = str_col->get_data();
+
+res.reserve(input_rows_count);
+for (size_t i = 0; i < input_rows_count; ++i) {
+if(null_map[i]) {
Review comment:
if (
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8377: [Bug] fe start fail
morningman commented on pull request #8377: URL: https://github.com/apache/incubator-doris/pull/8377#issuecomment-1060418509 Hi @BiteThet , this bug has been fixed in #8355 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt commented on pull request #8377: [Bug] fe start fail
BiteThet commented on pull request #8377: URL: https://github.com/apache/incubator-doris/pull/8377#issuecomment-1060506691 > Hi @BiteThet , this bug has been fixed in #8355 ok -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt closed pull request #8377: [Bug] fe start fail
BiteThet closed pull request #8377: URL: https://github.com/apache/incubator-doris/pull/8377 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt closed issue #8376: [Bug] fe start fail
BiteThet closed issue #8376: URL: https://github.com/apache/incubator-doris/issues/8376 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new issue #8380: [Bug] fix core dump on minmax_filter with decimal type
BiteThet opened a new issue #8380: URL: https://github.com/apache/incubator-doris/issues/8380 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? ```cpp start time: 2022年 03月 07日 星期一 16:45:23 CST WARNING: Logging before InitGoogleLogging() is written to STDERR I0307 16:45:23.564831 13352 env.cpp:46] Env init successfully. Minidump created at: /home/disk2/pxl/dev/baidu/bdg/doris/core/output/be/minidump/fdbac494-2ca6-4a14-92c2958e-a6d9a153.dmp *** Aborted at 1646642730 (unix time) try "date -d @1646642730" if you are using GNU date *** PC: @ 0x564a1bca5d41 doris::MinMaxNumFunc<>::insert() *** SIGSEGV (@0x0) received by PID 13352 (TID 0x7f5495f6c700) from PID 0; stack trace: *** @ 0x564a1dad7042 google::(anonymous namespace)::FailureSignalHandler() @ 0x7f54f35ff920 (unknown) @ 0x564a1bca5d41 doris::MinMaxNumFunc<>::insert() @ 0x564a1bcc324d doris::HashJoinNode::open() @ 0x564a1bec3484 doris::PlanFragmentExecutor::open_internal() @ 0x564a1bec4745 doris::PlanFragmentExecutor::open() @ 0x564a1bea7265 doris::FragmentExecState::execute() @ 0x564a1beabada doris::FragmentMgr::_exec_actual() @ 0x564a1beb724c std::_Function_handler<>::_M_invoke() @ 0x564a1c141299 doris::ThreadPool::dispatch_thread() @ 0x564a1c13667f doris::Thread::supervise_thread() @ 0x7f54f33b7851 start_thread @ 0x7f54f36b467d clone @0x0 (unknown) ``` ### What You Expected? fix it ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new pull request #8381: [Bug] fix core dump on minmax_filter with decimal type
BiteThet opened a new pull request #8381: URL: https://github.com/apache/incubator-doris/pull/8381 # Proposed changes Issue Number: close #8380 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt commented on issue #8317: [Bug] core dump on BrokerScanNode with clang
BiteThet commented on issue #8317: URL: https://github.com/apache/incubator-doris/issues/8317#issuecomment-1060523949 > Does it show the core dump reason, like segment fault? yes, the reason is segmentation fault. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a change in pull request #8322: [refactor] Impl of MemTracker, and related use
xinyiZzz commented on a change in pull request #8322:
URL: https://github.com/apache/incubator-doris/pull/8322#discussion_r820598899
##
File path: be/src/exprs/expr_context.cpp
##
@@ -371,10 +376,11 @@ Status ExprContext::get_const_value(RuntimeState* state,
Expr& expr, AnyVal** co
StringVal* sv = reinterpret_cast(*const_val);
if (!sv->is_null && sv->len > 0) {
// Make sure the memory is owned by this evaluator.
-char* ptr_copy =
reinterpret_cast(_pool->try_allocate(sv->len));
+Status rst;
+char* ptr_copy =
reinterpret_cast(_pool->try_allocate(sv->len, &rst));
if (ptr_copy == nullptr) {
-return _pool->mem_tracker()->MemLimitExceeded(
-state, "Could not allocate constant string value",
sv->len);
+RETURN_ALLOC_LIMIT_EXCEEDED(_pool->mem_tracker(), state,
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] dataroaring opened a new issue #8382: [Bug] variance is different with trino
dataroaring opened a new issue #8382: URL: https://github.com/apache/incubator-doris/issues/8382 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? variance's result is different with trino. https://github.com/trinodb/trino/blob/master/testing/trino-product-tests/src/main/resources/sql-tests/testcases/aggregate/variance.result ### What You Expected? may be a bug, and i am not sure. ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt closed issue #8262: [Bug][Vectorized] Be core dump on memtable
BiteThet closed issue #8262: URL: https://github.com/apache/incubator-doris/issues/8262 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] coderjiang commented on issue #5103: doris怎么删除昨天的数据
coderjiang commented on issue #5103:
URL:
https://github.com/apache/incubator-doris/issues/5103#issuecomment-1060563392
DELETE FROM ods_order_report WHERE purchase_date_local >=
DATE_ADD('2022-03-06 00:00:00', INTERVAL 1 day);
这语句也是不行的，会报错：
ERROR 1064 (HY000): errCode = 2, detailMessage = Right expr of binary
predicate should be value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] coderjiang commented on issue #5103: doris怎么删除昨天的数据
coderjiang commented on issue #5103:
URL:
https://github.com/apache/incubator-doris/issues/5103#issuecomment-1060564080
mysql> DELETE FROM ods_order_report WHERE purchase_date_local >=
DATE_ADD('2022-03-06 00:00:00', INTERVAL 1 day);
ERROR 1064 (HY000): errCode = 2, detailMessage = Right expr of binary
predicate should be value
mysql> select DATE_ADD('2022-03-06 00:00:00', INTERVAL 1 day);
+-+
| date_add('2022-03-06 00:00:00', INTERVAL 1 day) |
+-+
| 2022-03-07 00:00:00 |
+-+
1 row in set (0.00 sec)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #8364: [Function] Add compare_version function and ut
yangzhg commented on a change in pull request #8364:
URL: https://github.com/apache/incubator-doris/pull/8364#discussion_r820612758
##
File path: be/src/exprs/utility_functions.cpp
##
@@ -39,4 +39,82 @@ BooleanVal UtilityFunctions::sleep(FunctionContext* ctx,
const IntVal& seconds)
return BooleanVal(true);
}
+static void convert_to_string(doris_udf::StringVal& src, std::string& str_dst)
{
Review comment:
why called `convert_to_string` ？ just a concat ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #8364: [Function] Add compare_version function and ut
yangzhg commented on a change in pull request #8364:
URL: https://github.com/apache/incubator-doris/pull/8364#discussion_r820614209
##
File path: be/src/exprs/utility_functions.cpp
##
@@ -39,4 +39,82 @@ BooleanVal UtilityFunctions::sleep(FunctionContext* ctx,
const IntVal& seconds)
return BooleanVal(true);
}
+static void convert_to_string(doris_udf::StringVal& src, std::string& str_dst)
{
+for (int i = 0; i < src.len; i++) {
+str_dst += src.ptr[i];
+}
+return;
+}
+
+static BooleanVal split_string(std::string& str_src, std::vector&
str_vec_dst) {
Review comment:
there is already have split in the util, refer to
be/src/util/string_util.h or gutils
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new issue #8383: [Feature][Vectorized] support replace()
BiteThet opened a new issue #8383:
URL: https://github.com/apache/incubator-doris/issues/8383
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Description
support replace()
### Use case
```sql
MySQL [test]> select replace("http://www.baidu.com:9090";, "9090", "");
+--+
| replace('http://www.baidu.com:9090', '9090', '') |
+--+
| http://www.baidu.com:|
+--+
1 row in set (0.010 sec)
```
### Related issues
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new pull request #8384: [Feature][Vectorized] support replace()
BiteThet opened a new pull request #8384: URL: https://github.com/apache/incubator-doris/pull/8384 # Proposed changes @HappenLee Issue Number: close #8383 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a change in pull request #8322: [refactor] Impl of MemTracker, and related use
xinyiZzz commented on a change in pull request #8322:
URL: https://github.com/apache/incubator-doris/pull/8322#discussion_r820638296
##
File path: be/src/olap/task/engine_alter_tablet_task.cpp
##
@@ -18,13 +18,21 @@
#include "olap/task/engine_alter_tablet_task.h"
#include "olap/schema_change.h"
+#include "runtime/mem_tracker.h"
namespace doris {
using std::to_string;
EngineAlterTabletTask::EngineAlterTabletTask(const TAlterTabletReqV2& request)
-: _alter_tablet_req(request) {}
+: _alter_tablet_req(request) {
+_mem_tracker = MemTracker::create_tracker(
Review comment:
done
##
File path: be/src/olap/task/engine_batch_load_task.h
##
@@ -76,6 +76,8 @@ class EngineBatchLoadTask : public EngineTask {
AgentStatus* _res_status;
std::string _remote_file_path;
std::string _local_file_path;
+
+std::shared_ptr _mem_tracker;
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #8381: [Bug] fix core dump on minmax_filter with decimal type
HappenLee commented on a change in pull request #8381:
URL: https://github.com/apache/incubator-doris/pull/8381#discussion_r820653466
##
File path: be/src/exprs/minmax_predicate.h
##
@@ -42,9 +42,14 @@ class MinMaxNumFunc : public MinMaxFuncBase {
public:
MinMaxNumFunc() = default;
~MinMaxNumFunc() = default;
-virtual void insert(const void* data) {
-if (data == nullptr) return;
-const T val_data = *reinterpret_cast(data);
+void insert(const void* data) override {
+if (data == nullptr) {
+return;
+}
+
Review comment:
only core when int128_t, so maybe constexpr + sizeof.
and add comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee opened a new issue #8385: [Bug][Vectorized] Datetime type read error result
HappenLee opened a new issue #8385: URL: https://github.com/apache/incubator-doris/issues/8385 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version vectorized ### What's Wrong? get 2001 when year is ### What You Expected? get right data ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee opened a new pull request #8386: [Bug][Vectorized] Fix the datetime type read error and is_same set error in reader
HappenLee opened a new pull request #8386: URL: https://github.com/apache/incubator-doris/pull/8386 # Proposed changes Issue Number: close #8385 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] cambyzju commented on a change in pull request #8217: [feature][array-type]support select ARRAY data type on vectorized engine
cambyzju commented on a change in pull request #8217:
URL: https://github.com/apache/incubator-doris/pull/8217#discussion_r820707304
##
File path: be/src/vec/columns/column_array.cpp
##
@@ -0,0 +1,700 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+//
https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnArray.cpp
+// and modified by Doris
+
+#include // memcpy
+
+#include "vec/common/assert_cast.h"
+#include "vec/columns/collator.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/columns_common.h"
+#include "vec/columns/columns_number.h"
+
+namespace doris::vectorized {
+
+namespace ErrorCodes {
+extern const int NOT_IMPLEMENTED;
+extern const int BAD_ARGUMENTS;
+extern const int PARAMETER_OUT_OF_BOUND;
+extern const int SIZES_OF_COLUMNS_DOESNT_MATCH;
+extern const int LOGICAL_ERROR;
+extern const int TOO_LARGE_ARRAY_SIZE;
+}
+
+/** Obtaining array as Field can be slow for large arrays and consume vast
amount of memory.
+ * Just don't allow to do it.
+ * You can increase the limit if the following query:
+ * SELECT range(1000)
+ * will take less than 500ms on your machine.
+ */
+static constexpr size_t max_array_size_as_field = 100;
+
+ColumnArray::ColumnArray(MutableColumnPtr && nested_column, MutableColumnPtr
&& offsets_column)
+: data(std::move(nested_column)), offsets(std::move(offsets_column)) {
+const ColumnOffsets * offsets_concrete = typeid_cast(offsets.get());
+
+if (!offsets_concrete) {
+LOG(FATAL) << "offsets_column must be a ColumnUInt64";
+}
+
+if (!offsets_concrete->empty() && nested_column) {
+Offset last_offset = offsets_concrete->get_data().back();
+
+/// This will also prevent possible overflow in offset.
+if (nested_column->size() != last_offset) {
+LOG(FATAL) << "offsets_column has data inconsistent with
nested_column";
+}
+}
+
+/** NOTE
+ * Arrays with constant value are possible and used in implementation of
higher order functions (see FunctionReplicate).
+ * But in most cases, arrays with constant value are unexpected and code
will work wrong. Use with caution.
+ */
+}
+
+ColumnArray::ColumnArray(MutableColumnPtr && nested_column)
+: data(std::move(nested_column)) {
+if (!data->empty()) {
+LOG(FATAL) << "Not empty data passed to ColumnArray, but no offsets
passed";
+}
+
+offsets = ColumnOffsets::create();
+}
+
+std::string ColumnArray::get_name() const { return "Array(" +
get_data().get_name() + ")"; }
+
+MutableColumnPtr ColumnArray::clone_resized(size_t to_size) const {
+auto res = ColumnArray::create(get_data().clone_empty());
+
+if (to_size == 0)
+return res;
+size_t from_size = size();
+
+if (to_size <= from_size) {
+/// Just cut column.
+res->get_offsets().assign(get_offsets().begin(), get_offsets().begin()
+ to_size);
+res->get_data().insert_range_from(get_data(), 0, get_offsets()[to_size
- 1]);
+} else {
+/// Copy column and append empty arrays for extra elements.
+Offset offset = 0;
+if (from_size > 0) {
+res->get_offsets().assign(get_offsets().begin(),
get_offsets().end());
+res->get_data().insert_range_from(get_data(), 0,
get_data().size());
+offset = get_offsets().back();
+}
+
+res->get_offsets().resize(to_size);
+for (size_t i = from_size; i < to_size; ++i)
+res->get_offsets()[i] = offset;
+}
+
+return res;
+}
+
+size_t ColumnArray::size() const {
+return get_offsets().size();
+}
+
+Field ColumnArray::operator[](size_t n) const {
+size_t offset = offset_at(n);
+size_t size = size_at(n);
+
+if (size > max_array_size_as_field)
+LOG(FATAL) << "Array of size " << size << " is too large to be
manipulated as single field,"
+ << "maximum size " << max_array_size_as_field;
+
+Array res(size);
+
+for (size_t i = 0; i < size; ++i)
+res[i] = get_d
[GitHub] [incubator-doris] cambyzju commented on a change in pull request #8217: [feature][array-type]support select ARRAY data type on vectorized engine
cambyzju commented on a change in pull request #8217:
URL: https://github.com/apache/incubator-doris/pull/8217#discussion_r820727165
##
File path: be/src/vec/columns/column_array.cpp
##
@@ -0,0 +1,700 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+//
https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnArray.cpp
+// and modified by Doris
+
+#include // memcpy
+
+#include "vec/common/assert_cast.h"
+#include "vec/columns/collator.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/columns_common.h"
+#include "vec/columns/columns_number.h"
+
+namespace doris::vectorized {
+
+namespace ErrorCodes {
+extern const int NOT_IMPLEMENTED;
+extern const int BAD_ARGUMENTS;
+extern const int PARAMETER_OUT_OF_BOUND;
+extern const int SIZES_OF_COLUMNS_DOESNT_MATCH;
+extern const int LOGICAL_ERROR;
+extern const int TOO_LARGE_ARRAY_SIZE;
+}
+
+/** Obtaining array as Field can be slow for large arrays and consume vast
amount of memory.
+ * Just don't allow to do it.
+ * You can increase the limit if the following query:
+ * SELECT range(1000)
+ * will take less than 500ms on your machine.
+ */
+static constexpr size_t max_array_size_as_field = 100;
+
+ColumnArray::ColumnArray(MutableColumnPtr && nested_column, MutableColumnPtr
&& offsets_column)
+: data(std::move(nested_column)), offsets(std::move(offsets_column)) {
+const ColumnOffsets * offsets_concrete = typeid_cast(offsets.get());
+
+if (!offsets_concrete) {
+LOG(FATAL) << "offsets_column must be a ColumnUInt64";
+}
+
+if (!offsets_concrete->empty() && nested_column) {
+Offset last_offset = offsets_concrete->get_data().back();
+
+/// This will also prevent possible overflow in offset.
+if (nested_column->size() != last_offset) {
+LOG(FATAL) << "offsets_column has data inconsistent with
nested_column";
+}
+}
+
+/** NOTE
+ * Arrays with constant value are possible and used in implementation of
higher order functions (see FunctionReplicate).
+ * But in most cases, arrays with constant value are unexpected and code
will work wrong. Use with caution.
+ */
+}
+
+ColumnArray::ColumnArray(MutableColumnPtr && nested_column)
+: data(std::move(nested_column)) {
+if (!data->empty()) {
+LOG(FATAL) << "Not empty data passed to ColumnArray, but no offsets
passed";
+}
+
+offsets = ColumnOffsets::create();
+}
+
+std::string ColumnArray::get_name() const { return "Array(" +
get_data().get_name() + ")"; }
+
+MutableColumnPtr ColumnArray::clone_resized(size_t to_size) const {
+auto res = ColumnArray::create(get_data().clone_empty());
+
+if (to_size == 0)
+return res;
+size_t from_size = size();
+
+if (to_size <= from_size) {
+/// Just cut column.
+res->get_offsets().assign(get_offsets().begin(), get_offsets().begin()
+ to_size);
+res->get_data().insert_range_from(get_data(), 0, get_offsets()[to_size
- 1]);
+} else {
+/// Copy column and append empty arrays for extra elements.
+Offset offset = 0;
+if (from_size > 0) {
+res->get_offsets().assign(get_offsets().begin(),
get_offsets().end());
+res->get_data().insert_range_from(get_data(), 0,
get_data().size());
+offset = get_offsets().back();
+}
+
+res->get_offsets().resize(to_size);
+for (size_t i = from_size; i < to_size; ++i)
+res->get_offsets()[i] = offset;
+}
+
+return res;
+}
+
+size_t ColumnArray::size() const {
+return get_offsets().size();
+}
+
+Field ColumnArray::operator[](size_t n) const {
+size_t offset = offset_at(n);
+size_t size = size_at(n);
+
+if (size > max_array_size_as_field)
+LOG(FATAL) << "Array of size " << size << " is too large to be
manipulated as single field,"
+ << "maximum size " << max_array_size_as_field;
+
+Array res(size);
+
+for (size_t i = 0; i < size; ++i)
+res[i] = get_d
[GitHub] [incubator-doris] caiconghui opened a new pull request #8387: [Refactor](storage_engine) Use std::shared_lock to replace RWMutex
caiconghui opened a new pull request #8387: URL: https://github.com/apache/incubator-doris/pull/8387 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (No) 3. Has document been added or modified: (No) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8387: [Refactor](storage_engine) Use std::shared_lock to replace RWMutex
morningman commented on pull request #8387: URL: https://github.com/apache/incubator-doris/pull/8387#issuecomment-1060749080 Please describe the motivation and any possible performance impact, so that other reviewers can directly know what is going on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8336: [improvment](broker load) fix broker load check logic
morningman commented on pull request #8336: URL: https://github.com/apache/incubator-doris/pull/8336#issuecomment-1060793086 Seems like the code you changed in this PR does not related the problem you described in #8335 ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8340: Translate Chinese comment to English
morningman commented on pull request #8340: URL: https://github.com/apache/incubator-doris/pull/8340#issuecomment-1060795516 > You'd better cleanup your commits by removing useless commits like '第一次提交'. You can use git force push. That would be OK, because we using "Squash" merge. So the commit message will be reorganized finally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8340: Translate Chinese comment to English
morningman commented on pull request #8340: URL: https://github.com/apache/incubator-doris/pull/8340#issuecomment-1060796247 > There are a lot of Chinese comments, I prefer to clean together. I think we can simply merge this PR. And doing what you prefer later? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8275: [community] Modify doris connector release doc
morningman merged pull request #8275: URL: https://github.com/apache/incubator-doris/pull/8275 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8340: Translate Chinese comment to English
github-actions[bot] commented on pull request #8340: URL: https://github.com/apache/incubator-doris/pull/8340#issuecomment-1060797540 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (d9c2c2c -> c647def)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from d9c2c2c Revert "[refactor] remove unused new_in_predicate code (#8263)" (#8372) add c647def [community] Modify doris connector release doc (#8275) No new revisions were added by this update. Summary of changes: .../release-and-verify/release-doris-connectors.md | 82 ++ .../release-and-verify/release-doris-connectors.md | 78 +--- 2 files changed, 74 insertions(+), 86 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8324: [Feature](proc) Support proc dir for showing tablet health status
morningman commented on pull request #8324: URL: https://github.com/apache/incubator-doris/pull/8324#issuecomment-1060799805 Why don't just add new columns to `show proc "/statistic"`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8325: [typo]fix some typo in fe_config
morningman merged pull request #8325: URL: https://github.com/apache/incubator-doris/pull/8325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8350: [license] Organize third-party dependent licenses for bianry releases
morningman merged pull request #8350: URL: https://github.com/apache/incubator-doris/pull/8350 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8352: [improvement]support build with parallel parameter only
morningman merged pull request #8352: URL: https://github.com/apache/incubator-doris/pull/8352 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (c647def -> ca3be24)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from c647def [community] Modify doris connector release doc (#8275) add ca3be24 [typo]fix some typo in fe_config (#8325) No new revisions were added by this update. Summary of changes: docs/en/administrator-guide/config/fe_config.md| 45 - docs/zh-CN/administrator-guide/config/fe_config.md | 112 ++--- 2 files changed, 72 insertions(+), 85 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (ca3be24 -> 50a59f3)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from ca3be24 [typo]fix some typo in fe_config (#8325) add 50a59f3 [license] Organize third-party dependent licenses for bianry releases (#8350) No new revisions were added by this update. Summary of changes: NOTICE.txt |2 +- dist/LICENSE-dist.txt | 1561 dist/README| 30 + dist/licenses/LICENSE-CC0.txt | 29 + dist/licenses/LICENSE-CDDL-1.0.txt | 131 +++ dist/licenses/LICENSE-CDDL-1.1.txt | 77 ++ dist/licenses/LICENSE-CUP.txt |5 + dist/licenses/LICENSE-EDL-1.0.txt | 11 + dist/licenses/LICENSE-EPL-1.0.txt | 60 ++ dist/licenses/LICENSE-EPL-2.0.txt | 79 ++ dist/licenses/LICENSE-GPLv2-CE.txt | 131 +++ dist/licenses/LICENSE-JSch.txt | 30 + dist/licenses/LICENSE-MIT.txt |9 + dist/licenses/LICENSE-antlr4.txt | 52 + dist/licenses/LICENSE-asm.txt | 27 + dist/licenses/LICENSE-automaton.txt| 24 + dist/licenses/LICENSE-bitshuffle.txt | 21 + dist/licenses/LICENSE-boost.txt| 23 + dist/licenses/LICENSE-breakpad.txt | 132 +++ dist/licenses/LICENSE-brotli.txt | 19 + dist/licenses/LICENSE-bzip2.txt| 42 + dist/licenses/LICENSE-commons-compiler.txt | 31 + dist/licenses/LICENSE-curl.txt | 22 + dist/licenses/LICENSE-cyrus-sasl.txt | 44 + dist/licenses/LICENSE-fmt.txt | 11 + dist/licenses/LICENSE-gflag.txt| 28 + dist/licenses/LICENSE-glog.txt | 65 ++ dist/licenses/LICENSE-gperftools.txt | 28 + dist/licenses/LICENSE-gtest.txt| 28 + dist/licenses/LICENSE-hamcrest.txt | 27 + dist/licenses/LICENSE-javolution.txt | 27 + dist/licenses/LICENSE-jemalloc.txt | 26 + dist/licenses/LICENSE-jline.txt| 34 + dist/licenses/LICENSE-jta.txt | 245 + dist/licenses/LICENSE-kryo.txt | 10 + dist/licenses/LICENSE-leveldb.txt | 27 + dist/licenses/LICENSE-leveldbjni.txt | 27 + dist/licenses/LICENSE-libdivide.txt| 29 + dist/licenses/LICENSE-libevent.txt | 99 ++ dist/licenses/LICENSE-librdkafka.txt | 31 + dist/licenses/LICENSE-lz4.txt | 11 + dist/licenses/LICENSE-lzma.txt | 64 ++ dist/licenses/LICENSE-minlog.txt | 10 + dist/licenses/LICENSE-openssl.txt | 125 +++ dist/licenses/LICENSE-paranamer.txt| 28 + dist/licenses/LICENSE-pdqsort.txt | 16 + dist/licenses/LICENSE-protobuf.txt | 33 + dist/licenses/LICENSE-py4j.txt | 26 + dist/licenses/LICENSE-rapidjson.txt| 57 + dist/licenses/LICENSE-re2.txt | 27 + dist/licenses/LICENSE-snappy.txt | 54 + dist/licenses/LICENSE-unixodbc.txt | 511 + dist/licenses/LICENSE-xmlenc.txt | 23 + dist/licenses/LICENSE-xsimd.txt| 28 + dist/licenses/LICENSE-zlib.txt | 31 + dist/licenses/LICENSE-zstd-jni.txt | 26 + dist/licenses/LICENSE-zstd.txt | 30 + dist/tools/README | 45 + dist/tools/parse_licenses.py | 180 fe/fe-core/pom.xml | 110 +- fe/pom.xml | 217 ++-- fe/spark-dpp/pom.xml | 109 -- fs_brokers/apache_hdfs_broker/pom.xml | 13 + thirdparty/vars.sh | 10 +- 64 files changed, 4820 insertions(+), 298 deletions(-) create mode 100644 dist/LICENSE-dist.txt create mode 100644 dist/README create mode 100644 dist/licenses/LICENSE-CC0.txt create mode 100644 dist/licenses/LICENSE-CDDL-1.0.txt create mode 100644 dist/licenses/LICENSE-CDDL-1.1.txt create mode 100644 dist/licenses/LICENSE-CUP.txt create mode 100644 dist/licenses/LICENSE-EDL-1.0.txt create mode 100644 dist/licenses/LICENSE-EPL-1.0.txt create mode 100644 dist/licenses/LICENSE-EPL-2.0.txt create mode 100644 dist/licenses/LICENSE-GPLv2-CE.txt create mode 100644 dist/licenses/LICENSE-JSch.txt create mode 100644 dist/licenses/LICENSE-MIT.txt create mode 100644 dist/licenses/LICENSE-antlr4.txt create mode 100644 dist/licenses/LICENSE-asm.txt create mode 100644 dist/licenses/LICENSE-automaton.txt create mode 100644 dist/licenses/LICENSE-bitshuffle.txt create mode 100644 dist/licenses/LICENSE-boost.txt create mode 100644 dist/licenses/LICENSE-breakpad.txt create mode 100644 dist/licenses/LICENSE-brotli.txt create mode 100644 dist/licenses/LICENSE-bzip2.txt create mode 100644 dist/licenses/LICENSE-commons-
[incubator-doris] branch master updated (50a59f3 -> 089ef53)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 50a59f3 [license] Organize third-party dependent licenses for bianry releases (#8350) add 089ef53 [chore] fix build with parallel parameter only (#8352) No new revisions were added by this update. Summary of changes: build.sh | 13 - 1 file changed, 12 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #7529: Support remote storage, step2, only for be: hot data trans to cold data. clean cold data when drop table
morningman commented on pull request #7529: URL: https://github.com/apache/incubator-doris/pull/7529#issuecomment-1060820068 Please update the PR comment to describe the new implementation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8096: [optimize] optimze tablet read, avoid to create too much scanner for small tablet
github-actions[bot] commented on pull request #8096: URL: https://github.com/apache/incubator-doris/pull/8096#issuecomment-1060826763 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8305: [feature-wip][array-type] Support nested array insertion.
morningman commented on a change in pull request #8305:
URL: https://github.com/apache/incubator-doris/pull/8305#discussion_r820833353
##
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
##
@@ -373,6 +373,9 @@ private void toChildrenThrift(Column column, TColumn
tColumn) {
childrenTColumnType.setIndexLen(children.getOlapColumnIndexSize());
childrenTColumn.setColumnType(childrenTColumnType);
childrenTColumn.setIsAllowNull(children.isAllowNull());
+if (tColumn.getAggregationType() != null) {
+
childrenTColumn.setAggregationType(tColumn.getAggregationType());
Review comment:
why need to set agg type for child column?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8368: [feature-wip][array-type] Array data can be loaded in stream load.
morningman commented on a change in pull request #8368:
URL: https://github.com/apache/incubator-doris/pull/8368#discussion_r820839495
##
File path: be/src/util/array_parser.hpp
##
@@ -0,0 +1,215 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_BE_SRC_UTIL_ARRAY_PARSER_HPP
+#define DORIS_BE_SRC_UTIL_ARRAY_PARSER_HPP
Review comment:
```suggestion
#pragma once
```
##
File path: be/src/util/array_parser.hpp
##
@@ -0,0 +1,215 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_BE_SRC_UTIL_ARRAY_PARSER_HPP
+#define DORIS_BE_SRC_UTIL_ARRAY_PARSER_HPP
+
+#include
+
+#include
+
+#include "common/status.h"
+#include "exprs/anyval_util.h"
+#include "runtime/collection_value.h"
+#include "runtime/primitive_type.h"
+#include "runtime/types.h"
+#include "util/mem_util.hpp"
+
+namespace doris {
+
+template
+using ConstArray = typename rapidjson::GenericValue::ConstArray;
+
+template
+using ConstArrayIterator = typename ConstArray::ValueIterator;
+
+class ArrayParser {
+public:
+static Status parse(CollectionVal& array_val, FunctionContext* context,
+const StringVal& str_val) {
+rapidjson::Document document;
+if (document.Parse(reinterpret_cast(str_val.ptr),
str_val.len).HasParseError()) {
+return Status::RuntimeError("Failed to parse the json to array.");
+}
+if (document.IsNull()) {
+array_val = CollectionVal::null();
+return Status::OK();
+} else if (!document.IsArray()) {
+return Status::RuntimeError("Failed to parse the json to array.");
+}
+auto type_desc = convertToTypeDescriptor(context->get_return_type());
+return parse>(
+array_val, context,
+reinterpret_cast(&document)->GetArray(), type_desc);
+}
+
+private:
+static TypeDescriptor convertToTypeDescriptor(FunctionContext::TypeDesc
function_type_desc) {
Review comment:
```suggestion
static TypeDescriptor
_convert_to_type_descriptor(FunctionContext::TypeDesc function_type_desc) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] adonis0147 commented on a change in pull request #8305: [feature-wip][array-type] Support nested array insertion.
adonis0147 commented on a change in pull request #8305:
URL: https://github.com/apache/incubator-doris/pull/8305#discussion_r820847392
##
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
##
@@ -373,6 +373,9 @@ private void toChildrenThrift(Column column, TColumn
tColumn) {
childrenTColumnType.setIndexLen(children.getOlapColumnIndexSize());
childrenTColumn.setColumnType(childrenTColumnType);
childrenTColumn.setIsAllowNull(children.isAllowNull());
+if (tColumn.getAggregationType() != null) {
+
childrenTColumn.setAggregationType(tColumn.getAggregationType());
Review comment:
The parent aggregate type is `NONE` by default for array type. If we
don't set the aggregate type for the children column, after deserializing in
BE, the aggregate type of the children column is `TAggregationType::type::SUM`,
because the value of `TAggregationType::type::SUM` is zero.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] adonis0147 commented on a change in pull request #8305: [feature-wip][array-type] Support nested array insertion.
adonis0147 commented on a change in pull request #8305:
URL: https://github.com/apache/incubator-doris/pull/8305#discussion_r820847392
##
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
##
@@ -373,6 +373,9 @@ private void toChildrenThrift(Column column, TColumn
tColumn) {
childrenTColumnType.setIndexLen(children.getOlapColumnIndexSize());
childrenTColumn.setColumnType(childrenTColumnType);
childrenTColumn.setIsAllowNull(children.isAllowNull());
+if (tColumn.getAggregationType() != null) {
+
childrenTColumn.setAggregationType(tColumn.getAggregationType());
Review comment:
The parent aggregate type is `NONE` by default for array type.
If we don't set the aggregate type for the children column, after
deserializing in BE, the aggregate type of the children column is
`TAggregationType::type::SUM`, because the value of
`TAggregationType::type::SUM` is zero. That is incorrect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 commented on a change in pull request #8364: [Function] Add compare_version function and ut
carlvinhust2012 commented on a change in pull request #8364:
URL: https://github.com/apache/incubator-doris/pull/8364#discussion_r821251157
##
File path: be/src/exprs/utility_functions.cpp
##
@@ -39,4 +39,82 @@ BooleanVal UtilityFunctions::sleep(FunctionContext* ctx,
const IntVal& seconds)
return BooleanVal(true);
}
+static void convert_to_string(doris_udf::StringVal& src, std::string& str_dst)
{
Review comment:
just want to convert the type from 'StringVal' to 'string'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 commented on a change in pull request #8364: [Function] Add compare_version function and ut
carlvinhust2012 commented on a change in pull request #8364:
URL: https://github.com/apache/incubator-doris/pull/8364#discussion_r821251569
##
File path: be/src/exprs/utility_functions.cpp
##
@@ -39,4 +39,82 @@ BooleanVal UtilityFunctions::sleep(FunctionContext* ctx,
const IntVal& seconds)
return BooleanVal(true);
}
+static void convert_to_string(doris_udf::StringVal& src, std::string& str_dst)
{
+for (int i = 0; i < src.len; i++) {
+str_dst += src.ptr[i];
+}
+return;
+}
+
+static BooleanVal split_string(std::string& str_src, std::vector&
str_vec_dst) {
Review comment:
I will re-check this split function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-flink-connector] hf200012 merged pull request #12: [refactor]doris sink enable 2pc
hf200012 merged pull request #12: URL: https://github.com/apache/incubator-doris-flink-connector/pull/12 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-flink-connector] hf200012 commented on pull request #13: Feature/20220305 support flush when batch bytes reach threshold
hf200012 commented on pull request #13: URL: https://github.com/apache/incubator-doris-flink-connector/pull/13#issuecomment-1061321741 Resolve a conflict @bridgeDream -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 closed pull request #8364: [Function] Add compare_version function and ut
carlvinhust2012 closed pull request #8364: URL: https://github.com/apache/incubator-doris/pull/8364 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 commented on pull request #8364: [Function] Add compare_version function and ut
carlvinhust2012 commented on pull request #8364: URL: https://github.com/apache/incubator-doris/pull/8364#issuecomment-1061322368 > Why not `compare_version(ver1 , ver2)` return 0 if equal, 1 if ver1 > ver2, -1 if ver1 < ver2 ？ In addition, there is still a lot of room for optimization in the implementation of the program -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 commented on pull request #8364: [Function] Add compare_version function and ut
carlvinhust2012 commented on pull request #8364: URL: https://github.com/apache/incubator-doris/pull/8364#issuecomment-1061326259 > Why not `compare_version(ver1 , ver2)` return 0 if equal, 1 if ver1 > ver2, -1 if ver1 < ver2 ？ In addition, there is still a lot of room for optimization in the implementation of the program We discussed this approach with the business, but they still prefer to use the current approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 edited a comment on pull request #8364: [Function] Add compare_version function and ut
carlvinhust2012 edited a comment on pull request #8364: URL: https://github.com/apache/incubator-doris/pull/8364#issuecomment-1061326259 > Why not `compare_version(ver1 , ver2)` return 0 if equal, 1 if ver1 > ver2, -1 if ver1 < ver2 ？ In addition, there is still a lot of room for optimization in the implementation of the program -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 commented on pull request #8364: [Function] Add compare_version function and ut
carlvinhust2012 commented on pull request #8364: URL: https://github.com/apache/incubator-doris/pull/8364#issuecomment-1061326860 > > Why not `compare_version(ver1 , ver2)` return 0 if equal, 1 if ver1 > ver2, -1 if ver1 < ver2 ？ In addition, there is still a lot of room for optimization in the implementation of the program We discussed this approach with the business, but they still prefer to use the current approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 removed a comment on pull request #8364: [Function] Add compare_version function and ut
carlvinhust2012 removed a comment on pull request #8364: URL: https://github.com/apache/incubator-doris/pull/8364#issuecomment-1061326259 > Why not `compare_version(ver1 , ver2)` return 0 if equal, 1 if ver1 > ver2, -1 if ver1 < ver2 ？ In addition, there is still a lot of room for optimization in the implementation of the program -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #8336: [improvment](broker load) fix broker load check logic
yangzhg commented on a change in pull request #8336:
URL: https://github.com/apache/incubator-doris/pull/8336#discussion_r821258659
##
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java
##
@@ -232,16 +232,17 @@ public static String checkPath(String path,
StorageBackend.StorageType type) thr
if (Strings.isNullOrEmpty(path)) {
throw new AnalysisException("No dest path specified.");
}
-
+if (type == StorageBackend.StorageType.BROKER) {
+if (!path.toLowerCase().contains("bos") &&
!path.toLowerCase().contains("afs")
+&& !path.toLowerCase().contains("hdfs")) {
+throw new AnalysisException("Invalid export path. please use
valid 'HDFS://', 'AFS://' or 'BOS://' path.");
+}
+return path;
+}
Review comment:
How about a path `nvalid_schema://xxbos/xxafs_path/path-contians-hdfs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo opened a new issue #8388: [Enhancement] Avoid read index bitmap when no index is provided
wangbo opened a new issue #8388: URL: https://github.com/apache/incubator-doris/issues/8388 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description When a query hits index, then index bitmap is reset; Storage layer read rows by index bitmap. In the current implementation, even a query doesn't hit any index, storage still need to read rows by index bitmap. This could bring two costs: 1 Read data from roaring bitmap. 2 The foreach to read rowids from roaring bitmap. There is still optimization room for query without indexes. ### Solution _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Henry2SS commented on pull request #8336: [improvment](broker load) fix broker load check logic
Henry2SS commented on pull request #8336:
URL: https://github.com/apache/incubator-doris/pull/8336#issuecomment-1061349709
>
It is.
URI not support '{}' '[]'. And when the path contains '{}' '[]',
`URISyntaxException` will be thrown out.
This is the reason for the problem described in #8335.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Henry2SS removed a comment on pull request #8336: [improvment](broker load) fix broker load check logic
Henry2SS removed a comment on pull request #8336:
URL: https://github.com/apache/incubator-doris/pull/8336#issuecomment-1061349709
>
It is.
URI not support '{}' '[]'. And when the path contains '{}' '[]',
`URISyntaxException` will be thrown out.
This is the reason for the problem described in #8335.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on a change in pull request #8386: [Bug][Vectorized] Fix the datetime type read error and is_same set error in reader
wangbo commented on a change in pull request #8386: URL: https://github.com/apache/incubator-doris/pull/8386#discussion_r821275926 ## File path: be/src/vec/olap/vcollect_iterator.cpp ## @@ -129,7 +129,14 @@ bool VCollectIterator::LevelIteratorComparator::operator()(LevelIterator* lhs, L // for UNIQUE_KEYS just read the highest version and no need agg_update. // for AGG_KEYS if a version is deleted, the lower version no need to agg_update bool lower = (cmp_res != 0) ? (cmp_res < 0) : (lhs->version() < rhs->version()); Review comment: ```suggestion bool lower = lhs->version() < rhs->version(); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Henry2SS commented on pull request #8336: [improvment](broker load) fix broker load check logic
Henry2SS commented on pull request #8336:
URL: https://github.com/apache/incubator-doris/pull/8336#issuecomment-1061350644
> Seems like the code you changed in this PR does not related the problem
you described in #8335 ?
It is.
URI doesn't support '{}' '[]', and when a path contains '{}', '[]', a
`URISyntaxException` will be thrown out.
It is the reason for the problem described in #8335.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Henry2SS commented on a change in pull request #8336: [improvment](broker load) fix broker load check logic
Henry2SS commented on a change in pull request #8336:
URL: https://github.com/apache/incubator-doris/pull/8336#discussion_r821278406
##
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java
##
@@ -232,16 +232,17 @@ public static String checkPath(String path,
StorageBackend.StorageType type) thr
if (Strings.isNullOrEmpty(path)) {
throw new AnalysisException("No dest path specified.");
}
-
+if (type == StorageBackend.StorageType.BROKER) {
+if (!path.toLowerCase().contains("bos") &&
!path.toLowerCase().contains("afs")
+&& !path.toLowerCase().contains("hdfs")) {
+throw new AnalysisException("Invalid export path. please use
valid 'HDFS://', 'AFS://' or 'BOS://' path.");
+}
+return path;
+}
Review comment:
`checkPath` only do a pre-check, it checks the input style.
If the path is not a valid one, an error will be thrown out as expected.
```
type:ETL_RUN_FAIL; msg:errCode = 2, detailMessage = Broker list path failed.
```
And maybe change `path.contains` to `path.startsWith`, is it better? What's
your opinions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] adonis0147 commented on a change in pull request #8368: [feature-wip][array-type] Array data can be loaded in stream load.
adonis0147 commented on a change in pull request #8368:
URL: https://github.com/apache/incubator-doris/pull/8368#discussion_r821279648
##
File path: be/src/util/array_parser.hpp
##
@@ -0,0 +1,215 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_BE_SRC_UTIL_ARRAY_PARSER_HPP
+#define DORIS_BE_SRC_UTIL_ARRAY_PARSER_HPP
+
+#include
+
+#include
+
+#include "common/status.h"
+#include "exprs/anyval_util.h"
+#include "runtime/collection_value.h"
+#include "runtime/primitive_type.h"
+#include "runtime/types.h"
+#include "util/mem_util.hpp"
+
+namespace doris {
+
+template
+using ConstArray = typename rapidjson::GenericValue::ConstArray;
+
+template
+using ConstArrayIterator = typename ConstArray::ValueIterator;
+
+class ArrayParser {
+public:
+static Status parse(CollectionVal& array_val, FunctionContext* context,
+const StringVal& str_val) {
+rapidjson::Document document;
+if (document.Parse(reinterpret_cast(str_val.ptr),
str_val.len).HasParseError()) {
+return Status::RuntimeError("Failed to parse the json to array.");
+}
+if (document.IsNull()) {
+array_val = CollectionVal::null();
+return Status::OK();
+} else if (!document.IsArray()) {
+return Status::RuntimeError("Failed to parse the json to array.");
+}
+auto type_desc = convertToTypeDescriptor(context->get_return_type());
+return parse>(
+array_val, context,
+reinterpret_cast(&document)->GetArray(), type_desc);
+}
+
+private:
+static TypeDescriptor convertToTypeDescriptor(FunctionContext::TypeDesc
function_type_desc) {
Review comment:
Done. PTAL @morningman .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on pull request #8369: [docs] Update documentation configuration parameter `sink.batch.bytes…
hf200012 commented on pull request #8369: URL: https://github.com/apache/incubator-doris/pull/8369#issuecomment-1061355224 fix the code problem -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on a change in pull request #8386: [Bug][Vectorized] Fix the datetime type read error and is_same set error in reader
wangbo commented on a change in pull request #8386: URL: https://github.com/apache/incubator-doris/pull/8386#discussion_r821275926 ## File path: be/src/vec/olap/vcollect_iterator.cpp ## @@ -129,7 +129,14 @@ bool VCollectIterator::LevelIteratorComparator::operator()(LevelIterator* lhs, L // for UNIQUE_KEYS just read the highest version and no need agg_update. // for AGG_KEYS if a version is deleted, the lower version no need to agg_update bool lower = (cmp_res != 0) ? (cmp_res < 0) : (lhs->version() < rhs->version()); Review comment: ```suggestion bool lower = lhs->version() < rhs->version(); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on pull request #8384: [Feature][Vectorized] support replace()
HappenLee commented on pull request #8384: URL: https://github.com/apache/incubator-doris/pull/8384#issuecomment-1061363767 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on pull request #8387: [Refactor](storage_engine) Use std::shared_lock to replace RWMutex
caiconghui commented on pull request #8387: URL: https://github.com/apache/incubator-doris/pull/8387#issuecomment-1061365634 > Please describe the motivation and any possible performance impact, so that other reviewers can directly know what is going on. the motivation is to use the function of STL instead of wrapper of pthread_rwlock_t by ourselves, which is more portable. Exact behavior is not specified by the standard. But here are some clues: On POSIX systems shared_mutex will most likely be implemented on top of pthread_rwlock_t and implementations usually give preference to readers because of its requirement to support recursive read locks. reference as following: https://www.quora.com/Why-use-std-shared_mutex-instead-of-a-C-wrapper-of-pthread_rwlock_t https://stackoverflow.com/questions/33770500/when-to-use-c11-mutex-lock-unique-lock-shared-lock-etc anyway, I will make some performance test for the replacement -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on pull request #8324: [Feature](proc) Support proc dir for showing tablet health status
caiconghui commented on pull request #8324: URL: https://github.com/apache/incubator-doris/pull/8324#issuecomment-1061366922 > Why don't just add new columns to `show proc "/statistic"`? cannot find a suitable place to append this, which may cause /statistic info verbose and not easy to display -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lordk911 commented on issue #7502: Doris Roadmap 2022
lordk911 commented on issue #7502: URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1061367317 want to know when will vectorized query engine could be released. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8267: [fix](vectorization)Some small fix for SegmentIter Vectorization
github-actions[bot] commented on pull request #8267: URL: https://github.com/apache/incubator-doris/pull/8267#issuecomment-1061369587 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8307: [Vectorized][Function] support regexp&&sm4&&aes functions
github-actions[bot] commented on pull request #8307: URL: https://github.com/apache/incubator-doris/pull/8307#issuecomment-1061375913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8324: [Feature](proc) Support proc dir for showing tablet health status
morningman commented on pull request #8324: URL: https://github.com/apache/incubator-doris/pull/8324#issuecomment-1061394173 > So maybe it is better to move some column from `/statistic` to `/cluster_health`? such as `UnhealthyTabletNum` and `InconsistentTabletNum`? Otherwise we need to check them here and there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on pull request #8324: [Feature](proc) Support proc dir for showing tablet health status
caiconghui commented on pull request #8324: URL: https://github.com/apache/incubator-doris/pull/8324#issuecomment-1061396132 > > > > So maybe it is better to move some column from `/statistic` to `/cluster_health`? such as `UnhealthyTabletNum` and `InconsistentTabletNum`? Otherwise we need to check them here and there. move UnhealthyTabletNum | InconsistentTabletNum | CloningTabletNum | BadTabletNum | to /cluster_health is more reasonable. BadTabletNum is uncoverableTabletNum -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8267: [fix](vectorization)Some small fix for SegmentIter Vectorization
morningman merged pull request #8267: URL: https://github.com/apache/incubator-doris/pull/8267 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix](vectorization)Some small fix for SegmentIter Vectorization (#8267)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new d711d64 [fix](vectorization)Some small fix for SegmentIter
Vectorization (#8267)
d711d64 is described below
commit d711d64dda3b92f3b51154ee585bf0c92b9aed0d
Author: wangbo
AuthorDate: Tue Mar 8 13:13:17 2022 +0800
[fix](vectorization)Some small fix for SegmentIter Vectorization (#8267)
1. No longer using short-circuit to evaluate date type, because the cost of
read date type is small,
lazy materialization has higher costs.
2. Fix read hll/bitmap/date type error results.
---
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 5 +
be/src/vec/columns/column_complex.h| 16
be/src/vec/columns/column_vector.h | 4 ++--
3 files changed, 19 insertions(+), 6 deletions(-)
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index 2883e63..6a8febb 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -613,10 +613,7 @@ void SegmentIterator::_vec_init_lazy_materialization() {
_is_pred_column[cid] = true;
pred_column_ids.insert(cid);
-// for date type which can not be executed in a vectorized way,
using short circuit execution
-if (type == OLAP_FIELD_TYPE_VARCHAR || type ==
OLAP_FIELD_TYPE_CHAR ||
-type == OLAP_FIELD_TYPE_DECIMAL || type ==
OLAP_FIELD_TYPE_DATE ||
-predicate->is_in_predicate()) {
+if (type == OLAP_FIELD_TYPE_VARCHAR || type ==
OLAP_FIELD_TYPE_CHAR || type == OLAP_FIELD_TYPE_STRING ||
predicate->is_in_predicate()) {
short_cir_pred_col_id_set.insert(cid);
_short_cir_eval_predicate.push_back(predicate);
_is_all_column_basic_type = false;
diff --git a/be/src/vec/columns/column_complex.h
b/be/src/vec/columns/column_complex.h
index cd26c7f..df2b582 100644
--- a/be/src/vec/columns/column_complex.h
+++ b/be/src/vec/columns/column_complex.h
@@ -63,6 +63,7 @@ public:
}
void insert_many_binary_data(char* data_array, uint32_t* len_array,
uint32_t* start_offset_array, size_t num) override {
+resize(num);
if constexpr (std::is_same_v) {
for (size_t i = 0; i < num; i++) {
uint32_t len = len_array[i];
@@ -76,6 +77,21 @@ public:
*pvalue =
std::move(*reinterpret_cast(data_array + start_offset));
}
}
+} else if constexpr (std::is_same_v) {
+for (size_t i = 0; i < num; i++) {
+uint32_t len = len_array[i];
+uint32_t start_offset = start_offset_array[i];
+HyperLogLog* pvalue = &get_element(size() - 1);
+if (len != 0) {
+HyperLogLog value;
+value.deserialize(Slice(data_array + start_offset, len));
+*pvalue = std::move(value);
+} else {
+*pvalue =
std::move(*reinterpret_cast(data_array + start_offset));
+}
+}
+} else {
+LOG(FATAL) << "Unexpected type in column complex";
}
}
diff --git a/be/src/vec/columns/column_vector.h
b/be/src/vec/columns/column_vector.h
index c6b2311..216e8fd 100644
--- a/be/src/vec/columns/column_vector.h
+++ b/be/src/vec/columns/column_vector.h
@@ -179,7 +179,7 @@ public:
value |= *(unsigned char*)(cur_ptr);
vectorized::VecDateTimeValue date;
date.from_olap_date(value);
-data.push_back_without_reserve(date);
+this->insert_data(reinterpret_cast(&date), 0);
}
}
@@ -189,7 +189,7 @@ public:
const char* cur_ptr = data_ptr + value_size * i;
uint64_t value = *reinterpret_cast(cur_ptr);
vectorized::VecDateTimeValue date(value);
-data.push_back_without_reserve(date);
+this->insert_data(reinterpret_cast(&date), 0);
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #8306: [Vectorized][Feature] support regexp&&sm4&&aes function
morningman closed issue #8306: URL: https://github.com/apache/incubator-doris/issues/8306 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [feature](vectorize)(function) support regexp&&sm4&&aes functions (#8307)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 454b45b [feature](vectorize)(function) support regexp&&sm4&&aes
functions (#8307)
454b45b is described below
commit 454b45bea3cb5d0c6eb0c2391465a07a19aceed4
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Tue Mar 8 13:14:02 2022 +0800
[feature](vectorize)(function) support regexp&&sm4&&aes functions (#8307)
---
be/src/exprs/encryption_functions.cpp | 21 --
be/src/exprs/encryption_functions.h| 21 +-
be/src/exprs/string_functions.cpp | 4 +-
be/src/exprs/string_functions.h| 3 +
be/src/vec/CMakeLists.txt | 3 +
.../aggregate_function_percentile_approx.cpp | 1 -
be/src/vec/functions/function_encryption.cpp | 245 +
be/src/vec/functions/function_hex.cpp | 183 +++
be/src/vec/functions/function_regexp.cpp | 216 ++
be/src/vec/functions/function_string.cpp | 132 +--
be/src/vec/functions/function_string.h | 21 +-
be/src/vec/functions/math.cpp | 51 -
be/src/vec/functions/simple_function_factory.h | 6 +
be/test/vec/function/function_like_test.cpp| 59 +
be/test/vec/function/function_string_test.cpp | 241 +++-
gensrc/script/doris_builtins_functions.py | 74 +++
16 files changed, 1034 insertions(+), 247 deletions(-)
diff --git a/be/src/exprs/encryption_functions.cpp
b/be/src/exprs/encryption_functions.cpp
index 19ec1a7..a4d93b7 100644
--- a/be/src/exprs/encryption_functions.cpp
+++ b/be/src/exprs/encryption_functions.cpp
@@ -22,34 +22,13 @@
#include "runtime/string_value.h"
#include "runtime/tuple_row.h"
#include "util/debug_util.h"
-#include "util/encryption_util.h"
#include "util/md5.h"
#include "util/sm3.h"
-#include "util/string_util.h"
#include "util/url_coding.h"
namespace doris {
void EncryptionFunctions::init() {}
-StringCaseUnorderedMap aes_mode_map {
-{"AES_128_ECB", AES_128_ECB}, {"AES_192_ECB", AES_192_ECB},
-{"AES_256_ECB", AES_256_ECB}, {"AES_128_CBC", AES_128_CBC},
-{"AES_192_CBC", AES_192_CBC}, {"AES_256_CBC", AES_256_CBC},
-{"AES_128_CFB", AES_128_CFB}, {"AES_192_CFB", AES_192_CFB},
-{"AES_256_CFB", AES_256_CFB}, {"AES_128_CFB1", AES_128_CFB1},
-{"AES_192_CFB1", AES_192_CFB1}, {"AES_256_CFB1", AES_256_CFB1},
-{"AES_128_CFB8", AES_128_CFB8}, {"AES_192_CFB8", AES_192_CFB8},
-{"AES_256_CFB8", AES_256_CFB8}, {"AES_128_CFB128", AES_128_CFB128},
-{"AES_192_CFB128", AES_192_CFB128}, {"AES_256_CFB128", AES_256_CFB128},
-{"AES_128_CTR", AES_128_CTR}, {"AES_192_CTR", AES_192_CTR},
-{"AES_256_CTR", AES_256_CTR}, {"AES_128_OFB", AES_128_OFB},
-{"AES_192_OFB", AES_192_OFB}, {"AES_256_OFB", AES_256_OFB}};
-StringCaseUnorderedMap sm4_mode_map {{"SM4_128_ECB",
SM4_128_ECB},
- {"SM4_128_CBC",
SM4_128_CBC},
- {"SM4_128_CFB128",
SM4_128_CFB128},
- {"SM4_128_OFB",
SM4_128_OFB},
- {"SM4_128_CTR",
SM4_128_CTR}};
-
StringVal encrypt(FunctionContext* ctx, const StringVal& src, const StringVal&
key,
const StringVal& iv, EncryptionMode mode) {
if (src.len == 0 || src.is_null) {
diff --git a/be/src/exprs/encryption_functions.h
b/be/src/exprs/encryption_functions.h
index 8a7e45d..0046da8 100644
--- a/be/src/exprs/encryption_functions.h
+++ b/be/src/exprs/encryption_functions.h
@@ -22,13 +22,32 @@
#include "udf/udf.h"
#include "udf/udf_internal.h"
+#include "util/encryption_util.h"
+#include "util/string_util.h"
namespace doris {
class Expr;
struct ExprValue;
class TupleRow;
-
+static StringCaseUnorderedMap aes_mode_map {
+{"AES_128_ECB", AES_128_ECB}, {"AES_192_ECB", AES_192_ECB},
+{"AES_256_ECB", AES_256_ECB}, {"AES_128_CBC", AES_128_CBC},
+{"AES_192_CBC", AES_192_CBC}, {"AES_256_CBC", AES_256_CBC},
+{"AES_128_CFB", AES_128_CFB}, {"AES_192_CFB", AES_192_CFB},
+{"AES_256_CFB", AES_256_CFB}, {"AES_128_CFB1", AES_128_CFB1},
+{"AES_192_CFB1", AES_192_CFB1}, {"AES_256_CFB1", AES_256_CFB1},
+{"AES_128_CFB8", AES_128_CFB8}, {"AES_192_CFB8", AES_192_CFB8},
+{"AES_256_CFB8", AES_256_CFB8}, {"AES_128_CFB128", AES_128_CFB128},
+{"AES_192_CFB128", AES_192_CFB128}, {"AES_256_CFB128", AES_256_CFB128},
+{"AE
[GitHub] [incubator-doris] morningman merged pull request #8307: [Vectorized][Function] support regexp&&sm4&&aes functions
morningman merged pull request #8307: URL: https://github.com/apache/incubator-doris/pull/8307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8355: [improvement][fix](insert)(replay) support SHOW LAST INSERT stmt and fix json replay bug
morningman commented on a change in pull request #8355:
URL: https://github.com/apache/incubator-doris/pull/8355#discussion_r821328248
##
File path:
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowLastInsertStmt.java
##
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+// SHOW LAST INSERT
+public class ShowLastInsertStmt extends ShowStmt {
+private static final ShowResultSetMetaData META_DATA =
+ShowResultSetMetaData.builder()
+.addColumn(new Column("TransactionId",
ScalarType.createVarchar(128)))
+.addColumn(new Column("Label",
ScalarType.createVarchar(128)))
+.addColumn(new Column("Database",
ScalarType.createVarchar(128)))
+.addColumn(new Column("Table",
ScalarType.createVarchar(128)))
+.addColumn(new Column("TransactionStatus",
ScalarType.createVarchar(64)))
+.addColumn(new Column("LoadedRows",
ScalarType.createVarchar(128)))
+.addColumn(new Column("FilteredRows",
ScalarType.createVarchar(128)))
+.build();
+
+@Override
+public void analyze(Analyzer analyzer) {
Review comment:
No need to check priv here. Bacause `show last insert` can only view the
insert result of current session.
So if user does not have priv to insert, than there is no result to show.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8355: [improvement][fix](insert)(replay) support SHOW LAST INSERT stmt and fix json replay bug
morningman commented on a change in pull request #8355:
URL: https://github.com/apache/incubator-doris/pull/8355#discussion_r821328248
##
File path:
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowLastInsertStmt.java
##
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+// SHOW LAST INSERT
+public class ShowLastInsertStmt extends ShowStmt {
+private static final ShowResultSetMetaData META_DATA =
+ShowResultSetMetaData.builder()
+.addColumn(new Column("TransactionId",
ScalarType.createVarchar(128)))
+.addColumn(new Column("Label",
ScalarType.createVarchar(128)))
+.addColumn(new Column("Database",
ScalarType.createVarchar(128)))
+.addColumn(new Column("Table",
ScalarType.createVarchar(128)))
+.addColumn(new Column("TransactionStatus",
ScalarType.createVarchar(64)))
+.addColumn(new Column("LoadedRows",
ScalarType.createVarchar(128)))
+.addColumn(new Column("FilteredRows",
ScalarType.createVarchar(128)))
+.build();
+
+@Override
+public void analyze(Analyzer analyzer) {
Review comment:
No need to check priv here. Bacause `show last insert` can only view the
insert result of current session.
So if user does not have priv to insert, than there is no result to show.
Add this to the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 03/05: [fix](vectorization)Some small fix for SegmentIter Vectorization (#8267)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 9b6de9be424279ad5d33bfd74de9c8dfbfcc547e
Author: wangbo
AuthorDate: Tue Mar 8 13:13:17 2022 +0800
[fix](vectorization)Some small fix for SegmentIter Vectorization (#8267)
1. No longer using short-circuit to evaluate date type, because the cost of
read date type is small,
lazy materialization has higher costs.
2. Fix read hll/bitmap/date type error results.
---
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 5 +
be/src/vec/columns/column_complex.h| 16
be/src/vec/columns/column_vector.h | 4 ++--
3 files changed, 19 insertions(+), 6 deletions(-)
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index 2883e63..6a8febb 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -613,10 +613,7 @@ void SegmentIterator::_vec_init_lazy_materialization() {
_is_pred_column[cid] = true;
pred_column_ids.insert(cid);
-// for date type which can not be executed in a vectorized way,
using short circuit execution
-if (type == OLAP_FIELD_TYPE_VARCHAR || type ==
OLAP_FIELD_TYPE_CHAR ||
-type == OLAP_FIELD_TYPE_DECIMAL || type ==
OLAP_FIELD_TYPE_DATE ||
-predicate->is_in_predicate()) {
+if (type == OLAP_FIELD_TYPE_VARCHAR || type ==
OLAP_FIELD_TYPE_CHAR || type == OLAP_FIELD_TYPE_STRING ||
predicate->is_in_predicate()) {
short_cir_pred_col_id_set.insert(cid);
_short_cir_eval_predicate.push_back(predicate);
_is_all_column_basic_type = false;
diff --git a/be/src/vec/columns/column_complex.h
b/be/src/vec/columns/column_complex.h
index cd26c7f..df2b582 100644
--- a/be/src/vec/columns/column_complex.h
+++ b/be/src/vec/columns/column_complex.h
@@ -63,6 +63,7 @@ public:
}
void insert_many_binary_data(char* data_array, uint32_t* len_array,
uint32_t* start_offset_array, size_t num) override {
+resize(num);
if constexpr (std::is_same_v) {
for (size_t i = 0; i < num; i++) {
uint32_t len = len_array[i];
@@ -76,6 +77,21 @@ public:
*pvalue =
std::move(*reinterpret_cast(data_array + start_offset));
}
}
+} else if constexpr (std::is_same_v) {
+for (size_t i = 0; i < num; i++) {
+uint32_t len = len_array[i];
+uint32_t start_offset = start_offset_array[i];
+HyperLogLog* pvalue = &get_element(size() - 1);
+if (len != 0) {
+HyperLogLog value;
+value.deserialize(Slice(data_array + start_offset, len));
+*pvalue = std::move(value);
+} else {
+*pvalue =
std::move(*reinterpret_cast(data_array + start_offset));
+}
+}
+} else {
+LOG(FATAL) << "Unexpected type in column complex";
}
}
diff --git a/be/src/vec/columns/column_vector.h
b/be/src/vec/columns/column_vector.h
index c6b2311..216e8fd 100644
--- a/be/src/vec/columns/column_vector.h
+++ b/be/src/vec/columns/column_vector.h
@@ -179,7 +179,7 @@ public:
value |= *(unsigned char*)(cur_ptr);
vectorized::VecDateTimeValue date;
date.from_olap_date(value);
-data.push_back_without_reserve(date);
+this->insert_data(reinterpret_cast(&date), 0);
}
}
@@ -189,7 +189,7 @@ public:
const char* cur_ptr = data_ptr + value_size * i;
uint64_t value = *reinterpret_cast(cur_ptr);
vectorized::VecDateTimeValue date(value);
-data.push_back_without_reserve(date);
+this->insert_data(reinterpret_cast(&date), 0);
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 01/05: Revert "[refactor] remove unused new_in_predicate code (#8263)" (#8372)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit cadf19ae38bc2e4a88c27263bb703762695ccf4e
Author: Zhengguo Yang <780531...@qq.com>
AuthorDate: Mon Mar 7 15:55:38 2022 +0800
Revert "[refactor] remove unused new_in_predicate code (#8263)" (#8372)
This reverts commit 757e35744d4f6319e936fca84b4be13cf043a578.
---
be/src/common/daemon.cpp | 2 +
be/src/exprs/CMakeLists.txt | 1 +
be/src/exprs/new_in_predicate.cpp | 185
be/src/exprs/new_in_predicate.h | 348 ++
4 files changed, 536 insertions(+)
diff --git a/be/src/common/daemon.cpp b/be/src/common/daemon.cpp
index 04625e3..044feda 100644
--- a/be/src/common/daemon.cpp
+++ b/be/src/common/daemon.cpp
@@ -37,6 +37,7 @@
#include "exprs/json_functions.h"
#include "exprs/like_predicate.h"
#include "exprs/math_functions.h"
+#include "exprs/new_in_predicate.h"
#include "exprs/operators.h"
#include "exprs/string_functions.h"
#include "exprs/table_function/dummy_table_functions.h"
@@ -249,6 +250,7 @@ void Daemon::init(int argc, char** argv, const
std::vector& paths) {
StringFunctions::init();
ArrayFunctions::init();
CastFunctions::init();
+InPredicate::init();
MathFunctions::init();
EncryptionFunctions::init();
TimestampFunctions::init();
diff --git a/be/src/exprs/CMakeLists.txt b/be/src/exprs/CMakeLists.txt
index f266fd9..c71f23e 100644
--- a/be/src/exprs/CMakeLists.txt
+++ b/be/src/exprs/CMakeLists.txt
@@ -41,6 +41,7 @@ add_library(Exprs
expr.cpp
expr_context.cpp
in_predicate.cpp
+ new_in_predicate.cpp
bloomfilter_predicate.cpp
block_bloom_filter_avx_impl.cc
block_bloom_filter_impl.cc
diff --git a/be/src/exprs/new_in_predicate.cpp
b/be/src/exprs/new_in_predicate.cpp
new file mode 100644
index 000..b0a6390
--- /dev/null
+++ b/be/src/exprs/new_in_predicate.cpp
@@ -0,0 +1,185 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/new_in_predicate.h"
+
+#include
+
+#include "exprs/anyval_util.h"
+#include "runtime/string_value.hpp"
+
+namespace doris {
+
+void InPredicate::init() {}
+
+// Templated getter functions for extracting 'SetType' values from AnyVals
+template
+SetType get_val(const FunctionContext::TypeDesc* type, const T& x) {
+DCHECK(!x.is_null);
+return x.val;
+}
+
+template <>
+StringValue get_val(const FunctionContext::TypeDesc* type, const StringVal& x)
{
+DCHECK(!x.is_null);
+return StringValue::from_string_val(x);
+}
+
+template <>
+DateTimeValue get_val(const FunctionContext::TypeDesc* type, const
DateTimeVal& x) {
+return DateTimeValue::from_datetime_val(x);
+}
+
+template <>
+DecimalV2Value get_val(const FunctionContext::TypeDesc* type, const
DecimalV2Val& x) {
+return DecimalV2Value::from_decimal_val(x);
+}
+
+template
+void InPredicate::set_lookup_prepare(FunctionContext* ctx,
+ FunctionContext::FunctionStateScope
scope) {
+if (scope != FunctionContext::FRAGMENT_LOCAL) {
+return;
+}
+
+SetLookupState* state = new SetLookupState;
+state->type = ctx->get_arg_type(0);
+state->contains_null = false;
+for (int i = 1; i < ctx->get_num_args(); ++i) {
+DCHECK(ctx->is_arg_constant(i));
+T* arg = reinterpret_cast(ctx->get_constant_arg(i));
+if (arg->is_null) {
+state->contains_null = true;
+} else {
+state->val_set.insert(get_val(state->type, *arg));
+}
+}
+ctx->set_function_state(scope, state);
+}
+
+template
+void InPredicate::set_lookup_close(FunctionContext* ctx,
+ FunctionContext::FunctionStateScope scope) {
+if (scope != FunctionContext::FRAGMENT_LOCAL) {
+return;
+}
+SetLookupState* state =
+
reinterpret_cast*>(ctx->get_function_state(scope));
+delete state;
+}
+
+template
+BooleanVal InPredicate::templated_in(FunctionContext* ctx, const T& val, int
num_args,
+ const T* args) {
+if (val.is_null) {
+
[incubator-doris] 05/05: [improvement] Support show tablets stmt (#7970)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit d2b88edfdab7179f580e587c85007e27f7818cf1
Author: ChPi
AuthorDate: Sat Mar 5 15:25:57 2022 +0800
[improvement] Support show tablets stmt (#7970)
change `show tablet from tbl` to `show tablets from tbl`
---
.../Data Manipulation/SHOW TABLET.md | 13 ++---
.../Data Manipulation/SHOW TABLETS.md | 56 ++
.../Data Manipulation/SHOW TABLET.md | 33 ++---
.../{SHOW TABLET.md => SHOW TABLETS.md}| 46 +-
fe/fe-core/src/main/cup/sql_parser.cup | 4 +-
fe/fe-core/src/main/jflex/sql_scanner.flex | 1 +
6 files changed, 89 insertions(+), 64 deletions(-)
diff --git a/docs/en/sql-reference/sql-statements/Data Manipulation/SHOW
TABLET.md b/docs/en/sql-reference/sql-statements/Data Manipulation/SHOW
TABLET.md
index 17a559e..21ccbcc 100644
--- a/docs/en/sql-reference/sql-statements/Data Manipulation/SHOW TABLET.md
+++ b/docs/en/sql-reference/sql-statements/Data Manipulation/SHOW TABLET.md
@@ -26,17 +26,14 @@ under the License.
# SHOW TABLET
## Description
-This statement is used to display tablet-related information (for
administrators only)
-Grammar:
-SHOW TABLET
-[From [db-uu name]] table U name.; Table U Id]
+This statement is used to display information of the specified tablet (for
administrators only)
+Grammar:
+SHOW TABLET tablet_id
## example
-1. Display all tablet information in the specified table below the specified DB
-SHOW TABLET FROM example_db.table_name;
-2. Display parent level ID information of tablet with specified tablet ID of
1
-Performance board 1;
+// Display parent level ID information of tablet with specified tablet
id of 1
+SHOW TABLET 1;
## keyword
SHOW,TABLET
diff --git a/docs/en/sql-reference/sql-statements/Data Manipulation/SHOW
TABLETS.md b/docs/en/sql-reference/sql-statements/Data Manipulation/SHOW
TABLETS.md
new file mode 100644
index 000..0ed338d
--- /dev/null
+++ b/docs/en/sql-reference/sql-statements/Data Manipulation/SHOW TABLETS.md
@@ -0,0 +1,56 @@
+---
+{
+"title": "SHOW TABLETS",
+"language": "en"
+}
+---
+
+
+
+# SHOW TABLETS
+## Description
+This statement is used to display tablet-related information (for
administrators only)
+Grammar:
+SHOW TABLETS
+[FROM [db_name.]table_name] [partiton(partition_name_1,
partition_name_1)]
+[where [version=1] [and backendid=1] [and
state="NORMAL|ROLLUP|CLONE|DECOMMISSION"]]
+[order by order_column]
+[limit [offset,]size]
+
+## example
+// Display all tablets information in the specified table below the
specified DB
+SHOW TABLETS FROM example_db.table_name;
+
+SHOW TABLETS FROM example_db.table_name partition(p1, p2);
+
+// display 10 tablets information in the table
+SHOW TABLETS FROM example_db.table_name limit 10;
+
+SHOW TABLETS FROM example_db.table_name limit 5,10;
+
+// display the tablets that fulfill some conditions
+SHOW TABLETS FROM example_db.table_name where backendid=1 and
version=1 and state="NORMAL";
+
+SHOW TABLETS FROM example_db.table_name where backendid=1 order by
version;
+
+SHOW TABLETS FROM example_db.table_name where indexname="t1_rollup";
+
+## keyword
+SHOW,TABLETS,LIMIT
diff --git a/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/SHOW
TABLET.md b/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/SHOW
TABLET.md
index fc6fb77..d2de94f 100644
--- a/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/SHOW TABLET.md
+++ b/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/SHOW TABLET.md
@@ -26,40 +26,13 @@ under the License.
# SHOW TABLET
## description
-该语句用于显示 tablet 相关的信息（仅管理员使用）
+该语句用于显示指定tablet id 信息（仅管理员使用）
语法：
-SHOW TABLET
-[FROM [db_name.]table_name | tablet_id] [partiton(partition_name_1,
partition_name_1)]
-[where [version=1] [and backendid=1] [and
state="NORMAL|ROLLUP|CLONE|DECOMMISSION"]]
-[order by order_column]
-[limit [offset,]size]
+SHOW TABLET tablet_id
-现在show tablet命令支持按照按照以下字段进行过滤：partition, index name, version, backendid,
-state，同时支持按照任意字段进行排序，并且提供limit限制返回条数。
## example
-1. 显示指定 db 的下指定表所有 tablet 信息
-SHOW TABLET FROM example_db.table_name;
-
-// 获取partition p1和p2的tablet信息
-SHOW TABLET FROM example_db.table_name partition(p1, p2);
-
-// 获取10个结果
-SHOW TABLET FROM example_db.table_name limit 10;
-
-// 从偏移5开始获取10个结果
-SHOW TABLET FROM example_db.table_name limit 5,10;
-
-// 按照backendid/version/state字段进行过滤
-SHOW
[incubator-doris] 02/05: [chore] fix build with parallel parameter only (#8352)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 26f55dabcd87fff37180cce6ecbdd122f591f90d
Author: dataalive <99398130+dataal...@users.noreply.github.com>
AuthorDate: Mon Mar 7 23:19:19 2022 +0800
[chore] fix build with parallel parameter only (#8352)
---
build.sh | 13 -
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/build.sh b/build.sh
index 2c883c8..bc44fb8 100755
--- a/build.sh
+++ b/build.sh
@@ -121,6 +121,8 @@ BUILD_UI=
BUILD_SPARK_DPP=
CLEAN=
HELP=0
+PARAMETER_COUNT=$#
+PARAMETER_FLAG=0
if [ $# == 1 ] ; then
# default
BUILD_BE=1
@@ -146,11 +148,20 @@ else
--clean) CLEAN=1 ; shift ;;
-h) HELP=1; shift ;;
--help) HELP=1; shift ;;
--j) PARALLEL=$2; shift 2 ;;
+-j) PARALLEL=$2; PARAMETER_FLAG=1; shift 2 ;;
--) shift ; break ;;
*) echo "Internal error" ; exit 1 ;;
esac
done
+#only ./build.sh -j xx then build all
+if [[ ${PARAMETER_COUNT} -eq 3 ]] && [[ ${PARAMETER_FLAG} -eq 1 ]];then
+BUILD_BE=1
+BUILD_FE=1
+BUILD_BROKER=1
+BUILD_UI=1
+BUILD_SPARK_DPP=1
+CLEAN=0
+fi
fi
if [[ ${HELP} -eq 1 ]]; then
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 04/05: [feature](vectorize)(function) support regexp&&sm4&&aes functions (#8307)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 209c5a173b1335514af0f1e57f033b2c21803097
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Tue Mar 8 13:14:02 2022 +0800
[feature](vectorize)(function) support regexp&&sm4&&aes functions (#8307)
---
be/src/exprs/encryption_functions.cpp | 21 --
be/src/exprs/encryption_functions.h| 21 +-
be/src/exprs/string_functions.cpp | 4 +-
be/src/exprs/string_functions.h| 3 +
be/src/vec/CMakeLists.txt | 3 +
.../aggregate_function_percentile_approx.cpp | 1 -
be/src/vec/functions/function_encryption.cpp | 245 +
be/src/vec/functions/function_hex.cpp | 183 +++
be/src/vec/functions/function_regexp.cpp | 216 ++
be/src/vec/functions/function_string.cpp | 132 +--
be/src/vec/functions/function_string.h | 21 +-
be/src/vec/functions/math.cpp | 51 -
be/src/vec/functions/simple_function_factory.h | 6 +
be/test/vec/function/function_like_test.cpp| 59 +
be/test/vec/function/function_string_test.cpp | 241 +++-
gensrc/script/doris_builtins_functions.py | 74 +++
16 files changed, 1034 insertions(+), 247 deletions(-)
diff --git a/be/src/exprs/encryption_functions.cpp
b/be/src/exprs/encryption_functions.cpp
index 19ec1a7..a4d93b7 100644
--- a/be/src/exprs/encryption_functions.cpp
+++ b/be/src/exprs/encryption_functions.cpp
@@ -22,34 +22,13 @@
#include "runtime/string_value.h"
#include "runtime/tuple_row.h"
#include "util/debug_util.h"
-#include "util/encryption_util.h"
#include "util/md5.h"
#include "util/sm3.h"
-#include "util/string_util.h"
#include "util/url_coding.h"
namespace doris {
void EncryptionFunctions::init() {}
-StringCaseUnorderedMap aes_mode_map {
-{"AES_128_ECB", AES_128_ECB}, {"AES_192_ECB", AES_192_ECB},
-{"AES_256_ECB", AES_256_ECB}, {"AES_128_CBC", AES_128_CBC},
-{"AES_192_CBC", AES_192_CBC}, {"AES_256_CBC", AES_256_CBC},
-{"AES_128_CFB", AES_128_CFB}, {"AES_192_CFB", AES_192_CFB},
-{"AES_256_CFB", AES_256_CFB}, {"AES_128_CFB1", AES_128_CFB1},
-{"AES_192_CFB1", AES_192_CFB1}, {"AES_256_CFB1", AES_256_CFB1},
-{"AES_128_CFB8", AES_128_CFB8}, {"AES_192_CFB8", AES_192_CFB8},
-{"AES_256_CFB8", AES_256_CFB8}, {"AES_128_CFB128", AES_128_CFB128},
-{"AES_192_CFB128", AES_192_CFB128}, {"AES_256_CFB128", AES_256_CFB128},
-{"AES_128_CTR", AES_128_CTR}, {"AES_192_CTR", AES_192_CTR},
-{"AES_256_CTR", AES_256_CTR}, {"AES_128_OFB", AES_128_OFB},
-{"AES_192_OFB", AES_192_OFB}, {"AES_256_OFB", AES_256_OFB}};
-StringCaseUnorderedMap sm4_mode_map {{"SM4_128_ECB",
SM4_128_ECB},
- {"SM4_128_CBC",
SM4_128_CBC},
- {"SM4_128_CFB128",
SM4_128_CFB128},
- {"SM4_128_OFB",
SM4_128_OFB},
- {"SM4_128_CTR",
SM4_128_CTR}};
-
StringVal encrypt(FunctionContext* ctx, const StringVal& src, const StringVal&
key,
const StringVal& iv, EncryptionMode mode) {
if (src.len == 0 || src.is_null) {
diff --git a/be/src/exprs/encryption_functions.h
b/be/src/exprs/encryption_functions.h
index 8a7e45d..0046da8 100644
--- a/be/src/exprs/encryption_functions.h
+++ b/be/src/exprs/encryption_functions.h
@@ -22,13 +22,32 @@
#include "udf/udf.h"
#include "udf/udf_internal.h"
+#include "util/encryption_util.h"
+#include "util/string_util.h"
namespace doris {
class Expr;
struct ExprValue;
class TupleRow;
-
+static StringCaseUnorderedMap aes_mode_map {
+{"AES_128_ECB", AES_128_ECB}, {"AES_192_ECB", AES_192_ECB},
+{"AES_256_ECB", AES_256_ECB}, {"AES_128_CBC", AES_128_CBC},
+{"AES_192_CBC", AES_192_CBC}, {"AES_256_CBC", AES_256_CBC},
+{"AES_128_CFB", AES_128_CFB}, {"AES_192_CFB", AES_192_CFB},
+{"AES_256_CFB", AES_256_CFB}, {"AES_128_CFB1", AES_128_CFB1},
+{"AES_192_CFB1", AES_192_CFB1}, {"AES_256_CFB1", AES_256_CFB1},
+{"AES_128_CFB8", AES_128_CFB8}, {"AES_192_CFB8", AES_192_CFB8},
+{"AES_256_CFB8", AES_256_CFB8}, {"AES_128_CFB128", AES_128_CFB128},
+{"AES_192_CFB128", AES_192_CFB128}, {"AES_256_CFB128", AES_256_CFB128},
+{"AES_128_CTR", AES_128_CTR}, {"AES_192_CTR", AES_192_CTR},
+{"AES_256_CTR", AES_256_CTR}, {"AES_128_OFB", AES_128_OFB},
+{"AES_192_OFB", AES_192_OFB}, {"AES_25
[incubator-doris] branch dev-1.0.0 updated (d7f8a5d -> d2b88ed)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from d7f8a5d [feature](vec) Support update stmt in vec query engine (#8296)
new cadf19a Revert "[refactor] remove unused new_in_predicate code
(#8263)" (#8372)
new 26f55da [chore] fix build with parallel parameter only (#8352)
new 9b6de9b [fix](vectorization)Some small fix for SegmentIter
Vectorization (#8267)
new 209c5a1 [feature](vectorize)(function) support regexp&&sm4&&aes
functions (#8307)
new d2b88ed [improvement] Support show tablets stmt (#7970)
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/common/daemon.cpp | 2 +
be/src/exprs/CMakeLists.txt| 1 +
be/src/exprs/encryption_functions.cpp | 21 --
be/src/exprs/encryption_functions.h| 21 +-
be/src/exprs/new_in_predicate.cpp | 185 +++
be/src/exprs/new_in_predicate.h| 348 +
be/src/exprs/string_functions.cpp | 4 +-
be/src/exprs/string_functions.h| 3 +
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 5 +-
be/src/vec/CMakeLists.txt | 3 +
.../aggregate_function_percentile_approx.cpp | 1 -
be/src/vec/columns/column_complex.h| 16 +
be/src/vec/columns/column_vector.h | 4 +-
be/src/vec/functions/function_encryption.cpp | 245 +++
be/src/vec/functions/function_hex.cpp | 183 +++
be/src/vec/functions/function_regexp.cpp | 216 +
be/src/vec/functions/function_string.cpp | 132 +---
be/src/vec/functions/function_string.h | 21 +-
be/src/vec/functions/math.cpp | 51 ---
be/src/vec/functions/simple_function_factory.h | 6 +
be/test/vec/function/function_like_test.cpp| 59
be/test/vec/function/function_string_test.cpp | 241 +-
build.sh | 13 +-
.../Data Manipulation/SHOW TABLET.md | 13 +-
.../Data Manipulation/SHOW TABLETS.md | 56
.../Data Manipulation/SHOW TABLET.md | 33 +-
.../{SHOW TABLET.md => SHOW TABLETS.md}| 46 ++-
fe/fe-core/src/main/cup/sql_parser.cup | 4 +-
fe/fe-core/src/main/jflex/sql_scanner.flex | 1 +
gensrc/script/doris_builtins_functions.py | 74 ++---
30 files changed, 1690 insertions(+), 318 deletions(-)
create mode 100644 be/src/exprs/new_in_predicate.cpp
create mode 100644 be/src/exprs/new_in_predicate.h
create mode 100644 be/src/vec/functions/function_encryption.cpp
create mode 100644 be/src/vec/functions/function_hex.cpp
create mode 100644 be/src/vec/functions/function_regexp.cpp
create mode 100644 docs/en/sql-reference/sql-statements/Data Manipulation/SHOW
TABLETS.md
copy docs/zh-CN/sql-reference/sql-statements/Data Manipulation/{SHOW TABLET.md
=> SHOW TABLETS.md} (60%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
