[GitHub] [incubator-doris] 924060929 commented on a change in pull request #7645: [Optimize] Support adaptive runtime filter(#7546)
924060929 commented on a change in pull request #7645:
URL: https://github.com/apache/incubator-doris/pull/7645#discussion_r794276656
##
File path: be/src/exprs/runtime_filter.cpp
##
@@ -483,6 +536,63 @@ class RuntimePredicateWrapper {
_bloomfilter_func->merge(wrapper->_bloomfilter_func.get());
break;
}
+case RuntimeFilterType::IN_OR_BLOOM_FILTER: {
+auto real_filter_type = _is_bloomfilter
+ ? RuntimeFilterType::BLOOM_FILTER
+ : RuntimeFilterType::IN_FILTER;
+if (real_filter_type == RuntimeFilterType::IN_FILTER) {
+if (wrapper->_filter_type == RuntimeFilterType::IN_FILTER) {
// in merge in
+if (wrapper->_is_ignored_in_filter) {
+std::stringstream msg;
+msg << "fragment instance " <<
_fragment_instance_id.to_string()
+<< " can not ignore merge runtime filter(in filter
id "
+<< wrapper->_filter_id << ") when used
IN_OR_BLOOM_FILTER, ignore msg: "
+<< *(wrapper->get_ignored_in_filter_msg());
+return Status::InvalidArgument(msg.str());
+}
+_hybrid_set->insert(wrapper->_hybrid_set.get());
+if (_max_in_num >= 0 && _hybrid_set->size() >=
_max_in_num) {
+std::stringstream msg;
+msg << "fragment instance " <<
_fragment_instance_id.to_string()
+<< " change runtime filter to bloom filter(id=" <<
_filter_id
+<< ") because: in_num(" << _hybrid_set->size()
+<< ") >= max_in_num(" << _max_in_num << ")";
+LOG(INFO) << msg.str();
+change_to_bloom_filter();
+}
+// in merge bloom filter
+} else {
+std::stringstream msg;
+msg << "fragment instance " <<
_fragment_instance_id.to_string()
+<< " change runtime filter to bloom filter(id=" <<
_filter_id
+<< ") because: already exist a bloom filter";
+LOG(INFO) << msg.str();
Review comment:
already change to VLOG_DEBUG
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] 924060929 commented on a change in pull request #7645: [Optimize] Support adaptive runtime filter(#7546)
924060929 commented on a change in pull request #7645:
URL: https://github.com/apache/incubator-doris/pull/7645#discussion_r794276712
##
File path: be/src/exprs/runtime_filter.cpp
##
@@ -483,6 +536,63 @@ class RuntimePredicateWrapper {
_bloomfilter_func->merge(wrapper->_bloomfilter_func.get());
break;
}
+case RuntimeFilterType::IN_OR_BLOOM_FILTER: {
+auto real_filter_type = _is_bloomfilter
+ ? RuntimeFilterType::BLOOM_FILTER
+ : RuntimeFilterType::IN_FILTER;
+if (real_filter_type == RuntimeFilterType::IN_FILTER) {
+if (wrapper->_filter_type == RuntimeFilterType::IN_FILTER) {
// in merge in
+if (wrapper->_is_ignored_in_filter) {
+std::stringstream msg;
+msg << "fragment instance " <<
_fragment_instance_id.to_string()
+<< " can not ignore merge runtime filter(in filter
id "
+<< wrapper->_filter_id << ") when used
IN_OR_BLOOM_FILTER, ignore msg: "
+<< *(wrapper->get_ignored_in_filter_msg());
+return Status::InvalidArgument(msg.str());
+}
+_hybrid_set->insert(wrapper->_hybrid_set.get());
+if (_max_in_num >= 0 && _hybrid_set->size() >=
_max_in_num) {
+std::stringstream msg;
+msg << "fragment instance " <<
_fragment_instance_id.to_string()
+<< " change runtime filter to bloom filter(id=" <<
_filter_id
+<< ") because: in_num(" << _hybrid_set->size()
+<< ") >= max_in_num(" << _max_in_num << ")";
+LOG(INFO) << msg.str();
Review comment:
already change to VLOG_DEBUG
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] 924060929 commented on a change in pull request #7645: [Optimize] Support adaptive runtime filter(#7546)
924060929 commented on a change in pull request #7645:
URL: https://github.com/apache/incubator-doris/pull/7645#discussion_r794276864
##
File path: be/src/exprs/runtime_filter.cpp
##
@@ -439,9 +481,20 @@ class RuntimePredicateWrapper {
}
Status merge(const RuntimePredicateWrapper* wrapper) {
-DCHECK(_filter_type == wrapper->_filter_type);
-if (_filter_type != wrapper->_filter_type) {
-return Status::InvalidArgument("invalid filter type");
+bool can_not_merge_in_or_bloom
+= _filter_type == RuntimeFilterType::IN_OR_BLOOM_FILTER &&
+ (wrapper->_filter_type != RuntimeFilterType::IN_FILTER
+ && wrapper->_filter_type !=
RuntimeFilterType::BLOOM_FILTER);
+
+bool can_not_merge_other = _filter_type !=
RuntimeFilterType::IN_OR_BLOOM_FILTER
+ && _filter_type != wrapper->_filter_type;
+
+if (can_not_merge_in_or_bloom || can_not_merge_other) {
+std::stringstream msg;
Review comment:
already change to CHECK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] 924060929 commented on a change in pull request #7645: [Optimize] Support adaptive runtime filter(#7546)
924060929 commented on a change in pull request #7645:
URL: https://github.com/apache/incubator-doris/pull/7645#discussion_r794277036
##
File path: be/src/exprs/runtime_filter.cpp
##
@@ -351,6 +359,26 @@ class RuntimePredicateWrapper {
return Status::OK();
}
+Status change_to_bloom_filter() {
+if (_filter_type != RuntimeFilterType::IN_OR_BLOOM_FILTER) {
Review comment:
already change to return void and check by CHECK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] 924060929 commented on a change in pull request #7645: [Optimize] Support adaptive runtime filter(#7546)
924060929 commented on a change in pull request #7645:
URL: https://github.com/apache/incubator-doris/pull/7645#discussion_r794277103
##
File path: be/src/exprs/runtime_filter_slots.h
##
@@ -60,7 +60,27 @@ class RuntimeFilterSlotsBase {
runtime_filter->publish_finally();
};
-for (auto& filter_desc : _runtime_filter_descs) {
+// ordered vector: IN, IN_OR_BLOOM, others.
+// so we can ignore other filter if IN Predicate exists.
+std::vector
sorted_runtime_filter_descs(_runtime_filter_descs);
+auto compare_desc = [](TRuntimeFilterDesc& d1, TRuntimeFilterDesc& d2)
{
+if (d1.type == d2.type) {
+return 0;
+} else if (d1.type == TRuntimeFilterType::IN) {
+return -1;
+} else if (d2.type == TRuntimeFilterType::IN) {
+return 1;
+} else if (d1.type == TRuntimeFilterType::IN_OR_BLOOM) {
+return -1;
+} else if (d2.type == TRuntimeFilterType::IN_OR_BLOOM) {
+return 1;
+} else {
+return d1.type < d2.type ? -1 : 1;
+}
+};
+std::sort(sorted_runtime_filter_descs.begin(),
sorted_runtime_filter_descs.end(), compare_desc);
Review comment:
done
##
File path: be/src/exprs/runtime_filter.cpp
##
@@ -351,6 +359,26 @@ class RuntimePredicateWrapper {
return Status::OK();
}
+Status change_to_bloom_filter() {
+if (_filter_type != RuntimeFilterType::IN_OR_BLOOM_FILTER) {
+std::stringstream msg;
+msg << "Can not change to bloom filter because of runtime filter
type is "
+<< to_string(_filter_type);
+return Status::InvalidArgument(msg.str());
+}
+_is_bloomfilter = true;
+if (_hybrid_set->size() > 0) {
+auto it = _hybrid_set->begin();
+while (it->has_next()) {
+_bloomfilter_func->insert(it->get_value());
+it->next();
+}
+// release in filter
+_hybrid_set.reset(create_set(_column_return_type));
+}
+return Status::OK();
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] 924060929 commented on a change in pull request #7645: [Optimize] Support adaptive runtime filter(#7546)
924060929 commented on a change in pull request #7645: URL: https://github.com/apache/incubator-doris/pull/7645#discussion_r794277360 ## File path: docs/zh-CN/administrator-guide/runtime-filter.md ## @@ -138,6 +138,9 @@ set runtime_filter_type=7; - 默认只有右表数据行数少于1024才会下推（可通过session变量中的`runtime_filter_max_in_num`调整）。 - 目前IN predicate已实现合并方法。 - 当同时指定In predicate和其他filter，并且in的过滤数值没达到runtime_filter_max_in_num时，会尝试把其他filter去除掉。原因是In predicate是精确的过滤条件，即使没有其他filter也可以高效过滤，如果同时使用则其他filter会做无用功。目前仅在Runtime filter的生产者和消费者处于同一个fragment时才会有去除非in filter的逻辑。 +- **IN or Bloom Filter**: 根据右表的值的行数来判断使用IN predicate还是Bloom Filter +- 默认在右表数据行数少于1024时会使用IN predicate（可通过session变量中的`runtime_filter_max_in_num`调整） +- 否则使用Bloom filter - ** Review comment: 后续再加最佳实践 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] 924060929 commented on a change in pull request #7645: [Optimize] Support adaptive runtime filter(#7546)
924060929 commented on a change in pull request #7645: URL: https://github.com/apache/incubator-doris/pull/7645#discussion_r794277506 ## File path: docs/en/administrator-guide/runtime-filter.md ## @@ -138,6 +138,9 @@ set runtime_filter_type=7; - By default, only the number of data rows in the right table is less than 1024 will be pushed down (can be adjusted by `runtime_filter_max_in_num` in the session variable). - Currently IN predicate already implement a merge method. - When IN predicate and other filters are specified at the same time, and the filtering value of IN predicate does not reach runtime_filter_max_in_num will try to remove other filters. The reason is that IN predicate is an accurate filtering condition. Even if there is no other filter, it can filter efficiently. If it is used at the same time, other filters will do useless work. Currently, only when the producer and consumer of the runtime filter are in the same fragment can there be logic to remove the Non-IN predicate. +- **IN_OR_BLOOM FILTER**: Review comment: done ## File path: docs/zh-CN/administrator-guide/runtime-filter.md ## @@ -138,6 +138,9 @@ set runtime_filter_type=7; - 默认只有右表数据行数少于1024才会下推（可通过session变量中的`runtime_filter_max_in_num`调整）。 - 目前IN predicate已实现合并方法。 - 当同时指定In predicate和其他filter，并且in的过滤数值没达到runtime_filter_max_in_num时，会尝试把其他filter去除掉。原因是In predicate是精确的过滤条件，即使没有其他filter也可以高效过滤，如果同时使用则其他filter会做无用功。目前仅在Runtime filter的生产者和消费者处于同一个fragment时才会有去除非in filter的逻辑。 +- **IN or Bloom Filter**: 根据右表的值的行数来判断使用IN predicate还是Bloom Filter Review comment: done ## File path: docs/zh-CN/administrator-guide/runtime-filter.md ## @@ -138,6 +138,9 @@ set runtime_filter_type=7; - 默认只有右表数据行数少于1024才会下推（可通过session变量中的`runtime_filter_max_in_num`调整）。 - 目前IN predicate已实现合并方法。 - 当同时指定In predicate和其他filter，并且in的过滤数值没达到runtime_filter_max_in_num时，会尝试把其他filter去除掉。原因是In predicate是精确的过滤条件，即使没有其他filter也可以高效过滤，如果同时使用则其他filter会做无用功。目前仅在Runtime filter的生产者和消费者处于同一个fragment时才会有去除非in filter的逻辑。 +- **IN or Bloom Filter**: 根据右表的值的行数来判断使用IN predicate还是Bloom Filter Review comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] 924060929 commented on a change in pull request #7645: [Optimize] Support adaptive runtime filter(#7546)
924060929 commented on a change in pull request #7645:
URL: https://github.com/apache/incubator-doris/pull/7645#discussion_r794279227
##
File path:
fe/fe-core/src/test/java/org/apache/doris/qe/RuntimeFilterTypeHelperTest.java
##
@@ -50,6 +50,30 @@ public void testNormal() throws DdlException {
runtimeFilterType = "IN,BLOOM_FILTER,MIN_MAX";
Assert.assertEquals(new Long(7L),
RuntimeFilterTypeHelper.encode(runtimeFilterType));
+runtimeFilterType = "IN_OR_BLOOM_FILTER";
+Assert.assertEquals(new Long(8L),
RuntimeFilterTypeHelper.encode(runtimeFilterType));
+
+runtimeFilterType = "IN,IN_OR_BLOOM_FILTER";
+Assert.assertEquals(new Long(9L),
RuntimeFilterTypeHelper.encode(runtimeFilterType));
Review comment:
Just for test, we should guide user by doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] qidaye opened a new issue #7926: [Bug] Select into outfile occasional return error: `Empty partition info`
qidaye opened a new issue #7926: URL: https://github.com/apache/incubator-doris/issues/7926 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1ba20b1dbb4310c72f55f7834938e6ca40b21026 ### What's Wrong? When using select into outfile, the error `Empty partition info` is occasionally reported. ### What You Expected? Query normally. ### How to Reproduce? 1. Create table ```sql CREATE TABLE `t1` ( `id` int(11) NOT NULL DEFAULT "0" COMMENT "", `stream` text NOT NULL DEFAULT "" COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`id`, `stream`) COMMENT "OLAP" DISTRIBUTED BY HASH(`stream`) BUCKETS 1 PROPERTIES ( "replication_allocation" = "tag.location.default: 1", "in_memory" = "false", "storage_format" = "V2" ); ``` 2. Insert some data 3. Select into outfile ```sql select * from t1 INTO OUTFILE 'hdfs://path_to_file' FORMAT AS CSV PROPERTIES ( "broker.name" = 'xxx', "broker.username" = 'xxx', "broker.password" = 'xxx', "column_separator" = ',', "max_file_size" = '1024M' ); ``` 4. Query repeatly. ### Anything Else? It is because `DataSink::create_data_sink` creates `ResultFileSink`, and then executes `sink->init()`. `ResultFileSink` inherits from `DataStreamSender`, but does not rewrite the `init` method, which causes the method of the parent class `DataStreamSender` to be called. Because the parent class is not initialized, resulting in `_part_type` value is random, and an error will be reported occasionally. . ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] qidaye opened a new pull request #7927: [fix][query] Add init function in result_file_sink
qidaye opened a new pull request #7927: URL: https://github.com/apache/incubator-doris/pull/7927 # Proposed changes Issue Number: close #7926 Add `init` function in `result_file_sink` to fix the error `Empty partiton info`, which is occasional reported when using `SELECT INFO OUTFILE`. ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zuochunwei opened a new issue #7928: [Enhancement] cache PageDecoder's decode result instead of decompressed page data
zuochunwei opened a new issue #7928: URL: https://github.com/apache/incubator-doris/issues/7928 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description take BitShufflePageDecoder for example during the storage reading procdure: 1/3 time spent for disk read(reference IOTimer) 1/3 time for decode(got from DEBUG-timer result) 1/3 time for next_batch iterating(got from DEBUG-timer data) so, caching the data after decode but not after decompressed is more effecient, it's worth a try ### Solution _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 opened a new issue #7929: [Bug] The empty node + outer join + analytic function cause BE core
EmmyMiao87 opened a new issue #7929: URL: https://github.com/apache/incubator-doris/issues/7929 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version All versions before this pr fix ### What's Wrong? *** Aborted at 1643354333 (unix time) try "date -d @1643354333" if you are using GNU date *** PC: @ 0x146f036 doris::RowDescriptor::tuple_is_nullable() *** SIGSEGV (@0x0) received by PID 22713 (TID 0x7fe56fcf1700) from PID 0; stack trace: *** @ 0x209e942 google::(anonymous namespace)::FailureSignalHandler() @ 0x7fe6045d1630 (unknown) @ 0x146f036 doris::RowDescriptor::tuple_is_nullable() @ 0x1212d77 doris::TupleIsNullPredicate::prepare() @ 0x11da1c2 doris::Expr::prepare() @ 0x11e4820 doris::ExprContext::prepare() @ 0x11db0ae doris::Expr::prepare() @ 0x1bc3e13 doris::ResultSink::prepare_exprs() @ 0x1bc45b8 doris::ResultSink::prepare() @ 0x152fc7b doris::PlanFragmentExecutor::prepare() @ 0x14a851e doris::FragmentExecState::prepare() @ 0x14ac4ea doris::FragmentMgr::exec_plan_fragment() @ 0x14ae182 doris::FragmentMgr::exec_plan_fragment() @ 0x1575954 doris::PInternalServiceImpl<>::_exec_plan_fragment() @ 0x1575a0f doris::PInternalServiceImpl<>::exec_plan_fragment() @ 0x22115a7 brpc::policy::ProcessHttpRequest() @ 0x21de9c7 brpc::ProcessInputMessage() @ 0x21df88e brpc::InputMessenger::OnNewMessages() @ 0x2287dee brpc::Socket::ProcessEvent() @ 0x23305cf bthread::TaskGroup::task_runner() @ 0x23219d1 bthread_make_fcontext ### What You Expected? Return 0 rows ### How to Reproduce? step1: ``` CREATE TABLE `t1` ( `k1` int(11) NULL COMMENT "", `k2` int(11) NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`k1`, `k2`) COMMENT "OLAP" DISTRIBUTED BY HASH(`k1`) BUCKETS 1 PROPERTIES ( "replication_allocation" = "tag.location.default: 3", "in_memory" = "false", "storage_format" = "V2" ) ``` step2: ``` CREATE TABLE `t2` ( `j1` int(11) NULL COMMENT "", `j2` int(11) NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`j1`, `j2`) COMMENT "OLAP" DISTRIBUTED BY HASH(`j1`) BUCKETS 1 PROPERTIES ( "replication_allocation" = "tag.location.default: 3", "in_memory" = "false", "storage_format" = "V2" ) ``` step3: query and be core sometimes ``` select * from t1 left join (select max(j1) over() as x from t2)a on t1.k1=a.x where 1=1; ``` ### Anything Else? The explain of query as following: ``` MySQL [bpit]> desc verbose select * from t1 left join (select max(j1) over() as x from t2)a on t1.k1=a.x where 1=0; +--+ | Explain String | +--+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:`default_cluster:bpit`.`t1`.`k1` | `default_cluster:bpit`.`t1`.`k2` | if(TupleIsNull(1,5), NULL, ) | | PARTITION: UNPARTITIONED | | | | RESULT SINK | | | | 4:EMPTYSET | | tuple ids: 0 1 | | | ``` The empty set tuple ids is wrong: 0, 1 The right result is: 0, 1, 5 The tuple 5 is belongs to analytic function. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is
[GitHub] [incubator-doris] EmmyMiao87 opened a new pull request #7931: (fix)[planner] Fix the right tuple ids in empty set node
EmmyMiao87 opened a new pull request #7931: URL: https://github.com/apache/incubator-doris/pull/7931 # Proposed changes Issue Number: close #7929 ## Problem Summary: The tuple ids of the empty set node must be exactly the same as the tuple ids of the origin root node. In the issue, we found that once the tree where the root node is located has a window function, the tuple ids of the empty set node cannot be calculated correctly. This pr mostly fixes the problem. In order to calculate the correct tuple ids, the tuple ids obtained from the SelectStmt.getMaterializedTupleIds() function in the past are changed to directly use the tuple ids of the origin root node. Although we tried to fix #7929 by modifying the SelectStmt.getMaterializedTupleIds() function, this method can't get the tuple of the last correct window function. So we use other ways to construct tupleids of empty nodes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] spaces-X opened a new pull request #7932: [BUG] fix sql analyzed failed after increase the default bucket num
spaces-X opened a new pull request #7932: URL: https://github.com/apache/incubator-doris/pull/7932 # Proposed changes see #7910 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7931: (fix)[planner] Fix the right tuple ids in empty set node
github-actions[bot] commented on pull request #7931: URL: https://github.com/apache/incubator-doris/pull/7931#issuecomment-1024234292 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7927: [fix][query] Add init function in result_file_sink
github-actions[bot] commented on pull request #7927: URL: https://github.com/apache/incubator-doris/pull/7927#issuecomment-1024237483 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #7488: [Enhancement] Support show statement info when execute show proc '/current_queries' statement
morningman closed issue #7488: URL: https://github.com/apache/incubator-doris/issues/7488 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7487: [optimize](show) Support show statement info when execute show proc '/current_query_stmts' statement
morningman merged pull request #7487: URL: https://github.com/apache/incubator-doris/pull/7487 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [feature](show) add new statement show proc '/current_query_stmts' (#7487)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 22830ea [feature](show) add new statement show proc
'/current_query_stmts' (#7487)
22830ea is described below
commit 22830ea49899de91c7de77e37095fcf25fbf900a
Author: caiconghui <55968745+caicong...@users.noreply.github.com>
AuthorDate: Fri Jan 28 22:23:13 2022 +0800
[feature](show) add new statement show proc '/current_query_stmts' (#7487)
To show the the query statement at first level.
---
.../org/apache/doris/analysis/ShowProcStmt.java| 7 +--
.../common/proc/CurrentQueryFragmentProcNode.java | 20 ---
...ir.java => CurrentQueryStatementsProcNode.java} | 67 +-
.../common/proc/CurrentQueryStatisticsProcDir.java | 16 ++
.../org/apache/doris/common/proc/ProcService.java | 9 ++-
5 files changed, 26 insertions(+), 93 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowProcStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowProcStmt.java
index 561fbd0..fb00d8d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowProcStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowProcStmt.java
@@ -46,14 +46,9 @@ public class ShowProcStmt extends ShowStmt {
@Override
public void analyze(Analyzer analyzer) throws AnalysisException {
if
(!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(),
PrivPredicate.ADMIN)) {
-
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
-"ADMIN");
+
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
"ADMIN");
}
-
node = ProcService.getInstance().open(path);
-if (node == null) {
-ErrorReport.reportAnalysisException(ErrorCode.ERR_WRONG_PROC_PATH,
path);
-}
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryFragmentProcNode.java
b/fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryFragmentProcNode.java
index 619869e..166df29 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryFragmentProcNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryFragmentProcNode.java
@@ -19,11 +19,8 @@ package org.apache.doris.common.proc;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;
-import org.apache.doris.catalog.Catalog;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.qe.QueryStatisticsItem;
-import org.apache.doris.system.Backend;
-import org.apache.doris.thrift.TNetworkAddress;
import org.apache.doris.common.util.QueryStatisticsFormatter;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -52,23 +49,6 @@ public class CurrentQueryFragmentProcNode implements
ProcNodeInterface {
return requestFragmentExecInfos();
}
-private TNetworkAddress toBrpcHost(TNetworkAddress host) throws
AnalysisException {
-final Backend backend =
Catalog.getCurrentSystemInfo().getBackendWithBePort(
-host.getHostname(), host.getPort());
-if (backend == null) {
-throw new AnalysisException(new StringBuilder("Backend ")
-.append(host.getHostname())
-.append(":")
-.append(host.getPort())
-.append(" does not exist")
-.toString());
-}
-if (backend.getBrpcPort() < 0) {
-throw new AnalysisException("BRPC port isn't exist.");
-}
-return new TNetworkAddress(backend.getHost(), backend.getBrpcPort());
-}
-
private ProcResult requestFragmentExecInfos() throws AnalysisException {
final CurrentQueryInfoProvider provider = new
CurrentQueryInfoProvider();
final Collection
instanceStatisticsCollection
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryStatisticsProcDir.java
b/fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryStatementsProcNode.java
similarity index 51%
copy from
fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryStatisticsProcDir.java
copy to
fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryStatementsProcNode.java
index af42797..3b0474d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryStatisticsProcDir.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryStatementsProcNode.java
@@ -17,56 +17,30 @@
package org.apache.doris.common.proc;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lis
[GitHub] [incubator-doris] morningman commented on a change in pull request #7865: [fix](lateral-view) fix bugs of lateral view with CTE or where clause
morningman commented on a change in pull request #7865:
URL: https://github.com/apache/incubator-doris/pull/7865#discussion_r794547641
##
File path:
fe/fe-core/src/test/java/org/apache/doris/planner/TableFunctionPlanTest.java
##
@@ -470,4 +478,55 @@ public void aggColumnInOuterQuery() throws Exception {
Assert.assertTrue(explainString.contains("output slot id: 2"));
Assert.assertTrue(explainString.contains("tuple ids: 1 3"));
}
+
+@Test
+public void testLaterViewWithView() throws Exception {
+// test 1
+String createViewStr = "create view db1.v1 (k1,e1) as select k1,e1
from db1.table_for_view lateral view explode_split(k3,',') tmp as e1;";
+CreateViewStmt createViewStmt = (CreateViewStmt)
UtFrameUtils.parseAndAnalyzeStmt(createViewStr, ctx);
+Catalog.getCurrentCatalog().createView(createViewStmt);
+
+String sql = "select * from db1.v1;";
+String explainString = UtFrameUtils.getSQLPlanOrErrorMsg(ctx, sql,
true);
+Assert.assertTrue(explainString.contains("output slot id: 1 2"));
+// query again to see if it has error
+explainString = UtFrameUtils.getSQLPlanOrErrorMsg(ctx, sql, true);
+Assert.assertTrue(explainString.contains("output slot id: 1 2"));
+}
+
+@Test
+public void testLaterViewWithWhere() throws Exception {
+String sql = "select k1,e1 from db1.table_for_view lateral view
explode_split(k3,',') tmp as e1 where k1 in (select k2 from
db1.table_for_view);";
+String explainString = UtFrameUtils.getSQLPlanOrErrorMsg(ctx, sql,
true);
+Assert.assertTrue(explainString.contains("join op: LEFT SEMI JOIN
(BROADCAST)"));
+Assert.assertTrue(explainString.contains("equal join conjunct: `k1` =
`k2`"));
+Assert.assertTrue(!explainString.contains("equal join conjunct: `k2` =
`k2`"));
+}
+
+@Test
+public void testLaterViewWithCTE() throws Exception {
+String sql = "with tmp as (select k1,e1 from db1.table_for_view
lateral view explode_split(k3,',') tmp2 as e1) select * from tmp;";
+String explainString = UtFrameUtils.getSQLPlanOrErrorMsg(ctx, sql,
true);
+Assert.assertTrue(explainString.contains("table function:
explode_split(`default_cluster:db1`.`table_for_view`.`k3`, ',') "));
+}
+
+@Ignore
+// errCode = 2, detailMessage = Unknown column 'e1' in 'table list'
+public void testLaterViewWithCTEBug() throws Exception {
+String sql = "with tmp as (select * from db1.table_for_view where
k2=1) select k1,e1 from tmp lateral view explode_split(k3,',') tmp2 as e1;";
+String explainString = UtFrameUtils.getSQLPlanOrErrorMsg(ctx, sql,
true);
+Assert.assertTrue(!explainString.contains("Unknown column 'e1' in
'table list'"));
+}
+
+@Ignore
+// errCode = 2, detailMessage = Unknown column 'e1' in 'table list'
+public void testLaterViewUnknowColumnBug() throws Exception {
Review comment:
I leaves to you to fix it. LOL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7865: [fix](lateral-view) fix bugs of lateral view with CTE or where clause
morningman merged pull request #7865: URL: https://github.com/apache/incubator-doris/pull/7865 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #7823: [Bug] lateral view + subquery leads to the error predicate
morningman closed issue #7823: URL: https://github.com/apache/incubator-doris/issues/7823 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix](lateral-view) fix bugs of lateral view with CTE or where clause (#7865)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new f93ac89 [fix](lateral-view) fix bugs of lateral view with CTE or
where clause (#7865)
f93ac89 is described below
commit f93ac89a67765b7ced743ecaa1c511003ca4d1b6
Author: Mingyu Chen
AuthorDate: Fri Jan 28 22:24:23 2022 +0800
[fix](lateral-view) fix bugs of lateral view with CTE or where clause
(#7865)
fix bugs of lateral view with CTE or where clause.
The error case can be found in newly added tests in
`TableFunctionPlanTest.java`
But there are still some bugs not being fixed, so the unit test is
annotated with @Ignore
This PR contains the change is #7824 :
> Issue Number: close #7823
>
> After the subquery is rewritten, the rewritten stmt needs to be reset
> (that is, the content of the first analyze semantic analysis is cleared),
> and then the rewritten stmt can be reAnalyzed.
>
> The lateral view ref in the previous implementation forgot to implement
the reset function.
> This caused him to keep the first error message in the second analyze.
> Eventually, two duplicate tupleIds appear in the new stmt and are marked
with different tuple.
> From the explain string, the following syntax will have an additional
wrong join predicate.
> ```
> Query: explain select k1 from test_explode lateral view explode_split(k2,
",") tmp as e1 where k1 in (select k3 from tbl1);
> Error equal join conjunct: `k3` = `k3`
> ```
>
> This pr mainly adds the reset function of the lateral view
> to avoid possible errors in the second analyze
> when the lateral view and subquery rewrite occur at the same time.
---
.../org/apache/doris/analysis/LateralViewRef.java | 29 ++
.../java/org/apache/doris/analysis/SelectStmt.java | 11 +++-
.../java/org/apache/doris/analysis/TableRef.java | 65 +-
.../doris/planner/TableFunctionPlanTest.java | 59
4 files changed, 147 insertions(+), 17 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/LateralViewRef.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/LateralViewRef.java
index 5736d23..bee7032 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/LateralViewRef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/LateralViewRef.java
@@ -85,6 +85,11 @@ public class LateralViewRef extends TableRef {
isAnalyzed = true; // true now that we have assigned desc
}
+@Override
+public TableRef clone() {
+return new LateralViewRef(this.expr.clone(), this.viewName,
this.columnName);
+}
+
private void analyzeFunctionExpr(Analyzer analyzer) throws
AnalysisException {
fnExpr = (FunctionCallExpr) expr;
fnExpr.setTableFnCall(true);
@@ -178,5 +183,29 @@ public class LateralViewRef extends TableRef {
throw new AnalysisException("Subquery is not allowed in lateral
view");
}
}
+
+@Override
+public String toSql() {
+return "lateral view " + fnExpr.toSql() + " " + viewName + " as " +
columnName;
+}
+
+@Override
+public String toString() {
+return toSql();
+}
+
+@Override
+public void reset() {
+isAnalyzed = false;
+expr.reset();
+fnExpr = null;
+originSlotRefList = Lists.newArrayList();
+view = null;
+explodeSlotRef = null;
+// There is no need to call the reset function of @relatedTableRef
here.
+// The main reason is that @lateralViewRef itself is an attribute of
@relatedTableRef
+// The reset of @lateralViewRef happens in the reset() of
@relatedTableRef.
+}
}
+
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
index 22db67b..1c52f12 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
@@ -438,7 +438,7 @@ public class SelectStmt extends QueryStmt {
if (groupByClause != null && groupByClause.isGroupByExtension()) {
for (SelectListItem item : selectList.getItems()) {
if (item.getExpr() instanceof FunctionCallExpr &&
item.getExpr().fn instanceof AggregateFunction) {
-for (Expr expr: groupByClause.getGroupingExprs()) {
+for (Expr expr : groupByClause.getGroupingExprs()) {
if (item.getExpr().contains(expr)) {
throw new AnalysisException("column: " +
expr.toSql() + " cannot both in select list and "
+ "aggregate functions when using GROUPING
SETS
[GitHub] [incubator-doris] morningman closed issue #7877: [Enhancement] Some if conditions for comparing CatalogJournalVersion do not use the FeMetaVersion constant
morningman closed issue #7877: URL: https://github.com/apache/incubator-doris/issues/7877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [improvement](fe-meta-version)Some if conditions do not use the FeMetaVersion constant (#7879)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3a7bb7e [improvement](fe-meta-version)Some if conditions do not use
the FeMetaVersion constant (#7879)
3a7bb7e is described below
commit 3a7bb7e144e74634df039685bbe3d2263140d3f8
Author: blackstar-baba <535650...@qq.com>
AuthorDate: Fri Jan 28 22:25:17 2022 +0800
[improvement](fe-meta-version)Some if conditions do not use the
FeMetaVersion constant (#7879)
---
.../src/main/java/org/apache/doris/alter/AlterJob.java | 2 +-
.../src/main/java/org/apache/doris/catalog/MetaObject.java | 2 +-
.../src/main/java/org/apache/doris/catalog/Tablet.java | 3 ++-
.../main/java/org/apache/doris/common/FeMetaVersion.java | 14 ++
.../main/java/org/apache/doris/load/HadoopEtlJobInfo.java | 3 ++-
.../org/apache/doris/mysql/privilege/UserProperty.java | 2 +-
.../src/main/java/org/apache/doris/system/Backend.java | 4 ++--
7 files changed, 23 insertions(+), 7 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/AlterJob.java
b/fe/fe-core/src/main/java/org/apache/doris/alter/AlterJob.java
index 0371702..5b1d5e4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/AlterJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/AlterJob.java
@@ -306,7 +306,7 @@ public abstract class AlterJob implements Writable {
public synchronized void readFields(DataInput in) throws IOException {
// read common members as write in AlterJob.write().
// except 'type' member, which is read in AlterJob.read()
-if (Catalog.getCurrentCatalogJournalVersion() >= 4) {
+if (Catalog.getCurrentCatalogJournalVersion() >=
FeMetaVersion.VERSION_4) {
state = JobState.valueOf(Text.readString(in));
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MetaObject.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MetaObject.java
index b27d883..193987e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MetaObject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MetaObject.java
@@ -61,7 +61,7 @@ public class MetaObject implements Writable {
this.signature = in.readLong();
}
-if (Catalog.getCurrentCatalogJournalVersion() >= 6) {
+if (Catalog.getCurrentCatalogJournalVersion() >=
FeMetaVersion.VERSION_6) {
this.lastCheckTime = in.readLong();
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java
index 8154d6d..6a4e8a7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java
@@ -21,6 +21,7 @@ import org.apache.doris.catalog.Replica.ReplicaState;
import org.apache.doris.clone.TabletSchedCtx;
import org.apache.doris.clone.TabletSchedCtx.Priority;
import org.apache.doris.common.Config;
+import org.apache.doris.common.FeMetaVersion;
import org.apache.doris.common.Pair;
import org.apache.doris.common.io.Writable;
import org.apache.doris.resource.Tag;
@@ -341,7 +342,7 @@ public class Tablet extends MetaObject implements Writable {
}
}
-if (Catalog.getCurrentCatalogJournalVersion() >= 6) {
+if (Catalog.getCurrentCatalogJournalVersion() >=
FeMetaVersion.VERSION_6) {
checkedVersion = in.readLong();
checkedVersionHash = in.readLong();
isConsistent = in.readBoolean();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/FeMetaVersion.java
b/fe/fe-core/src/main/java/org/apache/doris/common/FeMetaVersion.java
index 371eeb0..e05ba86 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/FeMetaVersion.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/FeMetaVersion.java
@@ -18,6 +18,20 @@
package org.apache.doris.common;
public final class FeMetaVersion {
+
+public static final int VERSION_1 = 1;
+
+public static final int VERSION_2 = 2;
+
+public static final int VERSION_3 = 3;
+
+public static final int VERSION_4 = 4;
+
+public static final int VERSION_5 = 5;
+
+public static final int VERSION_6 = 6;
+
+public static final int VERSION_7 = 7;
// jira 1902
public static final int VERSION_8 = 8;
// jira 529
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/load/HadoopEtlJobInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/load/HadoopEtlJobInfo.java
index 8ada1bf..40f70d7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/HadoopEtlJobInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/HadoopEtlJobInfo.java
@@ -19,6 +19,7 @@ package org.apache.doris.load;
import org.apache.doris.catalog.Catalog;
import org.apa
[GitHub] [incubator-doris] morningman merged pull request #7879: [improvement](meta) Use meta_version constant to replace magic number when check catalog journal version
morningman merged pull request #7879: URL: https://github.com/apache/incubator-doris/pull/7879 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [docs] add function substring document (#7895)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new f9ac302 [docs] add function substring document (#7895)
f9ac302 is described below
commit f9ac30280755392b9b410bef08e0618968a4362c
Author: obobj
AuthorDate: Fri Jan 28 22:26:11 2022 +0800
[docs] add function substring document (#7895)
---
docs/.vuepress/sidebar/en.js | 1 +
docs/.vuepress/sidebar/zh-CN.js| 1 +
.../sql-functions/string-functions/substring.md| 77 ++
.../sql-functions/string-functions/substring.md| 72
4 files changed, 151 insertions(+)
diff --git a/docs/.vuepress/sidebar/en.js b/docs/.vuepress/sidebar/en.js
index 726f8f3..a6f1c52 100644
--- a/docs/.vuepress/sidebar/en.js
+++ b/docs/.vuepress/sidebar/en.js
@@ -369,6 +369,7 @@ module.exports = [
"starts_with",
"strleft",
"strright",
+ "substring",
"unhex",
{
title: "fuzzy match",
diff --git a/docs/.vuepress/sidebar/zh-CN.js b/docs/.vuepress/sidebar/zh-CN.js
index ebd0ab8..3f25bcd 100644
--- a/docs/.vuepress/sidebar/zh-CN.js
+++ b/docs/.vuepress/sidebar/zh-CN.js
@@ -373,6 +373,7 @@ module.exports = [
"starts_with",
"strleft",
"strright",
+ "substring",
"unhex",
{
title: "模糊匹配",
diff --git a/docs/en/sql-reference/sql-functions/string-functions/substring.md
b/docs/en/sql-reference/sql-functions/string-functions/substring.md
new file mode 100644
index 000..66f7b88
--- /dev/null
+++ b/docs/en/sql-reference/sql-functions/string-functions/substring.md
@@ -0,0 +1,77 @@
+---
+{
+"title": "substring",
+"language": "en"
+}
+---
+
+
+
+# substring
+## description
+### Syntax
+
+`VARCHAR substring(VARCHAR str, INT pos[, INT len])`
+
+The forms without a `len` argument return a substring from string `str`
starting at position `pos`.
+The forms with a `len` argument return a substring len characters long from
string `str`, starting at position pos.
+It is also possible to use a negative value for `pos`. In this case,
+the beginning of the substring is `pos` characters from the end of the string,
rather than the beginning.
+A negative value may be used for `pos` in any of the forms of this function.
+A value of 0 for `pos` returns an empty string.
+
+For all forms of SUBSTRING(),
+the position of the first character in the string from which the substring is
to be extracted is reckoned as 1.
+
+If len is less than 1, the result is the empty string.
+## example
+
+```
+mysql> select substring('abc1', -2);
++-+
+| substring('abc1', 2)|
++-+
+| bc1 |
++-+
+
+mysql> select substring('abc1', -2);
++-+
+| substring('abc1', -2) |
++-+
+| c1 |
++-+
+
+mysql> select substring('abc1', 5);
++-+
+| substring('abc1', 5)|
++-+
+| NULL|
++-+
+
+mysql> select substring('abc1def', 2, 2);
++-+
+| substring('abc1def', 2, 2) |
++-+
+| bc |
++-+
+```
+
+## keyword
+SUBSTRING
diff --git
a/docs/zh-CN/sql-reference/sql-functions/string-functions/substring.md
b/docs/zh-CN/sql-reference/sql-functions/string-functions/substring.md
new file mode 100644
index 000..cdbbc2b
--- /dev/null
+++ b/docs/zh-CN/sql-reference/sql-functions/string-functions/substring.md
@@ -0,0 +1,72 @@
+---
+{
+"title": "substring",
+"language": "zh-CN"
+}
+---
+
+
+
+# substring
+## description
+### Syntax
+
+`VARCHAR substring(VARCHAR str, INT pos[, INT len])`
+
+没有 `len` 参数时返回从位置 `pos` 开始的字符串 `str` 的一个子字符串，
+在有 `len` 参数时返回从位置 `pos` 开始的字符串 `str` 的一个长度为 `len` 子字符串，
+`pos` 参数可以使用负值，在这种情况下，子字符串是以字符串 `str` 末尾开始计算 `pos` 个字符，而不是开头,
+`pos` 的值为 0 返回一个空字符串。
+
+对于所有形式的 SUBSTRING()，要从中提取子字符串的字符串中第一个字符的位置为1。
+
+## example
+
+```
+mysql> select substring('abc1', -2);
++-+
+| substring('abc1', 2)|
++-+
+| bc1 |
++-+
+
+mysql> select substring('abc1', -2);
++-+
+| substring('abc1', -2) |
++-+
+| c1 |
++-+
+
+mysql> select substring('abc1', 5);
++-+
+| substring('abc1', 5)|
++-+
+| NULL
[GitHub] [incubator-doris] morningman merged pull request #7895: [docs] add function substring document
morningman merged pull request #7895: URL: https://github.com/apache/incubator-doris/pull/7895 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7898: [Benchmark] Change ssb create table column to NOT NULL
morningman merged pull request #7898: URL: https://github.com/apache/incubator-doris/pull/7898 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [refactor](benchmark) Change SSB create table column to NOT NULL (#7898)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6a89b89 [refactor](benchmark) Change SSB create table column to NOT
NULL (#7898)
6a89b89 is described below
commit 6a89b893a3f537f29ecde4025ae6eb7a5652d8a1
Author: HappenLee
AuthorDate: Fri Jan 28 22:26:55 2022 +0800
[refactor](benchmark) Change SSB create table column to NOT NULL (#7898)
---
tools/ssb-tools/create-tables.sql | 116 +++---
1 file changed, 58 insertions(+), 58 deletions(-)
diff --git a/tools/ssb-tools/create-tables.sql
b/tools/ssb-tools/create-tables.sql
index fc971b1..5e99f89 100644
--- a/tools/ssb-tools/create-tables.sql
+++ b/tools/ssb-tools/create-tables.sql
@@ -16,23 +16,23 @@
-- under the License.
CREATE TABLE `lineorder` (
- `lo_orderkey` bigint(20) NULL COMMENT "",
- `lo_linenumber` bigint(20) NULL COMMENT "",
- `lo_custkey` int(11) NULL COMMENT "",
- `lo_partkey` int(11) NULL COMMENT "",
- `lo_suppkey` int(11) NULL COMMENT "",
- `lo_orderdate` int(11) NULL COMMENT "",
- `lo_orderpriority` varchar(16) NULL COMMENT "",
- `lo_shippriority` int(11) NULL COMMENT "",
- `lo_quantity` bigint(20) NULL COMMENT "",
- `lo_extendedprice` bigint(20) NULL COMMENT "",
- `lo_ordtotalprice` bigint(20) NULL COMMENT "",
- `lo_discount` bigint(20) NULL COMMENT "",
- `lo_revenue` bigint(20) NULL COMMENT "",
- `lo_supplycost` bigint(20) NULL COMMENT "",
- `lo_tax` bigint(20) NULL COMMENT "",
- `lo_commitdate` bigint(20) NULL COMMENT "",
- `lo_shipmode` varchar(11) NULL COMMENT ""
+ `lo_orderkey` bigint(20) NOT NULL COMMENT "",
+ `lo_linenumber` bigint(20) NOT NULL COMMENT "",
+ `lo_custkey` int(11) NOT NULL COMMENT "",
+ `lo_partkey` int(11) NOT NULL COMMENT "",
+ `lo_suppkey` int(11) NOT NULL COMMENT "",
+ `lo_orderdate` int(11) NOT NULL COMMENT "",
+ `lo_orderpriority` varchar(16) NOT NULL COMMENT "",
+ `lo_shippriority` int(11) NOT NULL COMMENT "",
+ `lo_quantity` bigint(20) NOT NULL COMMENT "",
+ `lo_extendedprice` bigint(20) NOT NULL COMMENT "",
+ `lo_ordtotalprice` bigint(20) NOT NULL COMMENT "",
+ `lo_discount` bigint(20) NOT NULL COMMENT "",
+ `lo_revenue` bigint(20) NOT NULL COMMENT "",
+ `lo_supplycost` bigint(20) NOT NULL COMMENT "",
+ `lo_tax` bigint(20) NOT NULL COMMENT "",
+ `lo_commitdate` bigint(20) NOT NULL COMMENT "",
+ `lo_shipmode` varchar(11) NOT NULL COMMENT ""
)
PARTITION BY RANGE(`lo_orderdate`)
(PARTITION p1992 VALUES [("-2147483648"), ("19930101")),
@@ -48,14 +48,14 @@ PROPERTIES (
);
CREATE TABLE `customer` (
- `c_custkey` int(11) NULL COMMENT "",
- `c_name` varchar(26) NULL COMMENT "",
- `c_address` varchar(41) NULL COMMENT "",
- `c_city` varchar(11) NULL COMMENT "",
- `c_nation` varchar(16) NULL COMMENT "",
- `c_region` varchar(13) NULL COMMENT "",
- `c_phone` varchar(16) NULL COMMENT "",
- `c_mktsegment` varchar(11) NULL COMMENT ""
+ `c_custkey` int(11) NOT NULL COMMENT "",
+ `c_name` varchar(26) NOT NULL COMMENT "",
+ `c_address` varchar(41) NOT NULL COMMENT "",
+ `c_city` varchar(11) NOT NULL COMMENT "",
+ `c_nation` varchar(16) NOT NULL COMMENT "",
+ `c_region` varchar(13) NOT NULL COMMENT "",
+ `c_phone` varchar(16) NOT NULL COMMENT "",
+ `c_mktsegment` varchar(11) NOT NULL COMMENT ""
)
DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 10
PROPERTIES (
@@ -63,23 +63,23 @@ PROPERTIES (
);
CREATE TABLE `date` (
- `d_datekey` int(11) NULL COMMENT "",
- `d_date` varchar(20) NULL COMMENT "",
- `d_dayofweek` varchar(10) NULL COMMENT "",
- `d_month` varchar(11) NULL COMMENT "",
- `d_year` int(11) NULL COMMENT "",
- `d_yearmonthnum` int(11) NULL COMMENT "",
- `d_yearmonth` varchar(9) NULL COMMENT "",
- `d_daynuminweek` int(11) NULL COMMENT "",
- `d_daynuminmonth` int(11) NULL COMMENT "",
- `d_daynuminyear` int(11) NULL COMMENT "",
- `d_monthnuminyear` int(11) NULL COMMENT "",
- `d_weeknuminyear` int(11) NULL COMMENT "",
- `d_sellingseason` varchar(14) NULL COMMENT "",
- `d_lastdayinweekfl` int(11) NULL COMMENT "",
- `d_lastdayinmonthfl` int(11) NULL COMMENT "",
- `d_holidayfl` int(11) NULL COMMENT "",
- `d_weekdayfl` int(11) NULL COMMENT ""
+ `d_datekey` int(11) NOT NULL COMMENT "",
+ `d_date` varchar(20) NOT NULL COMMENT "",
+ `d_dayofweek` varchar(10) NOT NULL COMMENT "",
+ `d_month` varchar(11) NOT NULL COMMENT "",
+ `d_year` int(11) NOT NULL COMMENT "",
+ `d_yearmonthnum` int(11) NOT NULL COMMENT "",
+ `d_yearmonth` varchar(9) NOT NULL COMMENT "",
+ `d_daynuminweek` int(11) NOT NULL COMMENT "",
+ `d_daynuminmonth` int(11) NOT NULL COMMENT "",
+ `d_daynuminyear` int(11) NOT NULL COMMENT "",
+ `d_monthnuminyear` int(11) NOT NULL COMMENT "",
+ `d_weeknuminyear` int(11) NOT NULL COMMENT "",
+ `d_sellingseason` varchar(14) NOT NULL COMMENT "",
+ `d_lastdayinweekfl` int(11) NOT NULL COMMENT
[GitHub] [incubator-doris] morningman closed issue #7899: [Vectorzied][Bug] multi distinct function get wrong type
morningman closed issue #7899: URL: https://github.com/apache/incubator-doris/issues/7899 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7900: [Vectorized][Fix] fix bug multi distinct function get wrong type
morningman merged pull request #7900: URL: https://github.com/apache/incubator-doris/pull/7900 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7907: [docs] fix typo
morningman merged pull request #7907: URL: https://github.com/apache/incubator-doris/pull/7907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (6a89b89 -> 071be92)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 6a89b89 [refactor](benchmark) Change SSB create table column to NOT NULL (#7898) add 071be92 [fix](vectorized) fix bug multi distinct function get wrong type (#7900) No new revisions were added by this update. Summary of changes: be/src/vec/exec/vaggregation_node.cpp | 4 ++-- be/src/vec/exprs/vectorized_agg_fn.cpp | 2 +- be/src/vec/functions/function_conv.cpp | 2 +- fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java | 2 +- gensrc/script/doris_builtins_functions.py | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (071be92 -> 2d99041)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 071be92 [fix](vectorized) fix bug multi distinct function get wrong type (#7900) add 2d99041 [docs] fix typo (#7907) No new revisions were added by this update. Summary of changes: docs/en/faq/error.md| 2 +- docs/zh-CN/faq/error.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7912: fix(ui):fix home page spin bug #7170
github-actions[bot] commented on pull request #7912: URL: https://github.com/apache/incubator-doris/pull/7912#issuecomment-1024302481 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7857: [Fix] fix memory leak in be unit test
github-actions[bot] commented on pull request #7857: URL: https://github.com/apache/incubator-doris/pull/7857#issuecomment-1024414781 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7857: [Fix] fix memory leak in be unit test
morningman merged pull request #7857: URL: https://github.com/apache/incubator-doris/pull/7857 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #7856: [Vectorized][Fix] fix memory leak in be unit test under vectorized mode
morningman closed issue #7856: URL: https://github.com/apache/incubator-doris/issues/7856 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Fix] fix memory leak in be unit test (#7857)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new fb6e22f [Fix] fix memory leak in be unit test (#7857)
fb6e22f is described below
commit fb6e22f4ca97d5b945bf49a3c0a785bb9311db6d
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Sat Jan 29 01:00:38 2022 +0800
[Fix] fix memory leak in be unit test (#7857)
1. fix be unit test memory leak
2. ignore mindump test with ASAN test
---
be/CMakeLists.txt | 1 +
be/src/util/sm3.cpp| 4 ++
be/src/util/sm3.h | 2 +-
be/src/vec/functions/function.h| 4 ++
.../segment_v2/column_reader_writer_test.cpp | 73 ++
be/test/runtime/CMakeLists.txt | 2 +-
be/test/vec/aggregate_functions/agg_test.cpp | 3 +
be/test/vec/exprs/vexpr_test.cpp | 12
be/test/vec/function/function_bitmap_test.cpp | 21 +++
run-be-ut.sh | 2 +-
10 files changed, 96 insertions(+), 28 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index cfe4a46..15f8466 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -826,6 +826,7 @@ if (${MAKE_TEST} STREQUAL "ON")
add_subdirectory(${TEST_DIR}/udf)
add_subdirectory(${TEST_DIR}/util)
add_subdirectory(${TEST_DIR}/vec/core)
+add_subdirectory(${TEST_DIR}/vec/exec)
add_subdirectory(${TEST_DIR}/vec/exprs)
add_subdirectory(${TEST_DIR}/vec/function)
add_subdirectory(${TEST_DIR}/vec/runtime)
diff --git a/be/src/util/sm3.cpp b/be/src/util/sm3.cpp
index c2a6f8e..c124a99 100644
--- a/be/src/util/sm3.cpp
+++ b/be/src/util/sm3.cpp
@@ -27,6 +27,10 @@ SM3Digest::SM3Digest() {
EVP_DigestInit_ex(_ctx, _md, NULL);
}
+SM3Digest::~SM3Digest() {
+EVP_MD_CTX_free(_ctx);
+}
+
void SM3Digest::update(const void* data, size_t length) {
EVP_DigestUpdate(_ctx, data, length);
}
diff --git a/be/src/util/sm3.h b/be/src/util/sm3.h
index 9108410..d2e9398 100644
--- a/be/src/util/sm3.h
+++ b/be/src/util/sm3.h
@@ -28,7 +28,7 @@ namespace doris {
class SM3Digest {
public:
SM3Digest();
-
+~SM3Digest();
void update(const void* data, size_t length);
void digest();
diff --git a/be/src/vec/functions/function.h b/be/src/vec/functions/function.h
index cf00c08..0ea494a0 100644
--- a/be/src/vec/functions/function.h
+++ b/be/src/vec/functions/function.h
@@ -491,6 +491,10 @@ public:
return function->prepare(context, scope);
}
+Status close(FunctionContext* context, FunctionContext::FunctionStateScope
scope) override {
+return function->close(context, scope);
+}
+
bool is_suitable_for_constant_folding() const override {
return function->is_suitable_for_constant_folding();
}
diff --git a/be/test/olap/rowset/segment_v2/column_reader_writer_test.cpp
b/be/test/olap/rowset/segment_v2/column_reader_writer_test.cpp
index 10eaf63..7045708 100644
--- a/be/test/olap/rowset/segment_v2/column_reader_writer_test.cpp
+++ b/be/test/olap/rowset/segment_v2/column_reader_writer_test.cpp
@@ -126,31 +126,31 @@ void test_nullable_data(uint8_t* src_data, uint8_t*
src_is_null, int num_rows,
const TypeInfo* type_info = get_scalar_type_info(type);
// read and check
{
-// read and check
-ColumnReaderOptions reader_opts;
-FilePathDesc path_desc;
-path_desc.filepath = fname;
-std::unique_ptr reader;
-auto st = ColumnReader::create(reader_opts, meta, num_rows, path_desc,
&reader);
-ASSERT_TRUE(st.ok());
-
-ColumnIterator* iter = nullptr;
-st = reader->new_iterator(&iter);
-ASSERT_TRUE(st.ok());
-std::unique_ptr rblock;
-fs::BlockManager* block_manager =
fs::fs_util::block_manager(TStorageMedium::HDD);
-block_manager->open_block(path_desc, &rblock);
-
-ASSERT_TRUE(st.ok());
-ColumnIteratorOptions iter_opts;
-OlapReaderStatistics stats;
-iter_opts.stats = &stats;
-iter_opts.rblock = rblock.get();
-iter_opts.mem_tracker = std::make_shared();
-st = iter->init(iter_opts);
-ASSERT_TRUE(st.ok());
// sequence read
{
+ColumnReaderOptions reader_opts;
+FilePathDesc path_desc;
+path_desc.filepath = fname;
+std::unique_ptr reader;
+auto st = ColumnReader::create(reader_opts, meta, num_rows,
path_desc, &reader);
+ASSERT_TRUE(st.ok());
+
+ColumnIterator* iter = nullptr;
+st = reader->new_iterator(&iter);
+ASSERT_TRUE(st.ok());
+std::unique_ptr rblock;
+fs::BlockManager* blo
[incubator-doris] branch master updated: (fix)[planner] Fix the right tuple ids in empty set node (#7931)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1d900d8 (fix)[planner] Fix the right tuple ids in empty set node
(#7931)
1d900d8 is described below
commit 1d900d8605d41604762c50ea0e2eb909f0c31f27
Author: EmmyMiao87 <522274...@qq.com>
AuthorDate: Sat Jan 29 09:46:05 2022 +0800
(fix)[planner] Fix the right tuple ids in empty set node (#7931)
The tuple ids of the empty set node must be exactly the same as the tuple
ids of the origin root node.
In the issue, we found that once the tree where the root node is located
has a window function,
the tuple ids of the empty set node cannot be calculated correctly.
This pr mostly fixes the problem.
In order to calculate the correct tuple ids,
the tuple ids obtained from the SelectStmt.getMaterializedTupleIds()
function in the past
are changed to directly use the tuple ids of the origin root node.
Although we tried to fix #7929 by modifying the
SelectStmt.getMaterializedTupleIds() function,
this method can't get the tuple of the last correct window function.
So we use other ways to construct tupleids of empty nodes.
---
be/src/runtime/descriptors.cpp | 2 +-
.../java/org/apache/doris/analysis/SelectStmt.java | 5 +--
.../apache/doris/planner/SingleNodePlanner.java| 17 --
.../org/apache/doris/planner/QueryPlanTest.java| 38 ++
4 files changed, 49 insertions(+), 13 deletions(-)
diff --git a/be/src/runtime/descriptors.cpp b/be/src/runtime/descriptors.cpp
index 95d0a79..2e7041b 100644
--- a/be/src/runtime/descriptors.cpp
+++ b/be/src/runtime/descriptors.cpp
@@ -370,7 +370,7 @@ int RowDescriptor::get_row_size() const {
}
int RowDescriptor::get_tuple_idx(TupleId id) const {
-DCHECK_LT(id, _tuple_idx_map.size()) << "RowDescriptor: " <<
debug_string();
+CHECK_LT(id, _tuple_idx_map.size()) << "RowDescriptor: " << debug_string();
return _tuple_idx_map[id];
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
index 1c52f12..240d8ea 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
@@ -1732,9 +1732,10 @@ public class SelectStmt extends QueryStmt {
tupleIdList.addAll(tblRef.getMaterializedTupleIds());
}
}
-// Fixme(kks): get tuple id from analyticInfo is wrong, should get
from AnalyticEvalNode
+// Fixme(ml): get tuple id from analyticInfo is wrong, should get from
AnalyticEvalNode
+// Fixme(ml): The tuple id of AnalyticEvalNode actually is the
physical output tuple from analytic planner
// We materialize the agg tuple or the table refs together with the
analytic tuple.
-if (hasAnalyticInfo() && isEvaluateOrderBy()) {
+if (hasAnalyticInfo() && !isEvaluateOrderBy()) {
tupleIdList.add(analyticInfo.getOutputTupleId());
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
index f62f606..3b4dba7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
@@ -60,6 +60,7 @@ import org.apache.doris.common.FeConstants;
import org.apache.doris.common.Pair;
import org.apache.doris.common.Reference;
import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.VectorizedUtil;
import com.google.common.base.Preconditions;
import com.google.common.base.Predicate;
@@ -68,7 +69,6 @@ import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
-import org.apache.doris.common.util.VectorizedUtil;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -174,9 +174,11 @@ public class SingleNodePlanner {
* they are never unnested, and therefore the corresponding parent scan
should not
* materialize them.
*/
-private PlanNode createEmptyNode(QueryStmt stmt, Analyzer analyzer) {
+private PlanNode createEmptyNode(PlanNode inputPlan, QueryStmt stmt,
Analyzer analyzer) {
ArrayList tupleIds = Lists.newArrayList();
-stmt.getMaterializedTupleIds(tupleIds);
+if (inputPlan != null) {
+tupleIds = inputPlan.tupleIds;
+}
if (tupleIds.isEmpty()) {
// Constant selects do not have materialized tuples at this stage.
Preconditions.checkState(stmt instanceof SelectStmt,
@@ -298,14 +300,9 @@ public class Single
[GitHub] [incubator-doris] morningman closed issue #7929: [Bug] The empty node + outer join + analytic function cause BE core
morningman closed issue #7929: URL: https://github.com/apache/incubator-doris/issues/7929 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7931: (fix)[planner] Fix the right tuple ids in empty set node
morningman merged pull request #7931: URL: https://github.com/apache/incubator-doris/pull/7931 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7927: [fix][query] Add init function in result_file_sink
morningman merged pull request #7927: URL: https://github.com/apache/incubator-doris/pull/7927 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #7926: [Bug] Select into outfile occasional return error: `Empty partition info`
morningman closed issue #7926: URL: https://github.com/apache/incubator-doris/issues/7926 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix](query) Add init function for result_file_sink (#7927)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6a1a2a2 [fix](query) Add init function for result_file_sink (#7927)
6a1a2a2 is described below
commit 6a1a2a2ed5b7550500508199c93bb7e920799c4c
Author: qiye
AuthorDate: Sat Jan 29 10:08:57 2022 +0800
[fix](query) Add init function for result_file_sink (#7927)
Add init function in `result_file_sink` to fix the error "Empty partition
info",
which is occasional reported when using SELECT INFO OUTFILE.
---
be/src/runtime/result_file_sink.cpp | 4
be/src/runtime/result_file_sink.h | 1 +
2 files changed, 5 insertions(+)
diff --git a/be/src/runtime/result_file_sink.cpp
b/be/src/runtime/result_file_sink.cpp
index 697a452..974825b 100644
--- a/be/src/runtime/result_file_sink.cpp
+++ b/be/src/runtime/result_file_sink.cpp
@@ -70,6 +70,10 @@ ResultFileSink::~ResultFileSink() {
}
}
+Status ResultFileSink::init(const TDataSink& tsink) {
+return Status::OK();
+}
+
Status ResultFileSink::prepare_exprs(RuntimeState* state) {
// From the thrift expressions create the real exprs.
RETURN_IF_ERROR(Expr::create_expr_trees(state->obj_pool(), _t_output_expr,
&_output_expr_ctxs));
diff --git a/be/src/runtime/result_file_sink.h
b/be/src/runtime/result_file_sink.h
index f6d60ac..5ff64bc 100644
--- a/be/src/runtime/result_file_sink.h
+++ b/be/src/runtime/result_file_sink.h
@@ -49,6 +49,7 @@ public:
const std::vector& destinations,
ObjectPool* pool,
int sender_id, DescriptorTbl& descs);
virtual ~ResultFileSink();
+virtual Status init(const TDataSink& thrift_sink);
virtual Status prepare(RuntimeState* state);
virtual Status open(RuntimeState* state);
// send data in 'batch' to this backend stream mgr
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #7923: [storage]add Generic debug timer for debugging or profiling
morningman commented on pull request #7923: URL: https://github.com/apache/incubator-doris/pull/7923#issuecomment-1024808875 > Could you please add instruction for this tools in `docs/zh-CN/developer-guide/debug-tool.md` See #7924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7923: [storage]add Generic debug timer for debugging or profiling
github-actions[bot] commented on pull request #7923: URL: https://github.com/apache/incubator-doris/pull/7923#issuecomment-1024808891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7866: [fix](toolchain) Allow building when system contains libunwind
github-actions[bot] commented on pull request #7866: URL: https://github.com/apache/incubator-doris/pull/7866#issuecomment-1024810087 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui closed issue #7913: [Enhancement] Support that user can use show data skew statement instead of admin
caiconghui closed issue #7913: URL: https://github.com/apache/incubator-doris/issues/7913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui merged pull request #7914: [improvement](show) Support that user can use show data skew statement instead of admin
caiconghui merged pull request #7914: URL: https://github.com/apache/incubator-doris/pull/7914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (6a1a2a2 -> 4c7525c)
This is an automated email from the ASF dual-hosted git repository.
caiconghui pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from 6a1a2a2 [fix](query) Add init function for result_file_sink (#7927)
add 4c7525c [improvement](show) Support that user can use show data skew
statement instead of admin (#7914)
No new revisions were added by this update.
Summary of changes:
.gitignore | 5 +++
docs/.vuepress/sidebar/en.js | 2 +-
docs/.vuepress/sidebar/zh-CN.js| 2 +-
.../SHOW DATA SKEW.md} | 10 +++---
.../SHOW DATA SKEW.md} | 10 +++---
fe/fe-core/src/main/cup/sql_parser.cup | 8 ++---
.../org/apache/doris/analysis/CreateDbStmt.java| 1 -
...ShowDataSkewStmt.java => ShowDataSkewStmt.java} | 21 +--
.../org/apache/doris/catalog/MetadataViewer.java | 7 ++--
.../main/java/org/apache/doris/catalog/Tablet.java | 8 +
.../java/org/apache/doris/qe/ShowExecutor.java | 10 +++---
.../doris/analysis/AdminShowReplicaTest.java | 4 +--
.../apache/doris/analysis/CreateDbStmtTest.java| 8 ++---
.../doris/analysis/ShowTableCreationStmtTest.java | 7 ++--
.../org/apache/doris/planner/QueryPlanTest.java| 41 +++---
15 files changed, 72 insertions(+), 72 deletions(-)
rename
docs/en/sql-reference/sql-statements/{Administration/ADMIN-SHOW-DATA-SKEW.md =>
Data Manipulation/SHOW DATA SKEW.md} (85%)
rename
docs/zh-CN/sql-reference/sql-statements/{Administration/ADMIN-SHOW-DATA-SKEW.md
=> Data Manipulation/SHOW DATA SKEW.md} (84%)
rename
fe/fe-core/src/main/java/org/apache/doris/analysis/{AdminShowDataSkewStmt.java
=> ShowDataSkewStmt.java} (80%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo closed issue #7916: [Enhancement](Vectorized) Optinmize dict page decoder init
wangbo closed issue #7916: URL: https://github.com/apache/incubator-doris/issues/7916 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo merged pull request #7917: [typo](storage) Optinmize dict page decoder init
wangbo merged pull request #7917: URL: https://github.com/apache/incubator-doris/pull/7917 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix](Vectorized) optinmize dict page decoder init (#7917)
This is an automated email from the ASF dual-hosted git repository.
wangbo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new a72eaa2 [fix](Vectorized) optinmize dict page decoder init (#7917)
a72eaa2 is described below
commit a72eaa2b2e5ed00a1b8d422112e7a33cf81bb597
Author: Zeno Yang
AuthorDate: Sat Jan 29 11:47:57 2022 +0800
[fix](Vectorized) optinmize dict page decoder init (#7917)
this may cause mem leak
---
be/src/olap/rowset/segment_v2/column_reader.cpp | 20 ++--
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/be/src/olap/rowset/segment_v2/column_reader.cpp
b/be/src/olap/rowset/segment_v2/column_reader.cpp
index b336ad1..4f97218 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/column_reader.cpp
@@ -677,18 +677,18 @@ Status FileColumnIterator::_read_data_page(const
OrdinalPageIndexIterator& iter)
// PLAIN_ENCODING is supported for dict page right now
_dict_decoder.reset(new BinaryPlainPageDecoder(dict_data));
RETURN_IF_ERROR(_dict_decoder->init());
-}
-BinaryPlainPageDecoder* pd_decoder =
(BinaryPlainPageDecoder*)_dict_decoder.get();
-_dict_start_offset_array = new uint32_t[pd_decoder->_num_elems];
-_dict_len_array = new uint32_t[pd_decoder->_num_elems];
+auto* pd_decoder =
(BinaryPlainPageDecoder*)_dict_decoder.get();
+_dict_start_offset_array = new
uint32_t[pd_decoder->_num_elems];
+_dict_len_array = new uint32_t[pd_decoder->_num_elems];
-// todo(wb) padding dict value for SIMD comparison
-for (int i = 0; i < pd_decoder->_num_elems; i++) {
-const uint32_t start_offset = pd_decoder->offset(i);
-uint32_t len = pd_decoder->offset(i + 1) - start_offset;
-_dict_start_offset_array[i] = start_offset;
-_dict_len_array[i] = len;
+// todo(wb) padding dict value for SIMD comparison
+for (int i = 0; i < pd_decoder->_num_elems; i++) {
+const uint32_t start_offset = pd_decoder->offset(i);
+uint32_t len = pd_decoder->offset(i + 1) - start_offset;
+_dict_start_offset_array[i] = start_offset;
+_dict_len_array[i] = len;
+}
}
dict_page_decoder->set_dict_decoder(_dict_decoder.get(),
_dict_start_offset_array, _dict_len_array);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7903: [Enhancement] support build with libc++ && add some build config
github-actions[bot] commented on pull request #7903: URL: https://github.com/apache/incubator-doris/pull/7903#issuecomment-1024829631 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7866: [fix](toolchain) Allow building when system contains libunwind
morningman merged pull request #7866: URL: https://github.com/apache/incubator-doris/pull/7866 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix](toolchain) Allow building when system contains libunwind (#7866)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3904447 [fix](toolchain) Allow building when system contains
libunwind (#7866)
3904447 is described below
commit 3904447db87f11072f4125695a5f7c92658a1912
Author: Amos Bird
AuthorDate: Sat Jan 29 12:33:32 2022 +0800
[fix](toolchain) Allow building when system contains libunwind (#7866)
---
thirdparty/build-thirdparty.sh | 6 ++
thirdparty/patches/glog-0.4.0.patch | 12
2 files changed, 14 insertions(+), 4 deletions(-)
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index a73a4ab..eea591e 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -527,10 +527,8 @@ build_brpc() {
mkdir -p $BUILD_DIR && cd $BUILD_DIR
rm -rf CMakeCache.txt CMakeFiles/
LDFLAGS="-L${TP_LIB_DIR} -static-libstdc++ -static-libgcc" \
-${CMAKE_CMD} -G "${GENERATOR}" -DBUILD_SHARED_LIBS=0
-DCMAKE_INSTALL_PREFIX=$TP_INSTALL_DIR \
--DCMAKE_LIBRARY_PATH=$TP_INSTALL_DIR/lib64 \
--DBRPC_WITH_GLOG=ON -DWITH_GLOG=ON
-DGFLAGS_LIBRARY=$TP_INSTALL_DIR/lib/libgflags.a -DGLOG_LIB=$TP_INSTALL_DIR/lib
\
--DGFLAGS_INCLUDE_PATH=$TP_INSTALL_DIR/include
-DGLOG_LIB=$TP_INSTALL_DIR/lib/libglog.a
-DCMAKE_INCLUDE_PATH="$TP_INSTALL_DIR/include" \
+${CMAKE_CMD} -G "${GENERATOR}" -DBUILD_SHARED_LIBS=0 -DWITH_GLOG=ON
-DCMAKE_INSTALL_PREFIX=$TP_INSTALL_DIR \
+-DCMAKE_LIBRARY_PATH=$TP_INSTALL_DIR/lib64
-DCMAKE_INCLUDE_PATH="$TP_INSTALL_DIR/include" \
-DPROTOBUF_PROTOC_EXECUTABLE=$TP_INSTALL_DIR/bin/protoc ..
${BUILD_SYSTEM} -j $PARALLEL && ${BUILD_SYSTEM} install
}
diff --git a/thirdparty/patches/glog-0.4.0.patch
b/thirdparty/patches/glog-0.4.0.patch
index 12b6ea5..1ba0af2 100644
--- a/thirdparty/patches/glog-0.4.0.patch
+++ b/thirdparty/patches/glog-0.4.0.patch
@@ -338,3 +338,15 @@ diff -uprN a/src/logging.cc b/src/logging.cc
}
// Write to LOG file
+diff -uprN a/configure.ac b/configure.ac
+--- a/configure.ac 2022-01-24 22:47:23.776890869 +0800
b/configure.ac 2022-01-24 22:47:03.418891534 +0800
+@@ -38,7 +38,7 @@ AC_CHECK_HEADERS(sys/syscall.h)
+ # For backtrace with glibc.
+ AC_CHECK_HEADERS(execinfo.h)
+ # For backtrace with libunwind.
+-AC_CHECK_HEADERS(libunwind.h, ac_cv_have_libunwind_h=1,
ac_cv_have_libunwind_h=0)
++# AC_CHECK_HEADERS(libunwind.h, ac_cv_have_libunwind_h=1,
ac_cv_have_libunwind_h=0)
+ AC_CHECK_HEADERS(ucontext.h)
+ AC_CHECK_HEADERS(sys/utsname.h)
+ AC_CHECK_HEADERS(pwd.h)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo opened a new pull request #7933: [fix](storage) Using unique_ptr to refactor some class members
wangbo opened a new pull request #7933: URL: https://github.com/apache/incubator-doris/pull/7933 # Proposed changes Using unique_ptr to refactor some class members. Fix mem leak for ```SegmengIterator``` 's ```_pre_eval_block_predicate```. ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on pull request #7933: [fix](storage) Using unique_ptr to refactor some class members
wangbo commented on pull request #7933: URL: https://github.com/apache/incubator-doris/pull/7933#issuecomment-1024841849 This pr can work correctly in ```LSAN``` mode, using SSB sql. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7645: [Optimize] Support adaptive runtime filter(#7546)
github-actions[bot] commented on pull request #7645: URL: https://github.com/apache/incubator-doris/pull/7645#issuecomment-1024849158 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 opened a new pull request #7934: (improvement)[test] Combine multiple tests to use only one Doris cluster
EmmyMiao87 opened a new pull request #7934: URL: https://github.com/apache/incubator-doris/pull/7934 ## Problem Summary: This PR mainly includes the following two changes: 1. Shorten FE single measurement time In Doris's FE unit test, starting a Doris cluster is a time-consuming operation. In this PR, the unit tests of some small functions are merged into @QueryPlanTest, the same cluster is used centrally, so as to avoid the problem that the overall unit test time of FE is too long. 2. Refine the logic of "PR 7851" Although the function can be implemented correctly in PR #7851, the logic is not brief enough. This PR mainly succinct redundant code in terms of engineering implementation. Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on pull request #7934: (improvement)[test] Combine multiple tests to use only one Doris cluster
EmmyMiao87 commented on pull request #7934: URL: https://github.com/apache/incubator-doris/pull/7934#issuecomment-1024857027 #7837 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #7925: [optimization]Remove the old http module and completely replace it with http v2
morningman commented on pull request #7925: URL: https://github.com/apache/incubator-doris/pull/7925#issuecomment-1024859436 I think we can do this after next release version -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7933: [fix](storage) Using unique_ptr to refactor some class members
github-actions[bot] commented on pull request #7933: URL: https://github.com/apache/incubator-doris/pull/7933#issuecomment-1024860358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
