[GitHub] [incubator-doris] francisoliverlee closed pull request #4004: [Issue-fix-3963]Update doc for how to build fe on windows
francisoliverlee closed pull request #4004: URL: https://github.com/apache/incubator-doris/pull/4004 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] francisoliverlee commented on pull request #4004: [Issue-fix-3963]Update doc for how to build fe on windows
francisoliverlee commented on pull request #4004: URL: https://github.com/apache/incubator-doris/pull/4004#issuecomment-652861300 fix already This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] marising opened a new pull request #4005: LRU cache for sql/partition cache #2581
marising opened a new pull request #4005: URL: https://github.com/apache/incubator-doris/pull/4005 ## Features 1. Find the cache node by SQL Key, then find the corresponding partition data by Partition Key, and then decide whether to hit Cache by LastVersion and LastVersionTime 2. Refers to the classic cache algorithm LRU, which is the least recently used algorithm, using a three-layer data structure to achieve 3. The Cache elimination algorithm is implemented by ensuring the range of the partition as much as possible, to avoid the situation of partition discontinuity, which will reduce the hit rate of the Cache partition, 4. Use the two thresholds of maximum memory and elastic memory to control to avoid frequent elimination of data ## Cache fetch 1. HashMap guarantees to quickly find Cache nodes 2. Doubly linked list, put the most recently visited node at the bottom and the least visited at the top 3. The partition data under the Node node is stored in order according to the linked list, and sorted according to the partition key. Considering that the number of requested partitions will not be very large, the two ordered data are combined and the loop is used to find the partition. 4. Every access, will update the access time of the partition ## Cache update 1. Considering that the amount of updates will be relatively small, the Hash table is used here to find the corresponding partition. 2. Determine whether the updated version is higher than the existing version. If it is higher, it will be updated, otherwise it will not be updated. ## Cache pruning 1. The number below Part in the figure below represents the timestamp, and the timestamp of the most recent visit is saved 2. The entire algorithm uses a doubly linked list to find the nodes that have not been accessed recently, eliminate them, and then check them back and forth, and so on, until the memory reaches the standard 3. As shown in the figure below, find Node1 at the top, then find Part1, and eliminate, then find Part1 of Node2, and eliminate, thus eliminating the partition with timestamp 1-3 4. Node1 is cleaned up because none of the following parts  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui opened a new pull request #4006: [Feature] Support InPredicate in delete statement
caiconghui opened a new pull request #4006: URL: https://github.com/apache/incubator-doris/pull/4006 This PR is to add inPredicate support to delete statement, and add max_allowed_in_element_num_of_delete variable to limit element num of InPredicate in delete statement. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] WingsGo opened a new issue #4007: [Bug]Schema Change from int type to varchar failed because miss some logic check
WingsGo opened a new issue #4007: URL: https://github.com/apache/incubator-doris/issues/4007 **Describe the bug** Schema Change from int type to varchar failed because miss some logic check **To Reproduce** Steps to reproduce the behavior: 1. create a table with int type ``` CREATE TABLE `table2` ( `siteid` int(11) NULL DEFAULT "10" COMMENT "", `k` smallint(6) NULL COMMENT "", `v` int NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`siteid`, `k`) COMMENT "OLAP" DISTRIBUTED BY HASH(`siteid`) BUCKETS 2 PROPERTIES ( "replication_num" = "1", "in_memory" = "false", "storage_format" = "DEFAULT" ); ``` 2. do schema change `alter table table2 modify column v varchar(32); The reason why schema change failed is because in following judgement, we forget to add supported convert type in if statement. https://github.com/apache/incubator-doris/blob/5ade21b55d773b4160dfc0070f14bdc31a44981f/be/src/olap/schema_change.cpp#L340-L373 **Expected behavior** schema change to varchar type success. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui opened a new issue #4008: Delete statement need inPredicate support
caiconghui opened a new issue #4008: URL: https://github.com/apache/incubator-doris/issues/4008 **Is your feature request related to a problem? Please describe.** Now, delete stament doesn't support inPredicate. **Describe the solution you'd like** Support inPredicate in delete statement. mysql> SELECT * FROM test_alpha; +-+--+--+ | month | service_name | cluster_name | +-+--+--+ | 2020-04-01 00:00:00 | shop NN1 8 | x| | 2020-04-01 00:00:00 | shop NN1 8 | x| | 2020-04-01 00:00:00 | shop21 | x| | 2020-04-01 00:00:00 | shop22 | x| +-+--+--+ 4 rows in set (0.01 sec) mysql> SELECT * FROM test_alpha; +-+--+--+ | month | service_name | cluster_name | +-+--+--+ | 2020-04-01 00:00:00 | shop NN1 8 | x| | 2020-04-01 00:00:00 | shop NN1 8 | x| | 2020-04-01 00:00:00 | shop21 | x| | 2020-04-01 00:00:00 | shop22 | x| +-+--+--+ 4 rows in set (0.01 sec) mysql> delete from test_alpha partition p202004 where service_name not in ("shop21", "shop22"); Query OK, 0 rows affected (0.13 sec) {'label':'delete_6b3f32b8-18d2-4676-af72-64951de3a3cb', 'status':'VISIBLE', 'txnId':'10'} mysql> delete from test_beta partition p202004 where service_name not in ("shop21", "shop22"); Query OK, 0 rows affected (0.14 sec) {'label':'delete_54647f82-cabd-4f5d-9e56-efc56c23ee9c', 'status':'VISIBLE', 'txnId':'12'} mysql> select * from test_beta; +-+--+--+ | month | service_name | cluster_name | +-+--+--+ | 2020-04-01 00:00:00 | shop21 | x| | 2020-04-01 00:00:00 | shop22 | x| +-+--+--+ 2 rows in set (0.01 sec) mysql> select * from test_alpha; +-+--+--+ | month | service_name | cluster_name | +-+--+--+ | 2020-04-01 00:00:00 | shop21 | x| | 2020-04-01 00:00:00 | shop22 | x| +-+--+--+ 2 rows in set (0.01 sec) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] WingsGo opened a new pull request #4009: [Bug]Fix some schema change not work right
WingsGo opened a new pull request #4009: URL: https://github.com/apache/incubator-doris/pull/4009 This CL mainly fix some schema change to varchar type not work right because forget to logic check && Add ConvertTypeResolver to add supported convert type in order to avoid forget logic check This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] WingsGo commented on pull request #4009: [Bug]Fix some schema change not work right
WingsGo commented on pull request #4009: URL: https://github.com/apache/incubator-doris/pull/4009#issuecomment-653030178 For #4007 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on pull request #4006: [Feature] Support InPredicate in delete statement
caiconghui commented on pull request #4006: URL: https://github.com/apache/incubator-doris/pull/4006#issuecomment-653031083 for #4008 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #3985: [Doris On ES] [Bug-Fix][Refactor] Fix potential null pointer exception and refactor function process logic
morningman merged pull request #3985: URL: https://github.com/apache/incubator-doris/pull/3985 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #3984: [Bug of DOE]be accidental crashed
morningman closed issue #3984: URL: https://github.com/apache/incubator-doris/issues/3984 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Doris On ES] [Bug-Fix][Refactor] Fix potential null pointer exception and refactor function process logic (#3985)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new 1e813df [Doris On ES] [Bug-Fix][Refactor] Fix potential null pointer exception and refactor function process logic (#3985) 1e813df is described below commit 1e813df3fd3b1127d8be9449138fa087f49abc2f Author: Yunfeng,Wu AuthorDate: Thu Jul 2 22:32:16 2020 +0800 [Doris On ES] [Bug-Fix][Refactor] Fix potential null pointer exception and refactor function process logic (#3985) fix: https://github.com/apache/incubator-doris/issues/3984 1. add `conjunct.size` checking and `slot_desc nullptr` checking logic 2. For historical reasons, the function predicates are added one by one, I just refactor the processing make thelogic for function predicate processing more clearly --- be/src/exec/es/es_predicate.cpp | 177 +++- be/src/exec/es/es_predicate.h | 3 +- 2 files changed, 86 insertions(+), 94 deletions(-) diff --git a/be/src/exec/es/es_predicate.cpp b/be/src/exec/es/es_predicate.cpp index 7796201..4c799e1 100644 --- a/be/src/exec/es/es_predicate.cpp +++ b/be/src/exec/es/es_predicate.cpp @@ -226,11 +226,11 @@ static bool is_literal_node(const Expr* expr) { } Status EsPredicate::build_disjuncts_list(const Expr* conjunct) { +// process binary predicate if (TExprNodeType::BINARY_PRED == conjunct->node_type()) { if (conjunct->children().size() != 2) { return Status::InternalError("build disjuncts failed: number of childs is not 2"); } - SlotRef* slot_ref = nullptr; TExprOpcode::type op; Expr* expr = nullptr; @@ -243,7 +243,7 @@ Status EsPredicate::build_disjuncts_list(const Expr* conjunct) { if (TExprNodeType::SLOT_REF == conjunct->get_child(0)->node_type() || TExprNodeType::CAST_EXPR == conjunct->get_child(0)->node_type()) { expr = conjunct->get_child(1); -// process such as sub-query: select * from (select split_part(k, "_", 1) as new_field from case_replay_for_milimin) t where t.new_field > 1; +// process such as sub-query: select * from (select split_part(k, "_", 1) as new_field from table) t where t.new_field > 1; RETURN_ERROR_IF_EXPR_IS_NOT_SLOTREF(conjunct->get_child(0)); // process cast expr, such as: // k (float) > 2.0, k(int) > 3.2 @@ -283,93 +283,94 @@ Status EsPredicate::build_disjuncts_list(const Expr* conjunct) { _disjuncts.push_back(predicate); return Status::OK(); } - -if (is_match_func(conjunct)) { -Expr* expr = conjunct->get_child(1); -ExtLiteral literal(expr->type().type, _context->get_value(expr, NULL)); -vector query_conditions; -query_conditions.emplace_back(literal); -vector cols; //TODO -ExtPredicate* predicate = new ExtFunction( -TExprNodeType::FUNCTION_CALL, -conjunct->fn().name.function_name, -cols, -query_conditions); -if (_es_query_status.ok()) { -_es_query_status -= BooleanQueryBuilder::check_es_query(*(ExtFunction *)predicate); -if (!_es_query_status.ok()) { -delete predicate; -return _es_query_status; -} -} -_disjuncts.push_back(predicate); - -return Status::OK(); -} - -if (TExprNodeType::FUNCTION_CALL == conjunct->node_type() -&& (conjunct->fn().name.function_name == "is_null_pred" || conjunct->fn().name.function_name == "is_not_null_pred")) { -// such as sub-query: select * from (select split_part(k, "_", 1) as new_field from case_replay_for_milimin) t where t.new_field > 1; -// conjunct->get_child(0)->node_type() == TExprNodeType::FUNCTION_CALL, at present doris on es can not support push down function -RETURN_ERROR_IF_EXPR_IS_NOT_SLOTREF(conjunct->get_child(0)); -SlotRef* slot_ref = (SlotRef*)(conjunct->get_child(0)); -const SlotDescriptor* slot_desc = get_slot_desc(slot_ref); -bool is_not_null; -if (conjunct->fn().name.function_name == "is_null_pred") { -is_not_null = false; -} else { -is_not_null = true; -} -std::string col = slot_desc->col_name(); -if (_field_context.find(col) != _field_context.end()) { -col = _field_context[col]; -} -// use TExprNodeType::IS_NULL_PRED for BooleanQueryBuilder translate -ExtIsNullPredicate* predicate = new ExtIsNullPredicate(TExprNodeType::IS_NULL_PRED, col, slot_desc->type(), is_not_null); -_disjuncts.push_back(predicate); -return Status::OK(); -} - +// process func
[GitHub] [incubator-doris] morningman commented on pull request #4006: [Feature] Support InPredicate in delete statement
morningman commented on pull request #4006: URL: https://github.com/apache/incubator-doris/pull/4006#issuecomment-653043560 In fact, we are also trying to implement a batch delete function that can support the deletion of a large number of specified keys (including specifying multiple values in inPredicate) In your current implementation, if there are too many values in InPredicate (for example, tens of thousands), then each column must be looped through tens of thousands of conditions during query, which may be inefficient. Maybe you can discuss it with @yangzhg about it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #4010: [ShowIndex] Make Show Index stmt act same as MySQL behavior
morningman opened a new pull request #4010: URL: https://github.com/apache/incubator-doris/pull/4010 `SHOW INDEX FROM db2.tbl1 FROM db1;` will be same as `SHOW INDEX FROM db1.tbl1;` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #4002: [Bug] Cancel the query if OlapScanner prepare failed
morningman commented on a change in pull request #4002: URL: https://github.com/apache/incubator-doris/pull/4002#discussion_r449099844 ## File path: be/src/exec/olap_scan_node.cpp ## @@ -681,7 +681,11 @@ Status OlapScanNode::start_scan_thread(RuntimeState* state) { } OlapScanner* scanner = new OlapScanner( state, this, _olap_scan_node.is_preaggregation, _need_agg_finalize, *scan_range, scanner_ranges); +// add scanner to pool before doing prepare. +// so that scanner can be automatically deconstructed if prepare failed. _scanner_pool->add(scanner); Review comment: Why? the `scanner` here is created by `new` operator. How can it be deconstructed automatically? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #4005: LRU cache for sql/partition cache #2581
morningman commented on a change in pull request #4005: URL: https://github.com/apache/incubator-doris/pull/4005#discussion_r449113739 ## File path: be/src/common/config.h ## @@ -536,6 +536,15 @@ namespace config { // Whether to continue to start be when load tablet from header failed. CONF_Bool(ignore_load_tablet_failure, "false"); +// Set max cache's size of query results, the unit is M byte +CONF_Int32(cache_max_size, "256"); Review comment: ```suggestion CONF_Int32(query_cache_max_size_mb, "256"); ``` same suggestion for other 2 configs. ## File path: be/src/util/doris_metrics.cpp ## @@ -161,6 +158,10 @@ DorisMetrics::DorisMetrics() : _name("doris_be"), _hook_name("doris_metrics"), _ REGISTER_DORIS_METRIC(tablet_cumulative_max_compaction_score); REGISTER_DORIS_METRIC(tablet_base_max_compaction_score); +REGISTER_DORIS_METRIC(cache_memory_total); Review comment: ```suggestion REGISTER_DORIS_METRIC(query_cache_memory_total_mb); ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] liutang123 opened a new pull request #4011: Remove rs when version conflict when insert to a tablet
liutang123 opened a new pull request #4011: URL: https://github.com/apache/incubator-doris/pull/4011 For #3976 In this PR, when insert to a tablet, if the tablet has a different rs with the same version, remove it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] liutang123 commented on pull request #4011: Remove rs when version conflict when insert to a tablet
liutang123 commented on pull request #4011: URL: https://github.com/apache/incubator-doris/pull/4011#issuecomment-653102312 @chaoyli This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] liutang123 edited a comment on pull request #4011: Remove rs when version conflict when insert to a tablet
liutang123 edited a comment on pull request #4011: URL: https://github.com/apache/incubator-doris/pull/4011#issuecomment-653102312 @chaoyli Now, should we prohibit insertion when `OLAP_ERR_PUSH_VERSION_ALREADY_EXIST` appears? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] marising commented on a change in pull request #4005: LRU cache for sql/partition cache #2581
marising commented on a change in pull request #4005: URL: https://github.com/apache/incubator-doris/pull/4005#discussion_r449336062 ## File path: be/src/util/doris_metrics.cpp ## @@ -161,6 +158,10 @@ DorisMetrics::DorisMetrics() : _name("doris_be"), _hook_name("doris_metrics"), _ REGISTER_DORIS_METRIC(tablet_cumulative_max_compaction_score); REGISTER_DORIS_METRIC(tablet_base_max_compaction_score); +REGISTER_DORIS_METRIC(cache_memory_total); Review comment: query_cache_memory_total is byte,REGISTER_DORIS_METRIC(query_cache_memory_total); This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (1e813df -> a16236f)
This is an automated email from the ASF dual-hosted git repository. lichaoyong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 1e813df [Doris On ES] [Bug-Fix][Refactor] Fix potential null pointer exception and refactor function process logic (#3985) add a16236f [refactor] Remove useless return value of class RowsetGraph (#3977) No new revisions were added by this update. Summary of changes: be/src/olap/rowset_graph.cpp | 38 ++ be/src/olap/rowset_graph.h | 8 be/src/olap/tablet.cpp | 4 ++-- 3 files changed, 16 insertions(+), 34 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli merged pull request #3977: [refactor] Remove useless return value of class RowsetGraph
chaoyli merged pull request #3977: URL: https://github.com/apache/incubator-doris/pull/3977 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli merged pull request #3990: [shell] Fix BUILD_TYPE not used bug
chaoyli merged pull request #3990: URL: https://github.com/apache/incubator-doris/pull/3990 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (a16236f -> ab325f5)
This is an automated email from the ASF dual-hosted git repository. lichaoyong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from a16236f [refactor] Remove useless return value of class RowsetGraph (#3977) add ab325f5 [shell] Fix BUILD_TYPE not used bug (#3990) No new revisions were added by this update. Summary of changes: run-ut.sh | 13 - 1 file changed, 8 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wuyunfeng opened a new pull request #4012: [Doris On ES][refactor] refactor and enchanment ES sync meta logic
wuyunfeng opened a new pull request #4012: URL: https://github.com/apache/incubator-doris/pull/4012 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request #4011: Remove rs when version conflict when insert to a tablet
chaoyli commented on a change in pull request #4011: URL: https://github.com/apache/incubator-doris/pull/4011#discussion_r449348960 ## File path: be/src/olap/tablet.cpp ## @@ -826,6 +832,33 @@ void Tablet::_print_missed_versions(const std::vector& missed_versions) LOG(WARNING) << ss.str(); } +bool Tablet::_drop_rowset_if_version_conflict(const RowsetSharedPtr& rowset) { Review comment: There has a problem in this place. modify_rowsets() only change the TabletMeta in memory. But not change the meta in disk. So you should perform do_tablet_meta_checkpoint to save the meta and remove the rowsetmeta in RowsetMetaManager. ## File path: be/src/olap/tablet.cpp ## @@ -826,6 +832,33 @@ void Tablet::_print_missed_versions(const std::vector& missed_versions) LOG(WARNING) << ss.str(); } +bool Tablet::_drop_rowset_if_version_conflict(const RowsetSharedPtr& rowset) { +if (rowset == nullptr) { +return false; +} +Version insert_version = {rowset->start_version(), rowset->end_version()}; +Version max_version = _tablet_meta->max_version(); +if (insert_version != max_version) { +return false; +} + +RowsetSharedPtr exist_rs = get_rowset_by_version(insert_version); +// if a rs is created by doris 0.10 exists, remove it. +if (exist_rs != nullptr) { +vector to_add; +vector to_delete; +to_delete.push_back(exist_rs); +modify_rowsets(to_add, to_delete); +std::stringstream ss; +ss << "Remove rs " << exist_rs->rowset_id().to_string(); Review comment: LOG(WARNING) << "Remove rs " << exist_rs->rowset_id().to_string() << " in " << full_name() << " for version conflict with " << rowset->rowset_id().to_string() There is no necessity to use std::stringstream. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on pull request #4011: Remove rs when version conflict when insert to a tablet
chaoyli commented on pull request #4011: URL: https://github.com/apache/incubator-doris/pull/4011#issuecomment-653307162 > @chaoyli Now, should we prohibit insertion when `OLAP_ERR_PUSH_VERSION_ALREADY_EXIST` appears? Your function will add little cost to remove OLAP_ERR_PUSH_VERSION_ALREADY_EXIST. The cost will reflect on save the meta. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on pull request #4009: [Bug]Fix some schema change not work right
chaoyli commented on pull request #4009: URL: https://github.com/apache/incubator-doris/pull/4009#issuecomment-653312988 It's better add more discrete message in commit info like you issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] spaces-X commented on a change in pull request #4000: applyInnerJoinConditionReorganizeRule
spaces-X commented on a change in pull request #4000: URL: https://github.com/apache/incubator-doris/pull/4000#discussion_r449353484 ## File path: fe/src/main/java/org/apache/doris/analysis/SelectStmt.java ## @@ -816,9 +816,103 @@ protected void reorderTable(Analyzer analyzer) throws AnalysisException { // can not get AST only with equal join, MayBe cross join can help fromClause_.clear(); +List> misMatchJoinConditions = Lists.newArrayList(); +List tupleIds = Lists.newArrayList(); + for (Pair candidate : candidates) { -fromClause_.add(candidate.first); +TableRef tableRef = candidate.first; +fromClause_.add(tableRef); +tupleIds.add(tableRef.desc.getId()); +applyInnerJoinConditionReorganizeRule(tableRef, tupleIds, misMatchJoinConditions); +} + +Preconditions.checkState(misMatchJoinConditions.isEmpty()); +} + +/** + * After adjusting the order of tables, it is necessary to reselect the join condition of each table, + * otherwise some columns will not be recognized by the current table space + * @param tableRef: + * @param tupleIds: TableSpaces visible up to now + * @param misMatchJoinConditions: join conditions no matching yet, the first is fatherExpr, the second is sonExpr need to replace + */ +void applyInnerJoinConditionReorganizeRule(TableRef tableRef, List tupleIds, +List> misMatchJoinConditions) { +List> tempMisMatchConjuncts = Lists.newArrayList(); + +// Extract all join conditions from joinOnClause to misMatchJoinConditions, +// which cannot be bound by existing tableSpaces(tupleIds) +findUnboundedJoinConditions(tableRef.getOnClause(), tupleIds, misMatchJoinConditions); + +for (Pair exprPair : misMatchJoinConditions) { +Expr father = exprPair.first; +Expr expr = exprPair.second; +if (!expr.isBoundByTupleIds(tupleIds)) { +tempMisMatchConjuncts.add(new Pair(null, expr)); +if (father.getChild(0) == expr) { Review comment: father may be null here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] WingsGo commented on pull request #4009: [Bug]Fix some schema change not work right
WingsGo commented on pull request #4009: URL: https://github.com/apache/incubator-doris/pull/4009#issuecomment-653323967 In order to make review easiler, I will not add code format until review ended. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] WingsGo commented on pull request #4009: [Bug]Fix some schema change not work right
WingsGo commented on pull request #4009: URL: https://github.com/apache/incubator-doris/pull/4009#issuecomment-653323183 > It's better add more discrete message in commit info like you issue. Ok, done, and I tested the schema change supported in doc, but I found that convert LargetInt to Double is not support in code implement, maybe you can confirm that why it is added to doc in case It is supported but there is some bug exists? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] WingsGo edited a comment on pull request #4009: [Bug]Fix some schema change not work right
WingsGo edited a comment on pull request #4009: URL: https://github.com/apache/incubator-doris/pull/4009#issuecomment-653323183 > It's better add more discrete message in commit info like you issue. Ok, done, and I tested the schema change supported in doc, but I found that convert LargetInt to Double is not support in code implement, maybe you can help me confirm that why it is added to doc in case It is supported but there is some bug exists? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] funyeah commented on issue #3085: use datagrip connect fe error
funyeah commented on issue #3085: URL: https://github.com/apache/incubator-doris/issues/3085#issuecomment-653348056  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org