[GitHub] [incubator-doris] francisoliverlee closed pull request #4004: [Issue-fix-3963]Update doc for how to build fe on windows

2020-07-02 Thread GitBox


francisoliverlee closed pull request #4004:
URL: https://github.com/apache/incubator-doris/pull/4004


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] francisoliverlee commented on pull request #4004: [Issue-fix-3963]Update doc for how to build fe on windows

2020-07-02 Thread GitBox


francisoliverlee commented on pull request #4004:
URL: https://github.com/apache/incubator-doris/pull/4004#issuecomment-652861300


   fix already



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] marising opened a new pull request #4005: LRU cache for sql/partition cache #2581

2020-07-02 Thread GitBox


marising opened a new pull request #4005:
URL: https://github.com/apache/incubator-doris/pull/4005


   ## Features
   1. Find the cache node by SQL Key, then find the corresponding partition 
data by Partition Key, and then decide whether to hit Cache by LastVersion and 
LastVersionTime
   2. Refers to the classic cache algorithm LRU, which is the least recently 
used algorithm, using a three-layer data structure to achieve
   3. The Cache elimination algorithm is implemented by ensuring the range of 
the partition as much as possible, to avoid the situation of partition 
discontinuity, which will reduce the hit rate of the Cache partition,
   4. Use the two thresholds of maximum memory and elastic memory to control to 
avoid frequent elimination of data
   
   ## Cache fetch
   1. HashMap guarantees to quickly find Cache nodes
   2. Doubly linked list, put the most recently visited node at the bottom and 
the least visited at the top
   3. The partition data under the Node node is stored in order according to 
the linked list, and sorted according to the partition key. Considering that 
the number of requested partitions will not be very large, the two ordered data 
are combined and the loop is used to find the partition.
   4. Every access, will update the access time of the partition
   
   ## Cache update
   1. Considering that the amount of updates will be relatively small, the Hash 
table is used here to find the corresponding partition. 
   2. Determine whether the updated version is higher than the existing 
version. If it is higher, it will be updated, otherwise it will not be updated.
   
   ## Cache pruning
   1. The number below Part in the figure below represents the timestamp, and 
the timestamp of the most recent visit is saved
   2. The entire algorithm uses a doubly linked list to find the nodes that 
have not been accessed recently, eliminate them, and then check them back and 
forth, and so on, until the memory reaches the standard
   3. As shown in the figure below, find Node1 at the top, then find Part1, and 
eliminate, then find Part1 of Node2, and eliminate, thus eliminating the 
partition with timestamp 1-3
   4. Node1 is cleaned up because none of the following parts
   
   
![image](https://user-images.githubusercontent.com/8611398/86345693-730f9380-bc8e-11ea-909c-5cf8702f75bc.png)
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] caiconghui opened a new pull request #4006: [Feature] Support InPredicate in delete statement

2020-07-02 Thread GitBox


caiconghui opened a new pull request #4006:
URL: https://github.com/apache/incubator-doris/pull/4006


   This PR is to add inPredicate support to delete statement, and add 
max_allowed_in_element_num_of_delete variable to limit element num of 
InPredicate in delete statement. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] WingsGo opened a new issue #4007: [Bug]Schema Change from int type to varchar failed because miss some logic check

2020-07-02 Thread GitBox


WingsGo opened a new issue #4007:
URL: https://github.com/apache/incubator-doris/issues/4007


   **Describe the bug**
   Schema Change from int type to varchar failed because miss some logic check
   
   **To Reproduce**
   Steps to reproduce the behavior:
   1. create a table with int type
   
   ```
   CREATE TABLE `table2` (
 `siteid` int(11) NULL DEFAULT "10" COMMENT "",
 `k` smallint(6) NULL COMMENT "",
 `v` int NULL COMMENT ""
   ) ENGINE=OLAP
   DUPLICATE KEY(`siteid`, `k`)
   COMMENT "OLAP"
   DISTRIBUTED BY HASH(`siteid`) BUCKETS 2
   PROPERTIES (
   "replication_num" = "1",
   "in_memory" = "false",
   "storage_format" = "DEFAULT"
   );
   ```
   
   2. do schema change `alter table table2 modify column v varchar(32);
   
   The reason why schema change failed is because in following judgement, we 
forget to add supported convert type in if statement.
   
   
https://github.com/apache/incubator-doris/blob/5ade21b55d773b4160dfc0070f14bdc31a44981f/be/src/olap/schema_change.cpp#L340-L373
   
   **Expected behavior**
   schema change to varchar type success.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] caiconghui opened a new issue #4008: Delete statement need inPredicate support

2020-07-02 Thread GitBox


caiconghui opened a new issue #4008:
URL: https://github.com/apache/incubator-doris/issues/4008


   **Is your feature request related to a problem? Please describe.**
   Now, delete stament doesn't support inPredicate.
   
   **Describe the solution you'd like**
   Support inPredicate in delete statement.
   mysql> SELECT * FROM test_alpha;
   +-+--+--+
   | month   | service_name | cluster_name |
   +-+--+--+
   | 2020-04-01 00:00:00 | shop NN1 8   | x|
   | 2020-04-01 00:00:00 | shop NN1 8   | x|
   | 2020-04-01 00:00:00 | shop21   | x|
   | 2020-04-01 00:00:00 | shop22   | x|
   +-+--+--+
   4 rows in set (0.01 sec)
   
   mysql> SELECT * FROM test_alpha;
   +-+--+--+
   | month   | service_name | cluster_name |
   +-+--+--+
   | 2020-04-01 00:00:00 | shop NN1 8   | x|
   | 2020-04-01 00:00:00 | shop NN1 8   | x|
   | 2020-04-01 00:00:00 | shop21   | x|
   | 2020-04-01 00:00:00 | shop22   | x|
   +-+--+--+
   4 rows in set (0.01 sec)
   
   mysql> delete from test_alpha partition p202004 where service_name not in 
("shop21", "shop22");
   Query OK, 0 rows affected (0.13 sec)
   {'label':'delete_6b3f32b8-18d2-4676-af72-64951de3a3cb', 'status':'VISIBLE', 
'txnId':'10'}
   
   mysql> delete from test_beta partition p202004 where service_name not in 
("shop21", "shop22");
   Query OK, 0 rows affected (0.14 sec)
   {'label':'delete_54647f82-cabd-4f5d-9e56-efc56c23ee9c', 'status':'VISIBLE', 
'txnId':'12'}
   
   mysql> select * from test_beta;
   +-+--+--+
   | month   | service_name | cluster_name |
   +-+--+--+
   | 2020-04-01 00:00:00 | shop21   | x|
   | 2020-04-01 00:00:00 | shop22   | x|
   +-+--+--+
   2 rows in set (0.01 sec)
   
   mysql> select * from test_alpha;
   +-+--+--+
   | month   | service_name | cluster_name |
   +-+--+--+
   | 2020-04-01 00:00:00 | shop21   | x|
   | 2020-04-01 00:00:00 | shop22   | x|
   +-+--+--+
   2 rows in set (0.01 sec)
   
   
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] WingsGo opened a new pull request #4009: [Bug]Fix some schema change not work right

2020-07-02 Thread GitBox


WingsGo opened a new pull request #4009:
URL: https://github.com/apache/incubator-doris/pull/4009


   This CL mainly fix some schema change to varchar type not work right
   because forget to logic check && Add ConvertTypeResolver to add
   supported convert type in order to avoid forget logic check



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] WingsGo commented on pull request #4009: [Bug]Fix some schema change not work right

2020-07-02 Thread GitBox


WingsGo commented on pull request #4009:
URL: https://github.com/apache/incubator-doris/pull/4009#issuecomment-653030178


   For #4007 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] caiconghui commented on pull request #4006: [Feature] Support InPredicate in delete statement

2020-07-02 Thread GitBox


caiconghui commented on pull request #4006:
URL: https://github.com/apache/incubator-doris/pull/4006#issuecomment-653031083


   for #4008  



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] morningman merged pull request #3985: [Doris On ES] [Bug-Fix][Refactor] Fix potential null pointer exception and refactor function process logic

2020-07-02 Thread GitBox


morningman merged pull request #3985:
URL: https://github.com/apache/incubator-doris/pull/3985


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] morningman closed issue #3984: [Bug of DOE]be accidental crashed

2020-07-02 Thread GitBox


morningman closed issue #3984:
URL: https://github.com/apache/incubator-doris/issues/3984


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated: [Doris On ES] [Bug-Fix][Refactor] Fix potential null pointer exception and refactor function process logic (#3985)

2020-07-02 Thread morningman
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 1e813df  [Doris On ES] [Bug-Fix][Refactor] Fix potential null pointer 
exception and refactor function process logic (#3985)
1e813df is described below

commit 1e813df3fd3b1127d8be9449138fa087f49abc2f
Author: Yunfeng,Wu 
AuthorDate: Thu Jul 2 22:32:16 2020 +0800

[Doris On ES] [Bug-Fix][Refactor] Fix potential null pointer exception and 
refactor function process logic (#3985)

fix: https://github.com/apache/incubator-doris/issues/3984

1. add `conjunct.size` checking and `slot_desc nullptr` checking logic
2. For historical reasons, the function predicates are added one by one, I 
just refactor the processing make thelogic for function predicate processing 
more clearly
---
 be/src/exec/es/es_predicate.cpp | 177 +++-
 be/src/exec/es/es_predicate.h   |   3 +-
 2 files changed, 86 insertions(+), 94 deletions(-)

diff --git a/be/src/exec/es/es_predicate.cpp b/be/src/exec/es/es_predicate.cpp
index 7796201..4c799e1 100644
--- a/be/src/exec/es/es_predicate.cpp
+++ b/be/src/exec/es/es_predicate.cpp
@@ -226,11 +226,11 @@ static bool is_literal_node(const Expr* expr) {
 }
 
 Status EsPredicate::build_disjuncts_list(const Expr* conjunct) {
+// process binary predicate
 if (TExprNodeType::BINARY_PRED == conjunct->node_type()) {
 if (conjunct->children().size() != 2) {
 return Status::InternalError("build disjuncts failed: number of 
childs is not 2");
 }
-
 SlotRef* slot_ref = nullptr;
 TExprOpcode::type op;
 Expr* expr = nullptr;
@@ -243,7 +243,7 @@ Status EsPredicate::build_disjuncts_list(const Expr* 
conjunct) {
 if (TExprNodeType::SLOT_REF == conjunct->get_child(0)->node_type()
 || TExprNodeType::CAST_EXPR == 
conjunct->get_child(0)->node_type()) {
 expr = conjunct->get_child(1);
-// process such as sub-query: select * from (select split_part(k, 
"_", 1) as new_field from case_replay_for_milimin) t where t.new_field > 1;
+// process such as sub-query: select * from (select split_part(k, 
"_", 1) as new_field from table) t where t.new_field > 1;
 RETURN_ERROR_IF_EXPR_IS_NOT_SLOTREF(conjunct->get_child(0));
 // process cast expr, such as:
 // k (float) > 2.0, k(int) > 3.2
@@ -283,93 +283,94 @@ Status EsPredicate::build_disjuncts_list(const Expr* 
conjunct) {
 _disjuncts.push_back(predicate);
 return Status::OK();
 }
-
-if (is_match_func(conjunct)) {
-Expr* expr = conjunct->get_child(1);
-ExtLiteral literal(expr->type().type, _context->get_value(expr, NULL));
-vector query_conditions;
-query_conditions.emplace_back(literal);
-vector cols; //TODO
-ExtPredicate* predicate = new ExtFunction(
-TExprNodeType::FUNCTION_CALL,
-conjunct->fn().name.function_name,
-cols,
-query_conditions);
-if (_es_query_status.ok()) {
-_es_query_status 
-= BooleanQueryBuilder::check_es_query(*(ExtFunction 
*)predicate); 
-if (!_es_query_status.ok()) {
-delete predicate;
-return _es_query_status;
-}
-}
-_disjuncts.push_back(predicate);
-
-return Status::OK();
-}
-
-if (TExprNodeType::FUNCTION_CALL == conjunct->node_type()
-&& (conjunct->fn().name.function_name == "is_null_pred" || 
conjunct->fn().name.function_name == "is_not_null_pred")) {
-// such as sub-query: select * from (select split_part(k, "_", 1) as 
new_field from case_replay_for_milimin) t where t.new_field > 1;
-// conjunct->get_child(0)->node_type() == 
TExprNodeType::FUNCTION_CALL, at present doris on es can not support push down 
function 
-RETURN_ERROR_IF_EXPR_IS_NOT_SLOTREF(conjunct->get_child(0));
-SlotRef* slot_ref = (SlotRef*)(conjunct->get_child(0));
-const SlotDescriptor* slot_desc = get_slot_desc(slot_ref);
-bool is_not_null;
-if (conjunct->fn().name.function_name == "is_null_pred") {
-is_not_null = false;
-} else {
-is_not_null = true;
-}
-std::string col = slot_desc->col_name();
-if (_field_context.find(col) != _field_context.end()) {
-col = _field_context[col];
-}
-// use TExprNodeType::IS_NULL_PRED for BooleanQueryBuilder translate
-ExtIsNullPredicate* predicate = new 
ExtIsNullPredicate(TExprNodeType::IS_NULL_PRED, col, slot_desc->type(), 
is_not_null);
-_disjuncts.push_back(predicate);
-return Status::OK();
-}
-
+// process func

[GitHub] [incubator-doris] morningman commented on pull request #4006: [Feature] Support InPredicate in delete statement

2020-07-02 Thread GitBox


morningman commented on pull request #4006:
URL: https://github.com/apache/incubator-doris/pull/4006#issuecomment-653043560


   In fact, we are also trying to implement a batch delete function that can 
support the deletion of a large number of specified keys (including specifying 
multiple values in inPredicate)
   
   In your current implementation, if there are too many values in InPredicate 
(for example, tens of thousands), then each column must be looped through tens 
of thousands of conditions during query, which may be inefficient.
   
   Maybe you can discuss it with @yangzhg about it.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] morningman opened a new pull request #4010: [ShowIndex] Make Show Index stmt act same as MySQL behavior

2020-07-02 Thread GitBox


morningman opened a new pull request #4010:
URL: https://github.com/apache/incubator-doris/pull/4010


   `SHOW INDEX FROM db2.tbl1 FROM db1;` will be same as
   `SHOW INDEX FROM db1.tbl1;`



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] morningman commented on a change in pull request #4002: [Bug] Cancel the query if OlapScanner prepare failed

2020-07-02 Thread GitBox


morningman commented on a change in pull request #4002:
URL: https://github.com/apache/incubator-doris/pull/4002#discussion_r449099844



##
File path: be/src/exec/olap_scan_node.cpp
##
@@ -681,7 +681,11 @@ Status OlapScanNode::start_scan_thread(RuntimeState* 
state) {
 }
 OlapScanner* scanner = new OlapScanner(
 state, this, _olap_scan_node.is_preaggregation, 
_need_agg_finalize, *scan_range, scanner_ranges);
+// add scanner to pool before doing prepare.
+// so that scanner can be automatically deconstructed if prepare 
failed.
 _scanner_pool->add(scanner);

Review comment:
   Why? the `scanner` here is created by `new` operator. How can it be 
deconstructed automatically?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] morningman commented on a change in pull request #4005: LRU cache for sql/partition cache #2581

2020-07-02 Thread GitBox


morningman commented on a change in pull request #4005:
URL: https://github.com/apache/incubator-doris/pull/4005#discussion_r449113739



##
File path: be/src/common/config.h
##
@@ -536,6 +536,15 @@ namespace config {
 // Whether to continue to start be when load tablet from header failed.
 CONF_Bool(ignore_load_tablet_failure, "false");
 
+// Set max cache's size of query results, the unit is M byte
+CONF_Int32(cache_max_size, "256"); 

Review comment:
   ```suggestion
   CONF_Int32(query_cache_max_size_mb, "256"); 
   ```
   
   same suggestion for other 2 configs.

##
File path: be/src/util/doris_metrics.cpp
##
@@ -161,6 +158,10 @@ DorisMetrics::DorisMetrics() : _name("doris_be"), 
_hook_name("doris_metrics"), _
 REGISTER_DORIS_METRIC(tablet_cumulative_max_compaction_score);
 REGISTER_DORIS_METRIC(tablet_base_max_compaction_score);
 
+REGISTER_DORIS_METRIC(cache_memory_total);

Review comment:
   ```suggestion
   REGISTER_DORIS_METRIC(query_cache_memory_total_mb);
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] liutang123 opened a new pull request #4011: Remove rs when version conflict when insert to a tablet

2020-07-02 Thread GitBox


liutang123 opened a new pull request #4011:
URL: https://github.com/apache/incubator-doris/pull/4011


   For #3976 
   In this PR, when insert to a tablet, if the tablet has a different rs with 
the same version, remove it.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] liutang123 commented on pull request #4011: Remove rs when version conflict when insert to a tablet

2020-07-02 Thread GitBox


liutang123 commented on pull request #4011:
URL: https://github.com/apache/incubator-doris/pull/4011#issuecomment-653102312


   @chaoyli 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] liutang123 edited a comment on pull request #4011: Remove rs when version conflict when insert to a tablet

2020-07-02 Thread GitBox


liutang123 edited a comment on pull request #4011:
URL: https://github.com/apache/incubator-doris/pull/4011#issuecomment-653102312


   @chaoyli Now, should we prohibit insertion when 
`OLAP_ERR_PUSH_VERSION_ALREADY_EXIST` appears?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] marising commented on a change in pull request #4005: LRU cache for sql/partition cache #2581

2020-07-02 Thread GitBox


marising commented on a change in pull request #4005:
URL: https://github.com/apache/incubator-doris/pull/4005#discussion_r449336062



##
File path: be/src/util/doris_metrics.cpp
##
@@ -161,6 +158,10 @@ DorisMetrics::DorisMetrics() : _name("doris_be"), 
_hook_name("doris_metrics"), _
 REGISTER_DORIS_METRIC(tablet_cumulative_max_compaction_score);
 REGISTER_DORIS_METRIC(tablet_base_max_compaction_score);
 
+REGISTER_DORIS_METRIC(cache_memory_total);

Review comment:
   query_cache_memory_total is  
byte,REGISTER_DORIS_METRIC(query_cache_memory_total); 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated (1e813df -> a16236f)

2020-07-02 Thread lichaoyong
This is an automated email from the ASF dual-hosted git repository.

lichaoyong pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.


from 1e813df  [Doris On ES] [Bug-Fix][Refactor] Fix potential null pointer 
exception and refactor function process logic (#3985)
 add a16236f  [refactor] Remove useless return value of class RowsetGraph 
(#3977)

No new revisions were added by this update.

Summary of changes:
 be/src/olap/rowset_graph.cpp | 38 ++
 be/src/olap/rowset_graph.h   |  8 
 be/src/olap/tablet.cpp   |  4 ++--
 3 files changed, 16 insertions(+), 34 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] chaoyli merged pull request #3977: [refactor] Remove useless return value of class RowsetGraph

2020-07-02 Thread GitBox


chaoyli merged pull request #3977:
URL: https://github.com/apache/incubator-doris/pull/3977


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] chaoyli merged pull request #3990: [shell] Fix BUILD_TYPE not used bug

2020-07-02 Thread GitBox


chaoyli merged pull request #3990:
URL: https://github.com/apache/incubator-doris/pull/3990


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[incubator-doris] branch master updated (a16236f -> ab325f5)

2020-07-02 Thread lichaoyong
This is an automated email from the ASF dual-hosted git repository.

lichaoyong pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.


from a16236f  [refactor] Remove useless return value of class RowsetGraph 
(#3977)
 add ab325f5  [shell] Fix BUILD_TYPE not used bug (#3990)

No new revisions were added by this update.

Summary of changes:
 run-ut.sh | 13 -
 1 file changed, 8 insertions(+), 5 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] wuyunfeng opened a new pull request #4012: [Doris On ES][refactor] refactor and enchanment ES sync meta logic

2020-07-02 Thread GitBox


wuyunfeng opened a new pull request #4012:
URL: https://github.com/apache/incubator-doris/pull/4012


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] chaoyli commented on a change in pull request #4011: Remove rs when version conflict when insert to a tablet

2020-07-02 Thread GitBox


chaoyli commented on a change in pull request #4011:
URL: https://github.com/apache/incubator-doris/pull/4011#discussion_r449348960



##
File path: be/src/olap/tablet.cpp
##
@@ -826,6 +832,33 @@ void Tablet::_print_missed_versions(const 
std::vector& missed_versions)
 LOG(WARNING) << ss.str();
 }
 
+bool Tablet::_drop_rowset_if_version_conflict(const RowsetSharedPtr& rowset) {

Review comment:
   There has a problem in this place.
   modify_rowsets() only change the TabletMeta in memory. But not change the 
meta in disk.
   So you should perform do_tablet_meta_checkpoint to save the meta and remove 
the rowsetmeta in RowsetMetaManager.

##
File path: be/src/olap/tablet.cpp
##
@@ -826,6 +832,33 @@ void Tablet::_print_missed_versions(const 
std::vector& missed_versions)
 LOG(WARNING) << ss.str();
 }
 
+bool Tablet::_drop_rowset_if_version_conflict(const RowsetSharedPtr& rowset) {
+if (rowset == nullptr) {
+return false;
+}
+Version insert_version = {rowset->start_version(), rowset->end_version()};
+Version max_version = _tablet_meta->max_version();
+if (insert_version != max_version) {
+return false;
+}
+
+RowsetSharedPtr exist_rs = get_rowset_by_version(insert_version);
+// if a rs is created by doris 0.10 exists, remove it.
+if (exist_rs != nullptr) {
+vector to_add;
+vector to_delete;
+to_delete.push_back(exist_rs);
+modify_rowsets(to_add, to_delete);
+std::stringstream ss;
+ss << "Remove rs " << exist_rs->rowset_id().to_string();

Review comment:
   LOG(WARNING) << "Remove rs " << exist_rs->rowset_id().to_string()
<< " in " << full_name()
<< " for version conflict with " << 
rowset->rowset_id().to_string()
   There is no necessity to use std::stringstream.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] chaoyli commented on pull request #4011: Remove rs when version conflict when insert to a tablet

2020-07-02 Thread GitBox


chaoyli commented on pull request #4011:
URL: https://github.com/apache/incubator-doris/pull/4011#issuecomment-653307162


   > @chaoyli Now, should we prohibit insertion when 
`OLAP_ERR_PUSH_VERSION_ALREADY_EXIST` appears?
   
   Your function will add little cost to remove 
OLAP_ERR_PUSH_VERSION_ALREADY_EXIST.
   The cost will reflect on save the meta.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] chaoyli commented on pull request #4009: [Bug]Fix some schema change not work right

2020-07-02 Thread GitBox


chaoyli commented on pull request #4009:
URL: https://github.com/apache/incubator-doris/pull/4009#issuecomment-653312988


   It's better add more discrete message in commit info like you issue.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] spaces-X commented on a change in pull request #4000: applyInnerJoinConditionReorganizeRule

2020-07-02 Thread GitBox


spaces-X commented on a change in pull request #4000:
URL: https://github.com/apache/incubator-doris/pull/4000#discussion_r449353484



##
File path: fe/src/main/java/org/apache/doris/analysis/SelectStmt.java
##
@@ -816,9 +816,103 @@ protected void reorderTable(Analyzer analyzer) throws 
AnalysisException {
 
 // can not get AST only with equal join, MayBe cross join can help
 fromClause_.clear();
+List> misMatchJoinConditions = Lists.newArrayList();
+List tupleIds = Lists.newArrayList();
+
 for (Pair candidate : candidates) {
-fromClause_.add(candidate.first);
+TableRef tableRef = candidate.first;
+fromClause_.add(tableRef);
+tupleIds.add(tableRef.desc.getId());
+applyInnerJoinConditionReorganizeRule(tableRef, tupleIds, 
misMatchJoinConditions);
+}
+
+Preconditions.checkState(misMatchJoinConditions.isEmpty());
+}
+
+/**
+ *  After adjusting the order of tables, it is necessary to reselect the 
join condition of each table,
+ *  otherwise some columns will not be recognized by the current table 
space
+ * @param tableRef:
+ * @param tupleIds: TableSpaces visible up to now
+ * @param misMatchJoinConditions: join conditions no matching yet, the 
first is fatherExpr, the second is sonExpr need to replace
+ */
+void applyInnerJoinConditionReorganizeRule(TableRef tableRef, 
List tupleIds,
+List> misMatchJoinConditions) {
+List> tempMisMatchConjuncts = Lists.newArrayList();
+
+// Extract all join conditions from joinOnClause to 
misMatchJoinConditions,
+// which cannot be bound by existing tableSpaces(tupleIds)
+findUnboundedJoinConditions(tableRef.getOnClause(), tupleIds, 
misMatchJoinConditions);
+
+for (Pair exprPair : misMatchJoinConditions) {
+Expr father = exprPair.first;
+Expr expr = exprPair.second;
+if (!expr.isBoundByTupleIds(tupleIds)) {
+tempMisMatchConjuncts.add(new Pair(null, expr));
+if (father.getChild(0) == expr) {

Review comment:
   father may be null here.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] WingsGo commented on pull request #4009: [Bug]Fix some schema change not work right

2020-07-02 Thread GitBox


WingsGo commented on pull request #4009:
URL: https://github.com/apache/incubator-doris/pull/4009#issuecomment-653323967


   In order to make review easiler, I will not add code format until review 
ended.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] WingsGo commented on pull request #4009: [Bug]Fix some schema change not work right

2020-07-02 Thread GitBox


WingsGo commented on pull request #4009:
URL: https://github.com/apache/incubator-doris/pull/4009#issuecomment-653323183


   > It's better add more discrete message in commit info like you issue.
   
   Ok, done, and I tested the schema change supported in doc, but I found that 
convert LargetInt to Double is not support in code implement, maybe you can 
confirm that why it is added to doc in case It is supported but there is some 
bug exists?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] WingsGo edited a comment on pull request #4009: [Bug]Fix some schema change not work right

2020-07-02 Thread GitBox


WingsGo edited a comment on pull request #4009:
URL: https://github.com/apache/incubator-doris/pull/4009#issuecomment-653323183


   > It's better add more discrete message in commit info like you issue.
   
   Ok, done, and I tested the schema change supported in doc, but I found that 
convert LargetInt to Double is not support in code implement, maybe you can 
help me confirm that why it is added to doc in case It is supported but there 
is some bug exists?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[GitHub] [incubator-doris] funyeah commented on issue #3085: use datagrip connect fe error

2020-07-02 Thread GitBox


funyeah commented on issue #3085:
URL: 
https://github.com/apache/incubator-doris/issues/3085#issuecomment-653348056


   
![image](https://user-images.githubusercontent.com/19163232/86433315-fbdd0c80-bd2c-11ea-89d3-f7c7c4f80b24.png)
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org