[GitHub] [doris] yiguolei merged pull request #11414: [Bug](update) Update Stmt Fail use in predicate query
yiguolei merged PR #11414: URL: https://github.com/apache/doris/pull/11414 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #11413: [Bug] [Update] update stmt use in predicate cause npe
yiguolei closed issue #11413: [Bug] [Update] update stmt use in predicate cause npe URL: https://github.com/apache/doris/issues/11413 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Bug] Update Stmt Fail use in predicate query (#11414)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e41bfd8149 [Bug] Update Stmt Fail use in predicate query (#11414)
e41bfd8149 is described below
commit e41bfd8149e3c7896042d0179b818770bf6b2067
Author: HappenLee
AuthorDate: Tue Aug 2 15:14:36 2022 +0800
[Bug] Update Stmt Fail use in predicate query (#11414)
Co-authored-by: lihaopeng
---
fe/fe-core/src/main/java/org/apache/doris/analysis/UpdateStmt.java | 2 +-
regression-test/data/update/test_update_unique.out | 4 ++--
regression-test/suites/update/test_update_unique.groovy| 1 +
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/UpdateStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/UpdateStmt.java
index 3a2fbcffb7..23b182f9df 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/UpdateStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/UpdateStmt.java
@@ -198,8 +198,8 @@ public class UpdateStmt extends DdlStmt {
if (whereExpr == null) {
throw new AnalysisException("Where clause is required");
}
-whereExpr = analyzer.getExprRewriter().rewrite(whereExpr, analyzer,
ExprRewriter.ClauseType.WHERE_CLAUSE);
whereExpr.analyze(analyzer);
+whereExpr = analyzer.getExprRewriter().rewrite(whereExpr, analyzer,
ExprRewriter.ClauseType.WHERE_CLAUSE);
if (!whereExpr.getType().equals(Type.BOOLEAN)) {
throw new AnalysisException("Where clause is not a valid statement
return bool");
}
diff --git a/regression-test/data/update/test_update_unique.out
b/regression-test/data/update/test_update_unique.out
index dbe9d2d25f..6df338a346 100644
--- a/regression-test/data/update/test_update_unique.out
+++ b/regression-test/data/update/test_update_unique.out
@@ -1,7 +1,7 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
-- !select_uniq_table --
-1 2 1 2000-01-01
-2 2 1 2000-01-01
+1 2 1 1999-01-01
+2 2 1 1999-01-01
-- !desc_uniq_table --
k INT Yes true\N
diff --git a/regression-test/suites/update/test_update_unique.groovy
b/regression-test/suites/update/test_update_unique.groovy
index df0b2bc186..b695572d5d 100644
--- a/regression-test/suites/update/test_update_unique.groovy
+++ b/regression-test/suites/update/test_update_unique.groovy
@@ -32,6 +32,7 @@ suite("test_update_unique", "update") {
sql "insert into ${tbName} values(2, 1, 1, '2000-01-01');"
sql "UPDATE ${tbName} SET value1 = 2 WHERE k=1;"
sql "UPDATE ${tbName} SET value1 = value1+1 WHERE k=2;"
+sql "UPDATE ${tbName} SET date_value = '1999-01-01' WHERE k in (1,2);"
qt_select_uniq_table "select * from ${tbName} order by k"
qt_desc_uniq_table "desc ${tbName}"
sql "DROP TABLE ${tbName}"
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #11423: [regression](Nereids) add nereids syntax regression suite
hello-stephen commented on PR #11423: URL: https://github.com/apache/doris/pull/11423#issuecomment-1202108460 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luozenglin opened a new pull request, #11433: [fix](grouping sets) Fix the query result error caused by the groupin…
luozenglin opened a new pull request, #11433: URL: https://github.com/apache/doris/pull/11433 …g sets statement grouping as an expression # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg opened a new pull request, #11434: [improvement](fe) Remove constant keys in aggregation
mrhhsg opened a new pull request, #11434: URL: https://github.com/apache/doris/pull/11434 # Proposed changes Issue Number: close #xxx ## Problem summary The constant aggregation keys have no effect on the aggregation result and can be removed. Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11378: [fix](planner) Fix an issue where outputSmap's size could grow exponentially
github-actions[bot] commented on PR #11378: URL: https://github.com/apache/doris/pull/11378#issuecomment-1202126311 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 commented on pull request #11399: [refactor](agg) Abstract the hash operation into a method
starocean999 commented on PR #11399: URL: https://github.com/apache/doris/pull/11399#issuecomment-1202127617 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #11419: [Bug] (odbc vectorized) select null from a not null column do not get the right error report
yiguolei closed issue #11419: [Bug] (odbc vectorized) select null from a not null column do not get the right error report URL: https://github.com/apache/doris/issues/11419 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Bug](ODBC) fix vectorized null value error report in odbc scan node (#11420)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 38ffe685b5 [Bug](ODBC) fix vectorized null value error report in odbc
scan node (#11420)
38ffe685b5 is described below
commit 38ffe685b57885deb707440c2fbd8e4c4e829b8e
Author: HappenLee
AuthorDate: Tue Aug 2 15:44:12 2022 +0800
[Bug](ODBC) fix vectorized null value error report in odbc scan node
(#11420)
* [Bug](ODBC) fix vectorized null value error report in odbc scan node
Co-authored-by: lihaopeng
---
be/src/exec/odbc_scan_node.cpp | 6 --
be/src/exec/text_converter.hpp | 7 +++
be/src/vec/exec/vodbc_scan_node.cpp | 41 +++--
be/src/vec/exec/vodbc_scan_node.h | 7 ---
4 files changed, 28 insertions(+), 33 deletions(-)
diff --git a/be/src/exec/odbc_scan_node.cpp b/be/src/exec/odbc_scan_node.cpp
index 8e5837a957..c0cfd53bb4 100644
--- a/be/src/exec/odbc_scan_node.cpp
+++ b/be/src/exec/odbc_scan_node.cpp
@@ -201,8 +201,10 @@ Status OdbcScanNode::get_next(RuntimeState* state,
RowBatch* row_batch, bool* eo
slot_desc->col_name());
}
} else if (column_data.strlen_or_ind > column_data.buffer_length) {
-return Status::InternalError("nonnull column contains nullptr.
table={}, column={}",
- _table_name,
slot_desc->col_name());
+return Status::InternalError(
+"column value length longer than buffer length. "
+"table={}, column={}, buffer_length",
+_table_name, slot_desc->col_name(),
column_data.buffer_length);
} else {
RETURN_IF_ERROR(write_text_slot(static_cast(column_data.target_value_ptr),
column_data.strlen_or_ind,
slot_desc, state));
diff --git a/be/src/exec/text_converter.hpp b/be/src/exec/text_converter.hpp
index 14b9b52044..8baa90b6d5 100644
--- a/be/src/exec/text_converter.hpp
+++ b/be/src/exec/text_converter.hpp
@@ -310,15 +310,14 @@ inline bool TextConverter::write_vec_column(const
SlotDescriptor* slot_desc,
break;
}
-if (parse_result == StringParser::PARSE_FAILURE) {
+if (UNLIKELY(parse_result == StringParser::PARSE_FAILURE)) {
if (true == slot_desc->is_nullable()) {
-auto* nullable_column =
reinterpret_cast(col_ptr);
+auto* nullable_column =
reinterpret_cast(nullable_col_ptr);
size_t size = nullable_column->get_null_map_data().size();
doris::vectorized::NullMap& null_map_data =
nullable_column->get_null_map_data();
null_map_data[size - 1] = 1;
-} else {
-return false;
}
+return false;
}
return true;
}
diff --git a/be/src/vec/exec/vodbc_scan_node.cpp
b/be/src/vec/exec/vodbc_scan_node.cpp
index 4b75f332b1..d5a5bd0ec3 100644
--- a/be/src/vec/exec/vodbc_scan_node.cpp
+++ b/be/src/vec/exec/vodbc_scan_node.cpp
@@ -107,19 +107,6 @@ Status VOdbcScanNode::open(RuntimeState* state) {
return Status::OK();
}
-Status VOdbcScanNode::write_text_slot(char* value, int value_length,
SlotDescriptor* slot,
- RuntimeState* state) {
-if (!_text_converter->write_slot(slot, _tuple, value, value_length, true,
false,
- _tuple_pool.get())) {
-std::stringstream ss;
-ss << "Fail to convert odbc value:'" << value << "' to " <<
slot->type() << " on column:`"
- << slot->col_name() + "`";
-return Status::InternalError(ss.str());
-}
-
-return Status::OK();
-}
-
Status VOdbcScanNode::get_next(RuntimeState* state, Block* block, bool* eos) {
INIT_AND_SCOPE_GET_NEXT_SPAN(state->get_tracer(), _get_next_span,
"VOdbcScanNode::get_next");
VLOG_CRITICAL << get_scan_node_type() << "::GetNext";
@@ -173,7 +160,6 @@ Status VOdbcScanNode::get_next(RuntimeState* state, Block*
block, bool* eos) {
}
// Read one row from reader
-
for (int column_index = 0, materialized_column_index = 0;
column_index < column_size;
++column_index) {
auto slot_desc = tuple_desc->slots()[column_index];
@@ -186,12 +172,27 @@ Status VOdbcScanNode::get_next(RuntimeState* state,
Block* block, bool* eos) {
char* value_data =
static_cast(column_data.target_value_ptr);
int value_len = column_data.strlen_or_ind;
-if (!text_converter->write_column(slot_desc,
&columns[column_index], value_data,
- value_len, true, false)) {
-std::stringstream ss;
-ss << "Fail to
[GitHub] [doris] yiguolei merged pull request #11420: [Bug](ODBC) fix vectorized null value error report in odbc scan node
yiguolei merged PR #11420: URL: https://github.com/apache/doris/pull/11420 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11434: [improvement](fe) Remove constant keys in aggregation
morrySnow commented on code in PR #11434:
URL: https://github.com/apache/doris/pull/11434#discussion_r935225782
##
fe/fe-core/src/main/java/org/apache/doris/analysis/AggregateInfoBase.java:
##
@@ -95,6 +95,10 @@ protected AggregateInfoBase(AggregateInfoBase other) {
materializedSlots = Lists.newArrayList(other.materializedSlots);
}
+public void removeConstantKeys() {
+aggregateExprs.removeIf(expr -> expr.isConstant());
Review Comment:
👍🏻
```suggestion
groupingExprs.removeIf(Expr::isConstant);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SaintBacchus commented on pull request #11310: [feature-wip][multi-catalog]Support caseSensitive field name in file scan node
SaintBacchus commented on PR #11310: URL: https://github.com/apache/doris/pull/11310#issuecomment-1202139621 @morningman @stalary I had rebase this pr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature] (Nereids) add rule to merge consecutive project nodes (#11262)
This is an automated email from the ASF dual-hosted git repository.
huajianlan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new cd6fbd09bf [feature] (Nereids) add rule to merge consecutive project
nodes (#11262)
cd6fbd09bf is described below
commit cd6fbd09bf4b5d445e25b310ec123b062f3fae7f
Author: minghong
AuthorDate: Tue Aug 2 15:58:55 2022 +0800
[feature] (Nereids) add rule to merge consecutive project nodes (#11262)
Merge consecutive project nodes. For example:
logical plan tree:
```
project(a)
|
project(a,b)
|
project(a, b, c)
|
scan
```
transformed to:
```
project(a)
|
scan
```
---
.../doris/nereids/analyzer/UnboundRelation.java| 2 +-
.../org/apache/doris/nereids/rules/RuleType.java | 1 +
.../rules/rewrite/AggregateDisassemble.java| 21 +---
.../rewrite/logical/MergeConsecutiveProjects.java | 72 +
.../expressions/visitor/ExpressionReplacer.java| 40 +++
.../logical/MergeConsecutiveProjectsTest.java | 115 +
6 files changed, 230 insertions(+), 21 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java
index 4bd8413d0e..d00fb4266f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java
@@ -102,7 +102,7 @@ public class UnboundRelation extends LogicalLeaf implements
Unbound {
@Override
public String toString() {
-return "UnresolvedRelation" + "(" + StringUtils.join(nameParts, ".") +
")";
+return "UnboundRelation" + "(" + StringUtils.join(nameParts, ".") +
")";
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index e08270f08c..98c1cedd5f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -56,6 +56,7 @@ public enum RuleType {
REWRITE_JOIN_EXPRESSION(RuleTypeClass.REWRITE),
REORDER_JOIN(RuleTypeClass.REWRITE),
MERGE_CONSECUTIVE_FILTERS(RuleTypeClass.REWRITE),
+MERGE_CONSECUTIVE_PROJECTS(RuleTypeClass.REWRITE),
REWRITE_SENTINEL(RuleTypeClass.REWRITE),
// exploration rules
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java
index 96031d31de..8a0363e103 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java
@@ -24,7 +24,7 @@ import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.NamedExpression;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.expressions.functions.AggregateFunction;
-import
org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionReplacer;
import org.apache.doris.nereids.trees.plans.AggPhase;
import org.apache.doris.nereids.trees.plans.GroupPlan;
import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
@@ -135,23 +135,4 @@ public class AggregateDisassemble extends
OneRewriteRuleFactory {
);
}).toRule(RuleType.AGGREGATE_DISASSEMBLE);
}
-
-@SuppressWarnings("InnerClassMayBeStatic")
-private static class ExpressionReplacer
-extends DefaultExpressionRewriter> {
-private static final ExpressionReplacer INSTANCE = new
ExpressionReplacer();
-
-@Override
-public Expression visit(Expression expr, Map
substitutionMap) {
-// TODO: we need to do sub tree match and replace. but we do not
have semanticEquals now.
-//e.g. a + 1 + 2 in output expression should be replaced by
-//(slot reference to update phase out (a + 1)) + 2, if we do
group by a + 1
-// currently, we could only handle output expression same with
group by expression
-if (substitutionMap.containsKey(expr)) {
-return substitutionMap.get(expr);
-} else {
-return super.visit(expr, substitutionMap);
-}
-}
-}
}
diff --git
a/fe/fe-core/src
[doris-website] branch master updated: sql-block-rule,add more example
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 8c5d714fbfe sql-block-rule,add more example 8c5d714fbfe is described below commit 8c5d714fbfe0d02ca7fca93067321e0d3fee34f2 Author: jiafeng.zhang AuthorDate: Tue Aug 2 15:59:02 2022 +0800 sql-block-rule,add more example sql-block-rule,add more example --- .../Create/CREATE-SQL-BLOCK-RULE.md| 57 .../Create/CREATE-SQL-BLOCK-RULE.md| 61 +- 2 files changed, 72 insertions(+), 46 deletions(-) diff --git a/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md b/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md index 0790a9a3fee..7fc35132011 100644 --- a/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md +++ b/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md @@ -61,35 +61,48 @@ Parameter Description: 1. Create a block rule named test_rule - ```sql - mysql> CREATE SQL_BLOCK_RULE test_rule - -> PROPERTIES( - -> "sql"="select \\* from order_analysis;", - -> "global"="false", - -> "enable"="true" - -> ); - Query OK, 0 rows affected (0.01 sec) - +```sql +CREATE SQL_BLOCK_RULE test_rule +PROPERTIES( +"sql"="select \\* from order_analysis;", +"global"="false", +"enable"="true" +); + - When we execute the sql we just defined in the rule, an exception error will be returned. The example is as follows: +When we execute the sql we just defined in the rule, an exception error will be returned. The example is as follows: + +```sql +select * from order_analysis; +ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule + - ```sql - mysql> select * from order_analysis; - ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule - 2. Create test_rule2, limit the maximum number of scanned partitions to 30, and limit the maximum scan base to 10 billion rows. The example is as follows: ```sql - mysql> CREATE SQL_BLOCK_RULE test_rule2 - -> PROPERTIES ( - -> "partition_num" = "30", - -> "cardinality" = "100", - -> "global" = "false", - -> "enable" = "true" - -> ); - Query OK, 0 rows affected (0.01 sec) +CREATE SQL_BLOCK_RULE test_rule2 +PROPERTIES ( +"partition_num" = "30", +"cardinality" = "100", +"global" = "false", +"enable" = "true" +); +3. Create SQL BLOCK RULE with special chars + +```sql +CREATE SQL_BLOCK_RULE test_rule3 +PROPERTIES +( +"sql" = "select count\\(1\\) from db1.tbl1" +); +CREATE SQL_BLOCK_RULE test_rule4 +PROPERTIES +( +"sql" = "select \\* from db1.tbl1" +); +``` ### Keywords diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md index 17032f5afec..fa73c57e8c0 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md @@ -61,36 +61,49 @@ CREATE SQL_BLOCK_RULE rule_name 1. 创建一个名称为 test_rule 的阻止规则 - ```sql - mysql> CREATE SQL_BLOCK_RULE test_rule - -> PROPERTIES( - -> "sql"="select \\* from order_analysis;", - -> "global"="false", - -> "enable"="true" - -> ); - Query OK, 0 rows affected (0.01 sec) - ``` - - 当我们去执行刚才我们定义在规则里的sql时就会返回异常错误，示例如下： - - ```sql - mysql> select * from order_analysis; - ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule - ``` + ```sql + CREATE SQL_BLOCK_RULE test_rule + PROPERTIES( + "sql"="select \\* from order_analysis;", + "global"="false", + "enable"="true" + ); +``` + +当我们去执行刚才我们定义在规则里的sql时就会返回异常错误，示例如下： + +```sql +mysql> select * from order_analysis; +ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule +``` 2. 创建 test_rule2，将最大扫描的分区数量限制在30个，最大扫描基数限制在100亿行，示例如下： - ```sql - mysql> CREATE SQL_BLOCK_RULE test_rule2 - -> PROPERTIES +```sql +CREATE SQL_BLOCK_RULE test_rule2 +PROPERTIES ( - -> "partition_num" = "30"
[GitHub] [doris] 924060929 merged pull request #11262: [feature] (Nereids) add rule to merge consecutive project nodes
924060929 merged PR #11262: URL: https://github.com/apache/doris/pull/11262 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg merged pull request #11218: [WIP][feature] support `create`,`alter`,`refresh`,`drop` stmt syntax for multi table materialized view
yangzhg merged PR #11218: URL: https://github.com/apache/doris/pull/11218 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [WIP][feature] support `create`,`alter`,`refresh`,`drop` stmt syntax for multi table materialized view (#11218)
This is an automated email from the ASF dual-hosted git repository.
yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ff8a4ec2f2 [WIP][feature] support `create`,`alter`,`refresh`,`drop`
stmt syntax for multi table materialized view (#11218)
ff8a4ec2f2 is described below
commit ff8a4ec2f2c734b2dd7b267cb9cee20b016b8470
Author: Zhengguo Yang
AuthorDate: Tue Aug 2 16:03:36 2022 +0800
[WIP][feature] support `create`,`alter`,`refresh`,`drop` stmt syntax for
multi table materialized view (#11218)
* [WIP][feature] support `create`,`alter`,`refresh`,`drop` stmt for multi
table materialized view
---
fe/fe-core/src/main/cup/sql_parser.cup | 128 -
.../main/java/org/apache/doris/alter/Alter.java| 19 +++
.../doris/analysis/AlterMaterializedViewStmt.java | 55 +
.../CreateMultiTableMaterializedViewStmt.java | 72
.../doris/analysis/DropMaterializedViewStmt.java | 27 -
.../org/apache/doris/analysis/MVRefreshInfo.java | 75
.../analysis/MVRefreshIntervalTriggerInfo.java | 54 +
.../doris/analysis/MVRefreshTriggerInfo.java | 63 ++
.../analysis/RefreshMaterializedViewStmt.java | 76
.../main/java/org/apache/doris/catalog/Env.java| 15 +++
.../main/java/org/apache/doris/qe/DdlExecutor.java | 11 ++
fe/fe-core/src/main/jflex/sql_scanner.flex | 57 +
12 files changed, 619 insertions(+), 33 deletions(-)
diff --git a/fe/fe-core/src/main/cup/sql_parser.cup
b/fe/fe-core/src/main/cup/sql_parser.cup
index 131a337e7a..ad3b77fcc1 100644
--- a/fe/fe-core/src/main/cup/sql_parser.cup
+++ b/fe/fe-core/src/main/cup/sql_parser.cup
@@ -240,25 +240,25 @@ parser code {:
// Total keywords of doris
terminal String KW_ADD, KW_ADMIN, KW_AFTER, KW_AGGREGATE, KW_ALIAS, KW_ALL,
KW_ALTER, KW_AND, KW_ANTI, KW_APPEND, KW_AS, KW_ASC, KW_AUTHORS, KW_ARRAY,
-KW_BACKEND, KW_BACKUP, KW_BETWEEN, KW_BEGIN, KW_BIGINT, KW_BINLOG,
KW_BITMAP, KW_BITMAP_UNION, KW_QUANTILE_STATE, KW_QUANTILE_UNION, KW_BLOB,
KW_BOOLEAN, KW_BROKER, KW_BACKENDS, KW_BY, KW_BUILTIN,
+KW_BACKEND, KW_BACKUP, KW_BETWEEN, KW_BEGIN, KW_BIGINT, KW_BINLOG,
KW_BITMAP, KW_BITMAP_UNION, KW_QUANTILE_STATE, KW_QUANTILE_UNION, KW_BLOB,
KW_BOOLEAN, KW_BROKER, KW_BACKENDS, KW_BY, KW_BUILD, KW_BUILTIN,
KW_CANCEL, KW_CASE, KW_CAST, KW_CHAIN, KW_CHAR, KW_CHARSET, KW_CHECK,
KW_CLUSTER, KW_CLUSTERS, KW_CLEAN, KW_CURRENT_TIMESTAMP,
-KW_COLLATE, KW_COLLATION, KW_COLUMN, KW_COLUMNS, KW_COMMENT, KW_COMMIT,
KW_COMMITTED, KW_COMPACT,
+KW_COLLATE, KW_COLLATION, KW_COLUMN, KW_COLUMNS, KW_COMMENT, KW_COMMIT,
KW_COMMITTED, KW_COMPACT, KW_COMPLETE,
KW_CONFIG, KW_CONNECTION, KW_CONNECTION_ID, KW_CONSISTENT, KW_CONVERT,
KW_COUNT, KW_CREATE, KW_CREATION, KW_CROSS, KW_CUBE, KW_CURRENT,
KW_CURRENT_USER,
-KW_DATA, KW_DATABASE, KW_DATABASES, KW_DATE, KW_DATETIME, KW_DATEV2,
KW_DATETIMEV2, KW_DAY, KW_DECIMAL, KW_DECIMALV3, KW_DECOMMISSION, KW_DEFAULT,
KW_DESC, KW_DESCRIBE,
+KW_DATA, KW_DATABASE, KW_DATABASES, KW_DATE, KW_DATETIME, KW_DATEV2,
KW_DATETIMEV2, KW_DAY, KW_DECIMAL, KW_DECIMALV3, KW_DECOMMISSION, KW_DEFAULT,
KW_DEFERRED, KW_DEMAND, KW_DESC, KW_DESCRIBE,
KW_DELETE, KW_UPDATE, KW_DIAGNOSE, KW_DISK, KW_DISTINCT, KW_DISTINCTPC,
KW_DISTINCTPCSA, KW_DISTRIBUTED, KW_DISTRIBUTION, KW_DYNAMIC, KW_BUCKETS,
KW_DIV, KW_DOUBLE, KW_DROP, KW_DROPP, KW_DUPLICATE,
KW_ELSE, KW_ENABLE, KW_ENCRYPTKEY, KW_ENCRYPTKEYS, KW_END, KW_ENGINE,
KW_ENGINES, KW_ENTER, KW_ERRORS, KW_EVENTS, KW_EXCEPT, KW_EXCLUDE,
KW_EXISTS, KW_EXPORT, KW_EXTENDED, KW_EXTERNAL, KW_EXTRACT,
-KW_FALSE, KW_FEATURE, KW_FOLLOWER, KW_FOLLOWING, KW_FREE, KW_FROM,
KW_FIELDS, KW_FILE, KW_FILTER, KW_FIRST, KW_FLOAT, KW_FOR, KW_FORCE, KW_FORMAT,
KW_FRONTEND, KW_FRONTENDS, KW_FULL, KW_FUNCTION, KW_FUNCTIONS,
+KW_FAST, KW_FALSE, KW_FEATURE, KW_FOLLOWER, KW_FOLLOWING, KW_FREE,
KW_FROM, KW_FIELDS, KW_FILE, KW_FILTER, KW_FIRST, KW_FLOAT, KW_FOR, KW_FORCE,
KW_FORMAT, KW_FRONTEND, KW_FRONTENDS, KW_FULL, KW_FUNCTION, KW_FUNCTIONS,
KW_GLOBAL, KW_GRANT, KW_GRANTS, KW_GRAPH, KW_GROUP, KW_GROUPING,
KW_HASH, KW_HAVING, KW_HDFS, KW_HELP,KW_HLL, KW_HLL_UNION, KW_HOUR, KW_HUB,
-KW_IDENTIFIED, KW_IF, KW_IN, KW_INDEX, KW_INDEXES, KW_INFILE, KW_INSTALL,
+KW_IDENTIFIED, KW_IF, KW_IMMEDIATE, KW_IN, KW_INDEX, KW_INDEXES,
KW_INFILE, KW_INSTALL,
KW_INNER, KW_INSERT, KW_INT, KW_INTERMEDIATE, KW_INTERSECT, KW_INTERVAL,
KW_INTO, KW_IS, KW_ISNULL, KW_ISOLATION,
KW_JOB, KW_JOIN,
KW_KEY, KW_KEYS, KW_KILL,
KW_LABEL, KW_LARGEINT, KW_LAST, KW_LEFT, KW_LESS, KW_LEVEL, KW_LIKE,
KW_LIMIT, KW_LINK, KW_LIST, KW_LOAD,
KW_LOCAL, KW_LOCATION, KW_LOCK, KW_LOW_PRIORITY, KW_LATERAL,
KW_MAP, KW_MATERIALIZED, KW_MAX, KW_MAX_VALUE, KW_MERGE, KW_MIN,
KW_MINUTE, KW_MINUS, KW_MIG
[GitHub] [doris] kkfnui opened a new issue, #11435: [Feature] Support Unicode check in sqlalchemy
kkfnui opened a new issue, #11435:
URL: https://github.com/apache/doris/issues/11435
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Description
When use sqlalchemy with doris. There will be numbers wanging logs:
```
2022-08-02 08:05:05,358 WARN (doris-mysql-nio-pool-65575|435146)
[ConnectProcessor.handleQuery():226] Process one query failed because.
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage =
Syntax error in line 1:
...collated returns' AS CHAR CHARACTER SET utf8mb4) COLLA...
^
Encountered: CHARACTER
Expected: COMMA
at
org.apache.doris.qe.ConnectProcessor.analyze(ConnectProcessor.java:274)
~[palo-fe.jar:0.15-SNAPSHOT]
at
org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:200)
~[palo-fe.jar:0.15-SNAPSHOT]
at
org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:349)
~[palo-fe.jar:0.15-SNAPSHOT]
at
org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:538)
~[palo-fe.jar:0.15-SNAPSHOT]
at
org.apache.doris.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:50)
~[palo-fe.jar:0.15-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.doris.common.AnalysisException: errCode = 2,
detailMessage = Syntax error
at
org.apache.doris.analysis.SqlParser.unrecovered_syntax_error(SqlParser.java:2076)
~[palo-fe.jar:0.15-SNAPSHOT]
at java_cup.runtime.lr_parser.parse(lr_parser.java:616)
~[jflex-1.4.3.jar:?]
at
org.apache.doris.common.util.SqlParserUtils.getMultiStmts(SqlParserUtils.java:59)
~[palo-fe.jar:0.15-SNAPSHOT]
at
org.apache.doris.qe.ConnectProcessor.analyze(ConnectProcessor.java:265)
~[palo-fe.jar:0.15-SNAPSHOT]
... 7 more
```
And It is valid for mysql. So it makes warning log to useless.
### Use case
unicode test sql:
```
SELECT CAST('test collated returns' AS CHAR CHARACTER SET utf8) COLLATE
utf8mb4_bin AS anon_1;
```
https://github.com/sqlalchemy/sqlalchemy/blob/1536bc4664a248faf81c62326fe1be3dbe18b8cd/lib/sqlalchemy/dialects/mysql/mysqldb.py#L90
### Related issues
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11416: [doc](sql-block-rule) add more example
github-actions[bot] commented on PR #11416: URL: https://github.com/apache/doris/pull/11416#issuecomment-1202173778 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11416: [doc](sql-block-rule) add more example
github-actions[bot] commented on PR #11416: URL: https://github.com/apache/doris/pull/11416#issuecomment-1202173733 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] kkfnui commented on issue #11412: [Bug] doris子查询顺序会乱掉
kkfnui commented on issue #11412: URL: https://github.com/apache/doris/issues/11412#issuecomment-1202179107 好像没有规范定义这块吧？不过测试了几个， 1. spark-sql, 符合你的预期 2. presto， 符合你的预期 3. tidb， 符合你的预期 4. mysql， 符合你的预期 5. doris 1.0.0~1.1.1 不符合你的预期 6. flink-sql 不符合你的预期 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #10850: add json profile and add session context and remove olap limit
morrySnow commented on PR #10850: URL: https://github.com/apache/doris/pull/10850#issuecomment-1202192982 duplicated with a merged PR: #11279 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow closed pull request #10850: add json profile and add session context and remove olap limit
morrySnow closed pull request #10850: add json profile and add session context and remove olap limit URL: https://github.com/apache/doris/pull/10850 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 merged pull request #11423: [regression](Nereids) add nereids syntax regression suite
924060929 merged PR #11423: URL: https://github.com/apache/doris/pull/11423 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [regression](Nereids) add nereids syntax regression suite (#11423)
This is an automated email from the ASF dual-hosted git repository.
huajianlan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 945256f6a6 [regression](Nereids) add nereids syntax regression suite
(#11423)
945256f6a6 is described below
commit 945256f6a64c70aa5c7b987d04bfd29a36f71219
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Tue Aug 2 16:45:53 2022 +0800
[regression](Nereids) add nereids syntax regression suite (#11423)
add a new regression suite for test nereids syntax. It load data only and
has no test case now.
---
regression-test/suites/nereids_syntax/load.groovy | 184 ++
1 file changed, 184 insertions(+)
diff --git a/regression-test/suites/nereids_syntax/load.groovy
b/regression-test/suites/nereids_syntax/load.groovy
new file mode 100644
index 00..22103b19e3
--- /dev/null
+++ b/regression-test/suites/nereids_syntax/load.groovy
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("load") {
+sql """
+DROP TABLE IF EXISTS `customer`;
+"""
+
+sql """
+DROP TABLE IF EXISTS `dates`;
+"""
+
+sql """
+DROP TABLE IF EXISTS `lineorder`;
+"""
+
+sql """
+DROP TABLE IF EXISTS `part`;
+"""
+
+sql """
+DROP TABLE IF EXISTS `supplier`;
+"""
+
+sql """
+CREATE TABLE IF NOT EXISTS `customer` (
+ `c_custkey` int(11) NOT NULL COMMENT "",
+ `c_name` varchar(26) NOT NULL COMMENT "",
+ `c_address` varchar(41) NOT NULL COMMENT "",
+ `c_city` varchar(11) NOT NULL COMMENT "",
+ `c_nation` varchar(16) NOT NULL COMMENT "",
+ `c_region` varchar(13) NOT NULL COMMENT "",
+ `c_phone` varchar(16) NOT NULL COMMENT "",
+ `c_mktsegment` varchar(11) NOT NULL COMMENT ""
+)
+DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 10
+PROPERTIES (
+"replication_num" = "1"
+);
+"""
+
+sql """
+CREATE TABLE IF NOT EXISTS `dates` (
+ `d_datekey` int(11) NOT NULL COMMENT "",
+ `d_date` varchar(20) NOT NULL COMMENT "",
+ `d_dayofweek` varchar(10) NOT NULL COMMENT "",
+ `d_month` varchar(11) NOT NULL COMMENT "",
+ `d_year` int(11) NOT NULL COMMENT "",
+ `d_yearmonthnum` int(11) NOT NULL COMMENT "",
+ `d_yearmonth` varchar(9) NOT NULL COMMENT "",
+ `d_daynuminweek` int(11) NOT NULL COMMENT "",
+ `d_daynuminmonth` int(11) NOT NULL COMMENT "",
+ `d_daynuminyear` int(11) NOT NULL COMMENT "",
+ `d_monthnuminyear` int(11) NOT NULL COMMENT "",
+ `d_weeknuminyear` int(11) NOT NULL COMMENT "",
+ `d_sellingseason` varchar(14) NOT NULL COMMENT "",
+ `d_lastdayinweekfl` int(11) NOT NULL COMMENT "",
+ `d_lastdayinmonthfl` int(11) NOT NULL COMMENT "",
+ `d_holidayfl` int(11) NOT NULL COMMENT "",
+ `d_weekdayfl` int(11) NOT NULL COMMENT ""
+)
+DISTRIBUTED BY HASH(`d_datekey`) BUCKETS 1
+PROPERTIES (
+"replication_num" = "1"
+);
+"""
+
+sql """
+CREATE TABLE IF NOT EXISTS `lineorder` (
+ `lo_orderkey` bigint(20) NOT NULL COMMENT "",
+ `lo_linenumber` bigint(20) NOT NULL COMMENT "",
+ `lo_custkey` int(11) NOT NULL COMMENT "",
+ `lo_partkey` int(11) NOT NULL COMMENT "",
+ `lo_suppkey` int(11) NOT NULL COMMENT "",
+ `lo_orderdate` int(11) NOT NULL COMMENT "",
+ `lo_orderpriority` varchar(16) NOT NULL COMMENT "",
+ `lo_shippriority` int(11) NOT NULL COMMENT "",
+ `lo_quantity` bigint(20) NOT NULL COMMENT "",
+ `lo_extendedprice` bigint(20) NOT NULL COMMENT "",
+ `lo_ordtotalprice` bigint(20) NOT NULL COMMENT "",
+ `lo_discount` bigint(20) NOT NULL COMMENT "",
+ `lo_revenue` bigint(20) NOT NULL COMMENT "",
+ `lo_supplycost` bigint(20) NOT NULL COMMENT "",
+ `lo_tax` bigint(20) NOT NULL COMMENT "",
+ `lo_commitdate` bigint(20) NOT NULL COM
[GitHub] [doris] Toms1999 commented on a diff in pull request #10905: [extension] Mysql database import doris by external tables
Toms1999 commented on code in PR #10905:
URL: https://github.com/apache/doris/pull/10905#discussion_r935293300
##
extension/mysql_to_doris/all_tables.sh:
##
@@ -0,0 +1,142 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+
+# This script is used to will mysql databases import doris by external
+
+
+#reference configuration file
+source ./conf/mysql.conf
+source ./conf/doris.conf
+
+#define mysql database and doris database
+d_mysql=$1
+d_doris=$2
+
+#check args
+if [ ! -n "$1" ];then
+echo "please check source database"
+exit
+fi
+if [ ! -n "$2" ];then
+echo "please check sink database"
+exit
+fi
+
+#mkdir files to store tables and tables.sql
+mkdir -p files
+rm -rf ./files/tables
+rm -rf ./files/tables.sql
+
+#get tables from mysql databases
+echo "use $d_mysql; show tables;" |mysql -h$mysql -uroot -p$mysql_password
2>/dev/null >> ./files/tables
+
+#delete tables first line
+sed -i '1d' ./files/tables
+
+#reference tables to create tables.sql
+for table in $(awk -F '\n' '{print $1}' ./files/tables)
+do
+sed -i "/${table}view/d" ./files/tables
+echo "use $d_mysql; show create table ${table};" |mysql -h$mysql_host
-uroot -p$mysql_password 2>/dev/null >> ./files/tables.sql
+echo "print ${table} sql to tables.sql in the file dir"
+
+done
+
+echo '==start to transform mysql table for doris
extral table==='
+#adjust sql
+awk -F '\t' '{print $2}' ./files/tables.sql |awk '!(NR%2)' |awk '{print $0
";"}' > ./files/tables1.sql
+sed -i 's/\\n/\n/g' ./files/tables1.sql
+sed -n '/CREATE TABLE/,/ENGINE\=/p' ./files/tables1.sql > ./files/tables2.sql
+#delete tables special struct
+sed -i '/^ CON/d' ./files/tables2.sql
+sed -i '/^ KEY/d' ./files/tables2.sql
+rm -rf ./files/tables.sql
+rm -rf ./files/tables1.sql
+mv ./files/tables2.sql ./files/tables.sql
+
+#start transform tables struct
+sed -i '/ENGINE=/a) ENGINE=MYSQL\n COMMENT "MYSQL"\nPROPERTIES (\n"host" =
"ApacheDorisHostIp",\n"port" = "3306",\n"user" = "root",\n"password" =
"ApacheDorisHostPassword",\n"database" = "ApacheDorisDataBases",\n"table" =
"ApacheDorisTables");' ./files/tables.sql
+
+#delete match line
+sed -i '/ENGINT=/d' ./files/tables.sql
+sed -i '/PRIMARY KEY/d' ./files/tables.sql
+sed -i '/UNIQUE KEY/d' ./files/tables.sql
+#delete , at the beginning (
+sed -i '/,\s*$/{:loop; N; /,\(\s*\|\n\))/! bloop; s/,\s*[\n]\?\s*)/\n)/}'
./files/tables.sql
+
+#delete a line on keyword
+sed -i -e '$!N;/\n.*ENGINE=MYSQL/!P;D' ./files/tables.sql
+#replace mysql password、database、table、host
+for t_name in $(awk -F '\n' '{print $1}' ./files/tables)
+do
+sed -i "0,/ApacheDorisHostIp/s/ApacheDorisHostIp/${mysql}/"
./files/tables.sql
Review Comment:
I have completed the repair and test once
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11180: [Bugfix] Fix runtimefilter access violation when stub is nullptr
yiguolei merged PR #11180: URL: https://github.com/apache/doris/pull/11180 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [bugfix](runtimefilter)fix runtimefilter access violation when stub is nullptr (#11180)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1db8a2d136 [bugfix](runtimefilter)fix runtimefilter access violation
when stub is nullptr (#11180)
1db8a2d136 is described below
commit 1db8a2d136638b09b9867dfd8b5be457d9bcf3e0
Author: awakeljw <17329460+awake...@users.noreply.github.com>
AuthorDate: Tue Aug 2 16:57:17 2022 +0800
[bugfix](runtimefilter)fix runtimefilter access violation when stub is
nullptr (#11180)
---
be/src/runtime/runtime_filter_mgr.cpp | 25 +
1 file changed, 13 insertions(+), 12 deletions(-)
diff --git a/be/src/runtime/runtime_filter_mgr.cpp
b/be/src/runtime/runtime_filter_mgr.cpp
index dd58feb1e2..85d0fab248 100644
--- a/be/src/runtime/runtime_filter_mgr.cpp
+++ b/be/src/runtime/runtime_filter_mgr.cpp
@@ -237,32 +237,33 @@ Status RuntimeFilterMergeControllerEntity::merge(const
PMergeFilterRequest* requ
std::vector& targets = cntVal->target_info;
for (size_t i = 0; i < targets.size(); i++) {
rpc_contexts.emplace_back(new PPublishFilterRpcContext);
-rpc_contexts[i]->request = apply_request;
-rpc_contexts[i]->request.set_filter_id(request->filter_id());
-*rpc_contexts[i]->request.mutable_query_id() = request->query_id();
+size_t cur = rpc_contexts.size() - 1;
+rpc_contexts[cur]->request = apply_request;
+rpc_contexts[cur]->request.set_filter_id(request->filter_id());
+*rpc_contexts[cur]->request.mutable_query_id() =
request->query_id();
if (has_attachment) {
-
rpc_contexts[i]->cntl.request_attachment().append(request_attachment);
+
rpc_contexts[cur]->cntl.request_attachment().append(request_attachment);
}
-rpc_contexts[i]->cid = rpc_contexts[i]->cntl.call_id();
+rpc_contexts[cur]->cid = rpc_contexts[cur]->cntl.call_id();
// set fragment-id
-auto request_fragment_id =
rpc_contexts[i]->request.mutable_fragment_id();
-
request_fragment_id->set_hi(targets[i].target_fragment_instance_id.hi);
-
request_fragment_id->set_lo(targets[i].target_fragment_instance_id.lo);
+auto request_fragment_id =
rpc_contexts[cur]->request.mutable_fragment_id();
+
request_fragment_id->set_hi(targets[cur].target_fragment_instance_id.hi);
+
request_fragment_id->set_lo(targets[cur].target_fragment_instance_id.lo);
std::shared_ptr stub(
ExecEnv::GetInstance()->brpc_internal_client_cache()->get_client(
targets[i].target_fragment_instance_addr));
-VLOG_NOTICE << "send filter " <<
rpc_contexts[i]->request.filter_id()
+VLOG_NOTICE << "send filter " <<
rpc_contexts[cur]->request.filter_id()
<< " to:" <<
targets[i].target_fragment_instance_addr.hostname << ":"
<< targets[i].target_fragment_instance_addr.port
-<< rpc_contexts[i]->request.ShortDebugString();
+<< rpc_contexts[cur]->request.ShortDebugString();
if (stub == nullptr) {
rpc_contexts.pop_back();
continue;
}
-stub->apply_filter(&rpc_contexts[i]->cntl,
&rpc_contexts[i]->request,
- &rpc_contexts[i]->response, brpc::DoNothing());
+stub->apply_filter(&rpc_contexts[cur]->cntl,
&rpc_contexts[cur]->request,
+ &rpc_contexts[cur]->response,
brpc::DoNothing());
}
for (auto& rpc_context : rpc_contexts) {
brpc::Join(rpc_context->cid);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
yiguolei commented on code in PR #11400:
URL: https://github.com/apache/doris/pull/11400#discussion_r935298264
##
be/CMakeLists.txt:
##
@@ -517,7 +517,7 @@ set(CXX_FLAGS_DEBUG "${CXX_GCC_FLAGS} -O0")
# -O3: Enable all compiler optimizations
# -DNDEBUG: Turn off dchecks/asserts/debug only code.
set(CXX_FLAGS_RELEASE "${CXX_GCC_FLAGS} -O3 -DNDEBUG")
-SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O0 -fsanitize=address
-DADDRESS_SANITIZER")
+SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O3 -fsanitize=address
-DADDRESS_SANITIZER")
Review Comment:
@adonis0147 I think it makes no difference to a configuration or in
cmakelist. We have too many configurations, it is like code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 commented on a diff in pull request #11399: [refactor](agg) Abstract the hash operation into a method
Gabriel39 commented on code in PR #11399:
URL: https://github.com/apache/doris/pull/11399#discussion_r935299622
##
be/src/vec/exec/vaggregation_node.cpp:
##
@@ -706,6 +706,76 @@ bool AggregationNode::_should_expand_preagg_hash_tables() {
_agg_data._aggregated_method_variant);
}
+void AggregationNode::_emplace_into_hash_table(AggregateDataPtr* places,
ColumnRawPtrs& key_columns,
Review Comment:
```suggestion
void AggregationNode::_emplace_into_hash_table(AggregateDataPtr* places,
const ColumnRawPtrs& key_columns,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
adonis0147 commented on code in PR #11400:
URL: https://github.com/apache/doris/pull/11400#discussion_r935301365
##
be/CMakeLists.txt:
##
@@ -517,7 +517,7 @@ set(CXX_FLAGS_DEBUG "${CXX_GCC_FLAGS} -O0")
# -O3: Enable all compiler optimizations
# -DNDEBUG: Turn off dchecks/asserts/debug only code.
set(CXX_FLAGS_RELEASE "${CXX_GCC_FLAGS} -O3 -DNDEBUG")
-SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O0 -fsanitize=address
-DADDRESS_SANITIZER")
+SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O3 -fsanitize=address
-DADDRESS_SANITIZER")
Review Comment:
> @adonis0147 I think it makes no difference to a configuration or in
cmakelist. We have too many configurations, it is like code.
`run-be-ut.sh` use `ASAN`, this modification will affect the debugging
experience for BE UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
adonis0147 commented on code in PR #11400:
URL: https://github.com/apache/doris/pull/11400#discussion_r935301365
##
be/CMakeLists.txt:
##
@@ -517,7 +517,7 @@ set(CXX_FLAGS_DEBUG "${CXX_GCC_FLAGS} -O0")
# -O3: Enable all compiler optimizations
# -DNDEBUG: Turn off dchecks/asserts/debug only code.
set(CXX_FLAGS_RELEASE "${CXX_GCC_FLAGS} -O3 -DNDEBUG")
-SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O0 -fsanitize=address
-DADDRESS_SANITIZER")
+SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O3 -fsanitize=address
-DADDRESS_SANITIZER")
Review Comment:
> @adonis0147 I think it makes no difference to a configuration or in
cmakelist. We have too many configurations, it is like code.
`run-be-ut.sh` uses `ASAN`, this modification will affect the debugging
experience for BE UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caoliang-web opened a new pull request, #11437: [regression]SQL query conditions, if the data type is expanded with single quotes…
caoliang-web opened a new pull request, #11437: URL: https://github.com/apache/doris/pull/11437 ## Problem summary SQL query conditions, if the data type is expanded with single quotes, an error will occur case ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: Fix doc (#27)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 3138fa4691e Fix doc (#27) 3138fa4691e is described below commit 3138fa4691edf2f7c089b615e93bdb782e363bdc Author: Liqf <109049295+lemonlit...@users.noreply.github.com> AuthorDate: Tue Aug 2 17:17:52 2022 +0800 Fix doc (#27) * fix-document --- docs/admin-manual/config/fe-config.md | 56 +++--- .../current/admin-manual/config/fe-config.md | 14 +++--- 2 files changed, 35 insertions(+), 35 deletions(-) diff --git a/docs/admin-manual/config/fe-config.md b/docs/admin-manual/config/fe-config.md index 13c6e8374f5..6e0eaf4b728 100644 --- a/docs/admin-manual/config/fe-config.md +++ b/docs/admin-manual/config/fe-config.md @@ -199,7 +199,7 @@ Worker thread pool is not set by default, set according to your needs ### jetty_server_max_http_post_size -Default：100 * 1024 * 1024 （100MB） +Default：`100 * 1024 * 1024` （100MB） This is the maximum number of bytes of the file uploaded by the put or post method, the default value: 100MB @@ -811,7 +811,7 @@ MasterOnly：true ### max_bytes_per_broker_scanner -Default：3 * 1024 * 1024 * 1024L （3G） +Default：`3 * 1024 * 1024 * 1024L` （3G） IsMutable：true @@ -827,7 +827,7 @@ if set to false, auth check will be disable, in case some goes wrong with the ne ### tablet_stat_update_interval_second -Default：300，（5min） +Default：300（5min） update interval of tablet stat , All frontends will get tablet stat from all backends at each interval @@ -866,7 +866,7 @@ MasterOnly：true ### storage_min_left_capacity_bytes -Default： 2 * 1024 * 1024 * 1024 (2GB) +Default： `2 * 1024 * 1024 * 1024` (2GB) IsMutable：true @@ -896,7 +896,7 @@ Default：/lib/hadoop-client/hadoop/bin/hadoop ### dpp_bytes_per_reduce -Default：100 * 1024 * 1024L; // 100M +Default：`100 * 1024 * 1024L` (100M) ### dpp_default_cluster @@ -1001,7 +1001,7 @@ Maximal number of connections per FE. ### check_consistency_default_timeout_second -Default：600 （10分钟） +Default：600 （10 minutes） IsMutable：true @@ -1112,7 +1112,7 @@ After dropping database(table/partition), you can recover it by using RECOVER st ### storage_cooldown_second -Default：30 * 24 * 3600L （30day） +Default：`30 * 24 * 3600L` （30 days） When create a table(or partition), you can specify its storage medium(HDD or SSD). If set to SSD, this specifies the default duration that tablets will stay on SSD. After that, tablets will be moved to HDD automatically. You can set storage cooldown time in CREATE TABLE stmt. @@ -1124,7 +1124,7 @@ When create a table(or partition), you can specify its storage medium(HDD or SSD ### max_backend_down_time_second -Default：3600 （1hour） +Default：3600 （1 hour） IsMutable：true @@ -1134,7 +1134,7 @@ If a backend is down for *max_backend_down_time_second*, a BACKEND_DOWN event wi ### alter_table_timeout_second -Default：86400 （1day） +Default：86400 （1 day） IsMutable：true @@ -1200,7 +1200,7 @@ HIGH priority clone job's delay trigger time. ### clone_normal_priority_delay_second -Default：300 （5min） +Default：300 （5 minutes） IsMutable：true @@ -1210,7 +1210,7 @@ NORMAL priority clone job's delay trigger time ### clone_low_priority_delay_second -Default：600 （10min） +Default：600 （10 minutes） IsMutable：true @@ -1231,7 +1231,7 @@ Concurrency of LOW priority clone jobs. Concurrency of High priority clone jobs ### clone_job_timeout_second -Default：7200 (2小时) +Default：7200 (2 hours) IsMutable：true @@ -1241,7 +1241,7 @@ Default timeout of a single clone job. Set long enough to fit your replica size. ### clone_checker_interval_second -Default：300 （5min） +Default：300 （5 minutes） Clone checker's running interval @@ -1364,7 +1364,7 @@ Default spark home dir ### spark_load_default_timeout_second -Default：86400 (1天) +Default：86400 (1 day) IsMutable：true @@ -1380,7 +1380,7 @@ Default spark dpp version ### hadoop_load_default_timeout_second -Default：86400 * 3 (3天) +Default：86400 * 3 (3 days) IsMutable：true @@ -1400,7 +1400,7 @@ Min stream load timeout applicable to all type of load ### max_stream_load_timeout_second -Default：259200 （3天） +Default：259200 （3 days） IsMutable：true @@ -1410,7 +1410,7 @@ This configuration is specifically used to limit timeout setting for stream load ### max_load_timeout_second -Default：259200 （3天） +Default：259200 （3 days） IsMutable：true @@ -1450,7 +1450,7 @@ Default non-streaming mini load timeout ### broker_load_default_timeout_second -Default：14400 （4 hour） +Default：14400 （4 hours） IsMutable：true @@ -1623,7 +1623,7 @@ node(FE or BE) will be considered belonging to the same Palo cluster if they hav ### auth_token -Default：空
[GitHub] [doris] morrySnow commented on a diff in pull request #8862: [feature-wip](statistics) step5: show statistics job information
morrySnow commented on code in PR #8862:
URL: https://github.com/apache/doris/pull/8862#discussion_r935308090
##
fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsJob.java:
##
@@ -265,4 +273,86 @@ public static StatisticsJob fromAnalyzeStmt(AnalyzeStmt
stmt) throws AnalysisExc
Map properties = stmt.getProperties();
return new StatisticsJob(dbId, tblIds, tableIdToPartitionName,
tableIdToColumnName, properties);
}
+
+public List getShowInfo(@Nullable Long tableId) throws
AnalysisException {
+List result = Lists.newArrayList();
+
+result.add(Long.toString(id));
+
+SimpleDateFormat dateFormat = new SimpleDateFormat("-MM-dd
HH:mm:ss.SSS");
+result.add(TimeUtils.longToTimeString(createTime, dateFormat));
+result.add(startTime != -1L ? TimeUtils.longToTimeString(startTime,
dateFormat) : "N/A");
+result.add(finishTime != -1L ? TimeUtils.longToTimeString(finishTime,
dateFormat) : "N/A");
+
+StringBuilder sb = new StringBuilder();
+for (String errorMsg : errorMsgs) {
+sb.append(errorMsg).append("\n");
+}
+result.add(sb.toString());
+
+int totalTaskNum = 0;
+int finishedTaskNum = 0;
+Map> tblIdToCols = Maps.newHashMap();
+
+for (StatisticsTask task : tasks) {
+List statsDescs = task.getStatsDescs();
+
+if (!statsDescs.isEmpty()) {
+// The same task has the same stats properties
+StatsCategory statsCategory =
statsDescs.get(0).getStatsCategory();
+long tblId = statsCategory.getTableId();
+
+if (tableId == null || tableId == tblId) {
+totalTaskNum++;
+if (task.getTaskState() ==
StatisticsTask.TaskState.FINISHED) {
+finishedTaskNum++;
+}
+
+String col = statsCategory.getColumnName();
+if (Strings.isNullOrEmpty(col)) {
+continue;
+}
+if (tblIdToCols.containsKey(tblId)) {
+tblIdToCols.get(tblId).add(col);
+} else {
+Set cols = Sets.newHashSet();
+cols.add(col);
+tblIdToCols.put(tblId, cols);
+}
+}
+}
+}
+
+List scope = Lists.newArrayList();
+Database db = Env.getCurrentEnv()
+.getInternalDataSource().getDbOrAnalysisException(dbId);
+for (Long tblId : tblIds) {
+try {
+Table table = db.getTableOrAnalysisException(tblId);
+List baseSchema = table.getBaseSchema();
+Set cols = tblIdToCols.get(tblId);
+if (cols != null) {
+if (baseSchema.size() == cols.size()) {
+scope.add(table.getName() + "(*)");
+} else {
+scope.add(table.getName() + "(" +
StringUtils.join(cols.toArray(), ",") + ")");
Review Comment:
```suggestion
scope.add(table.getName() + "(" +
StringUtils.join(cols.toArray(), ", ") + ")");
```
##
fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsJob.java:
##
@@ -265,4 +273,86 @@ public static StatisticsJob fromAnalyzeStmt(AnalyzeStmt
stmt) throws AnalysisExc
Map properties = stmt.getProperties();
return new StatisticsJob(dbId, tblIds, tableIdToPartitionName,
tableIdToColumnName, properties);
}
+
+public List getShowInfo(@Nullable Long tableId) throws
AnalysisException {
+List result = Lists.newArrayList();
+
+result.add(Long.toString(id));
+
+SimpleDateFormat dateFormat = new SimpleDateFormat("-MM-dd
HH:mm:ss.SSS");
+result.add(TimeUtils.longToTimeString(createTime, dateFormat));
+result.add(startTime != -1L ? TimeUtils.longToTimeString(startTime,
dateFormat) : "N/A");
+result.add(finishTime != -1L ? TimeUtils.longToTimeString(finishTime,
dateFormat) : "N/A");
+
+StringBuilder sb = new StringBuilder();
+for (String errorMsg : errorMsgs) {
+sb.append(errorMsg).append("\n");
+}
+result.add(sb.toString());
+
+int totalTaskNum = 0;
+int finishedTaskNum = 0;
+Map> tblIdToCols = Maps.newHashMap();
+
+for (StatisticsTask task : tasks) {
+List statsDescs = task.getStatsDescs();
+
+if (!statsDescs.isEmpty()) {
+// The same task has the same stats properties
+StatsCategory statsCategory =
statsDescs.get(0).getStatsCategory();
+long tblId = statsCategory.getTableId();
+
+if (tableId == null || tableId == tblId) {
+totalTaskNum++;
+if (task.getTaskS
[GitHub] [doris] yiguolei commented on a diff in pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
yiguolei commented on code in PR #11400:
URL: https://github.com/apache/doris/pull/11400#discussion_r935321629
##
be/CMakeLists.txt:
##
@@ -517,7 +517,7 @@ set(CXX_FLAGS_DEBUG "${CXX_GCC_FLAGS} -O0")
# -O3: Enable all compiler optimizations
# -DNDEBUG: Turn off dchecks/asserts/debug only code.
set(CXX_FLAGS_RELEASE "${CXX_GCC_FLAGS} -O3 -DNDEBUG")
-SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O0 -fsanitize=address
-DADDRESS_SANITIZER")
+SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O3 -fsanitize=address
-DADDRESS_SANITIZER")
Review Comment:
Could you please provide the command line how to use the configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
yiguolei commented on code in PR #11400:
URL: https://github.com/apache/doris/pull/11400#discussion_r935322822
##
be/CMakeLists.txt:
##
@@ -517,7 +517,7 @@ set(CXX_FLAGS_DEBUG "${CXX_GCC_FLAGS} -O0")
# -O3: Enable all compiler optimizations
# -DNDEBUG: Turn off dchecks/asserts/debug only code.
set(CXX_FLAGS_RELEASE "${CXX_GCC_FLAGS} -O3 -DNDEBUG")
-SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O0 -fsanitize=address
-DADDRESS_SANITIZER")
+SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O3 -fsanitize=address
-DADDRESS_SANITIZER")
Review Comment:
The default asan use O0, it is very slow. I think the better method it to
use O3, then in most cases, it is very quickly to run both ut and P0. If the
developer find the bugs, then he or she could change the default value then run
it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
yiguolei commented on code in PR #11400:
URL: https://github.com/apache/doris/pull/11400#discussion_r935324833
##
be/CMakeLists.txt:
##
@@ -517,7 +517,7 @@ set(CXX_FLAGS_DEBUG "${CXX_GCC_FLAGS} -O0")
# -O3: Enable all compiler optimizations
# -DNDEBUG: Turn off dchecks/asserts/debug only code.
set(CXX_FLAGS_RELEASE "${CXX_GCC_FLAGS} -O3 -DNDEBUG")
-SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O0 -fsanitize=address
-DADDRESS_SANITIZER")
+SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O3 -fsanitize=address
-DADDRESS_SANITIZER")
Review Comment:
I don't think we should provide all the configuations that user could modify
as a configuration in command line. There are too many configurations that user
or developer want to modify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11399: [refactor](agg) Abstract the hash operation into a method
yiguolei merged PR #11399: URL: https://github.com/apache/doris/pull/11399 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [refactor](agg) Abstract the hash operation into a method" (#11399)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 842a5b8e24 [refactor](agg) Abstract the hash operation into a method"
(#11399)
842a5b8e24 is described below
commit 842a5b8e243f631d02d20cde68f2b6ed2d069567
Author: Jerry Hu
AuthorDate: Tue Aug 2 17:27:19 2022 +0800
[refactor](agg) Abstract the hash operation into a method" (#11399)
---
be/src/vec/exec/vaggregation_node.cpp | 271 +-
be/src/vec/exec/vaggregation_node.h | 3 +
2 files changed, 76 insertions(+), 198 deletions(-)
diff --git a/be/src/vec/exec/vaggregation_node.cpp
b/be/src/vec/exec/vaggregation_node.cpp
index 6de36c6d1e..1209750203 100644
--- a/be/src/vec/exec/vaggregation_node.cpp
+++ b/be/src/vec/exec/vaggregation_node.cpp
@@ -706,6 +706,76 @@ bool AggregationNode::_should_expand_preagg_hash_tables() {
_agg_data._aggregated_method_variant);
}
+void AggregationNode::_emplace_into_hash_table(AggregateDataPtr* places,
ColumnRawPtrs& key_columns,
+ const size_t num_rows) {
+std::visit(
+[&](auto&& agg_method) -> void {
+using HashMethodType = std::decay_t;
+using HashTableType = std::decay_t;
+using AggState = typename HashMethodType::State;
+AggState state(key_columns, _probe_key_sz, nullptr);
+
+_pre_serialize_key_if_need(state, agg_method, key_columns,
num_rows);
+
+std::vector hash_values;
+
+if constexpr (HashTableTraits::is_phmap) {
+if (hash_values.size() < num_rows)
hash_values.resize(num_rows);
+if constexpr
(ColumnsHashing::IsPreSerializedKeysHashMethodTraits<
+ AggState>::value) {
+for (size_t i = 0; i < num_rows; ++i) {
+hash_values[i] =
agg_method.data.hash(agg_method.keys[i]);
+}
+} else {
+for (size_t i = 0; i < num_rows; ++i) {
+hash_values[i] =
+
agg_method.data.hash(state.get_key_holder(i, _agg_arena_pool));
+}
+}
+}
+
+/// For all rows.
+for (size_t i = 0; i < num_rows; ++i) {
+AggregateDataPtr aggregate_data = nullptr;
+
+auto emplace_result = [&]() {
+if constexpr
(HashTableTraits::is_phmap) {
+if (LIKELY(i + HASH_MAP_PREFETCH_DIST < num_rows))
{
+if constexpr
(HashTableTraits::is_parallel_phmap) {
+
agg_method.data.prefetch_by_key(state.get_key_holder(
+i + HASH_MAP_PREFETCH_DIST,
_agg_arena_pool));
+} else
+agg_method.data.prefetch_by_hash(
+hash_values[i +
HASH_MAP_PREFETCH_DIST]);
+}
+
+return state.emplace_key(agg_method.data,
hash_values[i], i,
+ _agg_arena_pool);
+} else {
+return state.emplace_key(agg_method.data, i,
_agg_arena_pool);
+}
+}();
+
+/// If a new key is inserted, initialize the states of the
aggregate functions, and possibly something related to the key.
+if (emplace_result.is_inserted()) {
+/// exception-safety - if you can not allocate memory
or create states, then destructors will not be called.
+emplace_result.set_mapped(nullptr);
+
+aggregate_data = _agg_arena_pool.aligned_alloc(
+_total_size_of_aggregate_states,
_align_aggregate_states);
+_create_agg_status(aggregate_data);
+
+emplace_result.set_mapped(aggregate_data);
+} else
+aggregate_data = emplace_result.get_mapped();
+
+places[i] = aggregate_data;
+assert(places[i] != nullptr);
+}
+},
+_agg_data._aggregated_method_variant);
+}
+
Status AggregationNode::_pre_agg_with_serialized_key(doris::vectorized::Block*
in_block,
doris::vectorized::Block*
out_block) {
SCOPED_TIMER(_build_timer);
@@ -816,73 +886,7 @@ Status
AggregationNode::_pre_agg_with_serialized_key(doris::vecto
[GitHub] [doris] jollygrass opened a new issue, #11438: [Bug] After the output file compression is enabled in hive, the data read from hive external table is garbled
jollygrass opened a new issue, #11438:
URL: https://github.com/apache/doris/issues/11438
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Version
1.1.0
### What's Wrong?
After the output file compression is enabled in hive, the data read from
hive external table is garbled 。
-
当hive中开启hive.exec.compress.output后，doris中通过外表方式查询hive表数据，获取的数据乱码。
### What You Expected?
Query data correctly from hive external table.
### How to Reproduce?
in hive
set hive.exec.compress.output=true;
set
mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
drop table if exists dim_test;
create table dim_test (
key_1 string,
key_2 string
);
insert overwrite table dim_test values
('123456','测试'),('abc','123456'),('测试','abc');
select * from dim_test;
key_1 |key_2 |
--|--|
123456|测试|
abc |123456|
测试|abc |
in doris
drop table if exists dim_test;
CREATE TABLE `dim_test` (
key_1 varchar(100),
key_2 varchar(100)
)
ENGINE=HIVE
PROPERTIES (
'hive.metastore.uris' = 'thrift://192.168.1.201:9083',
'database' = 'bigdata_dev',
'table' = 'dim_test'
);
SELECT * from dim_test;
key_1 |key_2|
|-|
$ $$H123456|测试 |
abc |1 , |
测试 |abc |
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
adonis0147 commented on code in PR #11400:
URL: https://github.com/apache/doris/pull/11400#discussion_r935328844
##
be/CMakeLists.txt:
##
@@ -517,7 +517,7 @@ set(CXX_FLAGS_DEBUG "${CXX_GCC_FLAGS} -O0")
# -O3: Enable all compiler optimizations
# -DNDEBUG: Turn off dchecks/asserts/debug only code.
set(CXX_FLAGS_RELEASE "${CXX_GCC_FLAGS} -O3 -DNDEBUG")
-SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O0 -fsanitize=address
-DADDRESS_SANITIZER")
+SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O3 -fsanitize=address
-DADDRESS_SANITIZER")
Review Comment:
> The default asan use O0, it is very slow. I think the better method it to
use O3, then in most cases, it is very quickly to run both ut and P0. If the
developer find the bugs, then he or she could change the default value then run
it again.
It is not a big deal, we can add an argument or an environment variable to
the build.sh to control it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
adonis0147 commented on code in PR #11400:
URL: https://github.com/apache/doris/pull/11400#discussion_r935328844
##
be/CMakeLists.txt:
##
@@ -517,7 +517,7 @@ set(CXX_FLAGS_DEBUG "${CXX_GCC_FLAGS} -O0")
# -O3: Enable all compiler optimizations
# -DNDEBUG: Turn off dchecks/asserts/debug only code.
set(CXX_FLAGS_RELEASE "${CXX_GCC_FLAGS} -O3 -DNDEBUG")
-SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O0 -fsanitize=address
-DADDRESS_SANITIZER")
+SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O3 -fsanitize=address
-DADDRESS_SANITIZER")
Review Comment:
> The default asan use O0, it is very slow. I think the better method it to
use O3, then in most cases, it is very quickly to run both ut and P0. If the
developer find the bugs, then he or she could change the default value then run
it again.
It is not a big deal, we can add an argument or an environment variable to
the `build.sh` to control it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
adonis0147 commented on code in PR #11400:
URL: https://github.com/apache/doris/pull/11400#discussion_r935334238
##
be/CMakeLists.txt:
##
@@ -517,7 +517,7 @@ set(CXX_FLAGS_DEBUG "${CXX_GCC_FLAGS} -O0")
# -O3: Enable all compiler optimizations
# -DNDEBUG: Turn off dchecks/asserts/debug only code.
set(CXX_FLAGS_RELEASE "${CXX_GCC_FLAGS} -O3 -DNDEBUG")
-SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O0 -fsanitize=address
-DADDRESS_SANITIZER")
+SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O3 -fsanitize=address
-DADDRESS_SANITIZER")
Review Comment:
> Could you please provide the command line how to use the configuration?
It is simple. If we pass multiple optimization levels to gcc, only the last
one takes effect. We can just modify the
[L551](https://github.com/apache/doris/pull/11400/files#diff-3507aac2aff9b5fe5f66d28967f3aa848491d4ced2466f6bf201ab3a97531837L551).
Introduce a new variable and pass it from `build.sh` and `run-be-ut.sh`.
Reference:
> If you use multiple -O options, with or without level numbers, the last
such option is the one that is effective.
[Optimize-Options](https://gcc.gnu.org/onlinedocs/gcc-12.1.0/gcc/Optimize-Options.html#Optimize-Options)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
adonis0147 commented on code in PR #11400:
URL: https://github.com/apache/doris/pull/11400#discussion_r935338622
##
be/CMakeLists.txt:
##
@@ -517,7 +517,7 @@ set(CXX_FLAGS_DEBUG "${CXX_GCC_FLAGS} -O0")
# -O3: Enable all compiler optimizations
# -DNDEBUG: Turn off dchecks/asserts/debug only code.
set(CXX_FLAGS_RELEASE "${CXX_GCC_FLAGS} -O3 -DNDEBUG")
-SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O0 -fsanitize=address
-DADDRESS_SANITIZER")
+SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O3 -fsanitize=address
-DADDRESS_SANITIZER")
Review Comment:
> I don't think we should provide all the configuations that user could
modify as a configuration in command line. There are too many configurations
that user or developer want to modify.
We can use `custom_env.sh`, these file can affect the environment variables
before build, it is possible for user not to use an extra command argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
adonis0147 commented on code in PR #11400:
URL: https://github.com/apache/doris/pull/11400#discussion_r935338622
##
be/CMakeLists.txt:
##
@@ -517,7 +517,7 @@ set(CXX_FLAGS_DEBUG "${CXX_GCC_FLAGS} -O0")
# -O3: Enable all compiler optimizations
# -DNDEBUG: Turn off dchecks/asserts/debug only code.
set(CXX_FLAGS_RELEASE "${CXX_GCC_FLAGS} -O3 -DNDEBUG")
-SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O0 -fsanitize=address
-DADDRESS_SANITIZER")
+SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O3 -fsanitize=address
-DADDRESS_SANITIZER")
Review Comment:
> I don't think we should provide all the configuations that user could
modify as a configuration in command line. There are too many configurations
that user or developer want to modify.
We can use `custom_env.sh`, these file can affect the environment variables
before build, it is possible for users not to use an extra command argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
adonis0147 commented on code in PR #11400:
URL: https://github.com/apache/doris/pull/11400#discussion_r935338622
##
be/CMakeLists.txt:
##
@@ -517,7 +517,7 @@ set(CXX_FLAGS_DEBUG "${CXX_GCC_FLAGS} -O0")
# -O3: Enable all compiler optimizations
# -DNDEBUG: Turn off dchecks/asserts/debug only code.
set(CXX_FLAGS_RELEASE "${CXX_GCC_FLAGS} -O3 -DNDEBUG")
-SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O0 -fsanitize=address
-DADDRESS_SANITIZER")
+SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O3 -fsanitize=address
-DADDRESS_SANITIZER")
Review Comment:
> I don't think we should provide all the configuations that user could
modify as a configuration in command line. There are too many configurations
that user or developer want to modify.
We can use `custom_env.sh`. This file can affect the environment variables
before build, it is possible for users not to use an extra command argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] qinchaofeng opened a new pull request, #11439: [doc]fix-doc
qinchaofeng opened a new pull request, #11439: URL: https://github.com/apache/doris/pull/11439 # Proposed changes Revise the translation and supplement the content Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caoliang-web commented on a diff in pull request #10905: [extension] Mysql database import doris by external tables
caoliang-web commented on code in PR #10905:
URL: https://github.com/apache/doris/pull/10905#discussion_r935347149
##
extension/mysql_to_doris/all_tables.sh:
##
@@ -0,0 +1,150 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+
+# This script is used to will mysql databases import doris by external
+
+
+#reference configuration file
+source ./conf/mysql.conf
+source ./conf/doris.conf
+
+#define mysql database and doris database
+d_mysql=$1
+d_doris=$2
+
+#check args
+if [ ! -n "$1" ];then
+echo "please check source database"
+exit
+fi
+if [ ! -n "$2" ];then
+echo "please check sink database"
+exit
+fi
+
+#mkdir files to store tables and tables.sql
+mkdir -p files
+rm -rf ./files/tables
+rm -rf ./files/tables.sql
+
+#get tables from mysql databases
+echo "use $d_mysql; show tables;" |mysql -h$mysql_host -uroot
-p$mysql_password 2>/dev/null >> ./files/tables
+
+#delete tables first line
+sed -i '1d' ./files/tables
+
+#reference tables to create tables.sql
+for table in $(awk -F '\n' '{print $1}' ./files/tables)
+do
+sed -i "/${table}view/d" ./files/tables
+echo "use $d_mysql; show create table ${table};" |mysql -h$mysql_host
-uroot -p$mysql_password 2>/dev/null >> ./files/tables.sql
+echo "print ${table} sql to tables.sql in the file dir"
+
+done
+
+echo '==start to transform mysql table for doris
extral table=='
+#adjust sql
+awk -F '\t' '{print $2}' ./files/tables.sql |awk '!(NR%2)' |awk '{print $0
";"}' > ./files/tables1.sql
+sed -i 's/\\n/\n/g' ./files/tables1.sql
+sed -n '/CREATE TABLE/,/ENGINE\=/p' ./files/tables1.sql > ./files/tables2.sql
+#delete tables special struct
+sed -i '/^ CON/d' ./files/tables2.sql
+sed -i '/^ KEY/d' ./files/tables2.sql
+rm -rf ./files/tables.sql
+rm -rf ./files/tables1.sql
+mv ./files/tables2.sql ./files/tables.sql
+
+#start transform tables struct
+sed -i '/ENGINE=/a) ENGINE=MYSQL\n COMMENT "MYSQL"\nPROPERTIES (\n"host" =
"ApacheDorisHostIp",\n"port" = "3306",\n"user" = "root",\n"password" =
"ApacheDorisHostPassword",\n"database" = "ApacheDorisDataBases",\n"table" =
"ApacheDorisTables");' ./files/tables.sql
+
+#delete match line
+sed -i '/ENGINT=/d' ./files/tables.sql
+sed -i '/PRIMARY KEY/d' ./files/tables.sql
+sed -i '/UNIQUE KEY/d' ./files/tables.sql
+#delete , at the beginning (
+sed -i '/,\s*$/{:loop; N; /,\(\s*\|\n\))/! bloop; s/,\s*[\n]\?\s*)/\n)/}'
./files/tables.sql
+
+#delete a line on keyword
+sed -i -e '$!N;/\n.*ENGINE=MYSQL/!P;D' ./files/tables.sql
+#replace mysql password、database、table、host
+for t_name in $(awk -F '\n' '{print $1}' ./files/tables)
+do
+sed -i "0,/ApacheDorisHostIp/s/ApacheDorisHostIp/${mysql_host}/"
./files/tables.sql
+sed -i
"0,/ApacheDorisHostPassword/s/ApacheDorisHostPassword/${mysql_password}/"
./files/tables.sql
+sed -i "0,/ApacheDorisDataBases/s/ApacheDorisDataBases/${d_mysql}/"
./files/tables.sql
+sed -i "0,/ApacheDorisTables/s/ApacheDorisTables/${t_name}/"
./files/tables.sql
+
+done
+##
+#replace mysql type with doris
+sed -i 's/AUTO_INCREMENT//g' ./files/tables.sql
+sed -i 's/CHARACTER SET utf8 COLLATE utf8_bin//g' ./files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci//g'
./files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8mb4_bin//g' ./files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci//g'
./files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8_general_ci//g' ./files/tables.sql
+sed -i 's/CHARACTER SET utf8 COLLATE utf8_general_ci//g' ./files/tables.sql
+sed -i 's/DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP//g'
./files/tables.sql
+sed -i 's/DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP//g' ./files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8mb4_bin//g' ./files/tables.sql
+sed -i 's/DEFAULT CURRENT_TIMESTAMP
[GitHub] [doris] HappenLee opened a new pull request, #11440: [Bug](Date/Datetime) Invalid date/datetime to NULL
HappenLee opened a new pull request, #11440: URL: https://github.com/apache/doris/pull/11440 # Proposed changes Issue Number: close #11429 #9861 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11422: [Enhancement](TabletInvertIndex) use StampLock to instead of ReentrantReadWriteLock to get better performance
yiguolei merged PR #11422: URL: https://github.com/apache/doris/pull/11422 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #11421: [Enhancement] Improve the performance of meta management(Step-1)
yiguolei closed issue #11421: [Enhancement] Improve the performance of meta management(Step-1) URL: https://github.com/apache/doris/issues/11421 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (842a5b8e24 -> d6149e4777)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 842a5b8e24 [refactor](agg) Abstract the hash operation into a method" (#11399) add d6149e4777 [Enhancement](TabletInvertIndex) use StampLock to instead of ReentranReadWriteLock to get better performance (#11422) No new revisions were added by this update. Summary of changes: .../apache/doris/catalog/TabletInvertedIndex.java | 86 +++--- .../doris/common/proc/StatisticProcNode.java | 2 +- 2 files changed, 44 insertions(+), 44 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 commented on a diff in pull request #10905: [extension] Mysql database import doris by external tables
Toms1999 commented on code in PR #10905:
URL: https://github.com/apache/doris/pull/10905#discussion_r935358450
##
extension/mysql_to_doris/all_tables.sh:
##
@@ -0,0 +1,150 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+
+# This script is used to will mysql databases import doris by external
+
+
+#reference configuration file
+source ./conf/mysql.conf
+source ./conf/doris.conf
+
+#define mysql database and doris database
+d_mysql=$1
+d_doris=$2
+
+#check args
+if [ ! -n "$1" ];then
+echo "please check source database"
+exit
+fi
+if [ ! -n "$2" ];then
+echo "please check sink database"
+exit
+fi
+
+#mkdir files to store tables and tables.sql
+mkdir -p files
+rm -rf ./files/tables
+rm -rf ./files/tables.sql
+
+#get tables from mysql databases
+echo "use $d_mysql; show tables;" |mysql -h$mysql_host -uroot
-p$mysql_password 2>/dev/null >> ./files/tables
+
+#delete tables first line
+sed -i '1d' ./files/tables
+
+#reference tables to create tables.sql
+for table in $(awk -F '\n' '{print $1}' ./files/tables)
+do
+sed -i "/${table}view/d" ./files/tables
+echo "use $d_mysql; show create table ${table};" |mysql -h$mysql_host
-uroot -p$mysql_password 2>/dev/null >> ./files/tables.sql
+echo "print ${table} sql to tables.sql in the file dir"
+
+done
+
+echo '==start to transform mysql table for doris
extral table=='
+#adjust sql
+awk -F '\t' '{print $2}' ./files/tables.sql |awk '!(NR%2)' |awk '{print $0
";"}' > ./files/tables1.sql
+sed -i 's/\\n/\n/g' ./files/tables1.sql
+sed -n '/CREATE TABLE/,/ENGINE\=/p' ./files/tables1.sql > ./files/tables2.sql
+#delete tables special struct
+sed -i '/^ CON/d' ./files/tables2.sql
+sed -i '/^ KEY/d' ./files/tables2.sql
+rm -rf ./files/tables.sql
+rm -rf ./files/tables1.sql
+mv ./files/tables2.sql ./files/tables.sql
+
+#start transform tables struct
+sed -i '/ENGINE=/a) ENGINE=MYSQL\n COMMENT "MYSQL"\nPROPERTIES (\n"host" =
"ApacheDorisHostIp",\n"port" = "3306",\n"user" = "root",\n"password" =
"ApacheDorisHostPassword",\n"database" = "ApacheDorisDataBases",\n"table" =
"ApacheDorisTables");' ./files/tables.sql
+
+#delete match line
+sed -i '/ENGINT=/d' ./files/tables.sql
+sed -i '/PRIMARY KEY/d' ./files/tables.sql
+sed -i '/UNIQUE KEY/d' ./files/tables.sql
+#delete , at the beginning (
+sed -i '/,\s*$/{:loop; N; /,\(\s*\|\n\))/! bloop; s/,\s*[\n]\?\s*)/\n)/}'
./files/tables.sql
+
+#delete a line on keyword
+sed -i -e '$!N;/\n.*ENGINE=MYSQL/!P;D' ./files/tables.sql
+#replace mysql password、database、table、host
+for t_name in $(awk -F '\n' '{print $1}' ./files/tables)
+do
+sed -i "0,/ApacheDorisHostIp/s/ApacheDorisHostIp/${mysql_host}/"
./files/tables.sql
+sed -i
"0,/ApacheDorisHostPassword/s/ApacheDorisHostPassword/${mysql_password}/"
./files/tables.sql
+sed -i "0,/ApacheDorisDataBases/s/ApacheDorisDataBases/${d_mysql}/"
./files/tables.sql
+sed -i "0,/ApacheDorisTables/s/ApacheDorisTables/${t_name}/"
./files/tables.sql
+
+done
+##
+#replace mysql type with doris
+sed -i 's/AUTO_INCREMENT//g' ./files/tables.sql
+sed -i 's/CHARACTER SET utf8 COLLATE utf8_bin//g' ./files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci//g'
./files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8mb4_bin//g' ./files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci//g'
./files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8_general_ci//g' ./files/tables.sql
+sed -i 's/CHARACTER SET utf8 COLLATE utf8_general_ci//g' ./files/tables.sql
+sed -i 's/DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP//g'
./files/tables.sql
+sed -i 's/DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP//g' ./files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8mb4_bin//g' ./files/tables.sql
+sed -i 's/DEFAULT CURRENT_TIMESTAMP ON
[GitHub] [doris] qinchaofeng commented on issue #11412: [Bug] doris子查询顺序会乱掉
qinchaofeng commented on issue #11412: URL: https://github.com/apache/doris/issues/11412#issuecomment-1202282834 从数据库查询角度来说，这确实是个问题 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei opened a new pull request, #11441: [refactor](schema change)Remove delete from sc
yiguolei opened a new pull request, #11441: URL: https://github.com/apache/doris/pull/11441 # Proposed changes Issue Number: close #xxx ## Problem summary Currently, schema change will read a row block and call delete handler's is filter data to filter the row block. BUT rowset reader alreay filter the data during read so that there is no need to call delete handler to filter it again. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] qinchaofeng commented on issue #11412: [Bug] doris子查询顺序会乱掉
qinchaofeng commented on issue #11412: URL: https://github.com/apache/doris/issues/11412#issuecomment-1202286719 从数据库查询角度来说，这也许是个问题，但是写SQL的时候最好不要这样使用。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11436: [doc]fix-doc
github-actions[bot] commented on PR #11436: URL: https://github.com/apache/doris/pull/11436#issuecomment-1202287561 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11436: [doc]fix-doc
github-actions[bot] commented on PR #11436: URL: https://github.com/apache/doris/pull/11436#issuecomment-1202287595 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #11441: [refactor](schema change)Remove delete from sc
BiteThet commented on PR #11441: URL: https://github.com/apache/doris/pull/11441#issuecomment-1202290899 Better add some test about "schema change after delete" to regression-test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: stream load fix
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 8dc98907104 stream load fix 8dc98907104 is described below commit 8dc98907104d5eafc252406625a6da47218db68f Author: jiafeng.zhang AuthorDate: Tue Aug 2 18:26:29 2022 +0800 stream load fix --- docs/data-operate/import/import-way/stream-load-manual.md | 2 +- .../current/data-operate/import/import-way/stream-load-manual.md| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/data-operate/import/import-way/stream-load-manual.md b/docs/data-operate/import/import-way/stream-load-manual.md index 0b65f87aa46..7e92f1b1f9e 100644 --- a/docs/data-operate/import/import-way/stream-load-manual.md +++ b/docs/data-operate/import/import-way/stream-load-manual.md @@ -175,7 +175,7 @@ The number of rows in the original file = `dpp.abnorm.ALL + dpp.norm.ALL` The default two-phase bulk transaction commit is off. - > **Open method:** Configure `disable_stream_load_2pc=false` in be.conf (restart takes effect) and declare `two_phase_commit=true` in HEADER. + > **Open method:** Configure `disable_stream_load_2pc=false` in be.conf and declare `two_phase_commit=true` in HEADER. Example： diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/stream-load-manual.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/stream-load-manual.md index 12299fba999..25f8607ea5d 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/stream-load-manual.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/stream-load-manual.md @@ -186,7 +186,7 @@ Stream Load 由于使用的是 HTTP 协议，所以所有导入任务有关的 默认的两阶段批量事务提交为关闭。 - > **开启方式：** 在be.conf中配置`disable_stream_load_2pc=false`（重启生效） 并且 在 HEADER 中声明 `two_phase_commit=true` 。 + > **开启方式：** 在be.conf中配置`disable_stream_load_2pc=false` 并且 在 HEADER 中声明 `two_phase_commit=true` 。 示例： - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen opened a new pull request, #11442: [regression](unique-key-merge)add tpch_sf1_p1 test
hello-stephen opened a new pull request, #11442: URL: https://github.com/apache/doris/pull/11442 # Proposed changes Issue Number: close #xxx ## Problem summary tpch_sf1_p1 is writted to test unique key table merge correctly. It creates unique key table and sets bucket num to 1 in order to make sure that many rowsets will be created during loading and then the merge process will be triggered. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hf200012 merged pull request #11436: [doc]fix-doc
hf200012 merged PR #11436: URL: https://github.com/apache/doris/pull/11436 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: fix-doc (#11436)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 6e684c695d fix-doc (#11436) 6e684c695d is described below commit 6e684c695dda7514c81312f6d5f6d4c409b92b9f Author: Liqf <109049295+lemonlit...@users.noreply.github.com> AuthorDate: Tue Aug 2 18:38:30 2022 +0800 fix-doc (#11436) fe-config fix --- docs/en/docs/admin-manual/config/fe-config.md| 32 docs/zh-CN/docs/admin-manual/config/fe-config.md | 14 +-- 2 files changed, 23 insertions(+), 23 deletions(-) diff --git a/docs/en/docs/admin-manual/config/fe-config.md b/docs/en/docs/admin-manual/config/fe-config.md index 81809a9e4f..51a086cddc 100644 --- a/docs/en/docs/admin-manual/config/fe-config.md +++ b/docs/en/docs/admin-manual/config/fe-config.md @@ -199,7 +199,7 @@ Worker thread pool is not set by default, set according to your needs ### jetty_server_max_http_post_size -Default：100 * 1024 * 1024 （100MB） +Default：`100 * 1024 * 1024` （100MB） This is the maximum number of bytes of the file uploaded by the put or post method, the default value: 100MB @@ -811,7 +811,7 @@ MasterOnly：true ### max_bytes_per_broker_scanner -Default：3 * 1024 * 1024 * 1024L （3G） +Default：`3 * 1024 * 1024 * 1024L` （3G） IsMutable：true @@ -827,7 +827,7 @@ if set to false, auth check will be disable, in case some goes wrong with the ne ### tablet_stat_update_interval_second -Default：300，（5min） +Default：300（5min） update interval of tablet stat , All frontends will get tablet stat from all backends at each interval @@ -866,7 +866,7 @@ MasterOnly：true ### storage_min_left_capacity_bytes -Default： 2 * 1024 * 1024 * 1024 (2GB) +Default： `2 * 1024 * 1024 * 1024` (2GB) IsMutable：true @@ -896,7 +896,7 @@ Default：/lib/hadoop-client/hadoop/bin/hadoop ### dpp_bytes_per_reduce -Default：100 * 1024 * 1024L; // 100M +Default：`100 * 1024 * 1024L` (100M) ### dpp_default_cluster @@ -1001,7 +1001,7 @@ Maximal number of connections per FE. ### check_consistency_default_timeout_second -Default：600 （10分钟） +Default：600 （10 minutes） IsMutable：true @@ -1112,7 +1112,7 @@ After dropping database(table/partition), you can recover it by using RECOVER st ### storage_cooldown_second -Default：30 * 24 * 3600L （30day） +Default：`30 * 24 * 3600L` （30 days） When create a table(or partition), you can specify its storage medium(HDD or SSD). If set to SSD, this specifies the default duration that tablets will stay on SSD. After that, tablets will be moved to HDD automatically. You can set storage cooldown time in CREATE TABLE stmt. @@ -1124,7 +1124,7 @@ When create a table(or partition), you can specify its storage medium(HDD or SSD ### max_backend_down_time_second -Default：3600 （1hour） +Default：3600 （1 hours） IsMutable：true @@ -1134,7 +1134,7 @@ If a backend is down for *max_backend_down_time_second*, a BACKEND_DOWN event wi ### alter_table_timeout_second -Default：86400 （1day） +Default：86400 （1 day） IsMutable：true @@ -1653,7 +1653,7 @@ node(FE or BE) will be considered belonging to the same Palo cluster if they hav ### auth_token -Default：空 +Default：empty Cluster token used for internal authentication. @@ -1963,7 +1963,7 @@ Slow query contains all queries which cost exceed *qe_slow_log_ms* ### qe_slow_log_ms -Default：5000 （5秒） +Default：5000 （5 seconds） If the response time of a query exceed this threshold, it will be recorded in audit log as slow_query. @@ -2004,7 +2004,7 @@ Whether the plug-in is enabled, enabled by default ### label_keep_max_second -Default：3 * 24 * 3600 (3day) +Default：`3 * 24 * 3600` (3 days) IsMutable：true @@ -2026,7 +2026,7 @@ For some high-frequency load work, such as: INSERT, STREAMING LOAD, ROUTINE_LOAD ### history_job_keep_max_second -Default：7 * 24 * 3600 （7 day） +Default：`7 * 24 * 3600` （7 day） IsMutable：true @@ -2042,7 +2042,7 @@ Load label cleaner will run every *label_clean_interval_second* to clean the out ### delete_info_keep_max_second -Default：3 * 24 * 3600 (3day) +Default：`3 * 24 * 3600` (3day) IsMutable：true @@ -2134,7 +2134,7 @@ The valid ratio threshold of the difference between the version count of the slo ### min_bytes_indicate_replica_too_large -Default: 2 * 1024 * 1024 * 1024 (2G) +Default: `2 * 1024 * 1024 * 1024` (2G) Dynamically configured: true @@ -2176,7 +2176,7 @@ Whether it is a configuration item unique to the Master FE node: true The minimum data size required to commit a transaction. If the data size received by Fe is smaller than it, it will continue to wait for the next batch of data until the time exceeds `sync_commit_interval_second`. The default value is 15MB, if you want to modify this configuration, please make sure
[GitHub] [doris] hf200012 merged pull request #11416: [doc](sql-block-rule) add more example
hf200012 merged PR #11416: URL: https://github.com/apache/doris/pull/11416 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [doc](sql-block-rule) add more example (#11416)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 4850873f66 [doc](sql-block-rule) add more example (#11416) 4850873f66 is described below commit 4850873f66d7323d008aa594ddc267c9f16e3706 Author: Mingyu Chen AuthorDate: Tue Aug 2 18:39:13 2022 +0800 [doc](sql-block-rule) add more example (#11416) sql-block-rule ： add more example --- .../Create/CREATE-SQL-BLOCK-RULE.md| 66 +- .../Create/CREATE-SQL-BLOCK-RULE.md| 66 +- 2 files changed, 80 insertions(+), 52 deletions(-) diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md index 0790a9a3fe..a513baa7ec 100644 --- a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md +++ b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md @@ -61,35 +61,49 @@ Parameter Description: 1. Create a block rule named test_rule - ```sql - mysql> CREATE SQL_BLOCK_RULE test_rule - -> PROPERTIES( - -> "sql"="select \\* from order_analysis;", - -> "global"="false", - -> "enable"="true" - -> ); - Query OK, 0 rows affected (0.01 sec) - - - When we execute the sql we just defined in the rule, an exception error will be returned. The example is as follows: - - ```sql - mysql> select * from order_analysis; - ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule - +```sql +CREATE SQL_BLOCK_RULE test_rule +PROPERTIES( +"sql"="select \\* from order_analysis;", +"global"="false", +"enable"="true" +); + + +When we execute the sql we just defined in the rule, an exception error will be returned. The example is as follows: + +```sql +select * from order_analysis; +ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule + 2. Create test_rule2, limit the maximum number of scanned partitions to 30, and limit the maximum scan base to 10 billion rows. The example is as follows: - ```sql - mysql> CREATE SQL_BLOCK_RULE test_rule2 - -> PROPERTIES ( - -> "partition_num" = "30", - -> "cardinality" = "100", - -> "global" = "false", - -> "enable" = "true" - -> ); - Query OK, 0 rows affected (0.01 sec) - +```sql +CREATE SQL_BLOCK_RULE test_rule2 +PROPERTIES ( +"partition_num" = "30", +"cardinality" = "100", +"global" = "false", +"enable" = "true" +); + + +3. Create SQL BLOCK RULE with special chars + +```sql +CREATE SQL_BLOCK_RULE test_rule3 +PROPERTIES +( +"sql" = "select count\\(1\\) from db1.tbl1" +); + +CREATE SQL_BLOCK_RULE test_rule4 +PROPERTIES +( +"sql" = "select \\* from db1.tbl1" +); +``` ### Keywords diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md index 17032f5afe..1c41aa8d56 100644 --- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md +++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-SQL-BLOCK-RULE.md @@ -61,36 +61,50 @@ CREATE SQL_BLOCK_RULE rule_name 1. 创建一个名称为 test_rule 的阻止规则 - ```sql - mysql> CREATE SQL_BLOCK_RULE test_rule - -> PROPERTIES( - -> "sql"="select \\* from order_analysis;", - -> "global"="false", - -> "enable"="true" - -> ); - Query OK, 0 rows affected (0.01 sec) - ``` - - 当我们去执行刚才我们定义在规则里的sql时就会返回异常错误，示例如下： - - ```sql - mysql> select * from order_analysis; - ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule - ``` + ```sql + CREATE SQL_BLOCK_RULE test_rule + PROPERTIES( + "sql"="select \\* from order_analysis;", + "global"="false", + "enable"="true" + ); +``` + +当我们去执行刚才我们定义在规则里的sql时就会返回异常错误，示例如下： + +```sql +mysql> select * from order_analysis; +ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule +``` 2. 创建 test_rule2，将最大扫描的分区数量限制在30个，最大扫描基数限制在100亿行，示例如下： - ```sql - mysql> CREATE SQL_BLOCK_RULE test_rule2 - -> PROPERTIES +```sql +CREATE SQL_BLOCK_RULE test_rule2 +PROPERTIES ( - -> "partition_num" = "30", - -> "cardinality" = "100", - ->
[GitHub] [doris] Gabriel39 opened a new pull request, #11443: [WIP](decimalv3) WIP
Gabriel39 opened a new pull request, #11443: URL: https://github.com/apache/doris/pull/11443 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caoliang-web commented on a diff in pull request #10905: [extension] Mysql database import doris by external tables
caoliang-web commented on code in PR #10905:
URL: https://github.com/apache/doris/pull/10905#discussion_r935427739
##
extension/mysql_to_doris/user_define_tables.sh:
##
@@ -0,0 +1,144 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+
+# This script is used to will mysql tables import doris by external and by
user to define tables
+
+
+
+#reference configuration file
+source ./conf/mysql.conf
+source ./conf/doris.conf
+
+#define mysql database and doris database
+d_mysql=$1
+d_doris=$2
+
+#check args
+if [ ! -n "$1" ];then
+echo "please check source database"
+exit
+fi
+if [ ! -n "$2" ];then
+echo "please check sink database"
+exit
+fi
+
+#mkdir files to store tables and tables.sql
+mkdir -p user_files
+rm -rf ./user_files/tables
+rm -rf ./user_files/tables.sql
+
+#reference tables to create tables.sql
+for table in $(awk -F '\n' '{print $1}' ./conf/tables)
+do
+sed -i "/${table}view/d" ./conf/tables
+echo "use $d_mysql; show create table ${table};" |mysql -h$mysql_host
-uroot -p$mysql_password 2>/dev/null >> ./user_files/tables.sql
+echo "print ${table} sql to tables.sql in the user_file dir"
+done
+
+echo '==start to transform mysql table for doris
extral table=='
+
+#adjust sql
+awk -F '\t' '{print $2}' ./user_files/tables.sql |awk '!(NR%2)' |awk '{print
$0 ";"}' > ./user_files/tables1.sql
+sed -i 's/\\n/\n/g' ./user_files/tables1.sql
+sed -n '/CREATE TABLE/,/ENGINE\=/p' ./user_files/tables1.sql >
./user_files/tables2.sql
+
+#delete tables special struct
+sed -i '/^ CON/d' ./user_files/tables2.sql
+sed -i '/^ KEY/d' ./user_files/tables2.sql
+rm -rf ./user_files/tables.sql
+rm -rf ./user_files/tables1.sql
+mv ./user_files/tables2.sql ./user_files/tables.sql
+#start transform tables struct
+sed -i '/ENGINE=/a) ENGINE=MYSQL\n COMMENT "MYSQL"\nPROPERTIES (\n"host" =
"ApacheDorisHostIp",\n"port" = "3306",\n"user" = "root",\n"password" =
"ApacheDorisHostPassword",\n"database" = "ApacheDorisDataBases",\n"table" =
"ApacheDorisTables");' ./user_files/tables.sql
+
+#delete match line
+sed -i '/ENGINT=/d' ./user_files/tables.sql
+sed -i '/PRIMARY KEY/d' ./user_files/tables.sql
+sed -i '/UNIQUE KEY/d' ./user_files/tables.sql
+#delete , at the beginning (
+sed -i '/,\s*$/{:loop; N; /,\(\s*\|\n\))/! bloop; s/,\s*[\n]\?\s*)/\n)/}'
./user_files/tables.sql
+
+#delete a line on keyword
+sed -i -e '$!N;/\n.*ENGINE=MYSQL/!P;D' ./user_files/tables.sql
+
+#replace mysql password、database、table、host
+for t_name in $(awk -F '\n' '{print $1}' ./conf/tables)
+do
+sed -i "0,/ApacheDorisHostIp/s/ApacheDorisHostIp/${mysql_host}/"
./user_files/tables.sql
+sed -i
"0,/ApacheDorisHostPassword/s/ApacheDorisHostPassword/${mysql_password}/"
./user_files/tables.sql
+sed -i "0,/ApacheDorisDataBases/s/ApacheDorisDataBases/${d_mysql}/"
./user_files/tables.sql
+sed -i "0,/ApacheDorisTables/s/ApacheDorisTables/${t_name}/"
./user_files/tables.sql
+
+done
+#replace mysql type with doris
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/datetime/g' ./user_files/tables.sql
+sed -i 's/AUTO_INCREMENT//g' ./user_files/tables.sql
+sed -i 's/\//g' ./user_files/tables.sql
+sed -i 's/\//g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/enum([^)]*)/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci//g'
./user_files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8_general_ci//g
[GitHub] [doris] adonis0147 opened a new pull request, #11444: [enhancement](build) Support customizing extra compile flags
adonis0147 opened a new pull request, #11444: URL: https://github.com/apache/doris/pull/11444 # Proposed changes Add an environment variable `EXTRA_CXX_FLAGS` to customize the extra compile flags. ## Problem summary In our daily work, someones may want to change the compile flags to satisfy their requirements. This pr introduces a way to customize extra compile flags. E.g. Change the optimization level: ```shell BUILD_TYPE=ASAN EXTRA_CXX_FLAGS=-O3 ./build.sh --be ``` ```shell export EXTRA_CXX_FLAGS=-O3 BUILD_TYPE=ASAN ./build.sh --be ``` ```shell echo EXTRA_CXX_FLAGS=-O3 >> custom_env.sh BUILD_TYPE=ASAN ./build.sh --be ``` ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] suyanhanx opened a new pull request, #11445: fix(typo): in docs(zh-CN) how to be a committer
suyanhanx opened a new pull request, #11445: URL: https://github.com/apache/doris/pull/11445 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [x] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
adonis0147 commented on code in PR #11400:
URL: https://github.com/apache/doris/pull/11400#discussion_r935334238
##
be/CMakeLists.txt:
##
@@ -517,7 +517,7 @@ set(CXX_FLAGS_DEBUG "${CXX_GCC_FLAGS} -O0")
# -O3: Enable all compiler optimizations
# -DNDEBUG: Turn off dchecks/asserts/debug only code.
set(CXX_FLAGS_RELEASE "${CXX_GCC_FLAGS} -O3 -DNDEBUG")
-SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O0 -fsanitize=address
-DADDRESS_SANITIZER")
+SET(CXX_FLAGS_ASAN "${CXX_GCC_FLAGS} -O3 -fsanitize=address
-DADDRESS_SANITIZER")
Review Comment:
> Could you please provide the command line how to use the configuration?
It is simple. If we pass multiple optimization levels to gcc, only the last
one takes effect. We can just modify the
[L551](https://github.com/apache/doris/pull/11400/files#diff-3507aac2aff9b5fe5f66d28967f3aa848491d4ced2466f6bf201ab3a97531837L551).
Introduce a new variable and pass it from `build.sh` and `run-be-ut.sh`.
Reference:
> If you use multiple -O options, with or without level numbers, the last
such option is the one that is effective.
[Optimize-Options](https://gcc.gnu.org/onlinedocs/gcc-12.1.0/gcc/Optimize-Options.html#Optimize-Options)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #11400: [improvement](regresstion test) Improve performance of ASAN build by …
adonis0147 commented on PR #11400: URL: https://github.com/apache/doris/pull/11400#issuecomment-1202403760 @jacktengg @yiguolei #11444 introduces a way to change the default optimization level. The followings are the reasons why the default optimization level should be `-O0` or `-Og`: 1. For most users who want to build Doris from source, they can build it by `bash build.sh --be`. The build type of this command is `RELEASE` and the optimization level is `-O3`. All works fine as expect. 2. For most developers who develop Doris daily, `-O3` may harm the debugging experience. The standard edit-compile-debug cycle really matters which is the reason why `-Og` was introduced in `GCC`. 3. For other scenarios (workflows and etc.) , we can just modify the environment variables or deploy scripts to satisfy our requirement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 commented on a diff in pull request #10905: [extension] Mysql database import doris by external tables
Toms1999 commented on code in PR #10905:
URL: https://github.com/apache/doris/pull/10905#discussion_r935507013
##
extension/mysql_to_doris/user_define_tables.sh:
##
@@ -0,0 +1,144 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+
+# This script is used to will mysql tables import doris by external and by
user to define tables
+
+
+
+#reference configuration file
+source ./conf/mysql.conf
+source ./conf/doris.conf
+
+#define mysql database and doris database
+d_mysql=$1
+d_doris=$2
+
+#check args
+if [ ! -n "$1" ];then
+echo "please check source database"
+exit
+fi
+if [ ! -n "$2" ];then
+echo "please check sink database"
+exit
+fi
+
+#mkdir files to store tables and tables.sql
+mkdir -p user_files
+rm -rf ./user_files/tables
+rm -rf ./user_files/tables.sql
+
+#reference tables to create tables.sql
+for table in $(awk -F '\n' '{print $1}' ./conf/tables)
+do
+sed -i "/${table}view/d" ./conf/tables
+echo "use $d_mysql; show create table ${table};" |mysql -h$mysql_host
-uroot -p$mysql_password 2>/dev/null >> ./user_files/tables.sql
+echo "print ${table} sql to tables.sql in the user_file dir"
+done
+
+echo '==start to transform mysql table for doris
extral table=='
+
+#adjust sql
+awk -F '\t' '{print $2}' ./user_files/tables.sql |awk '!(NR%2)' |awk '{print
$0 ";"}' > ./user_files/tables1.sql
+sed -i 's/\\n/\n/g' ./user_files/tables1.sql
+sed -n '/CREATE TABLE/,/ENGINE\=/p' ./user_files/tables1.sql >
./user_files/tables2.sql
+
+#delete tables special struct
+sed -i '/^ CON/d' ./user_files/tables2.sql
+sed -i '/^ KEY/d' ./user_files/tables2.sql
+rm -rf ./user_files/tables.sql
+rm -rf ./user_files/tables1.sql
+mv ./user_files/tables2.sql ./user_files/tables.sql
+#start transform tables struct
+sed -i '/ENGINE=/a) ENGINE=MYSQL\n COMMENT "MYSQL"\nPROPERTIES (\n"host" =
"ApacheDorisHostIp",\n"port" = "3306",\n"user" = "root",\n"password" =
"ApacheDorisHostPassword",\n"database" = "ApacheDorisDataBases",\n"table" =
"ApacheDorisTables");' ./user_files/tables.sql
+
+#delete match line
+sed -i '/ENGINT=/d' ./user_files/tables.sql
+sed -i '/PRIMARY KEY/d' ./user_files/tables.sql
+sed -i '/UNIQUE KEY/d' ./user_files/tables.sql
+#delete , at the beginning (
+sed -i '/,\s*$/{:loop; N; /,\(\s*\|\n\))/! bloop; s/,\s*[\n]\?\s*)/\n)/}'
./user_files/tables.sql
+
+#delete a line on keyword
+sed -i -e '$!N;/\n.*ENGINE=MYSQL/!P;D' ./user_files/tables.sql
+
+#replace mysql password、database、table、host
+for t_name in $(awk -F '\n' '{print $1}' ./conf/tables)
+do
+sed -i "0,/ApacheDorisHostIp/s/ApacheDorisHostIp/${mysql_host}/"
./user_files/tables.sql
+sed -i
"0,/ApacheDorisHostPassword/s/ApacheDorisHostPassword/${mysql_password}/"
./user_files/tables.sql
+sed -i "0,/ApacheDorisDataBases/s/ApacheDorisDataBases/${d_mysql}/"
./user_files/tables.sql
+sed -i "0,/ApacheDorisTables/s/ApacheDorisTables/${t_name}/"
./user_files/tables.sql
+
+done
+#replace mysql type with doris
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/datetime/g' ./user_files/tables.sql
+sed -i 's/AUTO_INCREMENT//g' ./user_files/tables.sql
+sed -i 's/\//g' ./user_files/tables.sql
+sed -i 's/\//g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/enum([^)]*)/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/\/varchar(65533)/g' ./user_files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci//g'
./user_files/tables.sql
+sed -i 's/CHARACTER SET utf8mb4 COLLATE utf8_general_ci//g'
.
[GitHub] [doris] yangzhg commented on a diff in pull request #11186: [feature](cache) Add FileCache for RemoteFile
yangzhg commented on code in PR #11186: URL: https://github.com/apache/doris/pull/11186#discussion_r935530597 ## be/src/common/config.h: ## @@ -781,7 +781,10 @@ CONF_mBool(enable_function_pushdown, "false"); CONF_Int32(cooldown_thread_num, "5"); CONF_mInt64(generate_cooldown_task_interval_sec, "20"); CONF_Int32(concurrency_per_dir, "2"); -CONF_mInt64(cooldown_lag_time_sec, "10800"); // 3h +CONF_mInt64(cooldown_lag_time_sec, "10800");// 3h +CONF_mInt64(max_sub_cache_file_size, "1073741824"); // 1GB +CONF_mInt64(file_cache_alive_time_sec, "604800"); // 1 week +CONF_String(file_cache_type, ""); Review Comment: better add a comment to describe the possible value -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11445: typo: in docs(zh-CN) how to be a committer
github-actions[bot] commented on PR #11445: URL: https://github.com/apache/doris/pull/11445#issuecomment-1202444284 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11441: [refactor](schema change)Remove delete from sc
github-actions[bot] commented on PR #11441: URL: https://github.com/apache/doris/pull/11441#issuecomment-1202477171 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11441: [refactor](schema change)Remove delete from sc
github-actions[bot] commented on PR #11441: URL: https://github.com/apache/doris/pull/11441#issuecomment-1202477232 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new pull request, #11446: [enhancement](Nereids) support all join type in Nereids that could do join by HashJoinNode
morrySnow opened a new pull request, #11446: URL: https://github.com/apache/doris/pull/11446 # Proposed changes add and test join type: - inner join - left outer join - right outer join - left semi join - right semi join - left anti join - right anti join ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [x] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #11446: [enhancement](Nereids) support all join type in Nereids that could do join by HashJoinNode
morrySnow commented on PR #11446: URL: https://github.com/apache/doris/pull/11446#issuecomment-1202488680 @924060929 PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen opened a new pull request, #11447: [regression](unique-key-merge-test) add ssb_sf1_p1 test
hello-stephen opened a new pull request, #11447: URL: https://github.com/apache/doris/pull/11447 # Proposed changes Issue Number: close #xxx ## Problem summary ssb_sf1_p1 is writted to test unique key table merge correctly. It creates unique key table and sets bucket num to 1 in order to make sure that many rowsets will be created during loading and then the merge process will be triggered. It includes load data by INSERT INTO ... SELECT ... ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #11446: [enhancement](Nereids) support all join type in Nereids that could do join by HashJoinNode
924060929 commented on code in PR #11446:
URL: https://github.com/apache/doris/pull/11446#discussion_r935602174
##
regression-test/suites/nereids_syntax/join.groovy:
##
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("join") {
+sql """
+SET enable_nereids_planner=true
Review Comment:
```groovy
sql 'set enable_vectorized_engine=true'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #11448: [minor](log) add some log to observe the change of table's state.
morningman opened a new pull request, #11448: URL: https://github.com/apache/doris/pull/11448 # Proposed changes Issue Number: close #xxx ## Problem summary When upgrading to 1.x, some table's state may change to ROLLUP, Resulting in not able to create/drop/modify partition. I haven't find the root cause yet, so I add some log to observe the change of table's state. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #11388: [Community](readme)Update README.md
morningman merged PR #11388: URL: https://github.com/apache/doris/pull/11388 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (4850873f66 -> 7148e3dfaa)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 4850873f66 [doc](sql-block-rule) add more example (#11416) add 7148e3dfaa [Community](readme)Update README.md (#11388) No new revisions were added by this update. Summary of changes: README.md | 148 +++--- 1 file changed, 112 insertions(+), 36 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #11415: (feature-wip)[parquet-reader] file reader
morningman commented on code in PR #11415:
URL: https://github.com/apache/doris/pull/11415#discussion_r935694914
##
be/src/vec/exec/format/parquet/vparquet_reader.h:
##
@@ -48,40 +44,46 @@ namespace doris::vectorized {
class ParquetReader {
public:
-ParquetReader(FileReader* file_reader, int64_t batch_size, int32_t
num_of_columns_from_file,
+ParquetReader(FileReader* file_reader, int32_t num_of_columns_from_file,
int64_t range_start_offset, int64_t range_size);
+
~ParquetReader();
-virtual Status init_reader(const TupleDescriptor* tuple_desc,
- const std::vector&
tuple_slot_descs,
- const std::vector& conjunct_ctxs,
- const std::string& timezone) = 0;
-virtual Status next_batch(bool* eof) = 0;
+
+Status init_reader(const TupleDescriptor* tuple_desc,
+ const std::vector& tuple_slot_descs,
+ const std::vector& conjunct_ctxs, const
std::string& timezone);
+
+Status read_next_batch(Block* block);
+
+bool has_next() const { return !_batch_eof; };
+
//std::shared_ptr& statistics() { return _statistics; }
-void close() {};
-int64_t size(int64_t* size) { return _file_reader->size(); }
+void close();
+
+int64_t size() const { return _file_reader->size(); }
private:
-int64_t _get_row_group_start_offset(const tparquet::RowGroup& row_group);
+Status _column_indices(const std::vector&
tuple_slot_descs);
+void _init_row_group_reader();
+void _fill_block_data(std::vector columns);
+bool _has_page_index(std::vector columns);
+Status _process_page_index(std::vector columns);
private:
FileReader* _file_reader;
std::shared_ptr _file_metadata;
-//const int64_t _batch_size;
-//const int32_t _num_of_columns_from_file;
+std::shared_ptr _row_group_reader;
Review Comment:
In the new design, we should ensure that the row group reader is a reader
that can be processed independently in parallel.
##
be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:
##
@@ -17,4 +17,63 @@
#include "vparquet_group_reader.h"
-namespace doris::vectorized {} // namespace doris::vectorized
\ No newline at end of file
+namespace doris::vectorized {
+
+RowGroupReader::RowGroupReader(doris::FileReader* file_reader,
+ std::shared_ptr file_metadata,
+ std::vector column_ids)
+: _file_reader(file_reader),
+ _file_metadata(file_metadata),
+ _column_ids(column_ids),
+ _current_row_group(0) {
+DCHECK_LE(column_ids.size(), file_metadata->num_columns());
+_init_column_readers(column_ids);
+}
+
+void RowGroupReader::_init_column_readers(std::vector column_ids) {
+//for (int col_id: column_ids) {
+//
+//}
+}
+
+Status RowGroupReader::read_next_row_group(const int32_t* group_id) {
+int32_t total_group = _file_metadata->num_row_groups();
+while (_current_row_group < total_group) {
+bool filter_group = false;
+_process_row_group_filter(&filter_group);
+if (!filter_group) {
+group_id = &_current_row_group;
Review Comment:
missing `break`;
##
be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:
##
@@ -17,4 +17,63 @@
#include "vparquet_group_reader.h"
-namespace doris::vectorized {} // namespace doris::vectorized
\ No newline at end of file
+namespace doris::vectorized {
+
+RowGroupReader::RowGroupReader(doris::FileReader* file_reader,
+ std::shared_ptr file_metadata,
+ std::vector column_ids)
Review Comment:
```suggestion
const std::vector& column_ids)
```
##
be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:
##
@@ -17,4 +17,63 @@
#include "vparquet_group_reader.h"
-namespace doris::vectorized {} // namespace doris::vectorized
\ No newline at end of file
+namespace doris::vectorized {
+
+RowGroupReader::RowGroupReader(doris::FileReader* file_reader,
+ std::shared_ptr file_metadata,
+ std::vector column_ids)
+: _file_reader(file_reader),
+ _file_metadata(file_metadata),
+ _column_ids(column_ids),
+ _current_row_group(0) {
+DCHECK_LE(column_ids.size(), file_metadata->num_columns());
+_init_column_readers(column_ids);
+}
+
+void RowGroupReader::_init_column_readers(std::vector column_ids) {
+//for (int col_id: column_ids) {
+//
+//}
+}
+
+Status RowGroupReader::read_next_row_group(const int32_t* group_id) {
+int32_t total_group = _file_metadata->num_row_groups();
+while (_current_row_group < total_group) {
+bool filter_group = false;
+_process_row_group_filter(&filter_group);
[GitHub] [doris] BiteTheDDDDt commented on a diff in pull request #11396: [Bug](MaterializedView) Fix bug of light schema change do not set right unique id cause MV coredump
BiteThet commented on code in PR #11396: URL: https://github.com/apache/doris/pull/11396#discussion_r935697124 ## regression-test/conf/regression-conf.groovy: ## @@ -20,11 +20,11 @@ // **Note**: default db will be create if not exist defaultDb = "regression_test" -jdbcUrl = "jdbc:mysql://127.0.0.1:9030/?" +jdbcUrl = "jdbc:mysql://127.0.0.1:9031/?" Review Comment: need reset -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #11415: (feature-wip)[parquet-reader] file reader
wsjz commented on code in PR #11415:
URL: https://github.com/apache/doris/pull/11415#discussion_r935702185
##
be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:
##
@@ -17,4 +17,63 @@
#include "vparquet_group_reader.h"
-namespace doris::vectorized {} // namespace doris::vectorized
\ No newline at end of file
+namespace doris::vectorized {
+
+RowGroupReader::RowGroupReader(doris::FileReader* file_reader,
+ std::shared_ptr file_metadata,
+ std::vector column_ids)
+: _file_reader(file_reader),
+ _file_metadata(file_metadata),
+ _column_ids(column_ids),
+ _current_row_group(0) {
+DCHECK_LE(column_ids.size(), file_metadata->num_columns());
+_init_column_readers(column_ids);
+}
+
+void RowGroupReader::_init_column_readers(std::vector column_ids) {
+//for (int col_id: column_ids) {
+//
+//}
+}
+
+Status RowGroupReader::read_next_row_group(const int32_t* group_id) {
+int32_t total_group = _file_metadata->num_row_groups();
+while (_current_row_group < total_group) {
+bool filter_group = false;
+_process_row_group_filter(&filter_group);
+if (!filter_group) {
+group_id = &_current_row_group;
Review Comment:
It is a template implement this version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #11449: [dev-1.1.2](memtracker) Introducing new MemTracker
xinyiZzz opened a new pull request, #11449: URL: https://github.com/apache/doris/pull/11449 # Proposed changes Issue Number: close #xxx ## Problem summary On dev-1.1.2, both the old and the new MemTracker exist, and the new MemTracker is only used to track and limit query memory. On dev-1.2 later, expect to sync the new MemTracker on the master. ## Checklist(Required) 1. Does it affect the original behavior: - [x] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #11060: [improvement] Simplify scan node logic: removing _materialized_blocks…
morningman commented on code in PR #11060:
URL: https://github.com/apache/doris/pull/11060#discussion_r935712255
##
be/src/vec/exec/volap_scan_node.cpp:
##
@@ -502,17 +377,18 @@ void VOlapScanNode::scanner_thread(VOlapScanner* scanner)
{
int64_t raw_bytes_read = 0;
int64_t raw_bytes_threshold = config::doris_scanner_row_bytes;
bool get_free_block = true;
+bool reached_limit = false;
int num_rows_in_block = 0;
// Has to wait at least one full block, or it will cause a lot of schedule
task in priority
// queue, it will affect query latency and query concurrency for example
ssb 3.3.
while (!eos && raw_bytes_read < raw_bytes_threshold &&
((raw_rows_read < raw_rows_threshold && get_free_block) ||
num_rows_in_block < _runtime_state->batch_size())) {
-if (UNLIKELY(_transfer_done)) {
+if (UNLIKELY(state->is_cancelled())) {
eos = true;
status = Status::Cancelled("Cancelled");
-LOG(INFO) << "Scan thread cancelled, cause query done, maybe reach
limit.";
+LOG(INFO) << "Scan thread cancelled.";
Review Comment:
I think this log can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #11060: [improvement] Simplify scan node logic: removing _materialized_blocks…
morningman commented on PR #11060: URL: https://github.com/apache/doris/pull/11060#issuecomment-1202849217 Could you please explain more about the threading model of this olap scan node. For example: 1. How to determine the concurrency. 2. How to submit the scanner thread. 3. How to transfer the block from scanner to scan node. So that other developer can easily understand its logic -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] platoneko opened a new pull request, #11450: [feature](cold_on_s3) Show remote data usage
platoneko opened a new pull request, #11450: URL: https://github.com/apache/doris/pull/11450 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #8862: [feature-wip](statistics) step5: show statistics job information
weizhengte commented on code in PR #8862:
URL: https://github.com/apache/doris/pull/8862#discussion_r935838686
##
fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsJob.java:
##
@@ -265,4 +273,86 @@ public static StatisticsJob fromAnalyzeStmt(AnalyzeStmt
stmt) throws AnalysisExc
Map properties = stmt.getProperties();
return new StatisticsJob(dbId, tblIds, tableIdToPartitionName,
tableIdToColumnName, properties);
}
+
+public List getShowInfo(@Nullable Long tableId) throws
AnalysisException {
+List result = Lists.newArrayList();
+
+result.add(Long.toString(id));
+
+SimpleDateFormat dateFormat = new SimpleDateFormat("-MM-dd
HH:mm:ss.SSS");
+result.add(TimeUtils.longToTimeString(createTime, dateFormat));
+result.add(startTime != -1L ? TimeUtils.longToTimeString(startTime,
dateFormat) : "N/A");
+result.add(finishTime != -1L ? TimeUtils.longToTimeString(finishTime,
dateFormat) : "N/A");
+
+StringBuilder sb = new StringBuilder();
+for (String errorMsg : errorMsgs) {
+sb.append(errorMsg).append("\n");
+}
+result.add(sb.toString());
+
+int totalTaskNum = 0;
+int finishedTaskNum = 0;
+Map> tblIdToCols = Maps.newHashMap();
+
+for (StatisticsTask task : tasks) {
+List statsDescs = task.getStatsDescs();
+
+if (!statsDescs.isEmpty()) {
+// The same task has the same stats properties
+StatsCategory statsCategory =
statsDescs.get(0).getStatsCategory();
+long tblId = statsCategory.getTableId();
+
+if (tableId == null || tableId == tblId) {
+totalTaskNum++;
+if (task.getTaskState() ==
StatisticsTask.TaskState.FINISHED) {
+finishedTaskNum++;
+}
+
+String col = statsCategory.getColumnName();
+if (Strings.isNullOrEmpty(col)) {
+continue;
+}
+if (tblIdToCols.containsKey(tblId)) {
+tblIdToCols.get(tblId).add(col);
+} else {
+Set cols = Sets.newHashSet();
+cols.add(col);
+tblIdToCols.put(tblId, cols);
+}
+}
+}
+}
+
+List scope = Lists.newArrayList();
+Database db = Env.getCurrentEnv()
+.getInternalDataSource().getDbOrAnalysisException(dbId);
+for (Long tblId : tblIds) {
+try {
+Table table = db.getTableOrAnalysisException(tblId);
+List baseSchema = table.getBaseSchema();
+Set cols = tblIdToCols.get(tblId);
+if (cols != null) {
+if (baseSchema.size() == cols.size()) {
+scope.add(table.getName() + "(*)");
+} else {
+scope.add(table.getName() + "(" +
StringUtils.join(cols.toArray(), ",") + ")");
+}
+}
+} catch (AnalysisException e) {
+// catch this exception when table is dropped
+LOG.info("get table failed, tableId: " + tblId, e);
+}
+}
+
+result.add(StringUtils.join(scope.toArray(), ","));
+result.add(finishedTaskNum + "/" + totalTaskNum);
+
+if (totalTaskNum == finishedTaskNum) {
Review Comment:
a statistical job will be divided into multiple tasks to execute, so the job
status is finished means that all tasks have been finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #8862: [feature-wip](statistics) step5: show statistics job information
weizhengte commented on code in PR #8862:
URL: https://github.com/apache/doris/pull/8862#discussion_r935847810
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowAnalyzeStmt.java:
##
@@ -0,0 +1,366 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.OrderByPair;
+import org.apache.doris.mysql.privilege.PaloAuth;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSetMetaData;
+import org.apache.doris.statistics.StatisticsJob;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+/**
+ * ShowAnalyzeStmt is used to show statistics job info.
+ * syntax:
+ *SHOW ANALYZE
+ *[TABLE | ID]
+ *[
+ *WHERE
+ *[STATE =
["PENDING"|"SCHEDULING"|"RUNNING"|"FINISHED"|"FAILED"|"CANCELLED"]]
+ *]
+ *[ORDER BY ...]
+ *[LIMIT limit][OFFSET offset];
+ */
+public class ShowAnalyzeStmt extends ShowStmt {
+private static final String STATE_NAME = "state";
+private static final ImmutableList TITLE_NAMES = new
ImmutableList.Builder()
+.add("id")
+.add("create_time")
+.add("start_time")
+.add("finish_time")
+.add("error_msg")
+.add("scope")
+.add("progress")
+.add("state")
+.build();
+
+private List jobIds;
+private TableName dbTableName;
+private Expr whereClause;
+private LimitElement limitElement;
+private List orderByElements;
+
+// after analyzed
+private long dbId;
+private final Set tblIds = Sets.newHashSet();
+
+private String stateValue;
+private ArrayList orderByPairs;
+
+public ShowAnalyzeStmt() {
+}
+
+public ShowAnalyzeStmt(List jobIds) {
+this.jobIds = jobIds;
+}
+
+public ShowAnalyzeStmt(TableName dbTableName,
+ Expr whereClause,
+ List orderByElements,
+ LimitElement limitElement) {
+this.dbTableName = dbTableName;
+this.whereClause = whereClause;
+this.orderByElements = orderByElements;
+this.limitElement = limitElement;
+}
+
+public List getJobIds() {
+return jobIds;
+}
+
+public long getDbId() {
+Preconditions.checkArgument(isAnalyzed(),
+"The dbId must be obtained after the parsing is complete");
+return dbId;
+}
+
+public Set getTblIds() {
+Preconditions.checkArgument(isAnalyzed(),
+"The dbId must be obtained after the parsing is complete");
+return tblIds;
+}
+
+public String getStateValue() {
+Preconditions.checkArgument(isAnalyzed(),
+"The tbl name must be obtained after the parsing is complete");
+return stateValue;
+}
+
+public ArrayList getOrderByPairs() {
+Preconditions.checkArgument(isAnalyzed(),
+"The tbl name must be obtained after the parsing is complete");
+return orderByPairs;
+}
+
+public long getLimit() {
+if (limitElement != null && limitElement.hasLimit()) {
+return limitElement.getLimit();
+}
+return -1L;
+}
+
+public long getOffset() {
+if (limitElement != null && limitElement.hasOffset()) {
+return limitElement.getOffset();
+}
+return -1L;
+}
+
+@Override
+public void analyze(Analyzer analyzer) throws Ana
[GitHub] [doris] weizhengte commented on a diff in pull request #8862: [feature-wip](statistics) step5: show statistics job information
weizhengte commented on code in PR #8862:
URL: https://github.com/apache/doris/pull/8862#discussion_r935848320
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowAnalyzeStmt.java:
##
@@ -0,0 +1,366 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.OrderByPair;
+import org.apache.doris.mysql.privilege.PaloAuth;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSetMetaData;
+import org.apache.doris.statistics.StatisticsJob;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+/**
+ * ShowAnalyzeStmt is used to show statistics job info.
+ * syntax:
+ *SHOW ANALYZE
+ *[TABLE | ID]
+ *[
+ *WHERE
+ *[STATE =
["PENDING"|"SCHEDULING"|"RUNNING"|"FINISHED"|"FAILED"|"CANCELLED"]]
+ *]
+ *[ORDER BY ...]
+ *[LIMIT limit][OFFSET offset];
+ */
+public class ShowAnalyzeStmt extends ShowStmt {
+private static final String STATE_NAME = "state";
+private static final ImmutableList TITLE_NAMES = new
ImmutableList.Builder()
+.add("id")
+.add("create_time")
+.add("start_time")
+.add("finish_time")
+.add("error_msg")
+.add("scope")
+.add("progress")
+.add("state")
+.build();
+
+private List jobIds;
+private TableName dbTableName;
+private Expr whereClause;
+private LimitElement limitElement;
+private List orderByElements;
+
+// after analyzed
+private long dbId;
+private final Set tblIds = Sets.newHashSet();
+
+private String stateValue;
+private ArrayList orderByPairs;
+
+public ShowAnalyzeStmt() {
+}
+
+public ShowAnalyzeStmt(List jobIds) {
+this.jobIds = jobIds;
+}
+
+public ShowAnalyzeStmt(TableName dbTableName,
+ Expr whereClause,
+ List orderByElements,
+ LimitElement limitElement) {
+this.dbTableName = dbTableName;
+this.whereClause = whereClause;
+this.orderByElements = orderByElements;
+this.limitElement = limitElement;
+}
+
+public List getJobIds() {
+return jobIds;
+}
+
+public long getDbId() {
+Preconditions.checkArgument(isAnalyzed(),
+"The dbId must be obtained after the parsing is complete");
+return dbId;
+}
+
+public Set getTblIds() {
+Preconditions.checkArgument(isAnalyzed(),
+"The dbId must be obtained after the parsing is complete");
+return tblIds;
+}
+
+public String getStateValue() {
+Preconditions.checkArgument(isAnalyzed(),
+"The tbl name must be obtained after the parsing is complete");
+return stateValue;
+}
+
+public ArrayList getOrderByPairs() {
+Preconditions.checkArgument(isAnalyzed(),
+"The tbl name must be obtained after the parsing is complete");
+return orderByPairs;
+}
+
+public long getLimit() {
+if (limitElement != null && limitElement.hasLimit()) {
+return limitElement.getLimit();
+}
+return -1L;
+}
+
+public long getOffset() {
+if (limitElement != null && limitElement.hasOffset()) {
+return limitElement.getOffset();
+}
+return -1L;
+}
+
+@Override
+public void analyze(Analyzer analyzer) throws Ana
[GitHub] [doris] weizhengte commented on a diff in pull request #8862: [feature-wip](statistics) step5: show statistics job information
weizhengte commented on code in PR #8862:
URL: https://github.com/apache/doris/pull/8862#discussion_r935850716
##
fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsJobManager.java:
##
@@ -140,4 +150,84 @@ private void checkRestrict(long dbId, Set tableIds)
throws AnalysisExcepti
+ Config.cbo_max_statistics_job_num);
}
}
+
+public List> getAnalyzeJobInfos(ShowAnalyzeStmt showStmt)
throws AnalysisException {
+List> results = Lists.newArrayList();
+
+String stateValue = showStmt.getStateValue();
+StatisticsJob.JobState jobState = null;
+if (!Strings.isNullOrEmpty(stateValue)) {
+jobState = StatisticsJob.JobState.valueOf(stateValue);
+}
+
+// step 1: get job infos
+List jobIds = showStmt.getJobIds();
+if (jobIds != null && !jobIds.isEmpty()) {
+for (Long jobId : jobIds) {
+StatisticsJob statisticsJob = idToStatisticsJob.get(jobId);
+if (statisticsJob == null) {
+throw new AnalysisException("No such job id: " + jobId);
+}
+if (jobState == null || jobState ==
statisticsJob.getJobState()) {
+List showInfo =
statisticsJob.getShowInfo(null);
+results.add(showInfo);
+}
+}
+} else {
+long dbId = showStmt.getDbId();
+Set tblIds = showStmt.getTblIds();
+for (StatisticsJob statisticsJob : idToStatisticsJob.values()) {
+long jobDbId = statisticsJob.getDbId();
+if (jobDbId == dbId) {
+// check the state
+if (jobState == null || jobState ==
statisticsJob.getJobState()) {
+Set jobTblIds = statisticsJob.getTblIds();
+// get the intersection of two sets
+Set set = Sets.newHashSet();
+set.addAll(jobTblIds);
+set.retainAll(tblIds);
+for (long tblId : set) {
+List showInfo =
statisticsJob.getShowInfo(tblId);
+results.add(showInfo);
+}
+}
+}
+}
+}
+
+// step2: order the result
Review Comment:
the where condition is mainly to filter the job status(see jobState), which
has been processed above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [refactor](schema change)Remove delete from sc (#11441)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new de4466624d [refactor](schema change)Remove delete from sc (#11441)
de4466624d is described below
commit de4466624da9ef7eb76b3b70f8953c6ea6e89401
Author: yiguolei <676222...@qq.com>
AuthorDate: Wed Aug 3 03:29:41 2022 +0800
[refactor](schema change)Remove delete from sc (#11441)
* not need call delete handler to filter rows since they are filtered in
rowset reader
* need not call delete eval in schema change and remove related code
Co-authored-by: yiguolei
---
be/src/olap/collect_iterator.cpp | 27 +--
be/src/olap/collect_iterator.h | 1 -
be/src/olap/delete_handler.cpp | 13
be/src/olap/delete_handler.h | 16 +---
be/src/olap/olap_cond.cpp | 68 -
be/src/olap/olap_cond.h| 9 ---
be/src/olap/schema_change.cpp | 85 ++
be/test/olap/delete_handler_test.cpp | 9 ---
.../test_alter_table_column_with_delete.out| 12 +++
.../test_alter_table_column_with_delete.groovy | 63
10 files changed, 100 insertions(+), 203 deletions(-)
diff --git a/be/src/olap/collect_iterator.cpp b/be/src/olap/collect_iterator.cpp
index e9b5127022..dd98303108 100644
--- a/be/src/olap/collect_iterator.cpp
+++ b/be/src/olap/collect_iterator.cpp
@@ -195,7 +195,7 @@
CollectIterator::Level0Iterator::Level0Iterator(RowsetReaderSharedPtr rs_reader,
if (LIKELY(rs_reader->type() == RowsetTypePB::BETA_ROWSET)) {
_refresh_current_row = &Level0Iterator::_refresh_current_row_v2;
} else {
-_refresh_current_row = &Level0Iterator::_refresh_current_row_v1;
+LOG(FATAL) << "Not supported rowset type";
}
}
@@ -220,31 +220,6 @@ int64_t CollectIterator::Level0Iterator::version() const {
return _rs_reader->version().second;
}
-Status CollectIterator::Level0Iterator::_refresh_current_row_v1() {
-do {
-if (_row_block != nullptr && _row_block->has_remaining()) {
-size_t pos = _row_block->pos();
-_row_block->get_row(pos, &_row_cursor);
-if (_row_block->block_status() == DEL_PARTIAL_SATISFIED &&
-_reader->_delete_handler.is_filter_data(version(),
_row_cursor)) {
-_reader->_stats.rows_del_filtered++;
-_row_block->pos_inc();
-continue;
-}
-_current_row = &_row_cursor;
-return Status::OK();
-} else {
-auto res = _rs_reader->next_block(&_row_block);
-if (!res.ok()) {
-_current_row = nullptr;
-return res;
-}
-}
-} while (_row_block != nullptr);
-_current_row = nullptr;
-return Status::OLAPInternalError(OLAP_ERR_DATA_EOF);
-}
-
Status CollectIterator::Level0Iterator::_refresh_current_row_v2() {
do {
if (_row_block != nullptr && _row_block->has_remaining()) {
diff --git a/be/src/olap/collect_iterator.h b/be/src/olap/collect_iterator.h
index 1342987707..e765013666 100644
--- a/be/src/olap/collect_iterator.h
+++ b/be/src/olap/collect_iterator.h
@@ -139,7 +139,6 @@ private:
private:
Status (Level0Iterator::*_refresh_current_row)() = nullptr;
-Status _refresh_current_row_v1();
Status _refresh_current_row_v2();
RowsetReaderSharedPtr _rs_reader;
diff --git a/be/src/olap/delete_handler.cpp b/be/src/olap/delete_handler.cpp
index 716c84b504..f3a24731ab 100644
--- a/be/src/olap/delete_handler.cpp
+++ b/be/src/olap/delete_handler.cpp
@@ -310,19 +310,6 @@ Status DeleteHandler::init(const TabletSchema& schema,
return Status::OK();
}
-bool DeleteHandler::is_filter_data(const int64_t data_version, const
RowCursor& row) const {
-// According to semantics, the delete condition stored in _del_conds
should be an OR relationship,
-// so as long as the data matches one of the _del_conds, it will return
true.
-for (const auto& del_cond : _del_conds) {
-if (data_version <= del_cond.filter_version &&
-del_cond.del_cond->delete_conditions_eval(row)) {
-return true;
-}
-}
-
-return false;
-}
-
std::vector DeleteHandler::get_conds_version() {
std::vector conds_version;
for (const auto& cond : _del_conds) {
diff --git a/be/src/olap/delete_handler.h b/be/src/olap/delete_handler.h
index c04dbadcb5..f3f68eff5d 100644
--- a/be/src/olap/delete_handler.h
+++ b/be/src/olap/delete_handler.h
@@ -46,11 +46,7 @@ struct DeleteConditions {
//Status res;
//DeleteHandler delete_handler;
//res = delete_handler.init(tablet, condition_version);
-// 2
[GitHub] [doris] yiguolei merged pull request #11441: [refactor](schema change)Remove delete from sc
yiguolei merged PR #11441: URL: https://github.com/apache/doris/pull/11441 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11445: [typo](docs zh-CN) how to be a committer
github-actions[bot] commented on PR #11445: URL: https://github.com/apache/doris/pull/11445#issuecomment-1203353548 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11448: [minor](log) add some log to observe the change of table's state.
yiguolei merged PR #11448: URL: https://github.com/apache/doris/pull/11448 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
